Added:HTTP header tagging with DD_TRACE_HEADER_TAGS

docs/GettingStarted.md

@@ -1555,7 +1555,7 @@ run app
 | --- | ----------- | ------- |
 | `application` | Your Rack application. Required for `middleware_names`. | `nil` |
 | `distributed_tracing` | Enables [distributed tracing](#distributed-tracing) so that this service trace is connected with a trace of another service if tracing headers are received | `true` |
-| `headers` | Hash of HTTP request or response headers to add as tags to the `rack.request`. Accepts `request` and `response` keys with Array values e.g. `['Last-Modified']`. Adds `http.request.headers.*` and `http.response.headers.*` tags respectively. | `{ response: ['Content-Type', 'X-Request-ID'] }` |
+| `headers` | Hash of HTTP request or response headers to add as tags to the `rack.request`. Accepts `request` and `response` keys with Array values e.g. `['Last-Modified']`. Adds `http.request.headers.*` and `http.response.headers.*` tags respectively. This option overrides the global `DD_TRACE_HEADER_TAGS`, see [Applying header tags to root spans][header tags] for more information. | `{ response: ['Content-Type', 'X-Request-ID'] }` |
 | `middleware_names` | Enable this if you want to use the last executed middleware class as the resource name for the `rack` span. If enabled alongside the `rails` instrumention, `rails` takes precedence by setting the `rack` resource name to the active `rails` controller when applicable. Requires `application` option to use. | `false` |
 | `quantize` | Hash containing options for quantization. May include `:query` or `:fragment`. | `{}` |
 | `quantize.base` | Defines behavior for URL base (scheme, host, port). May be `:show` to keep URL base in `http.url` tag and not set `http.base_url` tag, or `nil` to remove URL base from `http.url` tag by default, leaving a path and setting `http.base_url`. Option must be nested inside the `quantize` option. | `nil` |
@@ -2073,7 +2073,7 @@ end
 | Key | Description | Default |
 | --- | ----------- | ------- |
 | `distributed_tracing` | Enables [distributed tracing](#distributed-tracing) so that this service trace is connected with a trace of another service if tracing headers are received | `true` |
-| `headers` | Hash of HTTP request or response headers to add as tags to the `sinatra.request`. Accepts `request` and `response` keys with Array values e.g. `['Last-Modified']`. Adds `http.request.headers.*` and `http.response.headers.*` tags respectively. | `{ response: ['Content-Type', 'X-Request-ID'] }` |
+| `headers` | Hash of HTTP request or response headers to add as tags to the `sinatra.request`. Accepts `request` and `response` keys with Array values e.g. `['Last-Modified']`. Adds `http.request.headers.*` and `http.response.headers.*` tags respectively. This option overrides the global `DD_TRACE_HEADER_TAGS`, see [Applying header tags to root spans][header tags] for more information. | `{ response: ['Content-Type', 'X-Request-ID'] }` |
 | `resource_script_names` | Prepend resource names with script name | `false` |
 
 ### Sneakers
@@ -2176,6 +2176,7 @@ For example, if `tracing.sampling.default_rate` is configured by [Remote Configu
 | `tracing.distributed_tracing.propagation_inject_style`  | `DD_TRACE_PROPAGATION_STYLE_INJECT`  | `['Datadog']`                                                     | Distributed tracing propagation formats to inject. Overrides `DD_TRACE_PROPAGATION_STYLE`. See [Distributed Tracing](#distributed-tracing) for more details.                                                                              |
 | `tracing.distributed_tracing.propagation_style`         | `DD_TRACE_PROPAGATION_STYLE` | `nil` | Distributed tracing propagation formats to extract and inject. See [Distributed Tracing](#distributed-tracing) for more details. |
 | `tracing.enabled`                                       | `DD_TRACE_ENABLED`             | `true`                                                            | Enables or disables tracing. If set to `false` instrumentation will still run, but no traces are sent to the trace agent.                                                                                                                 |
+| `tracing.header_tags`                                   | `DD_TRACE_HEADER_TAGS`         | `nil`                                                             | Record HTTP headers as span tags. See [Applying header tags to root spans][header tags] for more information.                                          |
 | `tracing.instrument(<integration-name>, <options...>)`  |                                |                                                                   | Activates instrumentation for a specific library. See [Integration instrumentation](#integration-instrumentation) for more details.                                                                                                       |
 | `tracing.log_injection`                                 | `DD_LOGS_INJECTION`            | `true`                                                            | Injects [Trace Correlation](#trace-correlation) information into Rails logs if present. Supports the default logger (`ActiveSupport::TaggedLogging`), `lograge`, and `semantic_logger`.                                                   |
 | `tracing.partial_flush.enabled`                         |                                | `false`                                                           | Enables or disables partial flushing. Partial flushing submits completed portions of a trace to the agent. Used when tracing instruments long running tasks (e.g. jobs) with many spans.                                                  |
@@ -2882,3 +2883,5 @@ As the implementation of `alias_method` exists within those libraries, Datadog g
 For libraries without a known workaround, consider removing the library using `alias` or `Module#alias_method` or separating libraries into different environments for testing.
 
 For any further questions or to report an occurence of this issue, please [reach out to Datadog support](https://docs.datadoghq.com/help)
+
+[header tags]: https://docs.datadoghq.com/tracing/configure_data_security/#applying-header-tags-to-root-spans

lib/datadog/tracing/configuration/ext.rb

@@ -7,6 +7,7 @@ module Configuration
       # e.g. Env vars, default values, enums, etc...
       module Ext
         ENV_ENABLED = 'DD_TRACE_ENABLED'
+        ENV_HEADER_TAGS = 'DD_TRACE_HEADER_TAGS'
         ENV_TRACE_ID_128_BIT_GENERATION_ENABLED = 'DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED'
 
         # @public_api

lib/datadog/tracing/configuration/http.rb

@@ -0,0 +1,75 @@
+# frozen_string_literal: true
+
+module Datadog
+  module Tracing
+    module Configuration
+      module HTTP
+        # Datadog tracing supports capturing HTTP request and response headers as span tags.
+        #
+        # The provided configuration String for this feature has to be pre-processed to
+        # allow for ease of utilization by each HTTP integration.
+        #
+        # This class process configuration, stores the result, and provides methods to
+        # utilize this configuration.
+        class HeaderTags
+          # @param header_tags [Array<String>] The list of strings from DD_TRACE_HEADER_TAGS.
+          def initialize(header_tags)
+            @request_headers = {}
+            @response_headers = {}
+            @header_tags = header_tags
+
+            header_tags.each do |header_tag|
+              header, tag = header_tag.split(':', 2)
+
+              next unless header # RBS type guard for `nil`
+
+              if tag
+                # When a custom tag name is provided, use that name for both
+                # request and response tags.
+                normalized_tag = Tracing::Metadata::Ext::HTTP::Headers.to_tag(tag, allow_nested: true)
+                request = response = normalized_tag
+              else
+                # Otherwise, use our internal pattern of
+                # "http.{request|response}.headers.{header}" as tag name.
+                request = Tracing::Metadata::Ext::HTTP::RequestHeaders.to_tag(header)
+                response = Tracing::Metadata::Ext::HTTP::ResponseHeaders.to_tag(header)
+              end
+
+              @request_headers[header] = request
+              @response_headers[header] = response
+            end
+          end
+
+          # Receives a {RequestHeaderCollection} with the request headers and returns
+          # a list of tag names and values that can be set in a span.
+          def request_tags(headers)
+            @request_headers.map do |header_name, span_tag|
+              # Case-insensitive search. {RequestHeaderCollection} already ensures case-insensitiveness.
+              header_value = headers[header_name]
+
+              [span_tag, header_value] if header_value
+            end.compact
+          end
+
+          # Receives a Hash with the response headers and returns
+          # a list of tag names and values that can be set in a span.
+          def response_tags(headers)
+            @response_headers.map do |header_name, span_tag|
+              # Case-insensitive search
+              # DEV: `String#casecmp?` can be used starting with Ruby 2.4. It's measurable faster than `String#casecmp`.
+              _, header_value = headers.find { |h, _| header_name.casecmp(h) == 0 }
+
+              [span_tag, header_value] if header_value
+            end.compact
+          end
+
+          # For easy configuration inspection,
+          # print the original configuration setting.
+          def to_s
+            @header_tags.join(',').to_s
+          end
+        end
+      end
+    end
+  end
+end

lib/datadog/tracing/configuration/settings.rb

@@ -1,4 +1,5 @@
 require_relative '../../tracing/configuration/ext'
+require_relative 'http'
 
 module Datadog
   module Tracing
@@ -9,6 +10,7 @@ module Configuration
       # rubocop:disable Metrics/BlockLength
       # rubocop:disable Metrics/CyclomaticComplexity
       # rubocop:disable Metrics/MethodLength
+      # rubocop:disable Metrics/PerceivedComplexity
       # rubocop:disable Layout/LineLength
       module Settings
         def self.extended(base)
@@ -173,6 +175,25 @@ def self.extended(base)
                 o.lazy
               end
 
+              # Comma-separated, case-insensitive list of header names that are reported in incoming and outgoing HTTP requests.
+              #
+              # Each header in the list can either be:
+              # * A header name, which is mapped to the respective tags `http.request.headers.<header name>` and `http.response.headers.<header name>`.
+              # * A key value pair, "header name:tag name", which is mapped to the span tag `tag name`.
+              #
+              # You can mix the two types of header declaration in the same list.
+              # Tag names will be normalized based on the [Datadog tag normalization rules](https://docs.datadoghq.com/getting_started/tagging/#defining-tags).
+              #
+              # @default `DD_TRACE_HEADER_TAGS` environment variable, otherwise `nil`
+              # @return [Array<String>]
+              option :header_tags do |o|
+                o.default { env_to_list(Configuration::Ext::ENV_HEADER_TAGS, nil, comma_separated_only: true) }
+                o.lazy
+                o.setter do |header_tags, _old_value|
+                  Configuration::HTTP::HeaderTags.new(header_tags) if header_tags
+                end
+              end
+
               # Enable 128 bit trace id generation.
               #
               # @default `DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED` environment variable, otherwise `false`
@@ -459,6 +480,7 @@ def self.extended(base)
       # rubocop:enable Metrics/BlockLength
       # rubocop:enable Metrics/CyclomaticComplexity
       # rubocop:enable Metrics/MethodLength
+      # rubocop:enable Metrics/PerceivedComplexity
       # rubocop:enable Layout/LineLength
     end
   end

lib/datadog/tracing/contrib/rack/header_collection.rb

@@ -19,6 +19,9 @@ def get(header_name)
               @env[Header.to_rack_header(header_name)]
             end
 
+            # Allows this class to have a similar API to a {Hash}.
+            alias [] get
+
             # Tests whether a header with the given name exists in the environment.
             def key?(header_name)
               @env.key?(Header.to_rack_header(header_name))

lib/datadog/tracing/contrib/rack/header_tagging.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+module Datadog
+  module Tracing
+    module Contrib
+      module Rack
+        # Matches Rack-style headers with a matcher and sets matching headers into a span.
+        module HeaderTagging
+          def self.tag_request_headers(span, env, configuration)
+            headers = Header::RequestHeaderCollection.new(env)
+
+            # Use global DD_TRACE_HEADER_TAGS if integration-level configuration is not provided
+            tags = if configuration.using_default?(:headers) && Datadog.configuration.tracing.header_tags
+                     Datadog.configuration.tracing.header_tags.request_tags(headers)
+                   else
+                     whitelist = configuration[:headers][:request] || []
+                     whitelist.each_with_object({}) do |header, result|
+                       header_value = headers.get(header)
+                       unless header_value.nil?
+                         header_tag = Tracing::Metadata::Ext::HTTP::RequestHeaders.to_tag(header)
+                         result[header_tag] = header_value
+                       end
+                     end
+                   end
+
+            span.set_tags(tags)
+          end
+
+          def self.tag_response_headers(span, headers, configuration)
+            # Use global DD_TRACE_HEADER_TAGS if integration-level configuration is not provided
+            tags = if configuration.using_default?(:headers) && Datadog.configuration.tracing.header_tags
+                     Datadog.configuration.tracing.header_tags.response_tags(headers)
+                   else
+                     whitelist = configuration[:headers][:response] || []
+                     whitelist.each_with_object({}) do |header, result|
+                       if headers.key?(header)
+                         result[Tracing::Metadata::Ext::HTTP::ResponseHeaders.to_tag(header)] = headers[header]
+                       else
+                         # Try a case-insensitive lookup
+                         uppercased_header = header.to_s.upcase
+                         matching_header = headers.keys.find { |h| h.upcase == uppercased_header }
+                         if matching_header
+                           result[Tracing::Metadata::Ext::HTTP::ResponseHeaders.to_tag(header)] = headers[matching_header]
+                         end
+                       end
+                     end
+                   end
+
+            span.set_tags(tags)
+          end
+        end
+      end
+    end
+  end
+end

lib/datadog/tracing/contrib/rack/middlewares.rb

@@ -9,6 +9,7 @@
 require_relative '../utils/quantization/http'
 require_relative 'ext'
 require_relative 'header_collection'
+require_relative 'header_tagging'
 require_relative 'request_queue'
 
 module Datadog
@@ -138,8 +139,6 @@ def call(env)
           # rubocop:disable Metrics/MethodLength
           def set_request_tags!(trace, request_span, env, status, headers, response, original_env)
             request_header_collection = Header::RequestHeaderCollection.new(env)
-            request_headers_tags = parse_request_headers(request_header_collection)
-            response_headers_tags = parse_response_headers(headers || {})
 
             # Since it could be mutated, it would be more accurate to fetch from the original env,
             # e.g. ActionDispatch::ShowExceptions middleware with Rails exceptions_app configuration
@@ -228,10 +227,8 @@ def set_request_tags!(trace, request_span, env, status, headers, response, origi
               request_span.set_tag(Tracing::Metadata::Ext::HTTP::TAG_USER_AGENT, user_agent)
             end
 
-            # Request headers
-            request_headers_tags.each do |name, value|
-              request_span.set_tag(name, value) if request_span.get_tag(name).nil?
-            end
+            HeaderTagging.tag_request_headers(request_span, env, configuration)
+            HeaderTagging.tag_response_headers(request_span, headers, configuration)
 
             # Response headers
             response_headers_tags.each do |name, value|
@@ -315,18 +312,12 @@ def parse_user_agent_header(headers)
             headers.get(Tracing::Metadata::Ext::HTTP::HEADER_USER_AGENT)
           end
 
-          def parse_request_headers(headers)
-            whitelist = configuration[:headers][:request] || []
-            whitelist.each_with_object({}) do |header, result|
-              header_value = headers.get(header)
-              unless header_value.nil?
-                header_tag = Tracing::Metadata::Ext::HTTP::RequestHeaders.to_tag(header)
-                result[header_tag] = header_value
-              end
+          def parse_response_headers(headers)
+            # Use global DD_TRACE_HEADER_TAGS if integration-level configuration is not provided
+            if configuration.using_default?(:headers) && Datadog.configuration.tracing.header_tags
+              return Datadog.configuration.tracing.header_tags.response_tags(headers)
             end
-          end
 
-          def parse_response_headers(headers)
             {}.tap do |result|
               whitelist = configuration[:headers][:response] || []
               whitelist.each do |header|

lib/datadog/tracing/contrib/sinatra/env.rb

@@ -1,6 +1,7 @@
 require 'time'
 
 require_relative '../../metadata/ext'
+require_relative '../rack/header_collection'
 require_relative 'ext'
 
 module Datadog
@@ -19,23 +20,6 @@ def set_datadog_span(env, span)
             env[Ext::RACK_ENV_SINATRA_REQUEST_SPAN] = span
           end
 
-          def request_header_tags(env, headers)
-            headers ||= []
-
-            {}.tap do |result|
-              headers.each do |header|
-                rack_header = header_to_rack_header(header)
-                if env.key?(rack_header)
-                  result[Tracing::Metadata::Ext::HTTP::RequestHeaders.to_tag(header)] = env[rack_header]
-                end
-              end
-            end
-          end
-
-          def header_to_rack_header(name)
-            "HTTP_#{name.to_s.upcase.gsub(/[-\s]/, '_')}"
-          end
-
           def route_path(env, use_script_names: Datadog.configuration.tracing[:sinatra][:resource_script_names])
             return unless env['sinatra.route']