[exporter/elasticsearch] Flesh out encoding log records with `mapping.mode: ecs` #31694

ycombinator · 2024-03-12T01:21:29Z

Description:

This PR is a follow up to #31553. In that PR, the foundations were laid for encoding log records with ECS fields when the user specifies mapping.mode: ecs in their exporter configuration.

This PR adds ECS conversions for several additional fields.

I expect there will be more followup PR(s) to add more conversions, especially ones that might not be straightforward.

Link to tracking issue: #29742

Testing: Added unit tests

Documentation: The mapping.mode: ecs configuration setting is already documented.

felixbarny · 2024-03-12T08:26:19Z

exporter/elasticsearchexporter/model.go

+		semconv.AttributeServiceName:           "service.name",
+		semconv.AttributeServiceVersion:        "service.version",


Looks like we're mapping it to the same value here. I think we can omit mappings where SemConv and ECS align.

++ done in 3ea4512c3e3451d73708fbed7d0ff2576cf3b304.

felixbarny · 2024-03-12T08:29:13Z

exporter/elasticsearchexporter/model.go

@@ -137,3 +182,17 @@ func scopeToAttributes(scope pcommon.InstrumentationScope) pcommon.Map {
 	}
 	return attrs
 }
+
+func mapLogAttributesToECS(document *objmodel.Document, attrs pcommon.Map, conversionMap map[string]string) {
+	attrs.Range(func(k string, v pcommon.Value) bool {


Based on the mappings so far, I'd expect the conversionMap to be smaller than the attributes. If you agree with that statement, it might make sense to iterate over the conversionMap instead. This may reduce the overhead a bit by having to iterate over fewer values.

Internally pcommon.Map is an array of key-value pairs (https://github.com/open-telemetry/opentelemetry-collector/blob/f0473ca0a55293b92374cc77e0275026537342da/pdata/pcommon/map.go#L26), so even if conversionMap is smaller we'd likely end up iterating attrs multiple times that way.

Special case: I did add an early return for the special case where conversionMap was empty: 47905d4e98ddccb243dcc3eadd2ac8ed1dc6e097.

axw

Mostly LGTM, just a few comments so far.

axw · 2024-03-13T04:29:29Z

exporter/elasticsearchexporter/model.go

+	resourceAttrsConversionMap := map[string]string{
+		semconv.AttributeServiceInstanceID:     "service.node.name",
+		semconv.AttributeDeploymentEnvironment: "service.environment",
+		semconv.AttributeTelemetrySDKName:      "agent.name",


In apm-data we've been setting agent.name to ${telemetry.sdk.name}/${telemetry.sdk.language}, when both are set. Should we be doing the same here? Also with elastic/apm-data@0e072ce we include telemetry.distro.name. If I were starting from scratch I wouldn't do that, as it's redundant information...

Implemented this logic, including that for telemetry.distro.name, in 4b9cca0bdf. Would you mind taking a look at the unit test cases, particularly the ones in TestEncodeLogECSModeAgentName to make sure I got the logic right? Thanks!

exporter/elasticsearchexporter/model.go

axw · 2024-03-13T05:09:38Z

exporter/elasticsearchexporter/model.go

+		semconv.AttributeHostName:              "host.hostname",
+		semconv.AttributeHostArch:              "host.architecture",
+		semconv.AttributeProcessExecutablePath: "process.executable",
+		semconv.AttributeOSType:                "os.platform",


Should we be vetting the values here? ECS expects os.platform to have one of a restricted set of values -- https://www.elastic.co/guide/en/ecs/current/ecs-os.html#field-os-type

For iOS and Android, I think we should set os.platform from os.name ("Android" or "iOS") rather than os.type ("linux" or "darwin")

See https://github.com/elastic/apm-data/blob/0e072ceaadae2ef8f61d75c3a27c47cc5ce81257/input/otlp/metadata.go#L320-L338

Implemented in d36a76fff8.

felixbarny · 2024-03-13T06:53:01Z

exporter/elasticsearchexporter/model.go

+	var document objmodel.Document
+
+	// First, try to map resource-level attributes to ECS fields.
+	resourceAttrsConversionMap := map[string]string{


I think what's missing is the translation of k8s.* SemConv resource attributes. See also https://github.com/elastic/apm-data/blob/0e072ceaadae2ef8f61d75c3a27c47cc5ce81257/input/otlp/metadata.go#L145-L165

Yep, I didn't include them as they're not really part of ECS. But as per the off-PR discussion with @axw and you, I will include them now for practical reasons.

Done in 468043f0494859176c0fd89d3c1d6c62466068d2.

gregkalapos · 2024-03-13T10:33:39Z

exporter/elasticsearchexporter/model.go

+	resourceAttrsConversionMap := map[string]string{
+		semconv.AttributeServiceInstanceID:     "service.node.name",
+		semconv.AttributeDeploymentEnvironment: "service.environment",
+		semconv.AttributeTelemetrySDKName:      "agent.name",
+		semconv.AttributeTelemetrySDKVersion:   "agent.version",
+		semconv.AttributeTelemetrySDKLanguage:  "service.language.name",
+		semconv.AttributeCloudPlatform:         "cloud.service.name",
+		semconv.AttributeContainerImageTags:    "container.image.tag",
+		semconv.AttributeHostName:              "host.hostname",
+		semconv.AttributeHostArch:              "host.architecture",
+		semconv.AttributeProcessExecutablePath: "process.executable",
+		semconv.AttributeOSType:                "os.platform",
+		semconv.AttributeOSDescription:         "os.full",
+	}


The mapping of resource level attributes can be shared across logs, trace, and metrics(not yet mapped in the exporter). Maybe some of the record level mapping can be shared as well.

Currently this code is in the encodeLogECSMode function.

No strong opinion from my side, and for the PR this is fine, but I think mid-term we should think about start sharing this logic across signals.

I moved this mapping out of this function and into package scope in bd90fbf4076586501b57b7799e6b12797cd30828 so, in the future, conversion code for other signals can use it too.

ycombinator · 2024-03-14T00:40:20Z

@axw @felixbarny @gregkalapos Thank you for your feedback. I have addressed all of it now and this PR is ready for another round of review. Thanks!

axw · 2024-03-14T03:22:00Z

exporter/elasticsearchexporter/model.go

+	// Set service language name in document.
+	if telemetrySdkLanguage == "" {
+		telemetrySdkLanguage = "unknown"
+	}
+	document.AddString("service.language.name", telemetrySdkLanguage)


Suggested change

// Set service language name in document.

if telemetrySdkLanguage == "" {

telemetrySdkLanguage = "unknown"

}

document.AddString("service.language.name", telemetrySdkLanguage)

One thing I think we should not carry across from apm-data is setting service.language.name when telemetry.sdk.language is unspecified. That's only really meaningful for application observability, and not all logs or metrics.

Maybe we can just skip setting service.language.name altogether for now, since it's included in agent.name?

@felixbarny WDYT?

One thing I think we should not carry across from apm-data is setting service.language.name when telemetry.sdk.language is unspecified.

Sounds good. But maybe we have to adapt some of the assumptions we have in the APM UI about this field always being present.

Maybe we can just skip setting service.language.name altogether for now, since it's included in agent.name?

The service.language.name attribute is used a lot in our UI. Since it directly translates to service.language.name I think I'd just add it to the resourceAttrsConversionMap.

The service.language.name attribute is used a lot in our UI. Since it directly translates to service.language.name I think I'd just add it to the resourceAttrsConversionMap.

This PR is focused on logs though, and I think the service.language.name usage in the (APM) UI is not relevant here. Could we start without it, and add it in later if there's a need?

let's do that

Removed code for setting service.language.name in cb5a3b88bb.

axw · 2024-03-14T03:25:20Z

exporter/elasticsearchexporter/model.go

+	semconv.AttributeTelemetrySDKName:      "",
+	semconv.AttributeTelemetryDistroName:   "",
+	semconv.AttributeTelemetrySDKLanguage:  "",
+	semconv.AttributeTelemetrySDKVersion:   "agent.version",


Suggested change

semconv.AttributeTelemetrySDKVersion: "agent.version",

semconv.AttributeTelemetrySDKVersion: "",

We'll need to handle telemetry.distro.version too: agent.version gets set to telemetry.distro.version if specified, and telemetry.sdk.version otherwise. Can't say I love this behaviour, but it's what exists in ECS and apm-data now...

Implemented in 04a681bd49.

axw · 2024-03-14T03:36:26Z

exporter/elasticsearchexporter/model.go

+	semconv.AttributeServiceInstanceID:     "service.node.name",
+	semconv.AttributeDeploymentEnvironment: "service.environment",
+	semconv.AttributeTelemetrySDKName:      "",
+	semconv.AttributeTelemetryDistroName:   "",


Suggested change

semconv.AttributeTelemetryDistroName: "",

semconv.AttributeTelemetryDistroName: "",

semconv.AttributeTelemetryDistroVersion: "",

Implemented as part of 04a681bd49.

axw

LGTM!

JaredTan95 · 2024-03-19T02:29:29Z

wait #31553 get merged.

.chloggen/exporter-elasticsarch-more-ecs-mode.yaml

…#32917) **Description:** <Describe what has changed.>  Fix a flaky test from #31694 due to non-deterministic JSON key order. **Link to tracking Issue:** Closes #32910

BenEfrati · 2024-05-27T06:38:42Z

Hi @ycombinator , @andrzej-stencel

There is a regression after this change - duplicate key in json:
before this change request to ELK was:

{
...
   "host":{
      "name":"hostname"
   },
   "os":{
      "description":"Windows 11 10.0",
      "type":"windows"
   },
...
}

after this change:

{
...
   "host":{
      "hostname":"hostname",
      "os":{
         "full":"Windows 11 10.0",
         "platform":"windows"
      }
   },
...
 "host":{
      "os":{
         "type":"windows"
      }
   },
...
}

in this example host key is duplicated, result in invalid json and error from ELK

 "error": {
  "reason": "failed to parse",
  "type": "mapper_parsing_exception"
  },
  "status": 400

Using same resource attributes:

Resource attributes:
host.name: Str(hostname)
os.description: Str(Windows 11 10.0)
os.type: Str(windows)

this is from debug exporter

opentelemetry-collector-contrib/exporter/elasticsearchexporter/internal/objmodel/objmodel.go

Lines 7 to 8 in fdbd368

    
           // The JSON parsing in Elasticsearch does not support parsing JSON documents 
        
           // with duplicate fields. The fields in the docuemt can be sort and duplicate entries

BenEfrati · 2024-05-27T06:40:58Z

exporter/elasticsearchexporter/model.go

+	if ecsHostOsType == "" {
+		return
+	}
+	document.AddString("host.os.type", ecsHostOsType)


in case host is already part of document it'll result in duplicate host key in json request:
Resource attributes:

host.name: Str(hostname) os.description: Str(Windows 11 10.0) os.type: Str(windows)

github-actions bot added the exporter/elasticsearch label Mar 12, 2024

github-actions bot requested a review from JaredTan95 March 12, 2024 01:21

felixbarny reviewed Mar 12, 2024

View reviewed changes

ycombinator force-pushed the exp-es-ecs-r2 branch from e02df27 to bd3f736 Compare March 12, 2024 19:42

ycombinator requested review from axw and felixbarny March 12, 2024 20:12

ycombinator mentioned this pull request Mar 12, 2024

[exporter/elasticsearch] handle ecs mode mapping #31553

Merged

ycombinator marked this pull request as ready for review March 12, 2024 21:19

ycombinator requested review from a team and atoulme and removed request for axw and felixbarny March 12, 2024 21:19

github-actions bot assigned songy23 Mar 12, 2024

ycombinator requested review from axw and felixbarny March 12, 2024 21:34

axw reviewed Mar 13, 2024

View reviewed changes

felixbarny reviewed Mar 13, 2024

View reviewed changes

gregkalapos reviewed Mar 13, 2024

View reviewed changes

ycombinator requested review from axw, gregkalapos and felixbarny March 14, 2024 00:40

axw reviewed Mar 14, 2024

View reviewed changes

ycombinator force-pushed the exp-es-ecs-r2 branch from d0f84d8 to b2313de Compare March 18, 2024 18:04

ycombinator requested a review from axw March 18, 2024 18:58

axw approved these changes Mar 19, 2024

View reviewed changes

felixbarny approved these changes Mar 25, 2024

View reviewed changes

carsonip approved these changes Mar 25, 2024

View reviewed changes

ycombinator force-pushed the exp-es-ecs-r2 branch from 04a681b to 5bdc40e Compare March 25, 2024 22:56

ycombinator added 11 commits May 1, 2024 15:10

Extract out resourceAttrsConversionMap to package-level variable

1047256

No need to pass time.Now() anymore

e514a05

Implement ability to skip keys during conversion

b4c9edc

Handle agent name and service language name

603109d

Handle conversion to ECS os.type field

fa31d70

Fixing OS conversions

42bd94c

Don't set service.language.name

e928d9a

Handle agent version

67e479d

Mention SemConv version in README

0aa3f92

Update test

0fa9353

Incorporate changes from prereq PR

d067b2e

ycombinator force-pushed the exp-es-ecs-r2 branch from 2638896 to d067b2e Compare May 1, 2024 22:10

andrzej-stencel reviewed May 7, 2024

View reviewed changes

.chloggen/exporter-elasticsarch-more-ecs-mode.yaml Outdated Show resolved Hide resolved

andrzej-stencel approved these changes May 7, 2024

View reviewed changes

update issue reference in changelog entry

e1dfd60

andrzej-stencel merged commit 3987074 into open-telemetry:main May 7, 2024
162 checks passed

github-actions bot added this to the next release milestone May 7, 2024

ycombinator deleted the exp-es-ecs-r2 branch May 7, 2024 07:58

crobert-1 mentioned this pull request May 7, 2024

Flaky test - TestExporter_PushEvent/publish_with_ecs_encoding #32910

Closed

carsonip mentioned this pull request May 7, 2024

[chore][exporter/elasticsearch] Fix TestExporter_PushEvent flaky test #32917

Merged

BenEfrati reviewed May 27, 2024

View reviewed changes

BenEfrati mentioned this pull request May 28, 2024

[exporter/elasticsearch] Regression: Duplicate Key in JSON After #31694 #33264

Closed

LucaLanziani mentioned this pull request Jun 2, 2024

Weekly Report: 2024-05-26 - 2024-06-02 #33329

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[exporter/elasticsearch] Flesh out encoding log records with `mapping.mode: ecs` #31694

[exporter/elasticsearch] Flesh out encoding log records with `mapping.mode: ecs` #31694

ycombinator commented Mar 12, 2024 •

edited

Loading

felixbarny Mar 12, 2024

ycombinator Mar 12, 2024

felixbarny Mar 12, 2024

axw Mar 12, 2024

ycombinator Mar 12, 2024 •

edited

Loading

axw left a comment

axw Mar 13, 2024

ycombinator Mar 14, 2024

axw Mar 13, 2024

ycombinator Mar 14, 2024

felixbarny Mar 13, 2024

ycombinator Mar 13, 2024

ycombinator Mar 13, 2024

gregkalapos Mar 13, 2024 •

edited

Loading

ycombinator Mar 13, 2024

ycombinator commented Mar 14, 2024

axw Mar 14, 2024

felixbarny Mar 14, 2024

axw Mar 14, 2024

felixbarny Mar 14, 2024

ycombinator Mar 18, 2024

axw Mar 14, 2024

ycombinator Mar 18, 2024

axw Mar 14, 2024

ycombinator Mar 18, 2024

axw left a comment

JaredTan95 commented Mar 19, 2024

BenEfrati commented May 27, 2024 •

edited

Loading

BenEfrati May 27, 2024

		semconv.AttributeServiceName: "service.name",
		semconv.AttributeServiceVersion: "service.version",

	semconv.AttributeTelemetrySDKVersion: "agent.version",
	semconv.AttributeTelemetrySDKVersion: "",

	semconv.AttributeTelemetryDistroName: "",
	semconv.AttributeTelemetryDistroName: "",
	semconv.AttributeTelemetryDistroVersion: "",

[exporter/elasticsearch] Flesh out encoding log records with mapping.mode: ecs #31694

[exporter/elasticsearch] Flesh out encoding log records with mapping.mode: ecs #31694

Conversation

ycombinator commented Mar 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ycombinator Mar 12, 2024 • edited Loading

Choose a reason for hiding this comment

axw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gregkalapos Mar 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ycombinator commented Mar 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

axw left a comment

Choose a reason for hiding this comment

JaredTan95 commented Mar 19, 2024

BenEfrati commented May 27, 2024 • edited Loading

Choose a reason for hiding this comment

[exporter/elasticsearch] Flesh out encoding log records with `mapping.mode: ecs` #31694

[exporter/elasticsearch] Flesh out encoding log records with `mapping.mode: ecs` #31694

ycombinator commented Mar 12, 2024 •

edited

Loading

ycombinator Mar 12, 2024 •

edited

Loading

gregkalapos Mar 13, 2024 •

edited

Loading

BenEfrati commented May 27, 2024 •

edited

Loading