loadbalancingexporter makes the collector accept data to produce a reject otelcol_receiver_refused_spans

[linliaoy]

Component(s)

exporter/loadbalancing

What happened?

Description

A lot of receiver refused span occurs when I use the svc resolution of k8s for loadbalancingexporter，I added a groupbytrace to fix this temporarily, but the current collector has a lot more cpu and memory footprint，exporter_send_failed_spacs will also increase。The other one that doesn't use loadbalancingexporter is good

Steps to Reproduce

Expected Result

Actual Result

Collector version

0.97.0

Environment information

Environment

OS: kernel 4.19.91
Compiler(if manually compiled): "go 1.21"

OpenTelemetry Collector configuration

receivers:
      otlp:
        protocols:
          grpc:
            endpoint: ${env:MY_POD_IP}:4317
          http:
            endpoint: ${env:MY_POD_IP}:4318
      otlp/mobile:
        protocols:
          grpc:
            endpoint: ${env:MY_POD_IP}:4315
            
          http:
            endpoint: ${env:MY_POD_IP}:4316
      
    extensions:
      health_check:
        path: "/health/status"
    processors:
      batch:
        send_batch_size: 10000
      memory_limiter:
        # 80% of maximum memory up to 2G 1\
        limit_mib: 800
        #limit_percentage: 10%
        # 25% of limit up to 2G    
        spike_limit_mib: 200    
        #spike_limit_percentage: 5%
        check_interval: 5s
      groupbytrace:
        wait_duration: 3s
        num_traces: 2000000
        num_workers: 10
    exporters:
      loadbalancing:
        protocol:
          otlp:
            tls:
              insecure: true
            sending_queue:
              num_consumers: 100
              queue_size: 100000
        resolver:
          k8s:
            service: backends-headless.observability
   
    service:
      telemetry:
        metrics:
          address: ":8888"
      extensions: [health_check]
      pipelines:
        traces:      
          receivers: [otlp]      
          #processors: [batch]
          processors: [memory_limiter,groupbytrace]
          exporters: [loadbalancing]
        traces/mobile:
          receivers: [otlp/mobile]      
          #processors: [batch]
          processors: [memory_limiter,groupbytrace]
          exporters: [loadbalancing]

Log output

No log output, only an increase on the otelcol_receiver_refused_spans value

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/loadbalancing: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[linliaoy]

I printed the error log on otlp Receiver Export 。
error： span error: couldn't find the exporter for the endpoint ""

[linliaoy]

lb.ring.item is empty

[linliaoy]

k8sResolver.resolve backends id empty

[linliaoy]

/label exporter/loadbalancing waiting-for-author

[github-actions]

Pinging code owners for exporter/loadbalancing: @jpkrohling. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[linliaoy]

When backends are empty, the pod update event is accompanied by the POD update event. The update event is deleted and then added. The intermediate time causes the exporter to not be found, resulting in refused_spans

[JaredTan95]

@linliaoy Can you turn on otel logs into debug level and try to capture some logs and paste them?

[linliaoy]

@JaredTan95 Is it OK if I print the log with fmt

[linliaoy]

@JaredTan95 I have changed the k8s update event update mechanism, and I have not rejected span for two days now

[jpkrohling]

Thank you! I left a couple of questions on the PR.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/loadbalancing: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.