Why trace clickhouse exporter always shows "Exporting Failed: The column Timestamp is not present"

[hueiyuan]

Component(s)

exporter/clickhouse

Describe the issue you're reporting

Hi everyone,
We try to export Otel trace data to clickhouse, but we found that the collector sometimes shows "Exporting Failed: The column Timestamp is not present". And after retry, which can flush to clickhouse successfully.
Does anyone have idea to solve it? Thank you.

Error Message Example

│ 2024-05-07T06:56:16.602Z    info    exporterhelper/retry_sender.go:118    Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": " │
│ traces", "name": "clickhouse/xxxx", "error": "PrepareContext:column Timestamp is not present in the table trace_table", "interval": "4.155902648s"} │

Version

• otel collector: 0.99.0
• clickhouse-go: 2.23

[github-actions]

Pinging code owners:

• exporter/clickhouse: @hanjm @dmitryax @Frapschen @SpencerTorres

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[SpencerTorres]

I have a few questions/ideas on why this might be happening:

• Is it possible that the export batch is trying to run before the table has been created? This would only happen on first start if otel_traces doesn't exist. It is recommended to manually create the required tables before running the exporter.
• Are you running any other table/database changes in the background? If some kind of schema migration tool is running, there could be a case where Timestamp is temporarily not present on the otel_traces table.
• Are you running in any kind of clustered setup where the INSERT would be routed to a destination that does not have the trace table fully created?
• Did otel_traces exist before you ran the exporter? In this case, it will rely on the existing table instead of overwriting it, which means you will need to ensure the schema you are using is compatible with the latest exporter version.

I noticed you are using trace_table (according to the logs). While this table may exist if you created it, it might not contain the correct columns. Please refer to the default_ddl folder of the ClickHouse exporter for a reference of what DDL is run automatically. In this case, the trace DDL would be relevant.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/clickhouse: @hanjm @dmitryax @Frapschen @SpencerTorres

See Adding Labels via Comments if you do not have permissions to add labels yourself.