This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Clickhouse Sink "upstream connect error or disconnect/reset before headers" after Clickhouse update #22116
Labels
Comments
|
Alright, after debugging and updating this post I seem to realise that this might be an Istio problem, as istio-proxy stills holds the old endpoints. Not going to delete this post as this might help somebody else |
|
Okay, this seems to be due to the fact that Hyper in Clickhouse Sink is doing connection pooling and envoy in Istio is as well. Is there any way to deactivate connection pooling in the Clickhouse Sink and let this be handled by Istio? See discussion here istio/istio#54539 |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
A note for the community
Problem
We are running Clickhouse and Vector in Kubernetes. Vector is receiving events from Kafka and is sinking those to Clickhouse. When updating Clickhouse, thereby rolling out a new Pod, data is incoming again without dropped data, however Vector is complaining with WARN messages. This doesn't go away on its own. A 503 seems to me, like Vector is trying to reach the old Clickhouse instances? Could this be possible? Failed requests are retried and written successfully. Requests seem to time out after 10 or 30s, as can be seen in the istio-proxy logs below. A restart of Vector helps, but this can't be the right solution.
2025-01-03T07:48:00.264670Z WARN sink{component_kind="sink" component_id=clickhouse_sink component_type=clickhouse}:request{request_id=15212892}: vector::sinks::util::retries: Internal log [Retrying after response.] has been suppressed 22 times. 2025-01-03T07:48:00.264683Z WARN sink{component_kind="sink" component_id=clickhouse_sink component_type=clickhouse}:request{request_id=15212892}: vector::sinks::util::retries: Retrying after response. reason=503 Service Unavailable: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection timeout internal_log_rate_limit=true 2025-01-03T07:48:00.264715Z WARN sink{component_kind="sink" component_id=clickhouse_sink component_type=clickhouse}:request{request_id=15212891}: vector::sinks::util::retries: Internal log [Retrying after response.] is being suppressed to avoid flooding. 2025-01-03T07:48:11.196311Z WARN sink{component_kind="sink" component_id=clickhouse_sink component_type=clickhouse}:request{request_id=15213093}: vector::sinks::util::retries: Internal log [Retrying after response.] has been suppressed 24 times. 2025-01-03T07:48:11.196327Z WARN sink{component_kind="sink" component_id=clickhouse_sink component_type=clickhouse}:request{request_id=15213093}: vector::sinks::util::retries: Retrying after response. reason=503 Service Unavailable: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection timeout internal_log_rate_limit=true 2025-01-03T07:48:11.196372Z WARN sink{component_kind="sink" component_id=clickhouse_sink component_type=clickhouse}:request{request_id=15213091}: vector::sinks::util::retries: Internal log [Retrying after response.] is being suppressed to avoid flooding. 2025-01-03T07:48:22.164840Z WARN sink{component_kind="sink" component_id=clickhouse_sink component_type=clickhouse}:request{request_id=15213341}: vector::sinks::util::retries: Internal log [Retrying after response.] has been suppressed 29 times. ### Configuration ```text clickhouse_sink: type: clickhouse compression: zstd healthcheck: enabled: true acknowledgements: enabled: true inputs: - clickhouse_tables endpoint: http://service-clickhouse-prod.clickhouse-prod.svc.cluster.local:8123 format: json_each_row database: ep_sdc_edge table: '{{ "{{kafkaSrcTopicToDbName}}" }}' skip_unknown_fields: true auth: strategy: basic user: ${DEST_CLICKHOUSE_USERNAME} password: ${DEST_CLICKHOUSE_PASSWORD} date_time_best_effort: true encoding: timestamp_format: rfc3339 batch: max_events: 2500 timeout_secs: 2Version
0.42.0-distroless-libc
Debug Output
Debug Logs of one request, that fails after rolling the Clickhouse Pods
istio-proxy logs outgoing from vector pod:
This doesn't seem to be a DNS issue, IPs, are correct:
Example Data
No response
Additional Context
No response
References
No response
The text was updated successfully, but these errors were encountered: