Looking for help with configuring okta oauth bearer token using librdkafka options in vector.

[CH-raghavendergogi]

Hi All,
I have been trying to configure following configuration in librdkafka_options for vector kafka sink but it seems to be not working. Could you please provide any working example with SASL_SSL and OAUTH bearer token? Also, please let me know how to check librdkafka version in vector as seems to be bundled inside
'sasl.mechanisms': 'OAUTHBEARER',
'sasl.oauthbearer.method': 'oidc',
'sasl.oauthbearer.client.id': args.client_id,
'sasl.oauthbearer.client.secret': args.client_secret,
'sasl.oauthbearer.token.endpoint.url': args.token_url,
'sasl.oauthbearer.scope': ' '.join(args.scopes)
Here are the logs:
limit=true
2024-07-26T15:19:12.439522Z ERROR sink{component_kind="sink" component_id=logging_kafka component_type=kafka}:request{request_id=5850}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] has been suppressed 9 times.
2024-07-26T15:19:12.439528Z ERROR sink{component_kind="sink" component_id=logging_kafka component_type=kafka}:request{request_id=5850}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true
2024-07-26T15:19:12.813388Z ERROR rdkafka::client: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
2024-07-26T15:19:13.440153Z ERROR sink{component_kind="sink" component_id=logging_kafka component_type=kafka}:request{request_id=5851}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] is being suppressed to avoid flooding.
2024-07-26T15:19:13.440245Z ERROR sink{component_kind="sink" component_id=logging_kafka component_type=kafka}:request{request_id=5851}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.
2024-07-26T15:19:14.582969Z INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="logging_kafka" time_remaining="54 seconds left"
2024-07-26T15:19:19.583030Z INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="logging_kafka" time_remaining="49 seconds left"
2024-07-26T15:19:19.812183Z INFO vector::app: Log level is enabled. level="info"
2024-07-26T15:19:19.813642Z INFO vector::app: Loading configs. paths=["/etc/vector/vector.yaml"]
2024-07-26T15:19:19.838447Z WARN sink{component_kind="sink" component_id=logging_kafka component_type=kafka}: librdkafka: librdkafka: CONFWARN [thrd:app]: Configuration property fetch.error.backoff.ms is a consumer property and will be ignored by this producer instance
2024-07-26T15:19:19.838548Z INFO vector::topology::running: Running healthchecks.
2024-07-26T15:19:19.838632Z INFO vector: Vector has started. debug="false" version="0.39.0" arch="aarch64" revision="73da9bb 2024-06-17 16:00:23.791735272"
2024-07-26T15:19:19.839642Z INFO vector::internal_events::api: API server running. address=0.0.0.0:8686 playground=http://0.0.0.0:8686/playground graphql=http://0.0.0.0:8686/graphql
2024-07-26T15:19:19.845558Z WARN librdkafka: librdkafka: CONFWARN [thrd:app]: Configuration property message.send.max.retries is a producer property and will be ignored by this consumer instance
2024-07-26T15:19:19.845566Z WARN librdkafka: librdkafka: CONFWARN [thrd:app]: Configuration property request.required.acks is a producer property and will be ignored by this consumer instance
2024-07-26T15:19:22.852996Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Meta data fetch error: OperationTimedOut (Local: Timed out) component_kind="sink" component_type="kafka" component_id=logging_kafka
2024-07-26T15:19:25.039866Z WARN librdkafka: librdkafka: FAIL [thrd:ssl://pkc-n98pk.us-west-2.aws.confluent.cloud:9092/bootstrap]: ssl://pkc-n98pk.us-west-2.aws.confluent.cloud:9092/bootstrap: Disconnected (after 5043ms in state UP)
2024-07-26T15:19:25.040153Z ERROR rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): ssl://pkc-n98pk.us-west-2.aws.confluent.cloud:9092/bootstrap: Disconnected (after 5043ms in state UP)
2024-07-26T15:19:25.040237Z ERROR rdkafka::client: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
2024-07-26T15:19:30.300260Z WARN librdkafka: librdkafka: FAIL [thrd:ssl://pkc-n98pk.us-west-2.aws.confluent.cloud:9092/bootstrap]: ssl://pkc-n98pk.us-west-2.aws.confluent.cloud:9092/bootstrap: Disconnected (after 5108ms in state UP, 1 identical error(s) suppressed)
2024-07-26T15:19:30.300444Z ERROR rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): ssl://pkc-n98pk.us-west-2.aws.confluent.cloud:9092/bootstrap: Disconnected (after 5108ms in state UP, 1 identical error(s) suppressed)
2024-07-26T15:19:30.300470Z ERROR rdkafka::client: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are dow

[CH-raghavendergogi]

Also, I see this when I configure sasl configuration under librdkafka_options..
2024-07-26T17:26:20.060605Z INFO vector::app: Log level is enabled. level="info"
2024-07-26T17:26:20.060905Z INFO vector::app: Loading configs. paths=["/etc/vector/vector.yaml"]
2024-07-26T17:26:20.065946Z ERROR vector::topology::builder: Configuration error. error=Sink "logging_kafka": creating kafka producer failed: Client config error: Configuration property "sasl.oauthbearer.client.id" not supported in this build: OAuth/OIDC depends on libcurl and OpenSSL which were not available at build time sasl.oauthbearer.client.id 0oafq15axz9WMbkJd1d7