Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Elastic-Agent Kafka settings #935

Merged
merged 1 commit into from
Feb 20, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,7 @@ outputs:
partition:
round_robin:
group_events: 1
topics:
- topic: '%{[fields.log_topic]}'
topic: 'elastic-agent'
headers: []
timeout: 30
broker_timeout: 30
Expand Down Expand Up @@ -196,7 +195,7 @@ include::output-shared-settings.asciidoc[tag=queue.mem.flush.timeout-setting]
[[output-kafka-topics-settings]]
== Topics settings

Use these options to dynamically set the Kafka topic for each {agent} event.
Use these options to set the Kafka topic for each {agent} event.

[cols="2*<a"]
|===
Expand All @@ -208,61 +207,9 @@ Use these options to dynamically set the Kafka topic for each {agent} event.

| The default Kafka topic used for produced events.

You can set the topic dynamically by using a format string to access any event field. For example, this configuration uses a custom field, `fields.log_topic`, to set the topic for each event:

[source,yaml]
------------------------------------------------------------------------------
topic: '%{[fields.log_topic]}'
------------------------------------------------------------------------------

// =============================================================================

|
[id="kafka-topics-setting"]
`topics`

| One or more topic processors including a condition, the event value to check against, and the resulting Kafka topic.

Events that don't match against any defined processor are set to the default topic.

Rule settings:

`topic`::
The topic format string to use. If this string contains field references, such as `%{[fields.name]}`, the fields must exist, or the rule fails.

`mappings`::
A dictionary that takes the value returned by `topic` and maps it to a new name.

`default`::
The default string value to use if `mappings` does not find a match.

`when`::
A condition that must succeed in order to execute the current rule. Refer to <<processor-conditions,conditions>> in the Elastic Agent processor syntax for condition descriptions. Currently the equals, contains, and regexp conditions are available.

As an example for setting up your processors, you might want to route log events based on severity. To do so, you can specify a default topic for all events not matched by other processors:

[source,yaml]
----
outputs:
kafka-output:
type: kafka
hosts:
- 'kafka1:9092'
- 'kafka2:9092'
- 'kafka3:9092'
topics:
- topic: 'critical-%{[agent.version]}'
when:
contains:
message: ' “CRITICAL”'
- topic: 'error-%{[agent.version]}'
when:
contains:
message: ' “ERR”'
- topic: '%{[fields.log_topic]}'
----

All non-critical and non-error events will then route to the default `%{[fields.log_topic]}` topic.

|===

Expand Down