The {es} output sends events directly to {es} by using the {es} HTTP API.
Compatibility: This output works with all compatible versions of {es}. See the Elastic Support Matrix.
This example configures an {es} output called default in the
elastic-agent.yml file:
outputs:
default:
type: elasticsearch
hosts: [127.0.0.1:9200]
username: elastic
password: changemeThis example is similar to the previous one, except that it uses the recommended token-based (API key) authentication:
outputs:
default:
type: elasticsearch
hosts: [127.0.0.1:9200]
api_key: "my_api_key"|
Note
|
Token-based authentication is required in an {serverless-full} environment. |
The elasticsearch output type supports the following settings, grouped by
category. Many of these settings have sensible defaults that allow you to run
{agent} with minimal configuration.
Setting |
|||
|
(list) The list of {es} nodes to connect to. The events are distributed to
these nodes in round robin order. If one node becomes unreachable, the event is
automatically sent to another node. Each {es} node can be defined as a
outputs:
default:
type: elasticsearch
hosts: ["10.45.3.2:9220", "10.45.3.1:9230"] (1)
protocol: https
path: /elasticsearch
|
||
|
(string) The name of the protocol {es} is reachable on. The options are:
|
||
|
(boolean) If set to Default: |
||
|
(string) Additional headers to send to proxies during CONNECT requests. |
||
|
(string) The URL of the proxy to use when connecting to the {es} servers. The
value may be either a complete URL or a |
When sending data to a secured cluster through the elasticsearch
output, {agent} can use any of the following authentication methods:
outputs:
default:
type: elasticsearch
hosts: ["https://myEShost:9200"]
username: "your-username"
password: "your-password"| Setting | Description |
|---|---|
|
(string) The basic authentication password for connecting to {es}. |
|
(string) The basic authentication username for connecting to {es}. This user needs the privileges required to publish events to {es}. Note that in an {serverless-full} environment you need to use token-based (API key) authentication. |
outputs:
default:
type: elasticsearch
hosts: ["https://myEShost:9200"]
api_key: "KnR6yE41RrSowb0kQ0HWoA"| Setting | Description |
|---|---|
|
(string) Instead of using a username and password, you can use {kibana-ref}/api-keys.html[API keys] to
secure communication with {es}. The value must be the ID of the API key and the
API key joined by a colon: |
outputs:
default:
type: elasticsearch
hosts: ["https://myEShost:9200"]
ssl.certificate: "/etc/pki/client/cert.pem"
ssl.key: "/etc/pki/client/cert.key"For a list of available settings, refer to [elastic-agent-ssl-configuration], specifically the settings under [common-ssl-options] and [client-ssl-options].
The following encryption types are supported:
-
aes128-cts-hmac-sha1-96
-
aes128-cts-hmac-sha256-128
-
aes256-cts-hmac-sha1-96
-
aes256-cts-hmac-sha384-192
-
des3-cbc-sha1-kd
-
rc4-hmac
Example output config with Kerberos password-based authentication:
outputs:
default:
type: elasticsearch
hosts: ["http://my-elasticsearch.elastic.co:9200"]
kerberos.auth_type: password
kerberos.username: "elastic"
kerberos.password: "changeme"
kerberos.config_path: "/etc/krb5.conf"
kerberos.realm: "ELASTIC.CO"The service principal name for the {es} instance is constructed from these options. Based on this configuration, the name would be:
HTTP/my-elasticsearch.elastic.co@ELASTIC.CO
Settings used to parse, filter, and transform data.
Setting |
Description |
||
|
(string) A format string value that specifies the {ref}/ingest.html[ingest pipeline] to write events to. outputs:
default:
type: elasticsearchoutput.elasticsearch:
hosts: ["http://localhost:9200"]
pipeline: my_pipeline_idYou can set the ingest pipeline dynamically by using a format string to
access any event field. For example, this configuration uses a custom field,
outputs:
default:
type: elasticsearch hosts: ["http://localhost:9200"]
pipeline: "%{[fields.log_type]}_pipeline"With this configuration, all events with
See the |
||
|
An array of pipeline selector rules. Each rule specifies the {ref}/ingest.html[ingest pipeline]
to use for events that match the rule. During publishing, {agent} uses the first
matching rule in the array. Rules can contain conditionals, format string-based
fields, and name mappings. If the Rule settings:
All the conditions supported by processors are also supported here. The following example sends events to a specific pipeline based on whether the
outputs:
default:
type: elasticsearch hosts: ["http://localhost:9200"]
pipelines:
- pipeline: "warning_pipeline"
when.contains:
message: "WARN"
- pipeline: "error_pipeline"
when.contains:
message: "ERR"The following example sets the pipeline by taking the name returned by the
outputs:
default:
type: elasticsearch
hosts: ["http://localhost:9200"]
pipelines:
- pipeline: "%{[fields.log_type]}"
mappings:
critical: "sev1_pipeline"
normal: "sev2_pipeline"
default: "sev3_pipeline"With this configuration, all events with |
Settings that modify the HTTP requests sent to {es}.
| Setting | Description |
|---|---|
|
Custom HTTP headers to add to each request created by the {es} output. Example: outputs:
default:
type: elasticsearch
headers:
X-My-Header: Header contentsSpecify multiple header values for the same header name by separating them with a comma. |
|
Dictionary of HTTP parameters to pass within the URL with index operations. |
|
(string) An HTTP path prefix that is prepended to the HTTP API calls. This is useful for the cases where {es} listens behind an HTTP reverse proxy that exports the API under a custom prefix. |
The memory queue keeps all events in memory.
The memory queue waits for the output to acknowledge or drop events. If the queue is full, no new events can be inserted into the memory queue. Only after the signal from the output will the queue free up space for more events to be accepted.
The memory queue is controlled by the parameters queue.mem.flush.min_events and queue.mem.flush.timeout. If
queue.mem.flush.timeout is 0s or queue.mem.flush.min_events is 0 or 1 then events can be sent by the output as
soon as they are available. If the output supports a bulk_max_size parameter it controls the
maximum batch size that can be sent.
If queue.mem.flush.min_events is greater than 1 and queue.mem.flush.timeout is greater than 0s, events will only
be sent to the output when the queue contains at least queue.mem.flush.min_events events or the
queue.mem.flush.timeout period has expired. In this mode the maximum size batch that that can be sent by the
output is queue.mem.flush.min_events. If the output supports a bulk_max_size parameter, values of
bulk_max_size greater than queue.mem.flush.min_events have no effect. The value of queue.mem.flush.min_events
should be evenly divisible by bulk_max_size to avoid sending partial batches to the output.
This sample configuration forwards events to the output if 512 events are available or the oldest available event has been waiting for 5s in the queue:
queue.mem.events: 4096
queue.mem.flush.min_events: 512
queue.mem.flush.timeout: 5sSetting |
Settings that may affect performance when sending data through the {es} output.
Use the preset option to automatically configure the group of performance tuning settings to optimize for throughput, scale, latency, or you can select a balanced set of performance specifications.
The performance tuning preset values take precedence over any settings that may be defined separately. If you want to change any setting, set preset to custom and specify the performance tuning settings individually.
| Setting | Description |
|---|---|
|
(string) The number of seconds to wait before trying to reconnect to {es}
after a network error. After waiting Default: |
|
(string) The maximum number of seconds to wait before attempting to connect to {es} after a network error. Default: |
|
(int) The maximum number of events to bulk in a single {es} bulk API index request. Events can be collected into batches. {agent} will split batches larger than
Specifying a larger batch size can improve performance by lowering the overhead of sending events. However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput. Setting Default: Default: |
|
(int) The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped. Set Default: |
|
Configures the full group of performance tuning settings to optimize your {agent} performance when sending data to an {es} output. Refer to [es-output-settings-performance-tuning-settings] for a table showing the group of values associated with any preset, and another table showing EPS (events per second) results from testing the different preset options. Performance tuning preset settings:
Default: |
|
(string) The HTTP request timeout in seconds for the {es} request. Default: |