You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In this example, we have defined three outputs: `elasticsearch1`, `elasticsearch2`, and `logstash`. We have also defined 6 total inputs, four writing to `elasticsearch1` and one each writing to `elasticsearch2` and `logstash`.
48
48
49
49
Agent has divided these inputs among four __components__ (running processes) according to their `type`. Each component is broken up into multiple __units__, an output unit that sends its data to the appropriate destination, and some number of input units that provide data to the output. For example the `filebeat1` component has three, two input and one output.
50
50
51
-
Now let's look at the same setup with the shipper enabled. The output configuration becomes:
52
-
53
-
```yml
54
-
outputs:
55
-
elasticsearch1:
56
-
type: "elasticsearch"
57
-
shipper.enabled: true
58
-
elasticsearch2:
59
-
type: "elasticsearch"
60
-
shipper.enabled: true
61
-
logstash:
62
-
type: "logstash"
63
-
shipper.enabled: true
64
-
```
65
-
66
-
and the rest of the configuration is unchanged.
67
-
68
-
69
-
70
-
This is the same as the previous scenario, but all outputs are configured to use the shipper. In this setup, there are shipper components whose job is to send the data upstream, and input components that send their data to a local shipper rather than managing their own independent queue and network connections.
71
-
72
-
A shipper component also has one output unit and one or more input units, but its input units correspond to the components that write to it rather than to individual data sources.
73
-
74
-
In this example there are _seven_ components, as Agent has created three shippers to manage the connections to each of the three outputs. The `shipper1` component has two input units, since it receives data from both `filebeat1` and `metricbeat` (with two individual data sources each). All this data is queued by the shipper and forwarded upstream to `elasticsearch1`.
75
-
76
51
## Agent policy format
77
52
78
53
The policy is specified in YML, and the basic layout is:
@@ -136,15 +111,7 @@ If present, this field determines whether the output is active. Defaults to true
136
111
137
112
#### `type` (string, required)
138
113
139
-
The output type. If `shipper.enabled: true` isn't set, this value must match one of the entries in the `outputs` field for its inputs' spec files. Otherwise, the `shippers` field for its inputs' spec file must include a shipper type that supports this output. See [Component Specs](component-specs.md) for more details.
140
-
141
-
#### `shipper.enabled` (boolean, removed)
142
-
143
-
If present, this field determines whether this output should be implemented by a Shipper component. Defaults to false. If `shipper.enabled` is `true` and some inputs targeting this output don't support the shipper, a warning should be displayed in the logs and in Fleet UI.
144
-
145
-
#### `shipper.*` (removed)
146
-
147
-
All configuration fields under `shipper` except `enabled` are reserved for possible future use, and cannot appear in output configurations. In the future, shipper outputs will be the default, and should be presented to the user as just "the output configuration," so the preference with new shipper features should be to evolve configurations by adding unique top-level fields as with any new output feature, rather than exposing the implementation. (However, this configuration subtree is reserved specifically so that we can revisit this policy if our needs change in the future.)
114
+
The output type.
148
115
149
116
#### `log_level` (string, removed)
150
117
@@ -158,32 +125,3 @@ The log level for this component. This field is removed from the raw configurati
158
125
#### `headers` (`map[string]string`)
159
126
160
127
Agent does not use this field itself, however if the output's `type` is `elasticsearch` then Agent will insert any headers it acquired during Fleet enrollment into this field.
161
-
162
-
163
-
### Shipper-specific fields
164
-
165
-
When components use the shipper, it results in units that don't correspond directly to a configuration entry in the policy. A component that writes to the shipper will be given an output unit that targets the shipper, and a shipper component will be given input units detailing the components that will connect to it.
166
-
167
-
#### Shipper output fields
168
-
169
-
The output unit of a component that writes to a shipper is given the following configuration:
170
-
171
-
- `type` (string): the shipper type
172
-
- `server` (string): the connection address of the shipper (a named socket on Darwin/Linux, a named pipe on Windows)
173
-
- `ssl.certificate_authorities` (string list): a list consisting of one element, which is the certificate authority the shipper will use to verify clients that connect to it. Each shipper instance is assigned its own unique certificate authority on startup.
174
-
- `ssl.certificate` (string): the certificate to present when connecting to the shipper, signed by the CA in `ssl.certificate_authorities`.
175
-
- `ssl.key` (string): the key for `ssl.certificate`
176
-
177
-
#### Shipper input fields
178
-
179
-
For each component that writes to a shipper, the shipper will be given an input unit with the following configuration:
180
-
181
-
- `id` (string): the id of the input unit, which is also the id of the originating component.
182
-
- `type` (string): the shipper type.
183
-
- `units` (list): a list of all configuration units in the originating component, with each containing:
184
-
* `id` (string): the unit id
185
-
* `config`: the full configuration tree for that unit
186
-
- `server` (string): the address the server should listen on for connections from this component (a named socket on Darwin/Linux, a named pipe on Windows). This value is the same for all units.
187
-
- `ssl.certificate_authorities` (string list): a list with one entry, which is this shipper's assigned certificate authority. This value is the same for all units. Clients connecting to the shipper will present certificates signed by this CA.
188
-
- `ssl.certificate` (string): the certificate this client will present when connecting to the shipper.
Copy file name to clipboardexpand all lines: docs/architecture.md
+2-2
Original file line number
Diff line number
Diff line change
@@ -1,11 +1,11 @@
1
1
# Elastic Agent V2 Architecture
2
2
3
3
## Overview
4
-
The Elastic Agent V2 architecture was introduced in the 8.6 version of the Elastic Agent. It was a large internal change of how the Elastic Agent was designed and how it operated. The change was performed to allow better status reporting of each running input, enabled support for shippers, improved the specification definition, and allowed for better parallel change actions.
4
+
The Elastic Agent V2 architecture was introduced in the 8.6 version of the Elastic Agent. It was a large internal change of how the Elastic Agent was designed and how it operated. The change was performed to allow better status reporting of each running input, improved the specification definition, and allowed for better parallel change actions.
5
5
## Component Model
6
6
The entire logic of the Elastic Agent V2 architecture works with what is called the Component Model. The Elastic Agent uses policy to generate a Component Model and then that Component Model is used to ensure a consistency between the currently observed state and the expected state (which is the Component Model).
7
7
### Component
8
-
A component to Elastic Agent is anything that the Elastic Agent must run, operate, and observe during the life of the Elastic Agent. Component is a rather generic term for a good reason. A component can be multiple different types of running code that works with the Elastic Agent. The first example of this is the elastic-agent-shipper which is not an input. A component also always has units, see units below for description of a unit.
8
+
A component to Elastic Agent is anything that the Elastic Agent must run, operate, and observe during the life of the Elastic Agent. Component is a rather generic term for a good reason. A component can be multiple different types of running code that works with the Elastic Agent. A component also always has units, see units below for description of a unit.
9
9
### Unit
10
10
A unit to Elastic Agent is a unique block of configuration that is passed to a component over the control protocol. The block of configuration only represents either one input or one output. It is possible for a component to be multiple units of inputs (aka. running multiple filestream inputs), or currently only one output, but designed to support the possibility of multiple outputs. Using multiple blocks of configuration into a set of units allows the Elastic Agent control protocol and the spawned component’s to communicate a unique set of changes as well as the status of each unit.
Copy file name to clipboardexpand all lines: docs/component-specs.md
+10-19
Original file line number
Diff line number
Diff line change
@@ -13,34 +13,29 @@ inputs:
13
13
...
14
14
- name: <input name 2>
15
15
...
16
-
shippers:
17
-
- name: <shipper name 1>
18
-
...
19
16
```
20
17
21
18
The `version` key must be present and must equal 2 (to distinguish from the older version 1 schema that is no longer supported).
22
19
23
-
`inputs`is a list of input types this component can run, and `shippers` is a list of shipper types this component can run. Each configured input and shipper also has its own list of `outputs` that it supports, but the spec file only tracks the list of supported types, and the rest comes from the [Agent policy](agent-policy.md).
24
-
25
-
Most configuration fields are shared between inputs and shippers. The next section lists all valid fields, noting where there are differences between the two cases.
20
+
`inputs`is a list of input types this component can run. Each configured input also has its own list of `outputs` that it supports, but the spec file only tracks the list of supported types, and the rest comes from the [Agent policy](agent-policy.md).
26
21
27
-
## Input / Shipper configuration fields
22
+
## Input configuration fields
28
23
29
24
### `name` (string, required)
30
25
31
-
The name of this input or shipper. This name must be unique for each platform, however two inputs or shippers that support different platforms can have the same name. This allows the configuration to vary between platforms.
26
+
The name of this input. This name must be unique for each platform, however two inputs that support different platforms can have the same name. This allows the configuration to vary between platforms.
32
27
33
28
### `aliases` (list of strings, input only)
34
29
35
30
Inputs may specify a list of alternate names that policies can use to refer to them. Any occurrence of these aliases in a policy configuration will be replaced with the value of `name`.
36
31
37
32
### `description` (string, required)
38
33
39
-
A short description of this input or shipper.
34
+
A short description of this input.
40
35
41
36
### `platforms` (list of strings, required)
42
37
43
-
The platforms this input or shipper supports. Must contain one or more of the following:
38
+
The platforms this input supports. Must contain one or more of the following:
44
39
- `container/amd64`
45
40
- `container/arm64`
46
41
- `darwin/amd64`
@@ -51,7 +46,7 @@ The platforms this input or shipper supports. Must contain one or more of the fo
51
46
52
47
### `outputs` (list of strings)
53
48
54
-
The output types this input or shipper supports. If this is an input, then inputs of this type can only target (non-shipper) output types in this list. If this is a shipper, then this shipper can only implement output types in this list.
49
+
The output types this input supports.
55
50
56
51
### `proxied_actions` (list of strings)
57
52
@@ -64,13 +59,9 @@ proxied_actions:
64
59
- UPGRADE
65
60
```
66
61
67
-
### `shippers` (list of strings, input only)
68
-
69
-
The shipper types this input supports. Inputs of this type can target any output type supported by the shippers in this list, as long as the output policy includes `shipper.enabled: true`. If an input supports more than one shipper implementing the same output type, then Agent will prefer the one that appears first in this list.
70
-
71
62
### `runtime.preventions`
72
63
73
-
The `runtime.preventions` field contains a list of [EQL conditions](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-syntax.html#eql-syntax-conditions) which should prevent the use of this input or shipper if any are true. Each prevention should include a `condition` in EQL syntax and a `message` that will be displayed if the condition prevents the use of a component.
64
+
The `runtime.preventions` field contains a list of [EQL conditions](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-syntax.html#eql-syntax-conditions) which should prevent the use of this input if any are true. Each prevention should include a `condition` in EQL syntax and a `message` that will be displayed if the condition prevents the use of a component.
74
65
75
66
Here are some example preventions taken from the Endpoint spec file:
76
67
@@ -94,9 +85,9 @@ The variables that can be accessed by a condition are:
94
85
- `user.root`: true if Agent is being run with root / administrator permissions.
95
86
- `install.in_default`: true if the Agent is installed in the default location or has been installed via deb or rpm.
96
87
97
-
### `command` (required for shipper)
88
+
### `command`
98
89
99
-
The `command` field determines how the component will be run. Shippers must include this field, while inputs must include either `command` or `service`. `command` consists of the following subfields:
90
+
The `command` field determines how the component will be run. Inputs must include either `command` or `service`. `command` consists of the following subfields:
100
91
101
92
#### `command.args` (list of strings)
102
93
@@ -150,7 +141,7 @@ Some components (particularly Beats) terminate when they receive a new configura
150
141
151
142
### `service` (input only)
152
143
153
-
Inputs that are run as a system service (like Endpoint Security) can use `service` instead of `command` to indicate that Agent should only monitor them, not manage their execution. `service` consists of the following subfields:
144
+
Inputs that are run as a system service (like Endpoint Security) can use `service` instead of `command` to indicate that Agent should only monitor them, not manage their execution. `service` consists of the following subfields:
0 commit comments