Skip to content

Commit 450098c

Browse files
AlexanderWertAlexanderWert
committed
EDOT Collector config diagrams
Signed-off-by: Alexander Wert <alexander.wert@elastic.co>
1 parent f321c9d commit 450098c

11 files changed

+520
-65
lines changed

‎docs/_config.yml

+4
Original file line numberDiff line numberDiff line change
@@ -75,3 +75,7 @@ edot_versions:
7575

7676
exclude:
7777
- "gen_edot_col_components/"
78+
79+
mermaid:
80+
# Version of mermaid library
81+
version: "11.5.0"

‎docs/_edot-collector/config/customize-data-ingestion.md

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
title: Customize Data Ingestion
3+
parent: Configuration
4+
layout: default
5+
nav_order: 5
6+
---
7+
8+
🚧 Coming soon
9+
10+
- data routing (example split data by K8s namespace into different data streams)
11+
- ECS mode (though not officially supported)

‎docs/_edot-collector/config/customize-logs-collection.md

+89
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
---
2+
title: Customize Logs
3+
parent: Configuration
4+
layout: default
5+
nav_order: 3
6+
---
7+
8+
# Customize Logs Collection
9+
10+
🚧 Coming soon
11+
12+
## Pre-processing Logs
13+
14+
- limitation: (as of 9.0) Elasticsearch Ingest Pipelinges are not (yet) applicable to OTel-native data
15+
- pre-processing of logs needs to happen in OTel collectors
16+
17+
### Parsing JSON logs
18+
19+
```yaml
20+
# ...
21+
receivers:
22+
filelog:
23+
# ...
24+
operators:
25+
# Parse body as JSON https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/json_parser.md
26+
- type: json_parser
27+
on_error: send_quiet
28+
parse_from: body
29+
parse_to: body
30+
31+
# ...
32+
```
33+
34+
### Setting custom fields
35+
36+
## Customizing logs parsing on Kubernetes
37+
38+
TODO: use K8s pods annotation to configure logs parsing, link Blog post
39+
40+
Daemonset collector config:
41+
42+
```yaml
43+
receivers:
44+
receiver_creator/logs:
45+
watch_observers: [k8s_observer]
46+
discovery:
47+
enabled: true
48+
receivers:
49+
50+
# ...
51+
52+
extensions:
53+
k8s_observer:
54+
55+
# ...
56+
57+
service:
58+
extensions: [k8s_observer]
59+
pipelines:
60+
```
61+
62+
- Make sure to remove / comment out the static file log receiver (or restrict the log file pattern) to avoid log duplication
63+
64+
Annotation of the pod
65+
66+
```yaml
67+
# ...
68+
metadata:
69+
annotations:
70+
io.opentelemetry.discovery.logs/enabled: "true"
71+
io.opentelemetry.discovery.logs/config: |
72+
operators:
73+
- id: container-parser
74+
type: container
75+
- id: json-parser
76+
type: json_parser
77+
on_error: send_quiet
78+
parse_from: body
79+
parse_to: body
80+
- id: custom-value
81+
type: add
82+
field: attributes.tag
83+
value: custom-value
84+
spec:
85+
containers:
86+
# ...
87+
```
88+
89+
## Using Processors / OTTL for Logs processing

‎docs/_edot-collector/config/customize-metrics-collection.md

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
title: Customize Metrics
3+
parent: Configuration
4+
layout: default
5+
nav_order: 4
6+
---
7+
8+
# Customize Metrics Collection
9+
10+
🚧 Coming soon

‎docs/_edot-collector/config/default-config-k8s.md

+226
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
---
2+
title: Default Config—Kubernetes
3+
parent: Configuration
4+
layout: default
5+
nav_order: 2
6+
---
7+
8+
# Default Configuration - EDOT Collectors on Kubernetes
9+
{: .no_toc }
10+
11+
The [Kubernetes setup](../../quickstart/index) utilizes the OpenTelemetry Operator to automate orchestration of EDOT Collectors:
12+
13+
* [EDOT Collector Cluster](#cluster-collector): Collection of cluster metrics.
14+
* [EDOT Collector Daemon](#daemonset-collector): Collection of node metrics, logs and application telemetry.
15+
* [EDOT Collector Gateway](#gateway-collectors): Pre-processing, aggregation and ingestion of data into Elastic.
16+
17+
<table>
18+
<thead>
19+
<tr>
20+
<td>Direct ingestion into Elasticsearch</td>
21+
<td>Managed OTLP Endpoint</td>
22+
</tr>
23+
</thead>
24+
<tbody>
25+
<tr>
26+
<td>
27+
<pre class="mermaid">
28+
flowchart LR
29+
cluster@{ shape: proc, label: "Cluster
30+
Collector
31+
fa:fa-microchip"} -->|otlp| gateway@{ shape: procs, label: "Gateway
32+
Collectors
33+
fa:fa-microchip"}
34+
35+
daemon@{ shape: procs, label: "Daemonset
36+
Collectors
37+
fa:fa-microchip"} -->|otlp| gateway
38+
39+
gateway ==>|_bulk| es@{ shape: db, label: "Elasticsearch" }
40+
41+
style es stroke:#33f,stroke-width:2px,color:#000;
42+
</pre>
43+
</td>
44+
<td>
45+
<pre class="mermaid">
46+
flowchart LR
47+
cluster@{ shape: proc, label: "Cluster
48+
Collector
49+
fa:fa-microchip"} -->|otlp| gateway@{ shape: procs, label: "Gateway
50+
Collectors
51+
fa:fa-microchip"}
52+
53+
daemon@{ shape: procs, label: "Daemonset
54+
Collectors
55+
fa:fa-microchip"} -->|otlp| gateway
56+
57+
gateway ==>|otlp| otlp@{ shape: display, label: "Managed
58+
OTLP endpoint" }
59+
60+
style otlp stroke:#33f,stroke-width:2px,color:#000;
61+
</pre>
62+
</td>
63+
</tr>
64+
</tbody>
65+
</table>
66+
67+
The following sections describe the default pipelines for the different roles of EDOT collectors in a Kubernetes setup.
68+
69+
- TOC
70+
{:toc}
71+
72+
## Pipeline - Cluster Collector
73+
74+
```mermaid
75+
flowchart LR
76+
k8s_cluster@{ shape: proc, label: "k8s_cluster
77+
fa:fa-right-to-bracket"} -->|M| k8sattributes@{ shape: proc, label: "k8sattributes
78+
fa:fa-gears"}
79+
80+
k8sattributes -->|M| resourcedetection@{ shape: procs, label: "resourcedetection
81+
fa:fa-gears"}
82+
83+
k8sobjects@{ shape: proc, label: "k8sobjects
84+
fa:fa-right-to-bracket"} -->|L| resourcedetection
85+
86+
resourcedetection -->|L/M| resource@{ shape: procs, label: "resource
87+
fa:fa-gears"}
88+
89+
resource -->|L/M| otlp_exporter@{ shape: proc, label: "OTLP
90+
fa:fa-right-from-bracket"}
91+
```
92+
93+
The main purpose of the `Cluster Collector` is to collect Kubernetes cluster-level metrics (using the [`k8s_cluster`] receiver) and cluster events ([`k8sobjects`] receiver) and forward them to the Gateway Collector through `OTLP`. The [`resource`] and [`resourcedetection`] processors enrich the cluster-level data with corresponding meta information.
94+
95+
96+
## Pipeline - Daemonset Collectors
97+
98+
```mermaid
99+
flowchart LR
100+
otlp@{ shape: proc, label: "OTLP
101+
fa:fa-right-to-bracket"} -->|T/L/M| batch@{ shape: proc, label: "batch
102+
fa:fa-gears"}
103+
104+
batch -->|T/L/M| resource@{ shape: proc, label: "resource
105+
fa:fa-right-from-bracket"}
106+
107+
resource -->|T/L/M| otlp_exporter@{ shape: proc, label: "OTLP
108+
fa:fa-right-from-bracket"}
109+
110+
%% logs pipeline
111+
filelog@{ shape: proc, label: "filelog
112+
fa:fa-right-to-bracket"} -->|L| batch_lm@{ shape: proc, label: "batch
113+
fa:fa-gears"}
114+
115+
batch_lm -->|L/M| k8sattributes@{ shape: proc, label: "k8sattributes
116+
fa:fa-gears"}
117+
118+
k8sattributes -->|L/M| resourcedetection@{ shape: procs, label: "resourcedetection
119+
fa:fa-gears"}
120+
121+
resourcedetection -->|L/M| resource@{ shape: procs, label: "resource
122+
fa:fa-gears"}
123+
124+
resource -->|L/M| otlp_exporter
125+
126+
%% system metrics pipeline
127+
kubletstats@{ shape: proc, label: "kubletstats
128+
fa:fa-right-to-bracket"} -->|M| batch_lm
129+
hostmetrics@{ shape: proc, label: "hostmetrics
130+
fa:fa-right-to-bracket"} -->|M| batch_lm
131+
```
132+
133+
The `Daemonset Collectors` gather telemetry associated with corresponding, individual Kubernetes nodes:
134+
135+
1. *Host metrics and container logs*
136+
137+
[`filelog`] and [`hostmetrics`] receivers are used to gather container logs and host metrics, respectively.
138+
The [`kubletstats`] receiver collects additional Kubernetes Node, Pod and Container metrics.
139+
The logs and metrics are batched for better performance ([`batch`] processor) and then enriched with meta information using the
140+
[`k8sattributes`], [`resourcedetection`] and [`resource`] processors.
141+
142+
2. *Application teleemtry through OTLP from OTel SDKs*
143+
144+
The `Daemonset Collectors` also receive the application telemetry from OTel SDKs that instrument services / pods running on
145+
corresponding Kubernetes nodes. The Daemonset Collectors receive that data through [`OTLP`], batch the data ([`batch`] processor)
146+
and pass it on to the Gateway Collector through the OTLP exporter.
147+
148+
## Pipeline - Gateway Collectors
149+
150+
The `Gateway Collectors` pipelines differ fundamentally between the two different deployment use cases *'Direct ingestion into Elasticsearch'*
151+
and using Elastic's *'Managed OTLP Endpoint'*.
152+
153+
### Direct ingestion into Elasticsearch
154+
155+
In *self-managed* and *Elastic Cloud Hosted* Stack deployment use cases the main purpose of the `Gateway Collector` is the central enrichment of data
156+
before the OpenTelemetry data is being ingested directly into Elasticsearch using the [`elasticsearch`] exporter.
157+
158+
```mermaid
159+
flowchart LR
160+
otlp@{ shape: proc, label: "OTLP
161+
fa:fa-right-to-bracket"} -->|T/L| batch@{ shape: proc, label: "batch
162+
fa:fa-gears"}
163+
batch -->|T| elastictrace@{ shape: proc, label: "elastictrace
164+
fa:fa-gears"}
165+
elastictrace -->|T| es_exporter@{ shape: proc, label: "elasticsearch
166+
fa:fa-right-from-bracket"}
167+
es_exporter -->|otel| otel@{ shape: framed-circle, label: "otel" }
168+
elastictrace -->|T| elasticapm@{ shape: hex, label: "elasticapm
169+
fa:fa-link"}
170+
elasticapm -->|M| es_exporter
171+
172+
batch -->|L| elasticapm
173+
174+
otlp -->|M| routing@{ shape: hex, label: "routing
175+
fa:fa-link"}
176+
routing -->|M| batch
177+
batch -->|L/M| es_exporter
178+
179+
routing -->|"M (infra)"| elasticinframetrics@{ shape: proc, label: "elasticinframetrics
180+
fa:fa-gears"}
181+
elasticinframetrics -->|M| attributes@{ shape: proc, label: "attributes
182+
fa:fa-gears"}
183+
attributes -->|M| resource@{ shape: proc, label: "resource
184+
fa:fa-gears"}
185+
resource -->|M| batch_ecs@{ shape: proc, label: "batch
186+
fa:fa-gears"}
187+
batch_ecs -->|M| es_exporter_ecs@{ shape: proc, label: "elasticsearch
188+
fa:fa-right-from-bracket"}
189+
es_exporter_ecs -->|ecs| ecs@{ shape: framed-circle, label: "ecs" }
190+
```
191+
192+
Hence, this Gateway Collector configuration comprises the pipelines for data enrichment of [application telemetry](./default-config-standalone#application--traces-collection-pipeline) and [host metrics](./default-config-standalone#host-metrics-collection-pipeline) (for details, refer to the linked descriptions of the corresponding standalone use cases).
193+
194+
The [`routing`] connector separates the infrastructure metrics from other metrics and routes them into the ECS-based pipeline, with ECS-compatibility exporter mode.
195+
Other metrics are exported in OTel-native format to Elasticsearch.
196+
197+
### Managed OTLP Endpoint
198+
199+
With the managed OTLP Endpoint, the Gateway Collector configuration simply pipes all the data from the [`OTLP`] receiver through a [`batch`] processor before the data is being exported through `OTLP` to the managed endpoint.
200+
201+
```mermaid
202+
flowchart LR
203+
otlp@{ shape: proc, label: "OTLP
204+
fa:fa-right-to-bracket"} -->|T/L/M| batch@{ shape: proc, label: "batch
205+
fa:fa-gears"}
206+
batch -->|T/L/M| otlp_exporter@{ shape: proc, label: "OTLP
207+
fa:fa-right-from-bracket"}
208+
```
209+
210+
With this scenario there's no need to do any Elastic-specific enrichment in your Kubernetes cluster, as all of that happens behind the managed OTLP endpoint.
211+
212+
[`hostmetrics`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver
213+
[`elasticsearch`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter
214+
[`elasticinframetrics`]: https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/elasticinframetricsprocessor
215+
[`elasticsearch`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter
216+
[`k8s_cluster`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver
217+
[`k8sobjects`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sobjectsreceiver
218+
[`resource`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourceprocessor
219+
[`k8sattributes`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/k8sattributesprocessor
220+
[`resourcedetection`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor
221+
[`filelog`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver
222+
[`hostmetrics`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver
223+
[`kubletstats`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kubletstatsreceiver
224+
[`batch`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/batchprocessor
225+
[`OTLP`]: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver
226+
[`routing`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/routingconnector

0 commit comments

Comments
 (0)