Skip to content

Commit f250533

Browse files
mlunadiamlunadia
authored
Merge pull request #1 from mdbirnstiehl/otel-docs-update
Create Elastic OpenTelemetry Collector docs
2 parents 3034076 + db85c0f commit f250533

9 files changed

+256
-2
lines changed

‎README.md

+35-2
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,35 @@
1-
# opentelemetry
2-
Get started with our Elastic Distros of OpenTelemetry
1+
# Elastic Distributions for OpenTelemetry
2+
3+
[OpenTelemetry](https://opentelemetry.io/docs/) is a vendor-neutral observability framework for collecting, processing, and exporting telemetry data.
4+
The Elastic Distribution for OpenTelemetry Collector (Elastic OTel Collector) is a supported, drop-in replacement distribution of the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) made up of mostly upstream components.
5+
You can send your telemetry data to Elastic Observability using OpenTelemetry the following ways:
6+
7+
- Collect and send logs and host metrics to [Elastic Cloud](https://cloud.elastic.co/) using the Elastic OTel Collector.
8+
- Instrument your applications and send logs, traces, and metrics to [Elastic Cloud](https://cloud.elastic.co/) using the Elastic Distributions for select [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/). Currently, Elastic provides distributions for the following language SDKs: Java, .NET, Node.js, and Python.
9+
- Build and configure a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or extend the [OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution to collect logs and metrics and send them to Elastic Observability.
10+
11+
This diagram provides a quick overview on how the different components work together. Refer to the [components](docs/collector-components.md) for a more in-depth look.
12+
13+
![Diagram of the OpenTelemetry flow](docs/images/elastic-otel-overview.png)
14+
15+
## Collect infrastructure data with the Elastic Distribution for OpenTelemetry Collector
16+
17+
These pages detail the components and how to configure the Elastic OTel Collector.
18+
19+
- [Components](docs/collector-components.md): Get details on the components used to receive, process, and export telemetry data.
20+
- [Guided onboarding](docs/guided-onboarding.md): Use the guided onboarding in Kibana or in a serverless Observability project to send data using the Elastic OTel Collector.
21+
- [Manual configurations](docs/manual-configuration.md): Manually configure the Elastic OTel Collector to send data to Elastic Observability.
22+
- [Limitations](docs/collector-limitations.md): Understand the current limitations of the Elastic OTel Collector.
23+
24+
## Collect application data with Elastic Distributions for OpenTelemetry language SDKs
25+
26+
Elastic offers several Distributions that extend [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/). The following languages are currently supported:
27+
28+
* [Java](https://github.com/elastic/elastic-otel-java)
29+
* [.NET](https://github.com/elastic/elastic-otel-dotnet)
30+
* [Node.js](https://github.com/elastic/elastic-otel-node)
31+
* [Python](https://github.com/elastic/elastic-otel-python)
32+
33+
## Configure a custom collector or the OpenTelemetry Collector Contrib distribution
34+
35+
[Configure a custom collector or the OpenTelemetry Collector Contrib distribution](docs/configure-upstream-collector.md): Build and configure a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or extend the [OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution to collect logs and metrics and send them to Elastic Observability.

‎docs/collector-components.md

+131
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
# Elastic Distribution for OpenTelemetry Collector components
2+
3+
The OpenTelemetry Collector uses the following components to receive, process, and export telemetry data:
4+
5+
- [Receivers](collector-components.md#receivers): Collect telemetry from your host.
6+
- [Processors](collector-components.md#processors): Modify or transform telemetry data before sending it to the exporters.
7+
- [Exporters](collector-components.md#exporters): Send data to the backends or destinations.
8+
- [Extensions](collector-components.md#extensions): Provide additional functionalities and capabilities.
9+
10+
The default configurations of the Elastic Distribution for the OpenTelemetry Collector follows these flows.
11+
12+
**MacOS and Linux Host metrics:**
13+
14+
```mermaid
15+
flowchart LR
16+
one["`Host metrics receiver`"]
17+
two["`Elastic infra metrics processor`"]
18+
three["`Attributes processor (dataset)`"]
19+
four["`Resource processor (process)`"]
20+
five["`Elasticsearch exporter`"]
21+
one --> two --> three --> four --> five
22+
23+
style one fill: #e6f9f7,stroke:#333,stroke-width:1px
24+
style two fill: ##f8e9e9,stroke:#333,stroke-width:1px
25+
style three fill: ##f8e9e9,stroke:#333,stroke-width:1px
26+
style four fill: ##f8e9e9,stroke:#333,stroke-width:1px
27+
style five fill:#e6f9f7,stroke:#333,stroke-width:1px
28+
```
29+
30+
**MacOS and Linux Logs**
31+
32+
```mermaid
33+
flowchart LR
34+
one["`File log receiver`"]
35+
two["`Resource detection processor`"]
36+
three["`Elasticsearch exporter`"]
37+
one --> two --> three
38+
39+
style one fill: #e6f9f7,stroke:#333,stroke-width:1px
40+
style two fill: ##f8e9e9,stroke:#333,stroke-width:1px
41+
style three fill:#e6f9f7,stroke:#333,stroke-width:1px
42+
```
43+
44+
**Kubernetes metrics**
45+
46+
```mermaid
47+
flowchart LR
48+
one["`Kubelet stats and host metrics receivers`"]
49+
two["`Kubernetes attributes processor`"]
50+
three["`Elastic infra metrics processor`"]
51+
four["`Resource detection processors (EKS, GCP, K8s)`"]
52+
five["`Resource processors (K8s, cloud)`"]
53+
six["`Attributes processor (dataset)`"]
54+
seven["`Resource processor`"]
55+
eight["`Elasticsearch exporter`"]
56+
one --> two --> three --> four --> five --> six --> seven --> eight
57+
58+
style one fill: #e6f9f7,stroke:#333,stroke-width:1px
59+
style two fill: ##f8e9e9,stroke:#333,stroke-width:1px
60+
style three fill: ##f8e9e9,stroke:#333,stroke-width:1px
61+
style four fill: ##f8e9e9,stroke:#333,stroke-width:1px
62+
style five fill: ##f8e9e9,stroke:#333,stroke-width:1px
63+
style six fill: ##f8e9e9,stroke:#333,stroke-width:1px
64+
style seven fill: ##f8e9e9,stroke:#333,stroke-width:1px
65+
style eight fill:#e6f9f7,stroke:#333,stroke-width:1px
66+
```
67+
68+
**Kubernetes, MacOS, and Linux logs**
69+
```mermaid
70+
flowchart LR
71+
one["`File log receiver`"]
72+
two["`Kubernetes attributes processor`"]
73+
three["`Resource detection processors (system, EKS, GCP)`"]
74+
four["`Resource processors (K8s, cloud)`"]
75+
five["`Attributes processor (k8s_logs_dataset)`"]
76+
six["`Elasticsearch and debug exporters`"]
77+
one --> two --> three --> four --> five --> six
78+
79+
style one fill: #e6f9f7,stroke:#333,stroke-width:1px
80+
style two fill: ##f8e9e9,stroke:#333,stroke-width:1px
81+
style three fill: ##f8e9e9,stroke:#333,stroke-width:1px
82+
style four fill: ##f8e9e9,stroke:#333,stroke-width:1px
83+
style five fill: ##f8e9e9,stroke:#333,stroke-width:1px
84+
style six fill: #e6f9f7,stroke:#333,stroke-width:1px
85+
```
86+
87+
Refer to the following tables for more information on the components included in the Elastic OTel Collector.
88+
Follow the links for OpenTelemetry documentation with more configuration details for each component.
89+
To set up the Elastic OTel Collector, get started using the [guided onboarding](guided-onboarding.md) docs or the [manual configuration](manual-configuration.md) docs.
90+
91+
## Receivers
92+
93+
| Component | Description |
94+
|---|---|
95+
| [`filelogreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/filelogreceiver/v0.105.0/receiver/filelogreceiver/README.md) | Collects logs from files on the local filesystem, supporting various formats and log rotation strategies. |
96+
| [`hostmetricsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/hostmetricsreceiver/v0.105.0/receiver/hostmetricsreceiver/README.md) | Collects metrics from the host machine, such as CPU, memory, disk, and network usage. |
97+
| [`httpcheckreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/httpcheckreceiver/v0.105.0/receiver/httpcheckreceiver/README.md) | Performs HTTP checks to monitor the availability and response time of web services. |
98+
| [`k8sclusterreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/k8sclusterreceiver/v0.105.0/receiver/k8sclusterreceiver/README.md) | Gathers metrics and metadata from a Kubernetes cluster. |
99+
| [`k8sobjectsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/k8sobjectsreceiver/v0.105.0/receiver/k8sobjectsreceiver/README.md) | Monitors changes to Kubernetes objects, and collects related metrics. |
100+
| [`kubeletstatsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/kubeletstatsreceiver/v0.105.0/receiver/kubeletstatsreceiver/README.md) | Collects metrics from the Kubelet, including node and pod-level resource usage. |
101+
| [`otlpreceiver`](https://github.com/open-telemetry/opentelemetry-collector/blob/receiver/otlpreceiver/v0.105.0/receiver/otlpreceiver/README.md) | Receives metrics, traces, and logs in OpenTelemetry Protocol (OTLP) format. |
102+
103+
## Processors
104+
105+
| Component | Description |
106+
|---|---|
107+
| [`elasticinframetricsprocessor`](https://github.com/elastic/opentelemetry-collector-components/blob/processor/elasticinframetricsprocessor/v0.7.1/processor/elasticinframetricsprocessor/README.md) | Processes infrastructure metrics to enhance and convert them for Elasticsearch. |
108+
| [`attributesprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/attributesprocessor/v0.105.0/processor/attributesprocessor/README.md) | Modifies telemetry data attributes. |
109+
| [`filterprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/filterprocessor/v0.105.0/processor/filterprocessor/README.md) | Filters telemetry data to include or exclude specific data points. |
110+
| [`k8sattributesprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/k8sattributesprocessor/v0.105.0/processor/k8sattributesprocessor/README.md) | Enhances telemetry data with Kubernetes-specific metadata. |
111+
| [`resourcedetectionprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/resourcedetectionprocessor/v0.105.0/processor/resourcedetectionprocessor/README.md) | Detects resource attributes and adds them to telemetry data. |
112+
| [`resourceprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/resourceprocessor/v0.105.0/processor/resourceprocessor/README.md) | Allows resource attributes to be modified in telemetry data. |
113+
| [`transformprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/transformprocessor/v0.105.0/processor/transformprocessor/README.md) | Transforms telemetry data modifying based on specified rules. |
114+
| [`batchprocessor`](https://github.com/open-telemetry/opentelemetry-collector/blob/processor/batchprocessor/v0.105.0/processor/batchprocessor/README.md) | Batches telemetry data to improve export performance and manage load on back-end systems. |
115+
116+
## Exporters
117+
118+
| Component | Description |
119+
|---|---|
120+
| [`elasticsearchexporter`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/exporter/elasticsearchexporter/v0.105.0/exporter/elasticsearchexporter/README.md) | Sends collected telemetry data to Elasticsearch for storage and analysis. |
121+
| [`fileexporter`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/exporter/fileexporter/v0.105.0/exporter/fileexporter/README.md) | Writes telemetry data to a file, useful for debugging or offline analysis. |
122+
| [`debugexporter`](https://github.com/open-telemetry/opentelemetry-collector/blob/exporter/debugexporter/v0.105.0/exporter/debugexporter/README.md) | Outputs telemetry data in a human-readable format for debugging purposes. |
123+
| [`otlpexporter`](https://github.com/open-telemetry/opentelemetry-collector/blob/exporter/otlpexporter/v0.105.0/exporter/otlpexporter/README.md) | Sends telemetry data in OTLP format to a specified endpoint. |
124+
| [`otlphttpexporter`](https://github.com/open-telemetry/opentelemetry-collector/blob/exporter/otlphttpexporter/v0.105.0/exporter/otlphttpexporter/README.md) | Sends telemetry data using HTTP with OTLP. |
125+
126+
## Extensions
127+
128+
| Component | Description |
129+
|---|---|
130+
| [`filestorage`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/storage/filestorage/v0.105.0/extension/storage/filestorage/README.md)| Provides file-based storage for temporary data, such as checkpoints and intermediate state. |
131+
| [`memorylimiterextension`](https://github.com/open-telemetry/opentelemetry-collector/blob/extension/memorylimiterextension/v0.105.0/extension/memorylimiterextension/README.md) | Limits the memory usage of the collector to prevent out-of-memory errors. |

‎docs/collector-limitations.md

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Elastic Distribution for OpenTelemetry Collector limitations
2+
3+
The Elastic Distribution for the OpenTelemetry Collector has the following limitations:
4+
5+
- Because of an upstream limitation, `host.network.*` metrics aren't present from the OpenTelemetry side.
6+
- `process.state` isn't present in the OpenTelemetry host metric. It's set to a dummy value of **Unknown** in the **State** column of the host processes table.
7+
- The Elasticsearch exporter handles the resource attributes, but **Host OS version** and **Operating system** may show as "N/A".
8+
- The CPU scraper needs to be enabled to collect the `systm.load.cores` metric, which affects the **Normalized Load** column in the **Hosts** table and the **Normalized Load** visualization on the host detailed view.
9+
- The [`hostmetrics receiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) doesn't support CPU and disk metrics on MacOS. These values will stay empty for collectors running on MacOS.
10+
- The console shows error Log messages when the [`hostmetrics receiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) can't access some of the process information due to permission issues.
11+
- The console shows mapping errors initially until mapping occurs.

‎docs/configure-upstream-collector.md

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# Configure a custom collector or the OpenTelemetry Collector Contrib distribution
2+
3+
You can build and configure a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or extend the [OpenTelemetry Collector Contrib ](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution to collect logs and metrics and send them to Elastic Observability.
4+
5+
For a more seamless experience, use the Elastic Distribution for the OpenTelemetry collector.
6+
Refer to the [guided onboarding](guided-onboarding.md) docs or the [manual configuration](manual-configuration.md) docs for more on configuring the Elastic OTel collector.
7+
8+
## Upstream collector configuration examples
9+
Refer to the OpenTelemetry documentation on [building a custom collector](https://opentelemetry.io/docs/collector/custom-collector/) for more on creating an upstream collector.
10+
Use the Elastic [example configurations](https://github.com/elastic/elastic-agent/tree/main/internal/pkg/otel/samples) as a reference when configuring your upstream collector.

‎docs/guided-onboarding.md

+32
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Collect logs and metrics using the guided onboarding
2+
The guided onboarding in Kibana or in a serverless Observability project walks you through collecting logs and metrics using the Elastic Distribution for OpenTelemetry Collector.
3+
To configure the Elastic OTel Collector manually, refer to the [manual configuration](manual-configuration.md) docs.
4+
5+
## Before you begin
6+
The onboarding has the following requirements and limitations:
7+
8+
- The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](https://www.elastic.co/docs/current/serverless/general/assign-user-roles).
9+
- Root privileges on the host are required to run the OpenTelemetry collector used in this quickstart.
10+
- The Elastic OTel Collector only works on Kubernetes, Linux, and MacOS systems.
11+
- Refer to [Elastic OpenTelemetry Collector limitations](collector-limitations.md) for known limitations when using the Elastic OTel Collector.
12+
13+
## Collect your logs and metrics
14+
15+
Follow these steps to collect logs and metrics using the Elastic OTel collector:
16+
17+
1. Open an [Elastic Cloud](cloud.elastic.co) deployment or a serverless Observability project.
18+
1. To open the guided onboarding, either:
19+
1. In an Elastic Cloud deployment, open Kibana, and go to **Observability** → **Add Data**.
20+
1. In a serverless Observability project, go to **Add Data**.
21+
1. Select **Collect and analyze logs**, and then select **OpenTelemetry**.
22+
1. Select the appropriate platform, and complete the following:
23+
1. For **MacOS and Linux**, copy the command, open a terminal on your host, and run the command to download and configure the OpenTelemetry collector.
24+
1. For **Kubernetes**, download the manifest.
25+
1. Copy the command under Step 2:
26+
1. For **MacOS and Linux**, run the command in your terminal to start the Elastic OTel collector.
27+
1. For **Kubernetes**, run the command from the directory where you downloaded the manifest to install the Elastic OTel Collector on every node of your cluster.
28+
29+
Logs are collected from setup onward, so you won't see logs that occurred before starting the Elastic OTel Collector.
30+
The default log path is `/var/log/*`. To update the path, modify `otel.yml`.
31+
32+
Under **Visualize your data**, you'll see links to **Logs Explorer** to view your logs and **Hosts** to view your host metrics.

‎docs/images/elastic-otel-overview.png

57.2 KB
Loading

‎docs/images/infrastructure-elastic-otel-flow.png

43.2 KB
Loading

‎docs/images/kubernetes-elastic-otel-flow.png

81.6 KB
Loading

0 commit comments

Comments
 (0)