Skip to content

Commit fb63391

Browse files
eedugonxrmx
eedugon
and
xrmx
authored
K8s auto-instrumentation details and example (Python) (#58)
* k8s otel python doc added * added python specific details * manual instrumentation link added --------- Co-authored-by: Riccardo Magliocchetti <riccardo.magliocchetti@gmail.com>
1 parent 27cace5 commit fb63391

File tree

2 files changed

+164
-1
lines changed

2 files changed

+164
-1
lines changed

‎docs/kubernetes/operator/instrumenting-applications.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -143,7 +143,7 @@ For details on instrumenting specific languages, refer to:
143143

144144
- [Instrumenting Java](./instrumenting-java.md)
145145
- [Instrumenting Python](./instrumenting-python.md)
146-
- [Instrumenting Dotnet](./instrumenting-dotnet.md)
146+
<!-- - [Instrumenting Dotnet](./instrumenting-dotnet.md) -->
147147

148148
### Namespace based annotations example
149149

‎docs/kubernetes/operator/instrumenting-python.md

+163
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
# Instrumenting Python applications with EDOT SDKs on Kubernetes
2+
3+
This document focuses on instrumenting Python applications on Kubernetes, using the OpenTelemetry Operator, Elastic Distribution of OpenTelemetry (EDOT) Collectors, and the [EDOT Python](https://github.com/elastic/elastic-otel-python) SDK.
4+
5+
- For general knowledge about the EDOT Python SDK, refer to the [getting started guide](https://github.com/elastic/elastic-otel-python/blob/main/docs/get-started.md).
6+
7+
- For Python auto-instrumentation specifics, refer to [OpenTelemetry Operator Python auto-instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#python).
8+
9+
- To manually instrument your Python application code (by customizing spans and metrics), refer to [EDOT Python manual instrumentation](https://github.com/elastic/elastic-otel-python/blob/main/docs/manual-instrumentation.md#manually-instrument-your-auto-instrumented-python-application).
10+
11+
- For general information about instrumenting applications on kubernetes, refer to [instrumenting applications](./instrumenting-applications.md).
12+
13+
## Supported environments and configuration
14+
15+
- EDOT Python container image supports `glibc` and `musl` based auto-instrumentation for Python 3.12.
16+
17+
- `musl` based containers instrumentation requires an [extra annotation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#annotations-python-musl) and operator v0.113.0+.
18+
19+
- To enable logs auto-instrumentation, refer to [auto-instrument python logs](https://opentelemetry.io/docs/kubernetes/operator/automatic/#auto-instrumenting-python-logs).
20+
21+
- To disable specific instrumentation libraries, refer to [excluding auto-instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#python-excluding-auto-instrumentation).
22+
23+
- For a full list of configuration options, refer to [Python specific configuration](https://opentelemetry.io/docs/zero-code/python/configuration/#python-specific-configuration).
24+
25+
- For Python specific limitations when using the OpenTelemetry operator, refer to [Python-specific topics](https://opentelemetry.io/docs/zero-code/python/operator/#python-specific-topics).
26+
27+
## Instrument a Python app with EDOT Python SDK on Kubernetes
28+
29+
In this example, you'll learn how to:
30+
31+
- Enable auto-instrumentation of a Python application using one of the following supported methods:
32+
- Adding an annotation to the deployment Pods.
33+
- Adding an annotation to the namespace.
34+
- Verify that auto-instrumentation libraries are injected and configured correctly.
35+
- Confirm data is flowing to **Kibana Observability**.
36+
37+
For this example, we assume the application you're instrumenting is a deployment named `python-app` running in the `python-ns` namespace.
38+
39+
1. Ensure you have successfully [installed the OpenTelemetry Operator](./README.md), and confirm that the following `Instrumentation` object exists in the system:
40+
41+
```bash
42+
$ kubectl get instrumentation -n opentelemetry-operator-system
43+
NAME AGE ENDPOINT
44+
elastic-instrumentation 107s http://opentelemetry-kube-stack-daemon-collector.opentelemetry-operator-system.svc.cluster.local:4318
45+
```
46+
> [!NOTE]
47+
> If your `Instrumentation` object has a different name or is created in a different namespace, you will have to adapt the annotation value in the next step.
48+
49+
2. Enable auto-instrumentation of the Python application using one of the following methods:
50+
51+
- Edit your application workload definition and include the annotation under `spec.template.metadata.annotations`:
52+
53+
```yaml
54+
spec:
55+
...
56+
template:
57+
metadata:
58+
labels:
59+
app: python-app
60+
annotations:
61+
instrumentation.opentelemetry.io/inject-python: opentelemetry-operator-system/elastic-instrumentation
62+
...
63+
```
64+
65+
- Alternatively, add the annotation at namespace level to apply auto-instrumentation in all Pods of the namespace:
66+
67+
```bash
68+
kubectl annotate namespace python-ns instrumentation.opentelemetry.io/inject-python=opentelemetry-operator-system/elastic-instrumentation
69+
```
70+
71+
3. Restart application:
72+
73+
Once the annotation has been set, restart the application to create new Pods and inject the instrumentation libraries:
74+
75+
```bash
76+
kubectl rollout restart deployment python-app -n python
77+
```
78+
79+
4. Verify the [auto-instrumentation resources](./instrumenting-applications.md#how-auto-instrumentation-works) are injected in the Pod:
80+
81+
Run a `kubectl describe` of one of your application pods and check:
82+
83+
- There should be an init container named `opentelemetry-auto-instrumentation-python` in the Pod:
84+
85+
```bash
86+
$ kubectl describe pod python-app-8d84c47b8-8h5z2 -n python
87+
...
88+
...
89+
Init Containers:
90+
opentelemetry-auto-instrumentation-python:
91+
Container ID: containerd://fdc86b3191e34ef5ec872853b14a950d0af1e36b0bc207f3d59bd50dd3caafe9
92+
Image: docker.elastic.co/observability/elastic-otel-python:0.3.0
93+
Image ID: docker.elastic.co/observability/elastic-otel-python@sha256:de7b5cce7514a10081a00820a05097931190567ec6e18a384ff7c148bad0695e
94+
Port: <none>
95+
Host Port: <none>
96+
Command:
97+
cp
98+
-r
99+
/autoinstrumentation/.
100+
/otel-auto-instrumentation-python
101+
State: Terminated
102+
Reason: Completed
103+
...
104+
```
105+
106+
- The main container has new environment variables, including `PYTHONPATH`:
107+
108+
```bash
109+
...
110+
Containers:
111+
python-app:
112+
...
113+
Environment:
114+
...
115+
PYTHONPATH: /otel-auto-instrumentation-python/opentelemetry/instrumentation/auto_instrumentation:/otel-auto-instrumentation-python
116+
OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
117+
OTEL_TRACES_EXPORTER: otlp
118+
OTEL_METRICS_EXPORTER: otlp
119+
OTEL_SERVICE_NAME: python-app
120+
OTEL_EXPORTER_OTLP_ENDPOINT: http://opentelemetry-kube-stack-daemon-collector.opentelemetry-operator-system.svc.cluster.local:4318
121+
...
122+
```
123+
124+
- The Pod has an `EmptyDir` volume named `opentelemetry-auto-instrumentation-python` mounted in both the main and the init containers in path `/otel-auto-instrumentation-python`:
125+
126+
```bash
127+
Init Containers:
128+
opentelemetry-auto-instrumentation-python:
129+
...
130+
Mounts:
131+
/otel-auto-instrumentation-python from opentelemetry-auto-instrumentation-python (rw)
132+
Containers:
133+
python-app:
134+
...
135+
Mounts:
136+
/otel-auto-instrumentation-python from opentelemetry-auto-instrumentation-python (rw)
137+
...
138+
Volumes:
139+
...
140+
opentelemetry-auto-instrumentation-python:
141+
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
142+
```
143+
144+
Ensure the environment variable `OTEL_EXPORTER_OTLP_ENDPOINT` points to a valid endpoint and there's network communication between the Pod and the endpoint.
145+
146+
5. Confirm data is flowing through in **Kibana**:
147+
148+
5. Confirm data is flowing to **Kibana**:
149+
150+
- Open **Observability** -> **Applications** -> **Service inventory**, and determine if:
151+
- The application appears in the list of services.
152+
- The application shows transactions and metrics.
153+
- If [python logs instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#auto-instrumenting-python-logs) is enabled, the application logs should appear in the Logs tab.
154+
155+
- For application container logs, open **Kibana Discover** and filter for your Pods' logs. In the provided example, we could filter for them with either of the following:
156+
- `k8s.deployment.name: "python-app"` (**adapt the query filter to your use case**)
157+
- `k8s.pod.name: python-app*` (**adapt the query filter to your use case**)
158+
159+
Note that the container logs are not provided by the instrumentation library, but by the DaemonSet collector deployed as part of the [operator installation](./README.md).
160+
161+
## Troubleshooting
162+
163+
- Refer to [troubleshoot auto-instrumentation](./troubleshoot-auto-instrumentation.md) for further analysis.

0 commit comments

Comments
 (0)