Merge main into 8.16 branch

README.md

@@ -23,11 +23,12 @@ These pages detail the components and how to configure the EDOT Collector.
 - [Manual configurations](docs/manual-configuration.md): Manually configure the EDOT Collector to send data to Elastic Observability.
 - [Limitations](docs/collector-limitations.md): Understand the current limitations of the EDOT Collector.
 
-## Kubernetes Observability using the EDOT Collector
+## Unified Kubernetes Observability with Elastic Distributions of OpenTelemetry
 
-- [Kubernetes guided onboarding](docs/kubernetes/operator/README.md): Use the guided onboarding to send Kubernetes logs, metrics, and application traces to Elasticsearch using the EDOT Collector and [OpenTelemetry Operator](https://github.com/open-telemetry/opentelemetry-operator/).
+- [Kubernetes guided onboarding](docs/kubernetes/operator/README.md): Use the guided onboarding to send Kubernetes logs, metrics, and application traces to Elasticsearch using the [OpenTelemetry Operator](https://github.com/open-telemetry/opentelemetry-operator/) to orchestrate the EDOT Collectors and EDOT SDK instances you will need.
+- [Instrumenting applications](docs/kubernetes/operator/instrumenting-applications.md) on Kubernetes: Collect application data using EDOT language SDKs.
 
-## Collect application data using the EDOT language SDKs
+## Collect application data using EDOT language SDKs
 
 Elastic offers several Distributions that extend [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/). The following languages are currently available:
 

docs/kubernetes/operator/README.md

@@ -21,7 +21,7 @@ This guide describes how to:
 
 - Elastic Stack (self-managed or [Elastic Cloud](https://www.elastic.co/cloud)) version 8.16.0 or higher, or an [Elasticsearch serverless](https://www.elastic.co/docs/current/serverless/elasticsearch/get-started) project.
 
-- A Kubernetes version supported by the OpenTelemetry Operator (refer to the operator's [compatibility matrix](https://github.com/open-telemetry/opentelemetry-operator?#compatibility-matrix) for more details).
+- A Kubernetes version supported by the OpenTelemetry Operator (refer to the operator's [compatibility matrix](https://github.com/open-telemetry/opentelemetry-operator/blob/main/docs/compatibility.md#compatibility-matrix) for more details).
 
 ## Compatibility Matrix
 
@@ -169,9 +169,11 @@ metadata:
     instrumentation.opentelemetry.io/inject-<LANGUAGE>: "opentelemetry-operator-system/elastic-instrumentation"
 ```
 
-where <LANGUAGE> is one of: `go` , `java`, `nodejs`, `python`, `dotnet`
+where ``<LANGUAGE>`` is one of: `go` , `java`, `nodejs`, `python`, `dotnet`
 
-For detailed instructions and examples on how to instrument applications in Kubernetes using the OpenTelemetry Operator, refer to this guide (TBD-add link and document).
+For detailed instructions and examples on how to instrument applications in Kubernetes using the OpenTelemetry Operator, refer to [Instrumenting applications](/docs/kubernetes/operator/instrumenting-applications.md).
+
+For troubleshooing details and verification steps, refer to [Troubleshooting auto-instrumentation](/docs/kubernetes/operator/troubleshoot-auto-instrumentation.md).
 
 ## Limitations
 

docs/kubernetes/operator/instrumenting-applications.md

@@ -0,0 +1,182 @@
+# Instrumenting applications with EDOT SDKs on Kubernetes
+
+Elastic Distributions of OpenTelemetry (EDOT) SDKs cover multiple languages:
+
+* [EDOT Java](https://github.com/elastic/elastic-otel-java)
+* [EDOT .NET](https://github.com/elastic/elastic-otel-dotnet)
+* [EDOT Node.js](https://github.com/elastic/elastic-otel-node)
+* [EDOT Python](https://github.com/elastic/elastic-otel-python)
+* [EDOT PHP](https://github.com/elastic/elastic-otel-php/)
+
+This section provides guidance and examples for applications instrumentation in a Kubernetes environment for all supported languages.
+
+In Kubernetes environments with the OpenTelemetry Operator, [**automatic (or zero-code) instrumentation**](https://opentelemetry.io/docs/kubernetes/operator/automatic/) simplifies the process by injecting and configuring instrumentation libraries into the targeted Pods.
+
+On the other hand, **manual instrumentation** with OpenTelemetry involves adding specific OpenTelemetry SDKs and APIs directly into your application’s code. This approach provides more granular control over what and how data is captured, allowing you to customize trace spans, metrics, and logging based on your application’s logic.
+
+## Table of contents
+
+- [Supported languages](#supported-languages)
+- [Prerequisites](#prerequisites)
+- [Auto-instrumentation basics](#auto-instrumentation-basics)
+- [Configuring auto-instrumentation](#configuring-auto-instrumentation)
+- [Advanced configuration](#advanced-configuration)
+- [Manual instrumentation](#manual-instrumentation)
+
+## Supported languages
+
+The following table illustrates the different languages supported by OpenTelemetry (OTel) and the Elastic Stack, the type of SDK/API used for instrumentation (either zero-code or source code dependencies), and the corresponding deployment types (on-premises, ESS, or serverless) for each language.
+
+| Language   | OTel SDK/API Type                                       | Deployment Model Support             |
+|------------|---------------------------------------------------------|-------------------------------------|
+| Java       | EDOT Java - **zero-code instrumentation**               | All deployment types                |
+| Node.js    | EDOT Node.js - **zero-code instrumentation**            | All deployment types                |
+| .NET       | EDOT .NET - **zero-code instrumentation**               | All deployment types                |
+| PHP        | EDOT PHP - source code dependencies                     | All deployment types                |
+| Python     | EDOT Python - **zero-code instrumentation**             | All deployment types                |
+| Swift      | EDOT Swift - source code dependencies                   | ESS, on-premises                    |
+| Android    | EDOT Android - source code dependencies                 | ESS, on-premises                    |
+| Javascript | Vanilla OTel RUM SDK/API - source code dependencies     | ESS, on-premises                    |
+| Rust       | Vanilla OTel Rust SDK/API - source code dependencies    | All deployment types                |
+| Ruby       | Vanilla OTel Ruby SDK/API - source code dependencies    | All deployment types                |
+| Go         | Vanilla OTel Go SDK/API - **zero-code instrumentation** | All deployment types                |
+| C++        | Vanilla OTel C++ SDK/API - source code dependencies     | All deployment types                |
+
+## Prerequisites
+
+Before starting with application auto-instrumentation, ensure the following prerequisites are in place for proper setup:
+
+- Install the OpenTelemetry operator and EDOT collectors following the [getting started guide](./README.md).
+- Ensure a valid `kind: Instrumentation` object exists in the cluster.
+
+## Auto-instrumentation basics
+
+Zero-code instrumentation is handled by the operator through `Instrumentation` objects, used to automatically inject the necessary SDKs and configuration into application workloads.
+
+If you followed the [getting started guide](./README.md) to install the operator, there should be an `Instrumentation` object with name `elastic-instrumentation` in namespace `opentelemetry-operator-system`:
+
+```bash
+kubectl get instrumentation -A
+NAMESPACE                       NAME                      AGE     ENDPOINT                                                                                                SAMPLER                    SAMPLER ARG
+opentelemetry-operator-system   elastic-instrumentation   5d20h   http://opentelemetry-kube-stack-daemon-collector.opentelemetry-operator-system.svc.cluster.local:4318   parentbased_traceidratio   1.0
+```
+
+The `Instrumentation` object stores important parameters:
+
+- The **exporter endpoint** represents the destination for the traces, in this case the HTTP receiver configured in the EDOT DaemonSet Collector. That endpoint has to be reachable by the Pods being instrumented.
+
+```yaml
+  exporter:
+    endpoint: http://opentelemetry-kube-stack-daemon-collector.opentelemetry-operator-system.svc.cluster.local:4318
+```
+
+- Language-specific **images** used by the operator to inject the appropriate library into each Pod.
+
+```yaml
+  dotnet:
+    image: docker.elastic.co/observability/elastic-otel-dotnet:edge
+  java:
+    image: docker.elastic.co/observability/elastic-otel-javaagent:1.0.0
+  nodejs:
+    image: docker.elastic.co/observability/elastic-otel-node:0.4.1
+  python:
+    image: docker.elastic.co/observability/elastic-otel-python:0.3.0
+```
+
+## Configuring auto-instrumentation
+
+To enable auto-instrumentation, add the corresponding language annotation to the **Pods** template (`spec.template.metadata.annotations`) in your Deployment or relevant workload object (StatefulSet, Job, CronJob, etc.).
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: myapp
+spec:
+  ...
+  template:
+    metadata:
+      annotations:
+        instrumentation.opentelemetry.io/inject-<LANGUAGE>: "opentelemetry-operator-system/elastic-instrumentation"
+      ...
+    spec:
+      containers:
+      - image: myapplication-image
+        name: app
+      ...        
+```
+
+where ``<LANGUAGE>`` is one of: `go` , `java`, `nodejs`, `python`, `dotnet`
+
+> [!NOTE]
+> Ensure you add the annotations at Pod level and not directly at the workload `spec` level (Deployment, Job, etc.).
+> Ensure the annotation value must points to an existing `Instrumentation` object.
+
+Alternatively, you can enable auto-instrumentation by adding the annotation at the **namespace level**. This approach automatically applies instrumentation to all Pods within the specified namespace.
+
+```yaml
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: mynamespace
+  annotations:
+    instrumentation.opentelemetry.io/inject-<LANGUAGE>: "opentelemetry-operator-system/elastic-instrumentation"
+```
+
+After adding annotations to Pods or Namespaces, the applications must be restarted for the instrumentation injection to take effect:
+
+```bash
+kubectl rollout restart deployment/my-deployment
+```
+
+In case you have multiple Instrumentation objects with different settings or images, ensure you point your Pods to the the desired `Instrumentation` objects in the annotations.
+
+The possible values for the annotation are detailed in the [Operator documentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#add-annotations-to-existing-deployments). For reference purposes, the values are:
+
+- `"true"`: to inject Instrumentation resource with default name from the current namespace.
+- `"my-instrumentation"`: to inject Instrumentation instance with name `"my-instrumentation"` in the current namespace.
+- `"my-other-namespace/my-instrumentation"`: to inject Instrumentation instance with name `"my-instrumentation"` from another namespace `"my-other-namespace"`.
+- `"false"`: do not inject.
+
+### Namespace based annotations example
+
+The following example creates a namespace with an annotation to instrument all Pods of the namespace with `java` libraries.
+
+```
+kubectl create namespace java-apps
+
+#Annotate app namespace
+kubectl annotate namespace java-apps instrumentation.opentelemetry.io/inject-java="opentelemetry-operator-system/elastic-instrumentation"
+
+# Run a java example application in the namespace
+kubectl run otel-test -n java-apps --env OTEL_INSTRUMENTATION_METHODS_INCLUDE="test.Testing[methodB]" --image docker.elastic.co/demos/apm/k8s-webhook-test
+```
+
+## Advanced configuration
+
+You can apply OTEL specific configuration to your applications at two different levels:
+- At Pod/container level, by using OTEL related environment variables.
+- At `Instrumentation` object level, for example configuring different settings per language.
+
+Use cases:
+- Change the library to be injected.
+- Change the exporter endpoint.
+- Apply certain logging level settings (OTEL_LOG_LEVEL).
+
+### Adding extra Instrumentation objects
+
+Consider also the creation of different `Instrumentation` objects for different purposes, such as:
+
+- Different configuration options for certain languages.
+- Trying out different versions of the SDKs.
+
+(TBD: add instructions and references about Instrumentation objects)
+
+
+## Manual instrumentation
+(TBD, in-progress)
+
+The manual instrumentation...
+Configuration requirements (does every language has its own requirements)?
+Exporter destination? HTTP vs OTLP? does each EDOT SDK support different protocols?
+

resources/kubernetes/operator/helm/values.yaml

@@ -23,19 +23,6 @@ defaultCRConfig:
   targetAllocator:
     enabled: false # Enable/disable the Operator's Target allocator.
                    # Refer to: https://github.com/open-telemetry/opentelemetry-operator/tree/main/cmd/otel-allocator
-  env:
-    - name: ELASTIC_AGENT_OTEL
-      value: '"true"'
-    - name: ELASTIC_ENDPOINT
-      valueFrom:
-        secretKeyRef:
-          name: elastic-secret-otel
-          key: elastic_endpoint
-    - name: ELASTIC_API_KEY
-      valueFrom:
-        secretKeyRef:
-          name: elastic-secret-otel
-          key: elastic_api_key
 
 clusterRole:
   rules:
@@ -59,6 +46,19 @@ collectors:
     #   requests:
     #     cpu: 100m
     #     memory: 500Mi
+    env:
+      - name: ELASTIC_AGENT_OTEL
+        value: '"true"'
+      - name: ELASTIC_ENDPOINT
+        valueFrom:
+          secretKeyRef:
+            name: elastic-secret-otel
+            key: elastic_endpoint
+      - name: ELASTIC_API_KEY
+        valueFrom:
+          secretKeyRef:
+            name: elastic-secret-otel
+            key: elastic_api_key
     config:
       exporters:
         # [Debug exporter](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/debugexporter/README.md)
@@ -219,11 +219,33 @@ collectors:
     # Configure the pods resources to control CPU and memory usage.
     resources:
       limits:
-        cpu: 100m
-        memory: 500Mi
+        cpu: 1500m
+        memory: 1500Mi
       requests:
         cpu: 100m
         memory: 500Mi
+    env:
+      # Work around for open /mounts error: https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/35990
+      - name: HOST_PROC_MOUNTINFO
+        value: ""
+      - name: ELASTIC_AGENT_OTEL
+        value: '"true"'
+      - name: ELASTIC_ENDPOINT
+        valueFrom:
+          secretKeyRef:
+            name: elastic-secret-otel
+            key: elastic_endpoint
+      - name: ELASTIC_API_KEY
+        valueFrom:
+          secretKeyRef:
+            name: elastic-secret-otel
+            key: elastic_api_key
+      - name: GOMAXPROCS
+        valueFrom:
+          resourceFieldRef:
+            resource: limits.cpu
+      - name: GOMEMLIMIT
+        value: "1025MiB"
     presets:
       logsCollection:
         enabled: true # Enable/disable the collection of node's logs.