processor-add_kubernetes_metadata.asciidoc

Add Kubernetes metadata

add_kubernetes_metadata

Tip	Inputs that collect logs and metrics use this processor by default, so you do not need to configure it explicitly.

The add_kubernetes_metadata processor annotates each event with relevant metadata based on which Kubernetes Pod the event originated from. At startup it detects an in_cluster environment and caches the Kubernetes-related metadata.

For events to be annotated with Kubernetes-related metadata, the Kubernetes configuration must be valid.

Each event is annotated with:

• Pod Name

• Pod UID

• Namespace

• Labels

In addition, the node and namespace metadata are added to the Pod metadata.

The add_kubernetes_metadata processor has two basic building blocks:

• Indexers

• Matchers

Indexers use Pod metadata to create unique identifiers for each one of the Pods. These identifiers help to correlate the metadata of the observed Pods with actual events. For example, the ip_port indexer can take a Kubernetes Pod and create identifiers for it based on all its pod_ip:container_port combinations.

Matchers use information in events to construct lookup keys that match the identifiers created by the indexers. For example, when the fields matcher takes ["metricset.host"] as a lookup field, it constructs a lookup key with the value of the field metricset.host. When one of these lookup keys matches with one of the identifiers, the event is enriched with the metadata of the identified Pod.

For more information about available indexers and matchers, plus some examples, refer to [kubernetes-indexers-and-matchers].

Examples

This configuration enables the processor when {agent} is run as a Pod in Kubernetes.

  - add_kubernetes_metadata:
      # Defining indexers and matchers manually is required for {beatname_lc}, for instance:
      #indexers:
      #  - ip_port:
      #matchers:
      #  - fields:
      #      lookup_fields: ["metricset.host"]
      #labels.dedot: true
      #annotations.dedot: true

This configuration enables the processor on an {agent} running as a process on the Kubernetes node:

  - add_kubernetes_metadata:
      host: <hostname>
      # If kube_config is not set, KUBECONFIG environment variable will be checked
      # and if not present it will fall back to InCluster
      kube_config: ${HOME}/.kube/config
      # Defining indexers and matchers manually is required for {beatname_lc}, for instance:
      #indexers:
      #  - ip_port:
      #matchers:
      #  - fields:
      #      lookup_fields: ["metricset.host"]
      #labels.dedot: true
      #annotations.dedot: true

This configuration disables the default indexers and matchers, and then enables different indexers and matchers:

  - add_kubernetes_metadata:
      host: <hostname>
      # If kube_config is not set, KUBECONFIG environment variable will be checked
      # and if not present it will fall back to InCluster
      kube_config: ~/.kube/config
      default_indexers.enabled: false
      default_matchers.enabled: false
      indexers:
        - ip_port:
      matchers:
        - fields:
            lookup_fields: ["metricset.host"]
      #labels.dedot: true
      #annotations.dedot: true

Configuration settings

processors.asciidoc

Name	Required	Default	Description
host	No		Node to scope {agent} to in case it cannot be accurately detected, as when running {agent} in host network mode.
scope	No	node	Whether the processor should have visibility at the node level (node) or at the entire cluster level (cluster).
namespace	No		Namespace to collect the metadata from. If no namespaces is specified, collects metadata from all namespaces.
add_resource_metadata	No		Filters and configuration for adding extra metadata to the event. This setting accepts the following settings: node or namespace: Labels and annotations filters for the extra metadata coming from node and namespace. By default all labels are included, but annotations are not. To change the default behavior, you can set include_labels, exclude_labels, and include_annotations. These settings are useful when storing labels and annotations that require special handling to avoid overloading the storage output. Wildcards are supported in these settings by using use_regex_include: true in combination with include_labels, and respectively by setting use_regex_exclude: true in combination with exclude_labels. To turn off enrichment of node or namespace metadata individually, set enabled: false. deployment: If the resource is pod and it is created from a deployment, the deployment name is not added by default. To enable this behavior, set deployment: true. cronjob: If the resource is pod and it is created from a cronjob, the cronjob name is not added by default. To enable this behavior, set cronjob: true. Expand this to see an example add_resource_metadata: namespace: include_labels: ["namespacelabel1"] # use_regex_include: false # use_regex_exclude: false # exclude_labels: ["namespacelabel2"] #labels.dedot: true #annotations.dedot: true node: # use_regex_include: false include_labels: ["nodelabel2"] include_annotations: ["nodeannotation1"] # use_regex_exclude: false # exclude_annotations: ["nodeannotation2"] #labels.dedot: true #annotations.dedot: true deployment: true cronjob: true
kube_config	No	KUBECONFIG environment variable, if present	Config file to use as the configuration for the Kubernetes client.
kube_client_options	No		Additional configuration options for the Kubernetes client. Currently client QPS and burst are supported. If this setting is not configured, the Kubernetes client’s default QPS and burst is used. Expand this to see an example kube_client_options: qps: 5 burst: 10
cleanup_timeout	No	60s	Time of inactivity before stopping the running configuration for a container.
sync_period	No		Timeout for listing historical resources.
labels.dedot	No	true	Whether to replace dots (.) in labels with underscores (_). annotations.dedot

indexers-and-matchers.asciidoc