ingest-pipeline-kubernetes.asciidoc

Using a custom ingest pipeline with the {k8s} Integration

This tutorial explains how to add a custom ingest pipeline to a {k8s} Integration in order to add specific metadata fields for deployments and cronjobs of pods.

Custom pipelines can be used to add custom data processing, like adding fields, obfuscating sensitive information, and more. Find more information in our tutorial about transforming data with custom ingest pipelines.

Metadata enrichment for Kubernetes

The {k8s} Integration is used to collect logs and metrics from Kubernetes clusters with {agent}. During the collection, the integration enhances the collected information with extra useful information that users can correlate with different Kubernetes assets. This additional information added on top of collected data, such as labels, annotations, ancestor names of Kubernetes assets, and others, are called metadata.

The {k8s} Provider offers the add_resource_metadata option to configure the metadata enrichment options.

For {agent} versions >[8.10.4], the default configuration for metadata enrichment is add_resource_metadata.deployment=false and add_resource_metadata.cronjob=false. This means that pods that are created from replicasets that belong to specific deployments would not be enriched with kubernetes.deployment.name. Additionally, pods that are created from jobs that belong to specific cronjobs, would not be enriched with kubernetes.cronjob.name.

Kubernetes Integration Policy > Collect Kubernetes metrics from Kube-state-metrics > Kubernetes Pod Metrics

Example: Enabling the enrichment through add_resource_metadata in a Managed {agent} Policy

Note: Enabling deployment and cronjob metadata enrichment leads to an increase of {agent}'s memory consumption. {agent} uses a local cache in order to keep records of the {k8s} assets from being discovered.

Add deployment and cronjob for {k8s} pods through ingest pipelines

As an alternative to keeping the feature enabled and using more memory resources for {agent}, users can make use of ingest pipelines to add the missing fields of kubernetes.deployment.name and kubernetes.cronjob.name.

Following the transforming data with custom ingest pipelines tutorial, navigate to state_pod datastream under: Kubernetes Integration Policy > Collect Kubernetes metrics from Kube-state-metrics > Kubernetes Pod Metrics.

Create the following custom ingest pipeline with two processors:

Processor for deployment

Processor for cronjob

The final metrics-kubernetes.state_pod@custom ingest pipeline:

[
  {
    "gsub": {
      "field": "kubernetes.replicaset.name",
      "pattern": "(?:.(?!-))+$",
      "replacement": "",
      "target_field": "kubernetes.deployment.name",
      "ignore_missing": true,
      "ignore_failure": true
    }
  },
  {
    "gsub": {
      "field": "kubernetes.job.name",
      "pattern": "(?:.(?!-))+$",
      "replacement": "",
      "target_field": "kubernetes.cronjob.name",
      "ignore_missing": true,
      "ignore_failure": true
    }
  }
]

Note: The ingest pipeline does not check for the actual existence of a deployment and cronjob ancestor, it only adds the specific values.