indexers-and-matchers.asciidoc

Indexers and matchers

The add_kubernetes_metadata processor has two basic building blocks:

• Indexers

• Matchers

Indexers

Indexers use Pod metadata to create unique identifiers for each one of the Pods.

Available indexers are:

container

Identifies the Pod metadata using the IDs of its containers.

ip_port

Identifies the Pod metadata using combinations of its IP and its exposed ports. When using this indexer, metadata is identified using the combination of ip:port for each of the ports exposed by all containers of the pod. The ip is the IP of the pod.

pod_name

Identifies the Pod metadata using its namespace and its name as namespace/pod_name.

pod_uid

Identifies the Pod metadata using the UID of the Pod.

Matchers

Matchers are used to construct the lookup keys that match with the identifiers created by indexes.

Available matchers are:

field_format

Looks up Pod metadata using a key created with a string format that can include event fields.

This matcher has an option format to define the string format. This string format can contain placeholders for any field in the event.

For example, the following configuration uses the ip_port indexer to identify the Pod metadata by combinations of the Pod IP and its exposed ports, and uses the destination IP and port in events as match keys:

- add_kubernetes_metadata:
    ...
    default_indexers.enabled: false
    default_matchers.enabled: false
    indexers:
      - ip_port:
    matchers:
      - field_format:
          format: '%{[destination.ip]}:%{[destination.port]}'

fields

Looks up Pod metadata using as key the value of some specific fields. When multiple fields are defined, the first one included in the event is used.

This matcher has an option lookup_fields to define the files whose value will be used for lookup.

For example, the following configuration uses the ip_port indexer to identify Pods, and defines a matcher that uses the destination IP or the server IP for the lookup, the first it finds in the event:

- add_kubernetes_metadata:
    ...
    default_indexers.enabled: false
    default_matchers.enabled: false
    indexers:
      - ip_port:
    matchers:
      - fields:
          lookup_fields: ['destination.ip', 'server.ip']

logs_path

Looks up Pod metadata using identifiers extracted from the log path stored in the log.file.path field.

This matcher has the following configuration settings:

logs_path

(Optional) Base path of container logs. If not specified, it uses the default logs path of the platform where Agent is running: for Linux - /var/lib/docker/containers/, Windows - C:\\ProgramData\\Docker\\containers. To change the default value: container ID must follow right after the logs_path - <log_path>/<container_id>, where container_id is a 64-character-long hexadecimal string.

resource_type

(Optional) Type of the resource to obtain the ID of. Valid resource_type:

• pod: to make the lookup based on the Pod UID. When resource_type is set to pod, logs_path must be set as well, supported path in this case:

  • /var/lib/kubelet/pods/ used to read logs from mounted into the Pod volumes, those logs end up under /var/lib/kubelet/pods/<pod UID>/volumes/<volume name>/…​ To use /var/lib/kubelet/pods/ as a log_path, /var/lib/kubelet/pods must be mounted into the filebeat Pods.

  • /var/log/pods/ Note: when using resource_type: 'pod' logs will be enriched only with Pod metadata: Pod id, Pod name, etc., not container metadata.

• container: to make the lookup based on the container ID, logs_path must be set to /var/log/containers/. It defaults to container.

To be able to use logs_path matcher agent’s input path must be a subdirectory of directory defined in logs_path configuration setting.

The default configuration is able to lookup the metadata using the container ID when the logs are collected from the default docker logs path (/var/lib/docker/containers/<container ID>/…​ on Linux).

For example the following configuration would use the Pod UID when the logs are collected from /var/lib/kubelet/pods/<pod UID>/…​.

- add_kubernetes_metadata:
    ...
    default_indexers.enabled: false
    default_matchers.enabled: false
    indexers:
      - pod_uid:
    matchers:
      - logs_path:
          logs_path: '/var/lib/kubelet/pods'
          resource_type: 'pod'