Filter to drop based on condition not working

[AnaVomvylas]

Hello,

I have this transformation applied in my values.yaml:

drop_kubelet_removestalestate:
  type: filter
  inputs:
    - datadog_agent.logs
  condition: 'contains(string!(.message), "RemoveStaleState") == false'

What I want to do is drop all logs that have the "RemoveStaleState" string in the message.
Here is an example event of what I want to drop:

{
  "ddsource": "kubelet",
  "ddtags": "",
  "hostname": "i-0fdf4636c7521e97c",
  "message": "{\"journald\":{\"PRIORITY\":\"6\",\"SYSLOG_FACILITY\":\"3\",\"SYSLOG_IDENTIFIER\":\"kubelet\",\"_BOOT_ID\":\"7f773d2c40724ce294bae234f80c82da\",\"_CAP_EFFECTIVE\":\"1ffffffffff\",\"_CMDLINE\":\"/usr/bin/kubelet --config /etc/kubernetes/kubelet/kubelet-config.json --kubeconfig /var/lib/kubelet/kubeconfig --container-runtime-endpoint unix:///run/containerd/containerd.sock --image-credential-provider-config /etc/eks/image-credential-provider/config.json --image-credential-provider-bin-dir /etc/eks/image-credential-provider --node-ip=abstractedforsecurity --pod-infra-container-image=602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/pause:3.5 --v=2 --hostname-override=ip-10-112-119-57.eu-west-1.compute.internal --cloud-provider=external --node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=arm64 --max-pods=17 --eviction-soft=nodefs.available\\u003c20% --eviction-soft-grace-period=nodefs.available=1m0s --eviction-max-pod-grace-period=120\",\"_COMM\":\"kubelet\",\"_EXE\":\"/usr/bin/kubelet\",\"_GID\":\"0\",\"_HOSTNAME\":\"abstractedforsecurity\",\"_MACHINE_ID\":\"ec2d00c9fc328b9a9a6335b682d5d75e\",\"_PID\":\"1547\",\"_STREAM_ID\":\"45b436960b6549acbf97fb1e711cfa67\",\"_SYSTEMD_CGROUP\":\"/runtime.slice/kubelet.service\",\"_SYSTEMD_SLICE\":\"runtime.slice\",\"_SYSTEMD_UNIT\":\"kubelet.service\",\"_TRANSPORT\":\"stdout\",\"_UID\":\"0\"},\"message\":\"I1129 13:37:32.121217    1547 memory_manager.go:354] \\\"RemoveStaleState removing state\\\" podUID=\\\"ded87050-9112-4567-b7df-1f5e3b2eae7e\\\" containerName=\\\"reserve-resources\\\"\"}",
  "service": "kubelet",
  "source_type": "datadog_agent",
  "status": "info",
  "timestamp": "2024-11-29T13:37:32.124Z"
}

I am running vector as a helm chart deployment, inside an AWS EKS cluster. I am also running datadog agent inside that cluster.
I am using v0.42.0 vector.

I used vector tap to get the example event above.

Any ideas why my transform is not working? Expected would be to drop the event but it is still kept and forwarded.

[jszwedko]

My first guess would be that your downstream components aren't using drop_kubelet_removestalestate as their input. Could you share the full configuration? I tested locally with this configuration:

sources:
  source0:
    type: stdin
    decoding:
      codec: json
transforms:
  transform0:
    type: filter
    inputs:
    - source0
    condition: 'contains(string!(.message), "RemoveStaleState") == false'
sinks:
  sink0:
    type: console
    inputs:
    - transform0
    encoding:
      codec: json

And the example input event you provided was dropped as expected.

[AnaVomvylas]

@jszwedko thank you for taking the time to look into this.

This is my full configuration:

  sources:
    datadog_agent:
      type: datadog_agent
      address: 0.0.0.0:8282
      multiple_outputs: true
      store_api_key: true
      disable_traces: true
  transforms:
    metrics_drop_tags:
      type: remap
      inputs:
        - datadog_agent.metrics
      source: |-
        del(.tags.kube_replica_set)
        del(.tags.kube_qos)
        del(.tags.kube_ownerref_kind)
        del(.tags.image_id)
        del(.tags.short_image)
    drop_status:
      type: remap
      inputs:
        - datadog_agent.logs
      source: |
        del(.status)
    drop_kubelet_removestalestate:
      type: filter
      inputs:
        - datadog_agent.logs
      condition: 'contains(string!(.message), "RemoveStaleState") == false'
  sinks:
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - metrics_drop_tags
      site: datadoghq.eu
    datadog_logs:
      type: datadog_logs
      inputs:
        - drop_status
        - drop_kubelet_removestalestate
      site: datadoghq.eu

The problematic transform is has valid inputs and is present in sinks. The rest of the transforms work fine.

[jszwedko]

I think the issue here is that you have both transforms as inputs to the sink:

      inputs:
        - drop_status
        - drop_kubelet_removestalestate

Because both inputs here have datadog_agent.logs as their input, that means you will "fan out" and then "fan in" such that will end up with duplicate events. I'm guessing that the events you expect to see dropped by the filter are actually being dropped, but those same events are also going through drop_status, where they won't be dropped' and then forwarded to the datadog_logs sink.

Visualizing the configuration with vector graph might make this clearer:

I think you may mean to chain the drop_status and drop_kubelet_removestalestate transforms to have them execute consecutively like:

  sources:
    datadog_agent:
      type: datadog_agent
      address: 0.0.0.0:8282
      multiple_outputs: true
      store_api_key: true
      disable_traces: true
  transforms:
    metrics_drop_tags:
      type: remap
      inputs:
        - datadog_agent.metrics
      source: |-
        del(.tags.kube_replica_set)
        del(.tags.kube_qos)
        del(.tags.kube_ownerref_kind)
        del(.tags.image_id)
        del(.tags.short_image)
    drop_status:
      type: remap
      inputs:
        - datadog_agent.logs
      source: |
        del(.status)
    drop_kubelet_removestalestate:
      type: filter
      inputs:
        - drop_status
      condition: 'contains(string!(.message), "RemoveStaleState") == false'
  sinks:
    datadog_metrics:
      type: datadog_metrics
      inputs:
        - metrics_drop_tags
      site: datadoghq.eu
    datadog_logs:
      type: datadog_logs
      inputs:
        - drop_kubelet_removestalestate
      site: datadoghq.eu

So that events pass through both transforms, serially, before going to the sink. Thus the graph looks like:

Does that make sense?

[AnaVomvylas]

Thank you for the detailed explanation. This did the trick!