need help! DONE with vrl

[maipal-c]

can anyone please explain what is wrong with my config -

api:
  address: 127.0.0.1:8686
  enabled: true
data_dir: /vector-data-dir
sinks:
  es_cluster:
    auth:
      password: ${ELASTICSEARCH_PASSWORD}
      strategy: basic
      user: ${ELASTICSEARCH_USER}
    bulk:
      index: |
        {{ .app }}-logs-%Y-%m-%d
    endpoints:
    - http://elasticsearch-master-hl.databases.svc.cluster.local:9200
    inputs:
    - add_dynamic_index
    type: elasticsearch
sources:
  k8s_logs:
    ignore_older_secs: 600
    namespace_annotation_fields:
      namespace_labels: ""
    node_annotation_fields:
      node_labels: ""
    pod_annotation_fields:
      container_id: ""
      container_image: ""
      pod_annotations: ""
      pod_ip: ""
    type: kubernetes_logs
transforms:
  add_dynamic_index:
    inputs:
    - k8s_logs
    source: |
      tmp = .kubernetes.pod_owner
      tmp_arr = split!(tmp, "/")
      resource_name = tmp_arr[1]
      if resource_name == null {
         resource_name = "unknown"
      }
      .app = join!([.kubernetes.pod_namespace, resource_name], separator: "-")
    type: remap

ALL i want to do is to have seprate index for per deployment/statefullset/replicaset

now the error on vector side -

2024-09-12T20:53:52.823203Z ERROR sink{component_kind="sink" component_id=es_cluster component_type=elasticsearch}:request{request_id=31}: vector::sinks::elasticsearch::service: Response contained errors. error_code="http_response_500" response=Response { status: 500, version: HTTP/1.1, headers: {"x-elastic-product": "Elasticsearch", "content-type": "application/json"}, body: b"{\"error\":{\"root_cause\":[{\"type\":\"json_e_o_f_exception\",\"reason\":\"Unexpected end-of-input in VALUE_STRING\\n at [Source: (byte[])\\\"{\\\"index\\\":{\\\"_index\\\":\\\"monitoring-prom-agent-kube-prometheus-stack-prometheus-shard-1-logs-2024-09-12\\\"; line: 1, column: 99]\"}],\"type\":\"json_e_o_f_exception\",\"reason\":\"Unexpected end-of-input in VALUE_STRING\\n at [Source: (byte[])\\\"{\\\"index\\\":{\\\"_index\\\":\\\"monitoring-prom-agent-kube-prometheus-stack-prometheus-shard-1-logs-2024-09-12\\\"; line: 1, column: 99]\"},\"status\":500}" }

error on es cluster -

[2024-09-12T20:54:47,014][WARN ][r.suppressed             ] [elasticsearch-master-0] path: /_bulk, params: {timeout=60s}, status: 500 com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input in VALUE_STRING
 at [Source: (byte[])"{"index":{"_index":"gitops-harbor-core-54cc84ff59-logs-2024-09-12"; line: 1, column: 66]

both logs have diff pod refrence but its the same error for all pods all logs, no new index at all on es cluster

[jszwedko]

Could you try using a proxy like https://mitmproxy.org/ to capture the request that Vector is sending? It seems like it might be malformed JSON, but I can't imagine why.

[maipal-c]

@jszwedko , i think i got the issue. okay so here you

REQ BODY JSON

{"index":{"_index":"security-vault-server-logs-2024-09-12
"}}
{"app":"security-vault-server","file":"/var/log/pods/security_vault-server-0_a8083f1d-c96f-4137-bfa1-7c7eb79a92f1/server/0.log","kubernetes":{"container_image_id":"docker.io/bitnami/vault@sha256:652a9dbe4ffa41a88c11c11ea1de607a1522b067f68caea3a9d8a44561a7025b","container_name":"server","pod_ips":["10.42.1.153"],"pod_labels":{"app.kubernetes.io/component":"server","app.kubernetes.io/instance":"vault","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"vault","app.kubernetes.io/part-of":"vault","app.kubernetes.io/version":"1.17.3","apps.kubernetes.io/pod-index":"0","controller-revision-hash":"vault-server-6c79c4d9f7","helm.sh/chart":"vault-1.4.18","statefulset.kubernetes.io/pod-name":"vault-server-0","vault-active":"true","vault-initialized":"true","vault-perf-standby":"false","vault-sealed":"false","vault-version":"1.17.3"},"pod_name":"vault-server-0","pod_namespace":"security","pod_node_name":"ip-10-10-145-125.us-east-2.compute.internal","pod_owner":"StatefulSet/vault-server","pod_uid":"a8083f1d-c96f-4137-bfa1-7c7eb79a92f1"},"message":"==> Vault server configuration:","source_type":"kubernetes_logs","stream":"stdout","timestamp":"2024-09-12T11:43:47.327402085Z"}
{"index":{"_index":"security-vault-server-logs-2024-09-12
"}}
{"app":"security-vault-server","file":"/var/log/pods/security_vault-server-0_a8083f1d-c96f-4137-bfa1-7c7eb79a92f1/server/0.log","kubernetes":{"container_image_id":"docker.io/bitnami/vault@sha256:652a9dbe4ffa41a88c11c11ea1de607a1522b067f68caea3a9d8a44561a7025b","container_name":"server","pod_ips":["10.42.1.153"],"pod_labels":{"app.kubernetes.io/component":"server","app.kubernetes.io/instance":"vault","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"vault","app.kubernetes.io/part-of":"vault","app.kubernetes.io/version":"1.17.3","apps.kubernetes.io/pod-index":"0","controller-revision-hash":"vault-server-6c79c4d9f7","helm.sh/chart":"vault-1.4.18","statefulset.kubernetes.io/pod-name":"vault-server-0","vault-active":"true","vault-initialized":"true","vault-perf-standby":"false","vault-sealed":"false","vault-version":"1.17.3"},"pod_name":"vault-server-0","pod_namespace":"security","pod_node_name":"ip-10-10-145-125.us-east-2.compute.internal","pod_owner":"StatefulSet/vault-server","pod_uid":"a8083f1d-c96f-4137-bfa1-7c7eb79a92f1"},"message":"","source_type":"kubernetes_logs","stream":"stdout","timestamp":"2024-09-12T11:43:47.327510348Z"}
{"index":{"_index":"security-vault-server-logs-2024-09-12
"}}

i guess there is something wrong in index name parsing from yaml
maybe inplace of this -

index: |
        {{ .app }}-logs-%Y-%m-%d

it should be this -

index: {{ .app }}-logs-%Y-%m-%d

the main reason why i did this, is basically helm chart values.yaml file & helm templating

bulk:
      index: |
               {{ `{{ .app }}-logs-%Y-%m-%d` }}

[jszwedko]

Ah, interesting! I'm surprised the encoding didn't escape the newline, but that does appear to be the issue. I think you want:

index: |-
               {{ `{{ .app }}-logs-%Y-%m-%d` }}

to avoid the trailing line break.

[jszwedko]

Or yes, index: {{ .app }}-logs-%Y-%m-%d would work too.

[maipal-c]

yea thanks, for me i am using argocd so following worked -

index: >-
               {{ printf "{{ .app }}" }}-logs-%Y-%m-%d

BUT i guess vector code should have a check for atleast index name validation.

[jszwedko]

I opened a PR to encode the parameters as JSON correctly: #21293. This would, I think, have made this error clearer (you would have seen an error log around invalid index). Agreed, though, the sink could do additional validation on the index name too to ensure it matches the rules mentioned in https://stackoverflow.com/a/41585755