log data in JSON format

[soulg24]

Good afternoon!
I'm working on setting up log transmission to OpenSearch using a JSON log format. My test attempts have been successful, but I need to find a way to configure Vector to send log data in JSON format without sending the entire log in one message.
I've tried specifying the type like this:

[sources.my_source]
type = "file"
include = ["/path/to/your/logs/log.json"]
file_type = "json"
I tried the encoding.codec = "json" option

However, I encountered an error. What is the correct way to do this?

[pront]

What does your log.json look like? Is it newline separated JSON strings?

[soulg24]

Как это log.json выглядит? Это строки JSON, разделенные новой строкой?

Yes, this is a json string.

{"timestamp": "2024-12-04 10:10:08.069925", "level": "debug", "Method": "POST", "URL": "http://testapp:8080/create/s", "request host": "xx.xx.xxx.xxx", "request user": "tech@user.com", "request param": "name='vmtest241204.jmeter-test.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ", "response": "204"}
{"timestamp": "2024-12-04 10:11:43.037671", "level": "debug", "Method": "POST", "URL": "http://testapp:8080/create/s", "request host": "xx.xx.xxx.xxx", "request user": "tech@user.com", "request param": "name='t24120412582-pg14stb-001.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ", "response": "204"}
{"timestamp": "2024-12-04 10:11:43.608739", "level": "debug", "Method": "POST", "URL": "http://testapp:8080/create/s", "request host": "xx.xx.xxx.xxx", "request user": "tech@user.com", "request param": "name='t24120412582-pg14stb-001.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ", "response": "204"}

[jszwedko]

I'm a bit confused here, neither file_type or encoding.codec are options that exist on the file source. Adding decoding options to the file source is being tracked by #9210.

If you want to parse the messages as JSON you will need to use a remap transform with the parse_json function.

[soulg24]

I'm a bit confused here, neither file_type or encoding.codec are options that exist on the file source. Adding decoding options to the file source is being tracked by #9210.

If you want to parse the messages as JSON you will need to use a remap transform with the parse_json function.

Sorry, English isn't my first language, so it's hard for me to explain. Could you provide a simple example of how to use the remap transform with the parse_json function? I want to convert the message into JSON format.

for the file source, I did not find how to specify the encoding

[jszwedko]

Correct, the file source does not support codecs like other sources do. This gap is being tracked by #9210

For the remap transform you can see the docs: https://vector.dev/docs/reference/configuration/transforms/remap/

In particular, you will want to use this function: https://vector.dev/docs/reference/vrl/examples/#parse_json

[pront]

Here is a sample config:

sources:
  source0:
    type: file
    include:
      - "/path/to/log.json"

transforms:
  transform0:
    type: remap
    inputs:
      - source0
    source: |
      .message = parse_json!(string!(.message))

sinks:
  console:
    type: console
    inputs: ["transform0"]
    encoding:
      codec: json
      json:
        pretty: true

The magic happens at .message = parse_json!(string!(.message)) it basically takes the bytes from file (line by line) and constructs an object map. And finally it overwrites the .message with the result.

You try this on the VRL playground. See example.

[soulg24]

Вот пример конфигурации:

sources:
  source0:
    type: file
    include:
      - "/path/to/log.json"

transforms:
  transform0:
    type: remap
    inputs:
      - source0
    source: |
      .message = parse_json!(string!(.message))

sinks:
  console:
    type: console
    inputs: ["transform0"]
    encoding:
      codec: json
      json:
        pretty: true

Магия происходит в .message = parse_json!(string!(.message)): по сути, он берёт байты из файла (строка за строкой) и создаёт карту объектов. И, наконец, он перезаписывает .message результатом.

Попробуйте это на игровой площадке VRL. Смотрите Пример.

That's fantastic! Thanks for providing the examples.