How to read entire JSON file and apply remap ?

[sankyfunky]

I have the use case where I have the valid JSON files are being generated in a folder.

I want to parse those file and extract events from it.

file01.json and other files are all same format like this.

{
    "scan_data": {
        "version": 1,
        "ip_info": "",
        "hostname": "examplehost.example.com"
    },
    "vulnerabilities": [
        {
            "id": 1001,
            "status": "",
            "secure": false,
            "details": "some details",
            "title": "ICMP Responses Logged"
        },
        {
            "id": 1002,
            "status": "",
            "secure": false,
            "details": "some details",
            "title": "Resolved DNS Name"
        }
    ]
}

events needs to be generated
event01

 {
        "version": 1,
        "ip_info": "",
        "hostname": "examplehost.example.com",
        "id": 1001,
        "status": "",
        "secure": false,
        "details": "some details",
        "title": ""title": "Resolved DNS Name""
}

event02

{
        "version": 1,
        "ip_info": "",
        "hostname": "examplehost.example.com",
        "id": 1002,
        "status": "",
        "secure": false,
        "details": "some details",
        "title": "ICMP Responses Logged"
}

Just like that. I want all events to be streamed.

how should I write my source and transform config.

this is what i have but it reads line by line and I can't parse json. Is it something available out of the box ?

data_dir = "/Users/folder/vector_data/"

[sources.file_source]
  type = "file"
  include = [ "/Users/folder/hs_logs/result/*.json" ]
  read_from = "beginning"

[transforms.asep_transform]
  type = "remap"
  inputs = ["file_source"]
  source = '''
    . = parse_json!(.message)
  '''

[sinks.console]
  type = "console"
  inputs = ["asep_transform"]
  target = "stdout"
  encoding.codec = "json"

[jszwedko]

Hi @sankyfunky ,

Vector sources are generally designed to read input line-by-line. Is it possible to change your input to have one event per line?

[sankyfunky]

unfortunately, I am on the consumer side. I can't change it to one event per line. Not for this use case. is that a limitation of vector ?

[jszwedko]

Unfortunately, yes, there's not a good way to read the whole contents of a file at once with Vector at the moment. It is designed to read streams of data. In the case of files, the stream is the lines in each file.

[jorgehermo9]

I just tried to get a config that would get the desired output and I think this would work

data_dir = "/Users/folder/vector_data/"

[sources.file_source]
type = "file"
include = ["/Users/folder/hs_logs/result/*.json"]
multiline.mode = "halt_with"
multiline.condition_pattern = "^}"
multiline.start_pattern = "^\\{"
multiline.timeout_ms = 100

[transforms.asep_transform]
type = "remap"
inputs = ["file_source"]
source = '''
    . = parse_json!(.message).vulnerabilities
  '''

[sinks.console]
type = "console"
inputs = ["asep_transform"]
target = "stdout"
encoding.codec = "json"

[jorgehermo9]

Note that this solution is not general and would just work in this specific case, and only if all the files have the same shape...

[sankyfunky]

I over simplified the json in the example above. But I tried different regex pattern, it is not accurately detecting the multiline.condition_pattern = "^}" in my case where I am having nested close curly braces "}".

Can we not detect all till EOF ?

[jorgehermo9]

can you try with condition_pattern: "^$"?
lets see if that works

[sankyfunky]

yeah..that is working to get the whole event at once. But now it gives an error at
parse_json!(.message)

2024-09-11T16:01:34.148448Z ERROR transform{component_kind="transform" component_id=asep_transform component_type=remap}: vector::internal_events::remap: Mapping failed with event. error="function call error for \"parse_json\" at (8:29): unable to parse json: EOF while parsing an object at line 142 column 5" error_type="conversion_failed" stage="processing" internal_log_rate_limit=true

I think parse_json is not able to filter EOF.

I tried strip_whitespace before parse_json. But it is also not working.
.message = strip_whitespace!(.message)

[jorgehermo9]

you can use result,err = parse_json(.message) (note the missing !) and safely abort processing if an error occurs for example. Maybe @jszwedko can be of more help here since I'm not sure