Vector File Source to SEQ Forward

[ghaffar3]

I have following vector.yaml file. My target is to pickup a file contents and push them to SEQ via http. One of the sinks in the below is the console. I have taken the console JSON file and pushed it to SEQ forwarder (SEQ) via Postman. And the event seems to be happily created. But when pushed through vector i receive errors

ERRORs from VECTOR:

2024-12-17T09:08:12.124624Z ERROR sink{component_kind="sink" component_id=send_to_seq component_type=http}:request{request_id=1}: vector::sinks::util::retries: Not retriable; dropping the request. reason="Http status: 400 Bad Request" internal_log_rate_limit=true
2024-12-17T09:08:12.125611Z ERROR sink{component_kind="sink" component_id=send_to_seq component_type=http}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=None request_id=1 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2024-12-17T09:08:12.128592Z ERROR sink{component_kind="sink" component_id=send_to_seq component_type=http}:request{request_id=1}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true

ERRORS FROM SEQ

{"@t":"2024-12-17T09:08:12.1220557Z","@m":"Rejecting payload due to invalid JSON, request body could not be parsed","@i":"7b7ae0e2","@l":"Debug","@x":"Newtonsoft.Json.JsonSerializationException: Deserialized JSON type 'Newtonsoft.Json.Linq.JArray' is not compatible with expected type 'Newtonsoft.Json.Linq.JObject'. Path '', line 1, position 363.\r\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateJToken(JsonReader reader, JsonContract contract)\r\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)\r\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)\r\n at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)\r\n at Seq.Forwarder.Web.Api.IngestionController.IngestRawFormat() in C:\\projects\\seq-forwarder\\src\\Seq.Forwarder\\Web\\Api\\IngestionController.cs:line 85","@tr":"b9ce0f00bfabb750ca80e821189b2b28","@sp":"b80d4c44681015c8","ClientHostIP":"127.0.0.1","ActionId":"1f7d2331-3067-4c83-add2-414b47a6c9d3","ActionName":"Seq.Forwarder.Web.Api.IngestionController.Ingest (seqfwd)","RequestId":"0HN8RI00BAD46:00000001","RequestPath":"/api/events/raw","ConnectionId":"0HN8RI00BAD46","MachineName":"DESKTOP-RNVC4PD","Application":"Seq Forwarder"}

It seems like the JSON format is not compatible, but i took the same json and pushed it via postman it works. Let me know if someone has a solution for this

Sources

sources:
command_logs:
type: file
include: ["C:/test/CommandLog0.txt"]
vector_logs:
type: internal_logs

Transforms

transforms:
clean_message:
type: lua
inputs: ["command_logs"]
version: "2"
hooks:
process: process
source: |
function process(event, emit)
local message = event.log.message
if message then
-- Remove null characters from the message
message = message:gsub("%z", "")
-- Construct the new event with explicit order
local clean_event = {
Events = {
{
-- Timestamp first
Timestamp = os.date("!%Y-%m-%dT%H:%M:%S") .. ".000Z",
Level = "Information",
MessageTemplate = "ABC LOG",
Properties = {
message = message
}
}
}
}
-- Emit the new cleaned-up event
emit({ log = clean_event })
end
end

Sinks

sinks:
clean_message_output:
type: console
inputs: ["clean_message"]
encoding:
codec: json
json:
pretty: true

send_to_seq:
type: http
inputs: ["clean_message"]
uri: "http://localhost:15341/api/events/raw?apikey=ABC" # Correct endpoint for SEQ
method: "post"
encoding:
codec: json
batch:
max_events: 1 # Send each log as an individual event
timeout_secs: 1 # Minimum delay for batching
request:
rate_limit_num: 1
rate_limit_duration_secs: 1
headers:
Content-Type: "application/json"`

[jszwedko]

The SEQ error makes me think it is expecting newline delimited JSON objects and Vector, with this configuration, will send an array. I'd suggest trying:

encoding:
  codec: json
framing:
  method: newline_delimited

On the sink.

Small note: it's easier to read YAML if it is formatted as code. Above you have included the configuration unformatted. See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax for the syntax.