Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SYSLOG parsing errors when parsing syslogs from fortgate firewall #21192

Open
vrajendr2132 opened this issue Sep 2, 2024 · 0 comments
Open
Labels
source: syslog Anything `syslog` source related type: bug A code related bug.

Comments

@vrajendr2132
Copy link

vrajendr2132 commented Sep 2, 2024

I'm trying to parse syslogs from fortinet firewalls using vector, when vector service is started it starts throwing error, please find the attached log file. Connectivity from fortinet to vector server is fine through 514 port.
vector-log.txt

The main error I'm getting is "Too many open files (os error 24)" and "Internal log [Events dropped] is being suppressed to avoid flooding." Current ulimit value is 1024 and I tried to increase it to 12000, but still getting the same error. Is there any specific syslog format required at the fortigate firewall end? In vector source I tried using both syslog and socket types.

=========================================================
The vector configuration I used to parse syslogs is pasted below:

Vector configuration in YAML

#Syslog source to collect logs from FortiGate firewall
sources:
fortigate_syslog:
type: "syslog"
address: "0.0.0.0:514"
mode: "tcp"
shutdown_timeout_secs: 120

transforms:
parse_fortigate_logs:
type: remap
inputs:
- fortigate_syslog
source: . |= parse_syslog!(.message)

#Sink to write logs to a file on the local server
sinks:
local_file:
type: file
inputs:
- parse_fortigate_logs
path: "/var/log/fortigate.log" # Path to the local file where logs will be stored
encoding:
codec: "json"
# shutdown_timeout_secs: 120

===============================================

log sample:

Aug 30 01:01:00 cl1pandasyslog1 vector[1541402]: 2024-08-30T05:01:00.112187Z ERROR sink{component_kind="sink" component_id=python_script_sink component_type=socket}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.

Aug 30 02:18:36 cl1pandasyslog1 vector[1542938]: 2024-08-30T06:18:36.891866Z ERROR source{component_kind="source" component_id=fortigate_syslog component_type=socket}: vector::internal_events::socket: Error receiving data. error=Incoming listener failed: Too many open files (os error 24) error_code="socket_receive" error_type="reader_failed" stage="receiving" mode=tcp internal_log_rate_limit=true

Can someone help me to resolve this issue.

@jszwedko jszwedko added type: bug A code related bug. source: syslog Anything `syslog` source related labels Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
source: syslog Anything `syslog` source related type: bug A code related bug.
Projects
None yet
Development

No branches or pull requests

2 participants