Vector - Syslog Logs #10780

ryansliceup · 2022-01-10T21:44:39Z

ryansliceup
Jan 10, 2022

Hello guys.
I am trying to capture "severity" from a Cisco device. Is this possible to do? Is there a way to capture the header or can I use regex to parse "Severity notice (5)".

Thanks,
Ryan

13:43:26.102526 IP (tos 0x0, ttl 254, id 1342, offset 0, flags [none], proto UDP (17), length 141)
10.100.1.1.62095 > 10.12.2.241.syslog: [udp sum ok] SYSLOG, length: 113
Facility local7 (23), Severity notice (5)
Msg: 1384: *Mar 4 00:49:56.339: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)
0x0000: 3c31 3839 3e31 3338 343a 202a 4d61 7220
0x0010: 2034 2030 303a 3439 3a35 362e 3333 393a
0x0020: 2025 5359 532d 352d 5052 4956 5f41 5554
0x0030: 485f 5041 5353 3a20 5072 6976 696c 6567
0x0040: 6520 6c65 7665 6c20 7365 7420 746f 2031
0x0050: 3520 6279 2063 7372 6f6f 7420 6f6e 2076
0x0060: 7479 3020 2831 302e 3132 2e32 2e32 3333
0x0070: 29

{"host":"10.100.1.1","message":"<189>1384: *Mar 4 00:49:56.339: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)","source_ip":"10.100.1.1","source_type":"syslog","timestamp":"2022-01-10T21:43:26.102749856Z"}

spencergilbert · 2022-01-12T17:58:58Z

spencergilbert
Jan 12, 2022
Maintainer

I don't believe we can capture the header today. Where is the event you've added at the bottom coming from, and has it been transformed at all?

0 replies

ryansliceup · 2022-01-12T19:28:27Z

ryansliceup
Jan 12, 2022
Author

Spencer, Okay if you cannot capture the header, i can get the severity based on another field in the message. I am thinking grok for this, but am open to suggestions. I am surprised no one has parsed cisco logs using vector before! Thanks, Ryan

…

On Wed, Jan 12, 2022 at 12:59 PM Spencer Gilbert ***@***.***> wrote: I don't believe we can capture the header today. Where is the event you've added at the bottom coming from, and has it been transformed at all? — Reply to this email directly, view it on GitHub <#10780 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWXADXFA4D2FWPODZ52NJ6DUVW6O3ANCNFSM5LUXBACA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

1 reply

spencergilbert Jan 12, 2022
Maintainer

I'm not sure if it was cisco but there have been some users in our discord that used grok/regex to parse "syslog-ish" messages coming from networking devices. If they have a defined format we'd be open to adding a parsing function for the message though.

ryansliceup · 2022-01-12T19:41:43Z

ryansliceup
Jan 12, 2022
Author

I do believe they do, let me do some inquiries and I will get back to you. Thanks, Ryan

…

On Wed, Jan 12, 2022 at 2:40 PM Spencer Gilbert ***@***.***> wrote: I'm not sure if it was cisco but there have been some users in our discord that used grok/regex to parse "syslog-ish" messages coming from networking devices. If they have a defined format we'd be open to adding a parsing function for the message though. — Reply to this email directly, view it on GitHub <#10780 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWXADXG3SG5SZ7QA2C2SK6TUVXKKLANCNFSM5LUXBACA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

0 replies

ryansliceup · 2022-01-13T12:27:33Z

ryansliceup
Jan 13, 2022
Author

When I try to use the grok filter this is what I get. Would you recommend I go on discord and ask?

[sources.in_udp]
type = "syslog"
address = "0.0.0.0:514"
mode = "udp"
host_key = "host"

[transforms.cisco_parser]
inputs = ["in_udp"]
type = "grok_parser"
pattern = "<%{INT:priority}>%{NUMBER:seq#}: *%{CISCOTIMESTAMP:log_date}: %{WORD:category}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:message}"
#1393: *Mar 5 16:00:36.461: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)

[sinks.kafka]
# General
type = "kafka"
inputs = ["in", "in_udp"]
bootstrap_servers = "10.12.2.243:9094"
topic = "vector-lines-0"
compression = "none"
healthcheck = true
encoding = "json"

[sinks.out]
inputs = ["in_udp"]
type = "console"
encoding = "text"

2022-01-13T12:26:06.251687Z WARN transform{component_kind="transform" component_id=cisco_parser component_type=grok_parser component_name=cisco_parser}: vector::internal_events::grok_parser: Grok pattern failed to match. field="<189>1397: *Mar 6 15:32:28.004: %SYS-5-PRIV_AUTH_PASS:[...]" internal_log_rate_secs=30
<189>1397: *Mar 6 15:32:28.004: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)

2 replies

spencergilbert Jan 13, 2022
Maintainer

I assume <189>1393: *Mar 5 16:00:36.461: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233) is the entire .message you're trying to parse? I can spend a few minutes trying to craft it correctly.

I'd also suggest replacing the grok_parser transform (which is deprecated) with a remap transform using the parse_grok() function from VRL

spencergilbert Jan 13, 2022
Maintainer

@ryansliceup it looks like this works, but my grok-fu isn't great:

$ parse_grok!(.message, "^<%{POSINT:syslog_pri}>%{NUMBER:seq}: [*]%{CISCOTIMESTAMP:timestamp}: [%]%{WORD:category}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:message}")
{ "category": "SYS", "message": "Privilege level set to 15 by csroot on vty0 (10.12.2.233)", "mnemonic": "PRIV_AUTH_PASS", "seq": "1393", "severity": "5", "syslog_pri": "189", "timestamp": "Mar 5 16:00:36.461" }

ryansliceup · 2022-01-14T12:43:23Z

ryansliceup
Jan 14, 2022
Author

Hello,
Thanks so much for the information, I appreciate it. As I am new to vector, is this right as it doesn't parse the severity, but it doesn't fail either. I have tried changing the inputs on the sink, and it did not help. What am I doing wrong, I am sure it is something silly.

Thanks,
Ryan

[sources.in_udp]
type = "syslog"
address = "0.0.0.0:514"
mode = "udp"
host_key = "host"

[transforms.cisco_parser]
inputs = ["in_udp"]
type = "remap"
source = '''
parse_grok!(.message, "^<%{POSINT:syslog_pri}>%{NUMBER:seq}: [*]%{CISCOTIMESTAMP:timestamp}: [%]%{WORD:category}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:message}")

'''
#1393: *Mar  5 16:00:36.461: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)

[sinks.kafka]
# General
type = "kafka"
#inputs = ["in_udp"]
inputs = ["cisco_parser"]
bootstrap_servers = "10.12.2.243:9094"
topic = "vector-lines-0"
compression = "none"
healthcheck = true
encoding = "json"

[sinks.out]
inputs = ["cisco_parser"]
type = "console"
encoding = "json"

1 reply

spencergilbert Jan 14, 2022
Maintainer

Can you share an example event from sources.in_udp (send it directly to sinks.out without parsing) as well as an example event that's currently getting printed by sinks.out (from the current cisco_parser input)?

ryansliceup · 2022-01-14T16:50:07Z

ryansliceup
Jan 14, 2022
Author

This is from the following configuration on the console. Is this what you asked? Thanks again for the information, and help!

Output:

<189>1405: *Mar  7 19:52:59.273: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)

Configuration:

[sources.in_udp]
    type = "syslog"
    address = "0.0.0.0:514"
    mode = "udp"
    host_key = "host"

[transforms.cisco_parser]
    inputs       = ["in_udp"]
    type         = "remap"
    source = '''
       parse_grok!(.message, "^<%{POSINT:syslog_pri}>%{NUMBER:seq}: [*]%{CISCOTIMESTAMP:timestamp}: [%]%{WORD:category}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:message}")

    '''
    #1393: *Mar  5 16:00:36.461: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)


[sinks.kafka]
    # General
    type = "kafka"
    #inputs = ["in_udp"]
    inputs = ["cisco_parser"]
    bootstrap_servers = "10.12.2.243:9094"
    topic = "vector-lines-0"
    compression = "none"
    healthcheck = true
    encoding = "json"

[sinks.out]
    inputs = ["in_udp"]
    type = "console"
    encoding = "text"

Then I have modified the configuration to change the inputs from the sink:
Configuration:

[sinks.kafka]
    # General
    type = "kafka"
    #inputs = ["in_udp"]
    inputs = ["cisco_parser"]
    bootstrap_servers = "10.12.2.243:9094"
    topic = "vector-lines-0"
    compression = "none"
    healthcheck = true
    encoding = "json"

[sinks.out]
    inputs = ["cisco_parser"]
    type = "console"
    encoding = "text"

Output:

<189>1406: *Mar  7 19:54:56.165: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)

2 replies

jszwedko Jan 14, 2022
Maintainer

Hi @ryansliceup !

You'll want to assign the result of the parse_grok call in your remap transform, otherwise VRL will just drop it. Try:

[transforms.cisco_parser]
    inputs       = ["in_udp"]
    type         = "remap"
    source = '''
       . = parse_grok!(.message, "^<%{POSINT:syslog_pri}>%{NUMBER:seq}: [*]%{CISCOTIMESTAMP:timestamp}: [%]%{WORD:category}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:message}")
    '''

You can also preserve the existing fields in . with something like:

[transforms.cisco_parser]
    inputs       = ["in_udp"]
    type         = "remap"
    source = '''
       .| = parse_grok!(.message, "^<%{POSINT:syslog_pri}>%{NUMBER:seq}: [*]%{CISCOTIMESTAMP:timestamp}: [%]%{WORD:category}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:message}")
    '''

I'd recommend trying both so you can see the differences.

StefanSa Jan 15, 2022

Hi there,
I have once found this with me (grok examples).

# IOS
"message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?:( %{NUMBER}:)? %{CISCOTIMESTAMPTZ:log_date}: %%{CISCO_REASON:facility}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}",
"message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?:( %{NUMBER}:)? %{CISCOTIMESTAMPTZ:log_date}: %%{CISCO_REASON:facility}-%{CISCO_REASON:facility_sub}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}",

# Nexus
"message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?: %{NEXUSTIMESTAMP:log_date}: %%{CISCO_REASON:facility}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}",
"message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?: %{NEXUSTIMESTAMP:log_date}: %%{CISCO_REASON:facility}-%{CISCO_REASON:facility_sub}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}"

ryansliceup · 2022-01-15T18:03:29Z

ryansliceup
Jan 15, 2022
Author

Thanks guys. You guys are awesome. Let me check this out!

…

On Sat, Jan 15, 2022, 5:44 AM StefanSa ***@***.***> wrote: Hi there, I have once found this with me (grok examples). # IOS "message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?:( %{NUMBER}:)? %{CISCOTIMESTAMPTZ:log_date}: %%{CISCO_REASON:facility}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}", "message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?:( %{NUMBER}:)? %{CISCOTIMESTAMPTZ:log_date}: %%{CISCO_REASON:facility}-%{CISCO_REASON:facility_sub}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}", # Nexus "message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?: %{NEXUSTIMESTAMP:log_date}: %%{CISCO_REASON:facility}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}", "message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?: %{NEXUSTIMESTAMP:log_date}: %%{CISCO_REASON:facility}-%{CISCO_REASON:facility_sub}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}" — Reply to this email directly, view it on GitHub <#10780 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWXADXFFYGWXF3JBWYGJF53UWFFXJANCNFSM5LUXBACA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

2 replies

ryansliceup Jan 18, 2022
Author

@StefanSa Is this in a different format? I am assuming i have to tweak this to plop it in the config?

StefanSa Jan 18, 2022

@ryansliceup
Yes you need to adjust that a bit, because that is an example of logstash, where grok is the same.

.| = parse_grok!(.message, "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?:( %{NUMBER}:)? [*]%{CISCOTIMESTAMP:timestamp}: %%{CISCO_REASON:facility}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}")

A good tool to debug grok is this one:
https://grokdebug.herokuapp.com/

jrehm-mmm · 2023-03-17T22:10:53Z

jrehm-mmm
Mar 17, 2023

@ryansliceup - did you end up implimenting this in your production environment? I'm considering switching our syslog handling over to vector and am quite surprised as well that there doesn't seem to be too much demand for help with parsing Cisco's plethora of syslog formatting!

0 replies

ryansliceup · 2023-03-19T00:57:04Z

ryansliceup
Mar 19, 2023
Author

I ended up putting rsyslog in front of it for Cisco. It parses better and I was able to aggregate all my servers easier. It worked for my use case at least.

…

On Fri, Mar 17, 2023, 6:11 PM jrehm-mmm ***@***.***> wrote: @ryansliceup <https://github.com/ryansliceup> - did you end up implimenting this in your production environment? I'm considering switching our syslog handling over to vector and am quite surprised as well that there doesn't seem to be too much demand for help with parsing Cisco's plethora of syslog formatting! — Reply to this email directly, view it on GitHub <#10780 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWXADXFDL3RUHHQFBECR46DW4TOPRANCNFSM5LUXBACA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Disclaimer: This email and its attachments are intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. Internet email is not a 100% secure transmission medium. Please understand and observe this lack of security when emailing us. Although we have taken steps to ensure that this email and its attachments are free from any virus, it is the recipient's sole responsibility, in keeping with good computing practice, to ensure that they are actually virus free.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vector - Syslog Logs #10780

{{title}}

Replies: 9 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Vector - Syslog Logs #10780

ryansliceup Jan 10, 2022

Replies: 9 comments · 8 replies

spencergilbert Jan 12, 2022 Maintainer

ryansliceup Jan 12, 2022 Author

spencergilbert Jan 12, 2022 Maintainer

ryansliceup Jan 12, 2022 Author

ryansliceup Jan 13, 2022 Author

spencergilbert Jan 13, 2022 Maintainer

spencergilbert Jan 13, 2022 Maintainer

ryansliceup Jan 14, 2022 Author

spencergilbert Jan 14, 2022 Maintainer

ryansliceup Jan 14, 2022 Author

jszwedko Jan 14, 2022 Maintainer

StefanSa Jan 15, 2022

ryansliceup Jan 15, 2022 Author

ryansliceup Jan 18, 2022 Author

StefanSa Jan 18, 2022

jrehm-mmm Mar 17, 2023

ryansliceup Mar 19, 2023 Author

ryansliceup
Jan 10, 2022

Replies: 9 comments 8 replies

spencergilbert
Jan 12, 2022
Maintainer

ryansliceup
Jan 12, 2022
Author

spencergilbert Jan 12, 2022
Maintainer

ryansliceup
Jan 12, 2022
Author

ryansliceup
Jan 13, 2022
Author

spencergilbert Jan 13, 2022
Maintainer

spencergilbert Jan 13, 2022
Maintainer

ryansliceup
Jan 14, 2022
Author

spencergilbert Jan 14, 2022
Maintainer

ryansliceup
Jan 14, 2022
Author

jszwedko Jan 14, 2022
Maintainer

ryansliceup
Jan 15, 2022
Author

ryansliceup Jan 18, 2022
Author

jrehm-mmm
Mar 17, 2023

ryansliceup
Mar 19, 2023
Author