Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiline Parser not working as expected #9796

Open
simcowhist opened this issue Jan 5, 2025 · 0 comments
Open

Multiline Parser not working as expected #9796

simcowhist opened this issue Jan 5, 2025 · 0 comments

Comments

@simcowhist
Copy link

Bug Report

Describe the bug
I have an issue with a setting up a multiline parser that captures logs properly, according to the documentation and forums this should work but the parser does not capture the full log.

To Reproduce
Set up an stdout using json as the format, have the input read a file containing the following log -

2024-12-24 01:45:32,717 [STP SmartThreadPool Thread #17136] ERROR customer.messages.MessageBL [(null)] - Internal Sender Error: GetMessageFromQueue [Exception:
=================================
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)
The network path was not found
   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling, SqlAuthenticationProviderManager sqlAuthProviderManager)
   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at customer.messages.MessageDAL.InsertMessage(MessageToServer_8267 message, Int32 dealerId, Hashtable phonesHashTable, String segmentLengthCalculationFormula, String segmentLengthCalculationFormula_ISO, String segmentsCalculationFormula, String infoPageSegmentCalculationFormula, ArrayList& projectIdList, Dictionary`2 quotasNeeded, Dictionary`2 messagesText, Int32& totalSegments, Decimal& totalSegmentsForBilling) in C:\JWS\DEBUG\projectInternalRSenderService\project\branches\47\libs\customer.messages\customer.messages.v1\MessageDAL.cs:line 1
   at 1.messages.MessageBL.GetMessageFromQueue(ProjectsInQueue projectsInQueue, Boolean isTestMode, Int32& activityCounter) in C:\JWS\DEBUG\projectInternalRSenderService\project\branches\47\libs\customer.messages\customer.messages.v1\MessageBL.cs:line 1
=================================] - Param[(null)]

Duplicate the log multiple times
See that the log does not fully parse, it will almost always miss a line

Expected behavior
The stdout output should print the full log including the =================================] - Param[(null)] line.

Your Environment

  • Version used:
  • Configuration:
[SERVICE]
    flush        1
    daemon       Off
    log_level    debug
    parsers_file parsers.conf
    plugins_file plugins.conf
    http_server  Off
    http_listen  0.0.0.0
    http_port    2020
    storage.metrics on
    storage.path /tmp/storage
    storage.sync normal
    storage.checksum off
    storage.backlog.mem_limit 5M


[INPUT]
    Name        tail
    Path        C:\customer\logs\iis\\\*.log
    Tag         iis-logs
    DB          C:\Program Files\fluent-bit\dbs\iis-logs.db
    Refresh_Interval 5
    Skip_Long_Lines Off
    Buffer_Chunk_Size 256k
    Buffer_Max_Size 512k
    Path_Key    app_path
    Read_From_Head true

[INPUT]
    Name        tail
    Path        C:\customer\logs\services\\\.log
    Tag         customer-services
    DB         C:\Program Files\fluent-bit\dbs\customer-services.db
    Refresh_Interval 2
    Skip_Long_Lines Off
    Buffer_Chunk_Size 256k
    Buffer_Max_Size 512k
    Path_Key    app_path
    Read_From_Head true

[INPUT]
    Name        tail
    Path        C:\customer\logs\web\\\.log
    Tag         web-services
    DB          C:\Program Files\fluent-bit\dbs\web-services.db
    Refresh_Interval 5
    Skip_Long_Lines Off
    Buffer_Chunk_Size 256k
    Buffer_Max_Size 512k
    Path_Key    app_path
    Read_From_Head true

[FILTER]
    Name multiline
    Match customer-services
    Multiline.Key_Content log
    Multiline.Parser multiline-regex
    Emitter_Mem_Buf_Limit 10M

[FILTER]
    Name multiline
    Match web-services
    Multiline.Key_Content log
    Multiline.Parser multiline-regex

[FILTER]
    Name multiline
    Match iis-logs
    Multiline.Key_Content log
    Multiline.Parser multiline-regex

[FILTER]
    Name parser
    Match iis-logs
    Key_Name log
    Parser app_name_parser
    Reserve_Data True
    Preserve_Key true

[FILTER]
    Name record_modifier
    Match iis-logs
    Record hostname ${HOSTNAME}
    Remove_key app_path

[FILTER]
    Name parser
    Match customer-services
    Key_Name app_path
    Parser app_name_parser
    Reserve_Data True
    Preserve_Key true

[FILTER]
    Name record_modifier
    Match customer-services
    Record hostname ${HOSTNAME}
    Remove_key app_path

[FILTER]
    Name parser
    Match web-services
    Key_Name app_path
    Parser app_name_parser
    Reserve_Data True
    Preserve_Key true

[FILTER]
    Name record_modifier
    Match web-services
    Record hostname ${HOSTNAME}
    Remove_key app_path


[OUTPUT]
    Name stdout
    Format json
    Match customer-services

  • Operating System and version:
  • Filters and plugins:
    Multiline Parser -
[MULTILINE_PARSER]
    name          multiline-regex
    type          regex
    flush_timeout 20000
    rule      "start_state"   "/^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})(.*)$/m" "cont"
    rule      "cont"          "/^(?!\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})(.*)$/m" "cont"


Is there anything wrong with the setup? I have tried changing the log, deleting lines, slimming it down, removing special characters, and still there are issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant