You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
I have an issue with a setting up a multiline parser that captures logs properly, according to the documentation and forums this should work but the parser does not capture the full log.
To Reproduce
Set up an stdout using json as the format, have the input read a file containing the following log -
2024-12-24 01:45:32,717 [STP SmartThreadPool Thread #17136] ERROR customer.messages.MessageBL [(null)] - Internal Sender Error: GetMessageFromQueue [Exception:
=================================
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)
The network path was not found
at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling, SqlAuthenticationProviderManager sqlAuthProviderManager)
at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at customer.messages.MessageDAL.InsertMessage(MessageToServer_8267 message, Int32 dealerId, Hashtable phonesHashTable, String segmentLengthCalculationFormula, String segmentLengthCalculationFormula_ISO, String segmentsCalculationFormula, String infoPageSegmentCalculationFormula, ArrayList& projectIdList, Dictionary`2 quotasNeeded, Dictionary`2 messagesText, Int32& totalSegments, Decimal& totalSegmentsForBilling) in C:\JWS\DEBUG\projectInternalRSenderService\project\branches\47\libs\customer.messages\customer.messages.v1\MessageDAL.cs:line 1
at 1.messages.MessageBL.GetMessageFromQueue(ProjectsInQueue projectsInQueue, Boolean isTestMode, Int32& activityCounter) in C:\JWS\DEBUG\projectInternalRSenderService\project\branches\47\libs\customer.messages\customer.messages.v1\MessageBL.cs:line 1
=================================] - Param[(null)]
Duplicate the log multiple times
See that the log does not fully parse, it will almost always miss a line
Expected behavior
The stdout output should print the full log including the =================================] - Param[(null)] line.
Your Environment
Version used:
Configuration:
[SERVICE]
flush 1
daemon Off
log_level debug
parsers_file parsers.conf
plugins_file plugins.conf
http_server Off
http_listen 0.0.0.0
http_port 2020
storage.metrics on
storage.path /tmp/storage
storage.sync normal
storage.checksum off
storage.backlog.mem_limit 5M
[INPUT]
Name tail
Path C:\customer\logs\iis\\\*.log
Tag iis-logs
DB C:\Program Files\fluent-bit\dbs\iis-logs.db
Refresh_Interval 5
Skip_Long_Lines Off
Buffer_Chunk_Size 256k
Buffer_Max_Size 512k
Path_Key app_path
Read_From_Head true
[INPUT]
Name tail
Path C:\customer\logs\services\\\.log
Tag customer-services
DB C:\Program Files\fluent-bit\dbs\customer-services.db
Refresh_Interval 2
Skip_Long_Lines Off
Buffer_Chunk_Size 256k
Buffer_Max_Size 512k
Path_Key app_path
Read_From_Head true
[INPUT]
Name tail
Path C:\customer\logs\web\\\.log
Tag web-services
DB C:\Program Files\fluent-bit\dbs\web-services.db
Refresh_Interval 5
Skip_Long_Lines Off
Buffer_Chunk_Size 256k
Buffer_Max_Size 512k
Path_Key app_path
Read_From_Head true
[FILTER]
Name multiline
Match customer-services
Multiline.Key_Content log
Multiline.Parser multiline-regex
Emitter_Mem_Buf_Limit 10M
[FILTER]
Name multiline
Match web-services
Multiline.Key_Content log
Multiline.Parser multiline-regex
[FILTER]
Name multiline
Match iis-logs
Multiline.Key_Content log
Multiline.Parser multiline-regex
[FILTER]
Name parser
Match iis-logs
Key_Name log
Parser app_name_parser
Reserve_Data True
Preserve_Key true
[FILTER]
Name record_modifier
Match iis-logs
Record hostname ${HOSTNAME}
Remove_key app_path
[FILTER]
Name parser
Match customer-services
Key_Name app_path
Parser app_name_parser
Reserve_Data True
Preserve_Key true
[FILTER]
Name record_modifier
Match customer-services
Record hostname ${HOSTNAME}
Remove_key app_path
[FILTER]
Name parser
Match web-services
Key_Name app_path
Parser app_name_parser
Reserve_Data True
Preserve_Key true
[FILTER]
Name record_modifier
Match web-services
Record hostname ${HOSTNAME}
Remove_key app_path
[OUTPUT]
Name stdout
Format json
Match customer-services
Operating System and version:
Filters and plugins:
Multiline Parser -
[MULTILINE_PARSER]
name multiline-regex
type regex
flush_timeout 20000
rule "start_state" "/^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})(.*)$/m" "cont"
rule "cont" "/^(?!\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})(.*)$/m" "cont"
Is there anything wrong with the setup? I have tried changing the log, deleting lines, slimming it down, removing special characters, and still there are issues.
The text was updated successfully, but these errors were encountered:
Bug Report
Describe the bug
I have an issue with a setting up a multiline parser that captures logs properly, according to the documentation and forums this should work but the parser does not capture the full log.
To Reproduce
Set up an stdout using json as the format, have the input read a file containing the following log -
Duplicate the log multiple times
See that the log does not fully parse, it will almost always miss a line
Expected behavior
The stdout output should print the full log including the =================================] - Param[(null)] line.
Your Environment
Multiline Parser -
Is there anything wrong with the setup? I have tried changing the log, deleting lines, slimming it down, removing special characters, and still there are issues.
The text was updated successfully, but these errors were encountered: