Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

out_datadog: Fix formatting issue when setting a static hostname #10104

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ddrthall
Copy link

@ddrthall ddrthall commented Mar 19, 2025

Providing a static hostname via the dd_hostname configuration options is currently bugged, with an off-by-one error in the message packing loop causing dropped attributes and/or malformed output json. This PR corrects the issue, incrementing the msgpack map size by one when a dd_hostname property is detected.


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Sorry, something went wrong.

@ddrthall
Copy link
Author

Example configuration file:

pipeline:
  inputs:
  - name: dummy
    tag: dummy
    dummy: |
      {
        "message": "From Fluent Bit",
        "test": "default"
      }

  outputs:
  - name: datadog
    match: "*"
    apikey: ${DD_API_KEY}
    dd_hostname: foobar

In the old version of the plugin, the "test" attribute would be dropped and not sent to datadog:
Screenshot 2025-03-19 at 1 08 49 PM

After this change, datadog now sees:
Screenshot 2025-03-19 at 1 10 07 PM

@ddrthall
Copy link
Author

Valgrind output:

==638690==
==638690== HEAP SUMMARY:
==638690==     in use at exit: 0 bytes in 0 blocks
==638690==   total heap usage: 10,305 allocs, 10,305 frees, 63,260,421 bytes allocated
==638690==
==638690== All heap blocks were freed -- no leaks are possible
==638690==
==638690== For lists of detected and suppressed errors, rerun with: -s
==638690== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

@ddrthall
Copy link
Author

Debug output:

[2025/03/19 13:19:16] [ info]  flush time     | 1.000000 seconds
[2025/03/19 13:19:16] [ info]  grace          | 5 seconds
[2025/03/19 13:19:16] [ info]  daemon         | 0
[2025/03/19 13:19:16] [ info] ___________
[2025/03/19 13:19:16] [ info]  inputs:
[2025/03/19 13:19:16] [ info]      dummy
[2025/03/19 13:19:16] [ info] ___________
[2025/03/19 13:19:16] [ info]  filters:
[2025/03/19 13:19:16] [ info] ___________
[2025/03/19 13:19:16] [ info]  outputs:
[2025/03/19 13:19:16] [ info]      datadog.0
[2025/03/19 13:19:16] [ info] ___________
[2025/03/19 13:19:16] [ info]  collectors:
[2025/03/19 13:19:16] [ info] [fluent bit] version=4.0.0, commit=cfc9ba95ef, pid=638863
[2025/03/19 13:19:16] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
[2025/03/19 13:19:16] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/03/19 13:19:16] [ info] [simd    ] disabled
[2025/03/19 13:19:16] [ info] [cmetrics] version=0.9.9
[2025/03/19 13:19:16] [ info] [ctraces ] version=0.6.1
[2025/03/19 13:19:16] [ info] [input:dummy:dummy.0] initializing
[2025/03/19 13:19:16] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2025/03/19 13:19:16] [debug] [dummy:dummy.0] created event channels: read=25 write=26
[2025/03/19 13:19:16] [debug] [datadog:datadog.0] created event channels: read=27 write=28
[2025/03/19 13:19:16] [debug] [output:datadog:datadog.0] scheme: http://
[2025/03/19 13:19:16] [debug] [output:datadog:datadog.0] uri: /api/v2/logs
[2025/03/19 13:19:16] [debug] [output:datadog:datadog.0] host: http-intake.logs.datad0g.com
[2025/03/19 13:19:16] [debug] [output:datadog:datadog.0] port: 80
[2025/03/19 13:19:16] [debug] [output:datadog:datadog.0] json_date_key: timestamp
[2025/03/19 13:19:16] [debug] [output:datadog:datadog.0] compress_gzip: 0
[2025/03/19 13:19:16] [ info] [sp] stream processor started
[2025/03/19 13:19:17] [debug] [task] created task=0xe955fc0199b0 id=0 OK
[2025/03/19 13:19:18] [debug] [task] created task=0xe955fc027150 id=1 OK
[2025/03/19 13:19:18] [debug] [upstream] KA connection #36 to http-intake.logs.datad0g.com:80 is connected
[2025/03/19 13:19:18] [debug] [http_client] not using http_proxy for header
[2025/03/19 13:19:18] [debug] [upstream] KA connection #35 to http-intake.logs.datad0g.com:80 is connected
[2025/03/19 13:19:18] [debug] [http_client] not using http_proxy for header
[2025/03/19 13:19:18] [debug] [output:datadog:datadog.0] http://http-intake.logs.datad0g.com, port=80, HTTP status=202 payload={}
[2025/03/19 13:19:18] [debug] [upstream] KA connection #35 to http-intake.logs.datad0g.com:80 is now available
[2025/03/19 13:19:18] [debug] [out flush] cb_destroy coro_id=1
[2025/03/19 13:19:18] [debug] [task] destroy task=0xe955fc027150 (task_id=1)
[2025/03/19 13:19:18] [debug] [output:datadog:datadog.0] http://http-intake.logs.datad0g.com, port=80, HTTP status=202 payload={}
[2025/03/19 13:19:18] [debug] [upstream] KA connection #36 to http-intake.logs.datad0g.com:80 is now available
[2025/03/19 13:19:18] [debug] [out flush] cb_destroy coro_id=0
[2025/03/19 13:19:18] [debug] [task] destroy task=0xe955fc0199b0 (task_id=0)
[2025/03/19 13:19:19] [debug] [task] created task=0xe955fc01a650 id=0 OK
[2025/03/19 13:19:19] [debug] [upstream] KA connection #35 to http-intake.logs.datad0g.com:80 has been assigned (recycled)
[2025/03/19 13:19:19] [debug] [http_client] not using http_proxy for header
[2025/03/19 13:19:19] [debug] [output:datadog:datadog.0] http://http-intake.logs.datad0g.com, port=80, HTTP status=202 payload={}
[2025/03/19 13:19:19] [debug] [upstream] KA connection #35 to http-intake.logs.datad0g.com:80 is now available
[2025/03/19 13:19:19] [debug] [out flush] cb_destroy coro_id=2
[2025/03/19 13:19:19] [debug] [task] destroy task=0xe955fc01a650 (task_id=0)
[2025/03/19 13:19:20] [debug] [task] created task=0xe955fc019870 id=0 OK
[2025/03/19 13:19:20] [debug] [upstream] KA connection #36 to http-intake.logs.datad0g.com:80 has been assigned (recycled)
[2025/03/19 13:19:20] [debug] [http_client] not using http_proxy for header
[2025/03/19 13:19:20] [debug] [output:datadog:datadog.0] http://http-intake.logs.datad0g.com, port=80, HTTP status=202 payload={}
[2025/03/19 13:19:20] [debug] [upstream] KA connection #36 to http-intake.logs.datad0g.com:80 is now available
[2025/03/19 13:19:20] [debug] [out flush] cb_destroy coro_id=3
[2025/03/19 13:19:20] [debug] [task] destroy task=0xe955fc019870 (task_id=0)```

…dd_hostname

Signed-off-by: Ryan Hall <ryan.hall@datadoghq.com>
@ddrthall ddrthall force-pushed the ryan.hall/fix-ddhostname branch from 4cc69e5 to bc7f154 Compare March 19, 2025 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants