out_azure_kusto: azure managed identity support added

[dceravigupta]

Currently, the Azure Kusto output plugin supports only Service Principal-based authentication. This PR introduces support for Managed Service Identity (MSI), including both system-assigned and user-assigned identities.
https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview

To implement this feature, I leveraged the existing OAuth2 support in Fluent Bit, as MSI follows the OAuth2.0 protocol (with the exception of the HTTP Request part), which significantly simplifies the implementation.

This feature has been asked multiple times in the past. Example:
#8763

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• [Done] Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• [Done] Documentation required for this feature
  Managed Identity support in Azure Kusto output plugin fluent-bit-docs#1578

Backporting

• [] Backport to latest stable release.

---

Example configuration:

User assigned Managed Service Identity:

[INPUT]
    Name tail
    Path logs
    Tag applicationlogs

[OUTPUT]
    Name azure_kusto
    Match applicationlogs
    managed_identity_client_id <MSI GUID>
    Ingestion_Endpoint <Azure Kusto ingestion URL>
    Database_Name <Database Name>
    Table_Name <Table Name>
    Ingestion_Mapping_Reference <Mapping>

System assigned Managed Service Identity:

[INPUT]
    Name tail
    Path logs
    Tag applicationlogs

[OUTPUT]
    Name azure_kusto
    Match applicationlogs
    managed_identity_client_id SYSTEM
    Ingestion_Endpoint <Azure Kusto ingestion URL>
    Database_Name <Database Name>
    Table_Name <Table Name>
    Ingestion_Mapping_Reference <Mapping>

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[patrick-stephens]

Please follow the contribution guidelines by updating your commit messages (and squash them ideally into a smaller amount) and resolve the failing DCO check. Until this is done we cannot review it.

[patrick-stephens]

I think we need unit tests at least as well for the new functionality.

[patrick-stephens]

It might be better to rebase your PR off master rather than include the merge commits

[dceravigupta]

Rebased from master and squashed all the changes into a single commit. Will work on adding unit tests next.

[dceravigupta]

@patrick-stephens, while adding standard json_invalid, json_long and json_small UTs for out_azure_kusto, I'm running into an issue where since we don't have a real azure kusto account configured for tests, the json_long and jsong_small tests are giving following errors:

$ ./bin/flb-rt-out_azure_kusto
Test json_invalid...                            [ OK ]
Test json_long...                               [2025/03/10 21:33:28] [error] [output:azure_kusto:azure_kusto.0] cannot create upstream connection
[2025/03/10 21:33:28] [error] [output:azure_kusto:azure_kusto.0] error getting ingestion storage resources
[2025/03/10 21:33:28] [error] [output:azure_kusto:azure_kusto.0] cannot load ingestion resources
[2025/03/10 21:33:28] [error] [output:azure_kusto:azure_kusto.0] cannot create upstream connection
[2025/03/10 21:33:28] [error] [output:azure_kusto:azure_kusto.0] error getting ingestion storage resources
[2025/03/10 21:33:28] [error] [output:azure_kusto:azure_kusto.0] cannot load ingestion resources
[2025/03/10 21:33:28] [error] [engine] chunk '34897-1741642407.288795466.flb' cannot be retried: task_id=0, input=lib.0 > output=azure_kusto.0
[ OK ]
Test json_small...                              [2025/03/10 21:33:30] [error] [output:azure_kusto:azure_kusto.0] cannot create upstream connection
[2025/03/10 21:33:30] [error] [output:azure_kusto:azure_kusto.0] error getting ingestion storage resources
[2025/03/10 21:33:30] [error] [output:azure_kusto:azure_kusto.0] cannot load ingestion resources
[ OK ]
SUCCESS: All unit tests have passed.

Test Config:

[INPUT]
    Name lib
    Tag test
	
[OUTPUT]
    Name azure_kusto
    Match test
    managed_identity_client_id SYSTEM
    Ingestion_Endpoint https://ingest-CLUSTER.kusto.windows.net
    Database_Name telemetrydb
    Table_Name logs

I believe, even if we configure a valid kusto account, they will continue to fail unless we run them on an actual Azure VM with managed identity configured?

What do you recommend in this case? Should I simply add json_invalid test alone or add all three of them?

[patrick-stephens]

@patrick-stephens, while adding standard json_invalid, json_long and json_small UTs for out_azure_kusto, I'm running into an issue where since we don't have a real azure kusto account configured for tests, the json_long and jsong_small tests are giving following errors:

$ ./bin/flb-rt-out_azure_kusto
Test json_invalid...                            [ OK ]
Test json_long...                               [2025/03/10 21:33:28] [error] [output:azure_kusto:azure_kusto.0] cannot create upstream connection
[2025/03/10 21:33:28] [error] [output:azure_kusto:azure_kusto.0] error getting ingestion storage resources
[2025/03/10 21:33:28] [error] [output:azure_kusto:azure_kusto.0] cannot load ingestion resources
[2025/03/10 21:33:28] [error] [output:azure_kusto:azure_kusto.0] cannot create upstream connection
[2025/03/10 21:33:28] [error] [output:azure_kusto:azure_kusto.0] error getting ingestion storage resources
[2025/03/10 21:33:28] [error] [output:azure_kusto:azure_kusto.0] cannot load ingestion resources
[2025/03/10 21:33:28] [error] [engine] chunk '34897-1741642407.288795466.flb' cannot be retried: task_id=0, input=lib.0 > output=azure_kusto.0
[ OK ]
Test json_small...                              [2025/03/10 21:33:30] [error] [output:azure_kusto:azure_kusto.0] cannot create upstream connection
[2025/03/10 21:33:30] [error] [output:azure_kusto:azure_kusto.0] error getting ingestion storage resources
[2025/03/10 21:33:30] [error] [output:azure_kusto:azure_kusto.0] cannot load ingestion resources
[ OK ]
SUCCESS: All unit tests have passed.

Test Config:

[INPUT]
    Name lib
    Tag test
	
[OUTPUT]
    Name azure_kusto
    Match test
    managed_identity_client_id SYSTEM
    Ingestion_Endpoint https://ingest-CLUSTER.kusto.windows.net
    Database_Name telemetrydb
    Table_Name logs

I believe, even if we configure a valid kusto account, they will continue to fail unless we run them on an actual Azure VM with managed identity configured?

What do you recommend in this case? Should I simply add json_invalid test alone or add all three of them?

Is there no way to simulate it? Relying on a managed service is not great for these unit tests as people want to run them locally/forks.

[dceravigupta]

Is there no way to simulate it? Relying on a managed service is not great for these unit tests as people want to run them locally/forks.

It is hard to simulate managed identity locally hence only keeping the test which works e2e locally.

[dceravigupta]

@patrick-stephens is there a way to trigger the pending checks against my PR? They haven't run in the last couple of days, so I'm wondering if they follow a predefined schedule or need to be triggered manually.

I'm ready to put my PR in review queue.

[patrick-stephens]

@patrick-stephens is there a way to trigger the pending checks against my PR? They haven't run in the last couple of days, so I'm wondering if they follow a predefined schedule or need to be triggered manually.

I'm ready to put my PR in review queue.

It's a security precaution to not run them for first time contributors so I've triggered them now