[8.17](backport #42401) Handle authorization errors in Kafka output

[mergify]

Proposed commit message

When there is an authorisation error in the Kafka output, the events are dropped and an error message is logged.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

## Disruptive User Impact
## Author's Checklist

How to test this PR locally

Run the tests added by this PR:

mage buildSystemTestBinary
go test -tags integration -run=TestAuthorisationErrors -v ./tests/integration

If you want to run any Beat yourself, you can modify the test so it only starts the mock and waits, then configure a Beat to connect to it.

Related issues

• Closes Kafka output causes high CPU usage when ACL is missconfigured #42343

## Use cases
## Screenshots
## Logs

---

This is an automatic backport of pull request #42401 done by Mergify.

[mergify]

This pull request has not been merged yet. Could you please review and merge it @belimawr? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @belimawr? 🙏

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/8.17/pr-42401 upstream/mergify/bp/8.17/pr-42401
git merge upstream/8.17
git push upstream mergify/bp/8.17/pr-42401

[mergify]

This pull request has not been merged yet. Could you please review and merge it @belimawr? 🙏

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[belimawr]

CI is failing due to a DNS issue while downloading Go modules: #43044

[belimawr]

Because of #43250, let's not merge it for now.

We might want to merge it once the fix for #43250 is ready so we don't get merge conflicts.

[mergify]

This pull request has not been merged yet. Could you please review and merge it @belimawr? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @belimawr? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @belimawr? 🙏

[pierrehilbert]

@belimawr What is the update for this backport?

[belimawr]

@belimawr What is the update for this backport?

It is the same as before (see #42844 (comment)). When I was talking with Fae about this issue, we understood that it was better not to merge this PR, so at least 8.17.x is not affected by #43250.

Let's have a chat if you want to further discuss this.

[pierrehilbert]

So why not closing that one?

[belimawr]

So why not closing that one?

The current idea is the the fix will build on top of those changes as they have been merged on main already, so once we have the fix, we can merge this PR and then backport the final fix. This way avoid merge conflicts caused by fixing #43250 on a codebase that includes this backport.

However, thinking about it, I can add a note to #43250 about this PR and close it. So at least we don't keep a pending backport open.

Thanks @pierrehilbert !