[windows] capture early start-up errors

[leehinman]

Describe the enhancement:

Capture standard error of elastic-agent when run as a Windows service. This isn't necessary under Linux since systemd will capture the standard error.

Describe a specific use case for the enhancement or feature:

There are several steps that happen before internal logging is started. If elastic-agent fails to start before internal logging is started the only place the error can be found is on the standard error of the elastic-agent process. When elastic-agent is run as a service under Windows, the standard error is not captured. This is a problem because elastic-agent can fail to start and there is no record of the failure. Adding this enhancement will allow us to "see" the error.

What is the definition of done?

When elastic-agent fails as a service under Windows the standard error of the process can be retrieved.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

For Windows, Lee suggested the best way to do this would be to have agent write to the windows event log.

[strawgate]

Does this run the risk of us creating a logging loop? Agent logs to Event Viewer, Winlogbeat reads Event Viewer, Winlogbeat crashes, Agent logs the crash to event viewer, winlogbeat reads the crash, etc....

[cmacknz]

We could narrow the scope of when we use the event log to only the period before we have our JSON logger setup and know the location those log files should be written to on disk.

This will encompass a point in time where no subprocess (like winlogbeat) are running, and wouldn't run the risk of a logging loop. Winlogbeat could read a record of previous agent crashes, but by the time winlogbeat is running agent is no longer writing to the event log.

We have similar problems with the monitoring filestream instance that needs some special handling and processors, I don't think we want to deal with any of that for regular uses of winlogbeat.

[leehinman]

We could narrow the scope of when we use the event log to only the period before we have our JSON logger setup and know the location those log files should be written to on disk.

I think we can make it even more limited in scope. If we just add writing an EventLog at

elastic-agent/internal/pkg/agent/cmd/run.go

Line 80 in b2532cf

fmt.Fprintf(streams.Err, "Error: %v\n%s\n", err, troubleshootMessage())

Then we only log to the EventLog if run fails and the error message would contain the info we would normally get on stderr if you were running from the command line. So at most we would only write one EventLog message, and that would only be if elastic-agent run failed.

[blakerouse]

I hit this before with WIndows and added this in the unprivileged work for Windows - https://github.com/elastic/elastic-agent/blame/main/internal/pkg/agent/cmd/run.go#L145

It doesn't cover all cases where it could fail, but it does a much better job then it did before. Logging to the Windows Event Log in the worse case scenario would be nice to have.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)