re-enable otel subcommand on Windows

[leehinman]

What does this PR do?

1. Re-enables otel subcommand on Windows
2. Moves launching go routine that processes Windows Service events earlier in boot process
3. Adds Windows machines to otel integration tests

Why is it important?

Previously when Windows was added, elastic-agent did not respond to the Windows Service manager quickly enough that it was starting and was deemed "unresponsive". Moving the go routine that responds to the Windows Service Manager earlier in the boot process should make this less likely. Given go's design of DLL loading and init code we can't eliminate this completely.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

None.

How to test this PR locally

mage integration:auth
mage integration:test

Related issues

• Closes [Windows] Service startup failing on CI with otel dependencies linked #4976
• Closes [Windows] Agent should respect Windows service manager requirements on startup #5710

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[mergify]

This pull request does not have a backport label. Could you fix it @leehinman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 4976_windows_startup upstream/4976_windows_startup
git merge upstream/main
git push upstream 4976_windows_startup

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[mauri870]

Codewise looks good, tests are passing so I assume the changes did fix the issue. I guess worst case scenario is that this starts flakying on CI.

[michalpristas]

nice optimisation, thanks for that @leehinman

[michalpristas]

👍

[jlind23]

@leehinman do you recall a reason why it was not backported to 8.17?

[mlunadia]

@jlind23 can we get a code snippet for the Windows onboarding flow?

[michalpristas]

this works on powershell
i kept it readable. you can concise later merging multiple commands into a single line

# Download and extract
$distroPath = "elastic-distro-8.17.2-windows-x86_64";$zipFile = "$distroPath.zip"


# Disable progress bar, it slows down the download
$ProgressPreference = 'SilentlyContinue';Invoke-WebRequest -Uri "https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.17.2-windows-x86_64.zip" -OutFile $zipFile;
New-Item -ItemType Directory -Force -Path $distroPath | Out-Null
Expand-Archive -Path $zipFile -DestinationPath $distroPath
Move-Item -Path "$distroPath\elastic-agent-8.17.2-windows-x86_64\*" -Destination $distroPath
Remove-Item -Path "$distroPath\elastic-agent-8.17.2-windows-x86_64" -Recurse
Remove-Item -Path $zipFile
Set-Location $distroPath

# Configure otel
Remove-Item -Path .\otel.yml -ErrorAction SilentlyContinue
Copy-Item .\otel_samples\platformlogs_hostmetrics.yml .\otel.yml
New-Item -ItemType Directory -Force -Path .\data\otelcol | Out-Null

# Replace environment variables in otel.yml
$content = Get-Content .\otel.yml
$content = $content -replace '\${env:STORAGE_DIR}', "$PWD\data\otelcol"
$content = $content -replace '\${env:ELASTIC_ENDPOINT}', "https://sample.eu-west-1.aws.qa.cld.elstc.co:443"
$content = $content -replace '\${env:ELASTIC_API_KEY}', "sampleApiKey=="
$content | Set-Content .\otel.yml

notes:

• uri is hardcoded, will need to be replaced
• comparing to linux we don't support arm or 32bit so defaulting to amd64 without an option to have this dynamic
• otel_resources are not packed in any of the zip files because we don't have any for windows. this needs to be fixed and published in future releases or published on github somewhere and fetched from there

cc @mlunadia

[mlunadia]

Amazing! @gbamparop here's the script for running EDOT on windows hosts so we can update the hosts OTel flow.

@michalpristas do we have one for Elastic Agent, so we can also add it as an option?

[michalpristas]

what do you have in mind for agent?

[mlunadia]

This is the current flow for Agent, is it just a matter of adding the icon or do we need a windows specific auto-detect script? @flash1293 maybe you can help?

[flash1293]

@mlunadia The auto-detect script is doing a bunch of stuff that relies on bash, so it's not a trivial change, that's why we didn't add it yet. Maybe it's possible to run it via the WSL, but I don't know enough about Windows to give a good opinion here.

https://github.com/elastic/kibana/blob/ef96cd5d0b9cba3f03a00ff8b6e1e93840119570/x-pack/solutions/observability/plugins/observability_onboarding/public/assets/auto_detect.sh#L3

I'm not sure whether it's worth translating it into powershell or something like this, this is a product question. We could think about a simplified flow based on powershell for windows, but the big question is whether we want it or not.

cc @gbamparop @akhileshpok how can we make this decision?

[flash1293]

Adding @michalpristas s script to the otel flow shouldn't be a problem

[michalpristas]

i was looking into autodetect and porting this to powershell is not that trivial. i think it should be possible but it means it is a normal small task for .net/powershell experienced person that needs to be planned. not something i can generate in one morning

[akhileshpok]

@mlunadia - We can and should enhance the OTel Host onboarding workflow in order to incorporate Windows support. In terms of the the Elastic Agent based onboarding workflow for Hosts, we will need functionality similar to our current auto-detect script for Linux before we can consider merging it.

[gbamparop]

Created https://github.com/elastic/observability-dev/issues/4317 so we can continue the discussion there.