Add wait for watcher

[pchila]

What does this PR do?

This PR adds a wait after launching the upgrade watcher to make sure that the process is up and running before restarting agent.

Why is it important?

We want to avoid race conditions that may happen in case of downgrade where after restart an older watcher that does not understand the new update marker format may break the whole agent installation.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes [Flaky Test]: TestStandaloneUpgrade/Upgrade_8.12.0-SNAPSHOT_to_8.13.0-SNAPSHOT_(privileged) – error installing package: failed to copy source directory... /opt/Elastic/Agent/elastic-agent: file exists #4204
• Closes The version of the watcher that gets the watcher lock is not deterministic. #4228

Use cases

Screenshots

Logs

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[AndersonQ]

if you capture the error you should get different errors, which will explain if it was a deadline exceeded or a context cancelled.

Suggested change

	case <-watcherContext.Done():
	log.Error("upgrade watcher did not start watching within %s or context has expired", waitTime)
	return ErrWatcherNotStarted
	case err := <-watcherContext.Done():
	log.Errorf("upgrade watcher did not start watching within %s or context has expired: %w", waitTime, err)
	return ErrWatcherNotStarted

[rdner]

I think there is a legit failure on Mac in one of the unit tests:

=== Failed
=== FAIL: internal/pkg/agent/application/upgrade TestWaitForWatcher/Runaround_path:_marker_is_jumping_around_and_landing_on_watching (0.10s)
    upgrade_test.go:856:
        	Error Trace:	/Users/admin/builds/bk-agent-prod-orka-1707490299318330142/elastic/elastic-agent/internal/pkg/agent/application/upgrade/upgrade_test.go:856
        	Error:      	Received unexpected error:
        	            	watcher did not start in time
        	            	context deadline exceeded
        	Test:       	TestWaitForWatcher/Runaround_path:_marker_is_jumping_around_and_landing_on_watching
        	Messages:   	waitForWatcher /tmp/TestWaitForWatcherRunaround_path_marker_is_jumping_around_and_landing_on_watching3782740199/001/.update-marker, [UPG_REQUESTED UPG_SCHEDULED UPG_DOWNLOADING UPG_EXTRACTING UPG_REPLACING UPG_RESTARTING UPG_WATCHING], 1ms, 100ms)
=== FAIL: internal/pkg/agent/application/upgrade TestWaitForWatcher (0.29s)

[pchila]

I think there is a legit failure on Mac in one of the unit tests:

=== Failed
=== FAIL: internal/pkg/agent/application/upgrade TestWaitForWatcher/Runaround_path:_marker_is_jumping_around_and_landing_on_watching (0.10s)
    upgrade_test.go:856:
        	Error Trace:	/Users/admin/builds/bk-agent-prod-orka-1707490299318330142/elastic/elastic-agent/internal/pkg/agent/application/upgrade/upgrade_test.go:856
        	Error:      	Received unexpected error:
        	            	watcher did not start in time
        	            	context deadline exceeded
        	Test:       	TestWaitForWatcher/Runaround_path:_marker_is_jumping_around_and_landing_on_watching
        	Messages:   	waitForWatcher /tmp/TestWaitForWatcherRunaround_path_marker_is_jumping_around_and_landing_on_watching3782740199/001/.update-marker, [UPG_REQUESTED UPG_SCHEDULED UPG_DOWNLOADING UPG_EXTRACTING UPG_REPLACING UPG_RESTARTING UPG_WATCHING], 1ms, 100ms)
=== FAIL: internal/pkg/agent/application/upgrade TestWaitForWatcher (0.29s)

Yeah, I already slowed the test down for MacOS and Windows but it seems that it is still a bit too fast...

[pchila]

@rdner I removed the backport v8.12.0 label from this PR in favor of #4238 which contains only the fix for the GRPC server shutdown.
I will still wait for https://buildkite.com/elastic/elastic-agent/builds/7004 to complete to verify that this fixed the problem as it was more apparent on this branch and then I will remove the commit with the GRPC fix from this branch

[rdner]

@pchila the backport label was for the watcher fix, we need to backport it because it's failing on 8.12 too, see one of the recent builds https://buildkite.com/elastic/elastic-agent/builds/7000#018d9c5d-a923-4f6a-90f5-218885912d91

Unless you think there is a strong reason why we should not backport this fix to 8.12, please set the label back.

[pchila]

I am not sure we can backport the fix for the watcher wait because it is built upon major changes contained in #4193

I will add the label back as a reminder but I doubt it will be a straightforward backport

[elastic-sonarqube]

Quality Gate failed

Failed conditions

37.5% Coverage on New Code (required ≥ 40%)

See analysis details on SonarQube

[pchila]

https://buildkite.com/elastic/elastic-agent/builds/7034 shows 4 failures:

• 2 are known issues: [Flaky Test]: TestStandaloneUpgradeWithGPGFallback – failed to verify SHA512 hash #4220 for TestStandaloneUpgradeWithGPGFallback [Flaky Test]: TestLogIngestionFleetManaged – Failed to remove install directory #4164 for TestStandaloneUpgradeFailsWhenUpgradeIsInProgress
• the remaining 2 are Fleet issues unrelated to this fix

Merging to feature branch