[Flaky Test]: 8.12 branch, Multiple tests – The service cannot accept control messages at this time

[rdner]

Failing test case

TestUpgrade*, TestStandaloneUpgrade*

Error message

The service cannot accept control messages at this time

Build

https://buildkite.com/elastic/elastic-agent/builds/7231#018dade7-b360-4571-b713-e7963dca3de7

OS

Windows

Stacktrace and notes

fixture.go:526: >> running binary with: [C:\Users\windows\AppData\Local\Temp\TestStandaloneDowngradeToSpecificSnapshotBuild493807473\001\elastic-agent-8.12.0-SNAPSHOT-windows-x86_64\elastic-agent.exe install --force --non-interactive]
    upgrade_downgrade_test.go:85: 
        	Error Trace:	C:/Users/windows/agent/testing/integration/upgrade_downgrade_test.go:85
        	Error:      	Received unexpected error:
        	            	failed to install start agent (err: error running agent install command: exit status 1) [output: Installing in non-interactive mode.
        	            	
[    ] Uninstalling current Elastic Agent  [0s] 
                                                

[    ] Stopping service  [0s] 
                                                

[    ] Failed to issue stop service  [0s] 
                                                

[    ] Failed to uninstall current Elastic Agent  [0s] Error uninstalling.  Printing logs
        	            	Error: error installing package: failed to uninstall Agent at (C:\Program Files\Elastic): failed to issue stop service (Elastic Agent): The service cannot accept control messages at this time.
        	            	For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.12/fleet-troubleshooting.html
        	            	]
        	Test:       	TestStandaloneDowngradeToSpecificSnapshotBuild
--- FAIL: TestStandaloneDowngradeToSpecificSnapshotBuild (46.78s)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[leehinman]

There is an earlier failure:

{"Time":"2024-02-15T18:16:37.0134639Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    fixture.go:731: Components were not modified from the fetched artifact\n"}
{"Time":"2024-02-15T18:16:37.0144004Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    fixture.go:526: \u003e\u003e running binary with: [C:\\Users\\windows\\AppData\\Local\\Temp\\TestUpgradeBrokenPackageVersion1822831636\\001\\elastic-agent-8.12.0-SNAPSHOT-windows-x86_64\\elastic-agent.exe version --binary-only --yaml]\n"}
{"Time":"2024-02-15T18:16:37.2355659Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    fixture.go:526: \u003e\u003e running binary with: [C:\\Users\\windows\\AppData\\Local\\Temp\\TestUpgradeBrokenPackageVersion1822831636\\002\\elastic-agent-8.10.4-windows-x86_64\\elastic-agent.exe version --binary-only --yaml]\n"}
{"Time":"2024-02-15T18:16:37.4699289Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    upgrader.go:205: Installing version \"8.12.0-SNAPSHOT\"\n"}
{"Time":"2024-02-15T18:16:37.4699289Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    fixture_install.go:103: [test TestUpgradeBrokenPackageVersion] Inside fixture install function\n"}
{"Time":"2024-02-15T18:16:37.4699289Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    fixture.go:526: \u003e\u003e running binary with: [C:\\Users\\windows\\AppData\\Local\\Temp\\TestUpgradeBrokenPackageVersion1822831636\\001\\elastic-agent-8.12.0-SNAPSHOT-windows-x86_64\\elastic-agent.exe install --force --non-interactive]\n"}
{"Time":"2024-02-15T18:17:11.559908Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    fixture.go:526: \u003e\u003e running binary with: [C:\\Program Files\\Elastic\\Agent\\elastic-agent.exe status --output json]\n"}
{"Time":"2024-02-15T18:17:11.7165526Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    upgrade_broken_package_test.go:83: package version files found: [data/elastic-agent-d0b9e4/package.version]\n"}
{"Time":"2024-02-15T18:17:11.7174456Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    upgrade_broken_package_test.go:88: removing package version file \"C:\\\\Program Files\\\\Elastic\\\\Agent\\\\data\\\\elastic-agent-d0b9e4\\\\package.version\"\n"}
{"Time":"2024-02-15T18:17:11.7175777Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    upgrader.go:240: Upgrading from version \"8.12.0-SNAPSHOT-d0b9e4834be63cadeca3c84c6d6cf1c034b5b35a\" to version \"8.10.4-a92ca32b922dc62e45b532600a8562ee1c82205f\"\n"}
{"Time":"2024-02-15T18:17:11.7175777Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    fixture.go:526: \u003e\u003e running binary with: [C:\\Program Files\\Elastic\\Agent\\elastic-agent.exe upgrade 8.10.4 --source-uri file://C:\\Users\\windows\\agent\\.agent-testing\\artifact --skip-verify]\n"}
{"Time":"2024-02-15T18:17:31.9613623Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    upgrader.go:284: waiting for upgrade watcher to start\n"}
{"Time":"2024-02-15T18:17:42.0173378Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    upgrader.go:289: upgrade watcher started\n"}
{"Time":"2024-02-15T18:17:52.0267431Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    fixture.go:526: \u003e\u003e running binary with: [C:\\Program Files\\Elastic\\Agent\\elastic-agent.exe status --output json]\n"}
{"Time":"2024-02-15T18:17:52.1379442Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    upgrader.go:325: waiting for upgrade watcher to finish\n"}
{"Time":"2024-02-15T18:18:52.1958844Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    upgrader.go:330: upgrade watcher finished\n"}
{"Time":"2024-02-15T18:18:52.1958844Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    fixture.go:526: \u003e\u003e running binary with: [C:\\Program Files\\Elastic\\Agent\\elastic-agent.exe status --output json]\n"}
{"Time":"2024-02-15T18:18:52.2854613Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    fixture_install.go:131: [test TestUpgradeBrokenPackageVersion] Inside fixture cleanup function\n"}
{"Time":"2024-02-15T18:18:52.2854613Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    fixture.go:526: \u003e\u003e running binary with: [C:\\Program Files\\Elastic\\Agent\\elastic-agent.exe uninstall --force]\n"}
{"Time":"2024-02-15T18:23:52.6022647Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"    fixture_install.go:181: \n"}
{"Time":"2024-02-15T18:23:52.6022647Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"        \tError Trace:\tC:/Users/windows/agent/pkg/testing/fixture_install.go:181\n"}
{"Time":"2024-02-15T18:23:52.6022647Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"        \t            \t\t\t\tC:/Program Files/Go/src/testing/testing.go:1150\n"}
{"Time":"2024-02-15T18:23:52.6022647Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"        \t            \t\t\t\tC:/Program Files/Go/src/testing/testing.go:1328\n"}
{"Time":"2024-02-15T18:23:52.6022647Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"        \t            \t\t\t\tC:/Program Files/Go/src/testing/testing.go:1570\n"}
{"Time":"2024-02-15T18:23:52.6022647Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"        \tError:      \tReceived unexpected error:\n"}
{"Time":"2024-02-15T18:23:52.6022647Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"        \t            \terror running uninstall command: exit status 1\n"}
{"Time":"2024-02-15T18:23:52.6022647Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"        \tTest:       \tTestUpgradeBrokenPackageVersion\n"}
{"Time":"2024-02-15T18:23:52.6022647Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"        \tMessages:   \tuninstalling agent failed. Output: \"\"\n"}
{"Time":"2024-02-15T18:23:55.4588901Z","Action":"output","Package":"github.com/elastic/elastic-agent/testing/integration(windows-amd64-2022-upgrade)(sudo)","Test":"TestUpgradeBrokenPackageVersion","Output":"--- FAIL: TestUpgradeBrokenPackageVersion (482.67s)\n"}

And if the 8.10.4 version is the one that is still running because it failed to uninstall we could run into this issue because 8.10.4 is using elastic-agent-libs 0.5.0, which has the bug where the service manager can only receive one request, it hangs after that, returning the dreaded "cannot accept control messages at this time" error message.

[rdner]

@leehinman any thoughts how we can avoid this failures on our CI? Do we have to use this version that contains the bug?

[leehinman]

@leehinman any thoughts how we can avoid this failures on our CI? Do we have to use this version that contains the bug?

I don't think we have to use 8.10.4, one oddity is the test calls PreviousMinor, which is returning the 8.10.4 version, which seems odd, I would have thought it would have been 8.11.X given it is starting from 8.12.0-SNAPSHOT. Probably a good idea to get @pchila or @blakerouse to chime in to see if this is expected behavior for PreviousMinor.

[rdner]

33 tests failed on 8.12 branch because of this issue in https://buildkite.com/elastic/elastic-agent/builds/7264#018dc584-6f9e-4d43-a528-9d4c165b2a00 (not sure about all 33 but most of them).

So, according to our changelog this issue is fixed in 8.11.0 https://www.elastic.co/guide/en/fleet/8.11/release-notes-8.11.0.html

I think we should reconsider using anything below 8.11.0 in our tests.

[rdner]

AFAIK, we no longer release 8.12.x, closing now.