[Flaky Test]: Multiple test cases fail because of the artifact API / CDN unavailability #4268

rdner · 2024-02-15T10:08:31Z

Any test case that downloads artifacts (e.g. TestStandaloneUpgrade*, TestUpgrade*, TestFleetManagedUpgrade*) may fail because the artifact API or CDN server is unavailable.

Here is a list with a few examples (the timestamp is recorded on the buildkite failure, so it's approximate):

Artifact API

2024-02-12T20:28:05.483Z

https://buildkite.com/elastic/elastic-agent/builds/7062#018d9ec4-9b4a-4877-b32e-4d9810e99d75

    fixture.go:632: >> running binary with: [/tmp/TestStandaloneUpgradeRetryDownload3070379000/001/elastic-agent-8.13.0-SNAPSHOT-linux-x86_64/elastic-agent version --binary-only --yaml]
    upgrade_standalone_retry_test.go:58: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_standalone_retry_test.go:58
        	Error:      	Received unexpected error:
        	            	failed to get a list of builds: 503: bad http status code
        	Test:       	TestStandaloneUpgradeRetryDownload

2024-02-13T21:24:31.816Z

https://buildkite.com/elastic/elastic-agent/builds/7141#018da41e-2c94-4883-a427-5120735369e5

    upgrade_broken_package_test.go:45: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_broken_package_test.go:45
        	Error:      	Received unexpected error:
        	            	error retrieving versions from Artifact API: getting versions: executing http request &{GET https://artifacts-api.elastic.co/v1/versions/ HTTP/1.1 1 1 map[]   0 [] false artifacts-api.elastic.co map[] map[]  map[]      0xc0000fb960}: Get "https://artifacts-api.elastic.co/v1/versions/": dial tcp 35.188.12.98:443: i/o timeout
        	Test:       	TestUpgradeBrokenPackageVersion

CDN

2024-02-13T16:11:14.713Z

https://buildkite.com/elastic/elastic-agent/builds/7119#018da2fc-1a7d-443b-9ed7-89566b637e75

fetcher_artifact.go:344: Downloading artifact progress 39.78%
    upgrade_standalone_test.go:77: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_standalone_test.go:77
        	            				/home/ubuntu/agent/testing/integration/upgrade_standalone_test.go:52
        	Error:      	Received unexpected error:
        	            	failed to prepare the startFixture: failed to download https://staging.elastic.co/7.17.18-bdb3703e/downloads/beats/elastic-agent/elastic-agent-7.17.18-linux-arm64.tar.gz: failed to write file /home/ubuntu/agent/.agent-testing/artifact/elastic-agent-7.17.18-linux-arm64.tar.gz: stream error: stream ID 1; INTERNAL_ERROR; received from peer
        	Test:       	TestStandaloneUpgrade/Upgrade_7.17.18_to_8.13.0-SNAPSHOT_(privileged)

2024-02-14T20:02:31.839Z

https://buildkite.com/elastic/elastic-agent/builds/7183#018da8f0-9957-45fc-9622-696d327bebf7

    fetcher_artifact.go:344: Downloading artifact progress 90.63%
    upgrade_fleet_test.go:406: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:406
        	            				/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:149
        	            				/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:127
        	Error:      	Received unexpected error:
        	            	failed to download https://snapshots.elastic.co/8.13.0-772867d3/downloads/beats/elastic-agent/elastic-agent-8.13.0-SNAPSHOT-linux-x86_64.tar.gz: failed to write file /tmp/TestFleetAirGappedUpgradePrivileged2800565667/002/downloads/beats/elastic-agent/beats/elastic-agent/elastic-agent-8.13.0-SNAPSHOT-linux-x86_64.tar.gz: read tcp 10.128.0.249:44254->34.120.127.130:443: read: connection reset by peer
        	Test:       	TestFleetAirGappedUpgradePrivileged
        	Messages:   	could not download agent 8.13.0-SNAPSHOT
--- FAIL: TestFleetAirGappedUpgradePrivileged (78.33s)

The text was updated successfully, but these errors were encountered:

elasticmachine · 2024-02-15T10:08:33Z

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

rdner · 2024-02-15T15:18:20Z

failed to get a list of builds: 503: bad http status code

This error from the artifact API was a known outage that lasted for 4 minutes.

rdner · 2024-02-20T08:23:19Z

Another failure due to a timeout on GET https://artifacts-api.elastic.co/v1/versions/8.13.0-SNAPSHOT/builds/

upgrade_uninstall_test.go:60: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_uninstall_test.go:60
        	Error:      	Received unexpected error:
        	            	failed to get a list of builds: getting builds for version 8.13.0-SNAPSHOT: executing http request &{GET https://artifacts-api.elastic.co/v1/versions/8.13.0-SNAPSHOT/builds/ HTTP/1.1 1 1 map[]   0 [] false artifacts-api.elastic.co map[] map[]  map[]      0x40002ae000}: Get "https://artifacts-api.elastic.co/v1/versions/8.13.0-SNAPSHOT/builds/": dial tcp 35.188.12.98:443: i/o timeout
        	Test:       	TestStandaloneUpgradeUninstallKillWatcher
--- FAIL: TestStandaloneUpgradeUninstallKillWatcher (43.64s)

https://buildkite.com/elastic/elastic-agent/builds/7268#018dc3d2-0f6e-438c-8510-835b6e4334af

rdner · 2024-02-20T09:06:01Z

I think we should implement retries in our artifact API client to mitigate this problem.

rdner · 2024-02-20T13:50:28Z

Another failure on GET https://artifacts-api.elastic.co/v1/versions/ in https://buildkite.com/elastic/elastic-agent/builds/7275#018dc65b-8d69-4b23-be5f-81d07defc3fd

=== RUN   TestStandaloneUpgradeRollback
    upgrade_rollback_test.go:55: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_rollback_test.go:55
        	Error:      	Received unexpected error:
        	            	error retrieving versions from Artifact API: getting versions: executing http request &{GET https://artifacts-api.elastic.co/v1/versions/ HTTP/1.1 1 1 map[]   0 [] false artifacts-api.elastic.co map[] map[]  map[]      0x4000258000}: Get "https://artifacts-api.elastic.co/v1/versions/": dial tcp 35.188.12.98:443: i/o timeout
        	Test:       	TestStandaloneUpgradeRollback
--- FAIL: TestStandaloneUpgradeRollback (30.01s)

rdner · 2024-02-21T09:26:17Z

Another failure in pretty much all the tests that use the artifact API, this time it returned 503 https://buildkite.com/elastic/elastic-agent/builds/7305#018dc85a-fece-4c24-b9b5-ce18d1435ddc

Error sample:

 upgrade_fleet_test.go:90: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:90
        	            				/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:49
        	Error:      	Received unexpected error:
        	            	failed to find snapshot URI for version 8.13.0-SNAPSHOT: failed to find package URL: https://artifacts-api.elastic.co/v1/search/8.13.0-SNAPSHOT/elastic-agent; bad status: 503 Service Unavailable
        	Test:       	TestFleetManagedUpgradeUnprivileged

rdner · 2024-02-21T21:21:40Z

Another failure Get "https://artifacts-api.elastic.co/v1/versions/": dial tcp 35.188.12.98:443: i/o timeout in https://buildkite.com/elastic/elastic-agent/builds/7339#018dcc31-3f59-49f1-a027-dc8e8e94c7ba

rdner · 2024-02-26T07:11:31Z

Another failure here https://buildkite.com/elastic/elastic-agent/builds/7480#018de2b8-47e5-4e01-843a-c294f7e3b56b

    fetcher_artifact.go:344: Downloading artifact progress 73.09%
    upgrade_fleet_test.go:406: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:406
        	            				/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:149
        	            				/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:127
        	Error:      	Received unexpected error:
        	            	failed to download https://snapshots.elastic.co/8.14.0-d950d4db/downloads/beats/elastic-agent/elastic-agent-8.14.0-SNAPSHOT-linux-arm64.tar.gz: failed to write file /tmp/TestFleetAirGappedUpgradePrivileged1764917254/002/downloads/beats/elastic-agent/beats/elastic-agent/elastic-agent-8.14.0-SNAPSHOT-linux-arm64.tar.gz: stream error: stream ID 7; INTERNAL_ERROR; received from peer
        	Test:       	TestFleetAirGappedUpgradePrivileged
        	Messages:   	could not download agent 8.14.0-SNAPSHOT
--- FAIL: TestFleetAirGappedUpgradePrivileged (653.81s)

rdner · 2024-03-06T07:48:47Z

We added retries for the artifact API #4348

rdner · 2024-03-06T13:55:39Z

Download got stuck at 50.99%

fetcher_artifact.go:344: Downloading artifact progress 50.99%
    upgrade_fleet_test.go:406: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:406
        	            				/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:149
        	            				/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:115
        	Error:      	Received unexpected error:
        	            	failed to download https://snapshots.elastic.co/8.14.0-f84a1ada/downloads/beats/elastic-agent/elastic-agent-8.14.0-SNAPSHOT-linux-arm64.tar.gz: failed to write file /tmp/TestFleetAirGappedUpgradeUnprivileged3011941863/002/downloads/beats/elastic-agent/beats/elastic-agent/elastic-agent-8.14.0-SNAPSHOT-linux-arm64.tar.gz: stream error: stream ID 7; INTERNAL_ERROR; received from peer
        	Test:       	TestFleetAirGappedUpgradeUnprivileged
        	Messages:   	could not download agent 8.14.0-SNAPSHOT

https://buildkite.com/elastic/elastic-agent/builds/7573#018e12ba-1e06-4cc8-9bfd-39a99763a99a

rdner · 2024-03-06T21:14:52Z

Another download failed:

fetcher_artifact.go:344: Downloading artifact progress 2.83%
    upgrade_fleet_test.go:90: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:90
        	            				/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:49
        	Error:      	Received unexpected error:
        	            	failed to download https://snapshots.elastic.co/8.14.0-78df864a/downloads/beats/elastic-agent/elastic-agent-8.14.0-SNAPSHOT-linux-arm64.tar.gz: failed to write file /home/ubuntu/agent/.agent-testing/artifact/elastic-agent-8.14.0-SNAPSHOT-linux-arm64.tar.gz: unexpected EOF
        	Test:       	TestFleetManagedUpgradeUnprivileged

https://buildkite.com/elastic/elastic-agent/builds/7588#018e1528-f7da-4e02-b81d-0165e3b5d780

rdner · 2024-03-07T08:25:14Z

Another failure of the artifact API:

upgrade_fleet_test.go:403: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:403
        	            				/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:149
        	            				/home/ubuntu/agent/testing/integration/upgrade_fleet_test.go:127
        	Error:      	Received unexpected error:
        	            	failed to find snapshot URI for version 8.14.0-SNAPSHOT: failed to find package URL: Get "https://artifacts-api.elastic.co/v1/search/8.14.0-SNAPSHOT/elastic-agent": read tcp 10.128.0.232:53734->35.188.12.98:443: read: connection reset by peer
        	Test:       	TestFleetAirGappedUpgradePrivileged
        	Messages:   	could not prepare fetcher to download agent 8.14.0-SNAPSHOT

https://buildkite.com/elastic/elastic-agent/builds/7588#018e159c-ee97-4fd6-899b-e378390a88e9

rdner · 2024-03-13T13:35:11Z

Another failure on the artifact API in https://buildkite.com/elastic/elastic-agent/builds/7753#018e3765-0a82-4e78-b265-4f8cc9024146

=== RUN   TestStandaloneUpgrade/Upgrade_8.14.0-SNAPSHOT_to_8.14.0-SNAPSHOT_(privileged)
    upgrade_standalone_test.go:74: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_standalone_test.go:74
        	            				/home/ubuntu/agent/testing/integration/upgrade_standalone_test.go:49
        	Error:      	Received unexpected error:
        	            	failed to prepare before exec: failed to find snapshot URI for version 8.14.0-SNAPSHOT: failed to find package URL: Get "https://artifacts-api.elastic.co/v1/search/8.14.0-SNAPSHOT/elastic-agent": dial tcp 35.188.12.98:443: i/o timeout
        	Test:       	TestStandaloneUpgrade/Upgrade_8.14.0-SNAPSHOT_to_8.14.0-SNAPSHOT_(privileged)
--- FAIL: TestStandaloneUpgrade/Upgrade_8.14.0-SNAPSHOT_to_8.14.0-SNAPSHOT_(privileged) (30.02s)

rdner · 2024-03-13T19:52:31Z

Lots of 503 responses from the artifact API in https://buildkite.com/elastic/elastic-agent/builds/7763#018e3881-3ee9-4e60-942d-51a1d0daad51

rdner · 2024-03-13T19:54:05Z

Another failure with 503 from the artifact API https://buildkite.com/elastic/elastic-agent/builds/7761#018e3870-a7be-4f6c-a28d-c9c7c54a4a90

rdner · 2024-03-21T18:03:39Z

Another failure https://buildkite.com/elastic/elastic-agent/builds/7900#018e617f-f132-46da-b9cd-625a6741682e

rdner · 2024-03-22T16:24:31Z

The artifact API failed in https://buildkite.com/elastic/elastic-agent/builds/7936#018e667e-1530-4a04-ab27-9e6bc0d60d25

failed to prepare before exec: failed to find snapshot URI for version 8.12.1: failed to find package URL: Get "https://artifacts-api.elastic.co/v1/search/8.12.1/elastic-agent": dial tcp 35.188.12.98:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

rdner · 2024-04-26T12:44:44Z

Since 2024-04-22 at 16:00 UTC we rolled out a new CDN proxy and it seems to have solved the stuck downloads.

The artifact API part of the issue will be addressed by #4458

amitkanfer · 2024-04-26T12:46:38Z

Well done @rdner

rdner added Team:Elastic-Agent Label for the Agent team flaky-test Unstable or unreliable test cases. labels Feb 15, 2024

rdner self-assigned this Feb 15, 2024

rdner changed the title ~~[Flaky Test]: Multiple test cases fail because of the artifact API unavailability~~ [Flaky Test]: Multiple test cases fail because of the artifact API / CDN unavailability Feb 15, 2024

rdner mentioned this issue Feb 23, 2024

Use a version file instead of artifact API in integration tests #4331

Merged

3 tasks

rdner mentioned this issue Mar 13, 2024

Better connection management when downloading artifacts #4409

Closed

rdner mentioned this issue Mar 13, 2024

Improve retry mechanism for HTTP clients (integration tests and agent upgrades) #4354

Closed

This was referenced Mar 14, 2024

Find another source for versions list other than the artifact API #4417

Closed

Review agent's upgrade tests and change used versions if necessary #4320

Closed

rdner mentioned this issue Mar 21, 2024

Stop using the artifact API in artifact downloads #4458

Closed

rdner closed this as completed Apr 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Flaky Test]: Multiple test cases fail because of the artifact API / CDN unavailability #4268

[Flaky Test]: Multiple test cases fail because of the artifact API / CDN unavailability #4268

rdner commented Feb 15, 2024 •

edited

Loading

elasticmachine commented Feb 15, 2024

rdner commented Feb 15, 2024

rdner commented Feb 20, 2024

rdner commented Feb 20, 2024

rdner commented Feb 20, 2024

rdner commented Feb 21, 2024

rdner commented Feb 21, 2024

rdner commented Feb 26, 2024

rdner commented Mar 6, 2024

rdner commented Mar 6, 2024

rdner commented Mar 6, 2024

rdner commented Mar 7, 2024

rdner commented Mar 13, 2024

rdner commented Mar 13, 2024

rdner commented Mar 13, 2024

rdner commented Mar 21, 2024

rdner commented Mar 22, 2024

rdner commented Apr 26, 2024

amitkanfer commented Apr 26, 2024

[Flaky Test]: Multiple test cases fail because of the artifact API / CDN unavailability #4268

[Flaky Test]: Multiple test cases fail because of the artifact API / CDN unavailability #4268

Comments

rdner commented Feb 15, 2024 • edited Loading

Artifact API

2024-02-12T20:28:05.483Z

2024-02-13T21:24:31.816Z

CDN

2024-02-13T16:11:14.713Z

2024-02-14T20:02:31.839Z

elasticmachine commented Feb 15, 2024

rdner commented Feb 15, 2024

rdner commented Feb 20, 2024

rdner commented Feb 20, 2024

rdner commented Feb 20, 2024

rdner commented Feb 21, 2024

rdner commented Feb 21, 2024

rdner commented Feb 26, 2024

rdner commented Mar 6, 2024

rdner commented Mar 6, 2024

rdner commented Mar 6, 2024

rdner commented Mar 7, 2024

rdner commented Mar 13, 2024

rdner commented Mar 13, 2024

rdner commented Mar 13, 2024

rdner commented Mar 21, 2024

rdner commented Mar 22, 2024

rdner commented Apr 26, 2024

amitkanfer commented Apr 26, 2024

rdner commented Feb 15, 2024 •

edited

Loading