Stop using the artifact API in artifact downloads

[rdner]

Describe the enhancement:

This is a follow up to #4417

We must stop using the artifact API in our artifact downloader here

elastic-agent/internal/pkg/agent/application/upgrade/artifact/download/snapshot/downloader.go

Line 127 in 8fc1874

artifactsURI := fmt.Sprintf("https://artifacts-api.elastic.co/v1/search/%s-SNAPSHOT/elastic-agent", version)

And the artifact fetcher in tests here

elastic-agent/pkg/testing/fetcher_artifact.go

Line 178 in 8fc1874

artifactsURI := fmt.Sprintf("https://artifacts-api.elastic.co/v1/search/%s/elastic-agent", version.VersionWithPrerelease())

Describe a specific use case for the enhancement or feature:

The artifact API is unstable and we cannot rely on it: recent failure example is https://buildkite.com/elastic/elastic-agent/builds/7900#018e617f-f132-46da-b9cd-625a6741682e

More examples can be found here #4268

Sometimes it starts returning 502 for several minutes and it fails our tests and might fail in production when upgrading to a snapshot (rare use-case).

What is the definition of done?

The artifact downloader in the agent and the artifact fetcher in tests are using an alternative source of versions not involving the artifact API.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[rdner]

We should be able to just use https://snapshots.elastic.co/ instead.

[rdner]

I talked to the team who owns the snapshot API.

TLDR;

It's possible to get a manifest for a certain version snapshot, e.g. 8.12.3-SNAPSHOT by doing this request https://snapshots.elastic.co/latest/8.12.3-SNAPSHOT.json and getting the build manifest from the manifest_url field of the response.

Which covers our exact use-case described in this issue.

Also, supported use-cases are:

• a particular build manifest e.g. 8.12.3-96ef37fe – https://snapshots.elastic.co/8.12.3-96ef37fe/manifest-8.12.3-SNAPSHOT.json
• latest for a branch, e.g. 8.12/8.13/master – https://snapshots.elastic.co/latest/8.13.json

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)