Find another source for versions list other than the artifact API

[rdner]

Describe the enhancement:
In the automation that updates the version file https://github.com/elastic/elastic-agent/blob/main/.github/workflows/bump-agent-versions.yml

We use the mage target that fetches all the available versions from the artifact API and filters them out based on set requirements:

elastic-agent/magefile.go

Lines 1578 to 1609 in 6135d60

	func (Integration) UpdateVersions(ctx context.Context) error {
	// test 2 current 8.x version, 1 previous 7.x version and 1 recent snapshot
	reqs := upgradetest.VersionRequirements{
	UpgradeToVersion: bversion.Agent,
	CurrentMajors: 2,
	PreviousMinors: 1,
	PreviousMajors: 1,
	RecentSnapshots: 1,
	}
	aac := tools.NewArtifactAPIClient(tools.WithLogFunc(log.Default().Printf))
	versions, err := upgradetest.FetchUpgradableVersions(ctx, aac, reqs)
	if err != nil {
	return fmt.Errorf("failed to fetch upgradable versions: %w", err)
	}
	versionFileData := upgradetest.AgentVersions{
	TestVersions: versions,
	}
	file, err := os.OpenFile(upgradetest.AgentVersionsFilename, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0644)
	if err != nil {
	return fmt.Errorf("failed to open %s for write: %w", upgradetest.AgentVersionsFilename, err)
	}
	defer file.Close()
	encoder := json.NewEncoder(file)
	encoder.SetIndent("", " ")
	err = encoder.Encode(versionFileData)
	if err != nil {
	return fmt.Errorf("failed to encode JSON to file %s: %w", upgradetest.AgentVersionsFilename, err)
	}
	return nil
	}

The artifact API is proven to be unreliable #4268 (occasional 503 outages, connectivity issues), also it returns versions that are not released yet as we can see in this PR #4414

The enhancement is to use git tags or any other source for fetching versions instead of the artifact API, this is a more stable source of versions.

Describe a specific use case for the enhancement or feature:

See #4414

What is the definition of done?

• The artifact API is not used for updating the versions file.
• There is a test that validates the versions list.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[rdner]

I missed that fact that the elastic-agent repository has only versions >=8.3.2. For previous versions (e.g. 7.17.* we test against) we would need the Beats repository.

So, it seems required that we git clone the Beats repository and list the tags there. If we're planning to diverge agent releases with Beats releases in the future, it's not going to work.

@cmacknz any thoughts/ideas?

[pchila]

At this point we can use the GH API for listing the releases, we can get those without cloning any additional repo, it should be easy to have a nice github client set up on a nodejs linux runner (or something like it) to run a couple of queries on the releases endpoint of both beats and elastic-agent repositories

[rdner]

@pchila how does it solve the problem of having diverged releases with Beats?
Accessing the Github API instead of git tags makes no difference, you still need to access 2 repositories and figure out versions of the agent.

[pchila]

@rdner it solves the problem of making an additional clone, querying the Github API does not require cloning a new repo just to fetch the tags.

[cmacknz]

There is a product versions API that lists every version ever made available for download. https://www.elastic.co/api/product_versions

That will have 7.17 in it, and might be easier than dealing with the Git tags and releases.

[cmacknz]

The enhancement is to use git tags for fetching versions instead of the artifact API, this is a more stable source of versions.

I think we will still need to use the artifacts API to know which snapshots exist. There is always a couple of days (maybe more) between the release of a minor and the creation of the snapshot for the next minor.

Although in that case the tests will just fail for the update PR until the snapshot exists so maybe we don't care.

[rdner]

Found that Kibana is using https://www.elastic.co/api/product_versions for getting agent versions:

https://github.com/elastic/kibana/blob/98c069ff950ce2632677e975533b793f19311ea3/src/dev/build/tasks/fetch_agent_versions_list.ts#L16-L41

Although, I would prefer to use a product ID (bltce270507523f4c56) instead of checking the title for a substring. Otherwise the alpha/beta/RC versions are also included.