chore: update Ubuntu base image from 20.04 to 24.04 (#5644) (#5697)

* feat: update Ubuntu base image from 20.04 to 24.04

This affects `elastic-agent` and `elastic-agent-complete` images,
which are both based on Ubuntu.
Other variants, such as `elastic-agent-cloud`, `elastic-agent-ubi`, `elastic-agent-wolfi`
are not affected by this change.

* fix: remove `ubuntu` user from Ubuntu 24.04 base image

The Ubuntu 24.04 image comes with user `ubuntu` with uid and gid 1000.
I propose to remove this user as described here:
https://askubuntu.com/questions/1513927/ubuntu-24-04-docker-images-now-includes-user-ubuntu-with-uid-gid-1000
and here: https://bugs.launchpad.net/cloud-images/+bug/2005129.

* add changelog entry

* add comment on user deletion

(cherry picked from commit bbe332a)

Co-authored-by: Andrzej Stencel <andrzej.stencel@elastic.co>

changelog/fragments/1727778754-update-base-ubuntu-from-20.04-to-24.04.yaml

@@ -0,0 +1,32 @@
+# Kind can be one of:
+# - breaking-change: a change to previously-documented behavior
+# - deprecation: functionality that is being removed in a later release
+# - bug-fix: fixes a problem in a previous version
+# - enhancement: extends functionality but does not break or fix existing behavior
+# - feature: new functionality
+# - known-issue: problems that we are aware of in a given version
+# - security: impacts on the security of a product or a user’s deployment.
+# - upgrade: important information for someone upgrading from a prior version
+# - other: does not fit into any of the other categories
+kind: enhancement
+
+# Change summary; a 80ish characters long description of the change.
+summary: Update Ubuntu base image from 20.04 to 24.04
+
+# Long description; in case the summary is not enough to describe the change
+# this field accommodate a description without length limits.
+# NOTE: This field will be rendered only for breaking-change and known-issue kinds at the moment.
+#description:
+
+# Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc.
+component: elastic-agent
+
+# PR URL; optional; the PR number that added the changeset.
+# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added.
+# NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number.
+# Please provide it if you are adding a fragment for a different PR.
+#pr: https://github.com/owner/repo/1234
+
+# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of).
+# If not present is automatically filled by the tooling with the issue linked to the PR number.
+#issue: https://github.com/owner/repo/1234

dev-tools/packaging/packages.yml

@@ -225,11 +225,11 @@ shared:
   - &docker_ubuntu_spec
     docker_variant: 'basic'
     extra_vars:
-      from: '--platform=linux/amd64 ubuntu:20.04'
+      from: '--platform=linux/amd64 ubuntu:24.04'
   - &docker_ubuntu_arm_spec
     docker_variant: 'basic'
     extra_vars:
-      from: '--platform=linux/arm64 ubuntu:20.04'
+      from: '--platform=linux/arm64 ubuntu:24.04'
 
   - &docker_ubi_spec
     docker_variant: 'ubi'

dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl

@@ -74,6 +74,9 @@ RUN for iter in {1..10}; do microdnf update -y && microdnf install -y tar gzip f
 {{- end }}
 
 {{- if contains .from "ubuntu" }}
+# Delete the existing "ubuntu" user to prevent conflict with the user that is added later.
+# See https://bugs.launchpad.net/cloud-images/+bug/2005129/comments/2.
+RUN touch /var/mail/ubuntu && chown ubuntu /var/mail/ubuntu && userdel -r ubuntu
 RUN for iter in {1..10}; do \
         apt-get update -y && \
         DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends --yes ca-certificates curl gawk xz-utils && \