Track and report upgrade details

[ycombinator]

Describe the enhancement:

Elastic Agent should track details about an ongoing upgrade and report them to Fleet Server.

Describe a specific use case for the enhancement or feature:

To understand where upgrades fail or get stuck without progressing.

What is the definition of done?

• Elastic Agent is able to track the various upgrade states (defined below) representing details about an ongoing upgrade and correctly transition between these states.
• Elastic Agent is able to report a change in an upgrade state to Fleet Server via a new upgrade_details field in the check-in API.
• For backwards compatibility, Elastic Agent continues to acknowledge the UPGRADE action with Fleet Server, via the acknowledgements API, as before.

Details

In the future, Elastic Agents will send details about an ongoing upgrade to Fleet Server. These details will be communicated via a new upgrade_details field in check-in API requests.

The proposed structure of the upgrade_details field is:

{
  "upgrade_details": { // new field; present when upgrade is in progress
    "target_version": "8.12.0", // version being upgraded to; always present
    "action_id": "xxxxxxxx", // ID of the UPGRADE action
    "state": "UPG_*",
    "metadata": {
      "scheduled_at": "2023-08-09T10:11:12Z", // when state == "UPG_SCHEDULED"
      "download_percent": 16.4, // when state == "UPG_DOWNLOADING"
      "failed_state": "UPG_*" // when state == "UPG_FAILED"
      "error_msg": "" // when state == "UPG_FAILED"
    }
  }
}

Where upgrade_details.state is expected to hold one of the following values:

State	Meaning
UPG_REQUESTED	Upgrade requested by user
UPG_SCHEDULED	Upgrade scheduled for <date/time>
UPG_DOWNLOADING	Downloading new Agent artifact version
UPG_EXTRACTING	Extracting new Agent artifact version
UPG_REPLACING	Replacing old Agent artifact version with new one version
UPG_RESTARTING	Starting new Agent version
UPG_WATCHING	Monitoring new Agent version
UPG_ROLLBACK	Upgrade unsuccessful; rolling back to Agent version
UPG_FAILED	Upgrade failed due to error from state

The possible transitions between these states are:

The various points in the upgrade workflow at which these state transitions would occur are:

sequenceDiagram
    actor U as User
    participant UI as Fleet UI
    participant ES
    participant FS as Fleet Server
    participant A as Agent
    participant UW as Upgrade Watcher
    participant UM as Upgrader Marker

    U->>UI: Initiate upgrade
    UI->>ES: Update Agent doc in `.fleet-agents`<br />set `upgrade_started_at`
    UI->>UI: Show Agent status as "updating"
    UI->>ES: Create new doc in `.fleet-actions` for `UPGRADE` action
    A->>FS: Check-in request
    FS->>ES: Read pending actions from .fleet-actions
    FS->>A: Check-in response
    A-->>FS: Check-in request: `upgrade_details.state_id` = `UPG_REQUESTED`
    A->>A: Queue upgrade action
    alt If upgrade is scheduled for future
        A-->>FS: Check-in request: `upgrade_details.state_id` = `UPG_SCHEDULED`
    end
    alt If upgrade start fails
       A->>FS: Ack failed upgrade
       A-->>FS: Check-in request: `upgrade_details.state_id` = `UPG_ERROR` with `upgrade_details.failed_state_id` = `UPG_REQUESTED` or `UPG_SCHEDULED`
       FS->>ES: Update Agent doc in `.fleet-agents`<br />set `upgrade_status` = "failed"
       FS-->>ES: Set `upgrade_details` = value of `upgrade_details` from check-in request
       UI-->>UI: Agent status = "upgrade failed"
       UI->>UI: Agent status remains as "updating" (bug) (fallback)
    else
       opt If previous upgrades found
          A->>FS: Ack previous upgrades
          A->>A: Remove previous upgrades from queue
       end
       A-->>FS: Check-in request: `upgrade_details.state_id` = `UPG_DOWNLOADING`
       A->>A: Download new Agent artifact
       opt If download fails
          A-->>FS: Check-in request: `upgrade_details.state_id` = `UPG_ERROR` with `upgrade_details.failed_state_id` = `UPG_DOWNLOADING`
          FS-->>ES: Set `upgrade_details` = value of `upgrade_details` from check-in request
          UI-->>UI: Agent status = "upgrade failed"
       end
       A-->>FS: Check-in request: `upgrade_details.state_id` = `UPG_EXTRACTING`
       A->>A: Extract new Agent artifact
       opt If extraction fails
          A-->>FS: Check-in request: `upgrade_details.state_id` = `UPG_ERROR` with `upgrade_details.failed_state_id` = `UPG_EXTRACTING`
          FS-->>ES: Set `upgrade_details` = value of `upgrade_details` from check-in request
          UI-->>UI: Agent status = "upgrade failed"
       end
       A-->>FS: Check-in request: `upgrade_details.state_id` = `UPG_REPLACING`
       A->>A: Replace current Agent artifact with new one
       opt If extraction fails
          A-->>FS: Check-in request: `upgrade_details.state_id` = `UPG_ERROR` with `upgrade_details.failed_state_id` = `UPG_REPLACING`
          FS-->>ES: Set `upgrade_details` = value of `upgrade_details` from check-in request
          UI-->>UI: Agent status = "upgrade failed"
       end
       A->>UM: Create
       A->>UW: Start
       A-->>FS: Check-in request: `upgrade_details.state_id` = `UPG_RESTARTING`
       A->>A: Rexec to start new Agent artifact
       opt If restart fails
          A-->>FS: Check-in request: `upgrade_details.state_id` = `UPG_ERROR` with `upgrade_details.failed_state_id` = `UPG_RESTARTING`
          FS-->>ES: Set `upgrade_details` = value of `upgrade_details` from check-in request
          UI-->>UI: Agent status = "upgrade failed"
       end
       A-->>FS: Check-in request: `upgrade_details.state_id` = `UPG_WATCHING`
       UI-->>UI: Show Agent status as "upgrade watching"
       UW->>UW: Watch new Agent
   end
   opt Success
       A->>FS: Ack successful upgrade
       FS->>ES: Write successful ack in `.fleet-actions-results`
       FS->>ES: Update Agent doc in `.fleet-agents`<br />set `upgrade_status` = null<br />`upgraded_at` = <now><br />`upgrade_started_at` = null
       UI->>UI: Show Agent status as "healthy" (fallback)
       UW->>UM: Remove
       A-->>UM: Watch for removal
       A-->>FS: Check-in request: remove `upgrade_details` field
       FS-->>ES: Update Agent doc in `.fleet-agents`<br />remove `upgrade_details` field
       UI-->>UI: Show Agent status as "healthy"
   end
   opt Rollback
       UW->>UM: Write `upgrade_details.state_id` = `UPG_ROLLBACK`
       UW->>A: Start
       A-->>UM: Read `upgrade_details.state_id`
       A-->>FS: Check-in request: `upgrade_details.state_id` = value from UM
       A->>FS: Ack failed upgrade
       FS-->>ES: Set `upgrade_details` = value of `upgrade_details` from check-in request
       FS->>ES: Update Agent doc in `.fleet-agents`<br />set `upgrade_status` = null<br />`upgraded_at = <now> (fallback)
       UI->>UI: Show Agent status as "healthy" (fallback)
       UW->>UM: Remove
       A-->>UM: Watch for removal
       A-->>FS: Check-in request: remove `upgrade_details` field
       FS-->>ES: Update Agent doc in `.fleet-agents`<br />remove `upgrade_details` field
       UI-->>UI: Show Agent status as "healthy"
   end

Loading

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[ycombinator]

I'm going to start implementing this enhancement now. I'm breaking it down into the following sub-tasks with the idea of each sub-task resulting in a single PR, to make it easier to review and also track incremental progress.

• When in Fleet-managed mode, whenever there is a state transition, call check-in API call to Fleet, with upgrade details in new upgrade_details field. See Track and report upgrade details #3119 (comment)
• Track current upgrade details within Agent for all upgrade states (except UPG_SCHEDULED and UPG_ROLLBACK; see follow-up items below): Track upgrade details #3527
• Include the current upgrade details in the output of elastic-agent status: Include upgrade details in output of elastic-agent status during ongoing upgrade #3615
• Include the current upgrade details in the output of elastic-agent diagnostics and diagnostics requests from Fleet: Include upgrade details in diagnostics bundle during ongoing upgrade #3624
• Follow up: Track UPG_ROLLBACK state. This is being done as a follow up as it requires data to be exchanged between the Upgrade Watcher process and the "main" Elastic Agent process: [Upgrade Details] Implement tracking of UPG_ROLLBACK state #3694
• Follow up: When in Fleet-managed mode, track UPG_SCHEDULED state. This is being done as a follow up as the scheduling happens in the Fleet Dispatcher code, which is a separate component from the Upgrader where most of the rest of the upgrade states are tracked: [Upgrade Details] Implement tracking of UPG_SCHEDULED state #3757
• When in Fleet-managed mode, include the current upgrade details in next call to check-in API via upgrade_details field: Send upgrade details to Fleet Server in check-in API requests  #3528
• Regression testing: ensure that Elastic Agent continues to acknowledge the UPGRADE action with Fleet Server via the acknowledgements API.
• Cleanup: generate UPG_* constants for use in Agent code from corresponding Fleet OpenAPI definition. See also Track upgrade details #3527 (comment).

[cmacknz]

When in Fleet-managed mode, whenever there is a state transition, call check-in API call to Fleet, with upgrade details in new upgrade_details field.

The impact of the increased check ins will need to go through scale testing, so you'll need to update Horde or perhaps take on some of #2169.

I would suggest you make this possible but don't actually change the check in frequency so we can release this without impacting the system stability, and increase the checkin frequency separately.

[blakerouse]

Tracking and reporting upgrade details is completed in 8.12.

[amitkanfer]

well done team! Happy to see this getting closed. 🚀