Document migrating agents to a new cluster (#528)

* Document migrating agents to a new cluster

* Cleanup

* Reorder ToC

Author: kilfoyle

docs/en/ingest-management/fleet/fleet-manage-agents.asciidoc

@@ -24,10 +24,13 @@ To manage your {agent}s, go to *Management > {fleet} > Agents* in {kib}. On the
 |<<upgrade-elastic-agent>>
 |Upgrade {agent}s to the latest version.
 
-|<<filter-agent-list-by-tags>>
-|Add tags to {agent}, then use the tags to filter the Agents list in {fleet}.
+|<<migrate-elastic-agent>>
+|Migrate {agent}s from one cluster to another.
 
 |<<monitor-elastic-agent>>
 |Monitor {fleet}-managed {agent}s by viewing agent status, logs, and metrics.
 
+|<<filter-agent-list-by-tags>>
+|Add tags to {agent}, then use the tags to filter the Agents list in {fleet}.
+
 |===

docs/en/ingest-management/fleet/migrate-elastic-agent.asciidoc

@@ -0,0 +1,177 @@
+[[migrate-elastic-agent]]
+= Migrate {fleet}-managed {agent}s from one cluster to another
+
+++++
+<titleabbrev>Migrate {agent}s</titleabbrev>
+++++
+
+There are situations where you may need to move your installed {agent}s from being managed in one cluster to being managed in another cluster. 
+
+For a seamless migration, we advise that you create an identical agent policy in the new cluster that is configured in the same manner as the original cluster. There are a few methods to do this.
+
+This guide takes you through the steps to migrate your {agent}s by snapshotting a source cluster and restoring it on a target cluster. These instructions assume that you have an {ecloud} deployment, but they can be applied to on-premise clusters as well.
+
+[discrete]
+[[migrate-elastic-agent-take-snapshot]]
+== Take a snapshot of the source cluster
+
+Refer to the full {ref}/snapshot-restore.html[Snapshot and restore] documentation for full details. In short, to create a new snapshot in an {ecloud} deployment:
+
+. In {kib}, open the main menu, then click *Manage this deployment*.
+. In the deployment menu, select *Snapshots*.
+. Click *Take snapshot now*.
++
+[role="screenshot"]
+image::images/migrate-agent-take-snapshot.png[Deployments Snapshots page]
+
+[discrete]
+[[migrate-elastic-agent-create-target]]
+== Create a new target cluster from the snapshot
+
+You can create a new cluster based on the snapshot taken in the previous step, and then migrate your {agent}s and {fleet} to the new cluster. For best results, it's recommended that the new target cluster be at the same version as the cluster that the agents are migrating from.
+
+. Open the {ecloud} console and select *Create deployment*.
+. Select *Restore snapshot data*.
+. In the *Restore from* field, select your source deployment.
+. Choose your deployment settings, and, optimally, choose the same {stack} version as the source cluster.
+. Click *Create deployment*.
++
+[role="screenshot"]
+image::images/migrate-agent-new-deployment.png[Create a deployment page]
+
+[discrete]
+[[migrate-elastic-agent-target-settings]]
+== Update settings in the target cluster
+
+when the target cluster is available you'll need to adjust a few settings. Take some time to examine the {fleet} setup in the new cluster.
+
+. Open the {kib} menu and select *Fleet*.
+. On the *Agents* tab, your agents should visible, however they'll appear as `Offline`. This is because these agents have not yet enrolled in the new, target cluster, and are still enrolled in the original, source cluster.
++
+[role="screenshot"]
+image::images/migrate-agent-agents-offline.png[Agents tab in Fleet showing offline agents]
+
+. Open the {fleet} *Settings* tab.
+. Examine the configurations captured there for {fleet}. Note that these settings are scopied from the snapshot of the source cluster and may not have a meaning in the target cluster, so they need to be modified accordingly.
++
+In the following example, both the *Fleet Server hosts* and the *Outputs* settings are copied over from the source cluster:
++
+[role="screenshot"]
+image::images/migrate-agent-host-output-settings.png[Settings tab in Fleet showing source deployment host and output settings]
++
+The next steps explain how to obtain the relevant {fleet-server} host and {es} output details applicable to the new target cluster in {ecloud}.
+
+[discrete]
+[[migrate-elastic-agent-elasticsearch-output]]
+=== Modify the {es} output
+
+. In the new target cluster on {ecloud}, in the *Outputs* section, on the {fleet} *Settings* tab, you will find an internal output named `Elastic Cloud internal output`. The host address is in the form:
++
+`https://<cluster-id-target>.containerhost:9244`
++
+Record this `<cluster-id-target>` from the target cluster. In the example shown, the ID is `fcccb85b651e452aa28703a59aea9b00`.
+
+. Also in the *Outputs* section, notice that the default {es} output (that was copied over from the source cluster) is also in the form:
++
+`https://<cluster-id-source>.<cloud endpoint address>:443`.
++
+Modify the {es} output so that the cluster ID is the same as that for `Elastic Cloud internal output`. In this example we also rename the output to `New Elasticsearch`.
++
+[role="screenshot"]
+image::images/migrate-agent-elasticsearch-output.png[Outputs section showing the new Elasticsearch host setting]
++
+In this example, the `New Elasticsearch` output and the `Elastic Cloud internal output` now have the same cluster ID, namely `fcccb85b651e452aa28703a59aea9b00`.
+
+You have now created an {es} output that agents can use to write data to the new, target cluster. For on-premise environments not using {ecloud}, you should similarly be able to use the host address of the new cluster.
+
+[discrete]
+[[migrate-elastic-agent-fleet-host]]
+=== Modify the {fleet-server} host
+
+Like the {es} host, the {fleet-server} host has also changed with the new target cluster. Note that if you're deploying {fleet-server} on premise, the host has probably not changed address and this setting does not need to be modified. We still recommend that you ensure the agents are able to reach the the on-premise {fleet-server} host (which they should be able to as they were able to connect to it prior to the migration).
+
+The {ecloud} {fleet-server} host has a similar format to the {es} output:
+
+`https://<deployment-id>.fleet.<domain>.io`
+
+To configure the correct {ecloud} {fleet-server} host you will need to find the target cluster's full `deployment-id`, and use it to replace the original `deployment-id` that was copied over from the source cluster. 
+
+The easiest way to find the `deployment-id` is from the deployment URL:
+
+. From the {kib} menu select *Manage this deployment*.
+. Copy the deployment ID from the URL in your browser's address bar.
++
+[role="screenshot"]
+image::images/migrate-agent-deployment-id.png[Deployment management page, showing the browser URL]
++
+In this example, the new deployment ID is `eed4ae8e2b604fae8f8d515479a16b7b`.
++
+Using that value for `deployment-id`, the new {fleet-server} host URL is:
++
+`https://eed4ae8e2b604fae8f8d515479a16b7b.fleet.us-central1.gcp.cloud.es.io:443`
+
+. In the target cluster, under *Fleet server hosts*, replace the original host URL with the new value.
++
+[role="screenshot"]
+image::images/migrate-agent-fleet-server-host.png[Fleet server hosts showing the new host URL]
+
+[discrete]
+[[migrate-elastic-agent-confirm-policy]]
+=== Confirm your policy settings
+
+Now that the {fleet} settings are correctly set up, it pays to ensure that the {agent} policy is also correctly pointing to the correct entities.
+
+. In the target cluster, go to *Fleet -> Agent policies*.
+. Select a policy to verify.
+. Open the *Settings* tab.
+. Ensure that *Fleet Server*, *Output for integrations*, and *Output for agent monitoring* are all set to the newly created entities.
++
+[role="screenshot"]
+image::images/migrate-agent-policy-settings.png[An agent policy's settings showing the newly created entities]
+
+NOTE: If you modified the {fleet-server} and the output in place these would have been updated accordingly. However if new entities are created, then ensure that the correct ones are referenced here.
+
+[discrete]
+[[migrate-elastic-agent-migrated-policies]]
+== Agent policies in the new target cluster
+
+By creating the new target cluster from a snapshot, all of your policies should have been created along with all of the agents. These agents will be offline due to the fact that the actual agents are not checking in with the new, target cluster (yet) and are still communicating with the source cluster.
+
+The agents can now be re-enrolled into these policies and migrated over to the new, target cluster.
+
+[discrete]
+[[migrate-elastic-agent-migrated-agents]]
+== Migrate {agent}s to the new target cluster
+
+In order to ensure that all required API keys are correctly created, the agents in your current cluster need to be re-enrolled into the new, target cluster.
+
+This is best performed one policy at a time. For a given policy, you need to capture the enrollment token and the URL for the agent to connect to. You can find these by running the in-product steps to add a new agent.
+
+. On the target cluster, open *Fleet* and select *Add agent*.
+. Select your newly created policy.
+. In the section *Install {agent} on your host*, find the sample install command. This contains the details you'll need to enroll the agents, namely the enrollment token and the {fleet-server} URL.
+. Copy the portion of the install command containing these values. That is, `–url=<path> –enrollment-token=<token for the new policy>`.
++
+[role="screenshot"]
+image::images/migrate-agent-install-command.png[Install command from the Add Agent UI]
+
+. On the host machines where the current agents are installed, enroll the agents again using this copied URL and the enrollment token:
++
+[source,shell]
+----
+sudo elastic-agent enroll --url=<fleet server url> --enrollment-token=<token for the new policy>
+----
++
+The command output should be like the following:
++
+[role="screenshot"]
+image::images/migrate-agent-install-command-output.png[Install command output]
+
+. The agent on each host will now check into the new {fleet-server} and appear in the new target cluster. In the source cluster, the agents will go offline as they won't be sending any check-ins.
++
+[role="screenshot"]
+image::images/migrate-agent-newly-enrolled-agents.png[Newly enrolled agents in the target cluster]
+
+. Repeat this procedure for each {agent} policy.
+
+If all has gone well, you've successfully migrated your {fleet}-managed {agent}s to a new cluster.

docs/en/ingest-management/images/migrate-agent-agents-offline.png

@@ -125,10 +125,12 @@ include::fleet/set-inactivity-timeout.asciidoc[leveloffset=+3]
 
 include::fleet/upgrade-elastic-agent.asciidoc[leveloffset=+3]
 
-include::fleet/filter-agent-list-by-tags.asciidoc[leveloffset=+3]
+include::fleet/migrate-elastic-agent.asciidoc[leveloffset=+3]
 
 include::fleet/monitor-elastic-agent.asciidoc[leveloffset=+3]
 
+include::fleet/filter-agent-list-by-tags.asciidoc[leveloffset=+3]
+
 include::agent-policies.asciidoc[leveloffset=+2]
 
 include::create-agent-policies-no-UI.asciidoc[leveloffset=+3]