Remove current APM soak infrastructure and decide on the next steps

[lahsivjar]

The current APM soak setup has not been working for a few months due to upgrade process being broken causing the clusters to be in a bad state. To fix this, the clusters will require to be recreated. In addition, now the clusters should be created as a team cluster rather than an individual cluster. We also need to take this opportunity to decide if we want to keep the soak testing active with the current state where the input data is not updated. Current soak testing infrastructure has 2 components:

1. A couple of ESS clusters (created using oblt-cli)
2. Load generator VMs running on ESS (created using terraform)

Follow tasks are required:

• Delete the ESS clusters
• Delete the load generator VMs
• Create follow-up issues if required
• Add a public note on the decommission of soak testing

[simitt]

@lahsivjar please update this task with current status and either finish it if you have room or move it out of the iteration.

[lahsivjar]

The ADR is remaining, I will move this to the next iteration.

[raultorrecilla]

Given the upcoming deadlines with HOtel, Azure and GCP I am not moving this to next iteration 106. Putting it in iteration 108 as I expect 107 to be also quite full.

@lahsivjar @simitt let me know if this NEEDS to happen before, but is marked as low prio as of now.

[simitt]

I don't see harm in moving to the next iteration.