Feature/multi loader logs collection

[nosnelmil]

Summary

Extends multi-loader by collecting key logs from nodes in the cluster for the Knative platform. Users can optionally collect the following logs:

• TOP (resource usage metrics)
• Prometheus snapshots
• Logs from the Activator Pod
• Logs from the Autoscaler Pod

Implementation Notes ⚒️

• Added an additional Metrics field in the multi-loader config, accepting an array with any of the following values: top, prometheus, activator, autoscaler.
• Introduced optional fields: MasterNode, ActivatorNode, AutoscalerNode, and WorkerNodes to allow users to manually specify IPs instead of relying on multi-loader to determine them (mostly unnecessary in typical scenarios).
• Uses kubectl to automatically determine node IPs and classify them based on their roles.
• Resets TOP metrics for all nodes before starting any experiment.
• Collects Activator Pod logs from:
  /var/log/pods/knative-serving_activator-*/activator/*
• Collects Autoscaler Pod logs from:
  /var/log/pods/knative-serving_autoscaler-*/autoscaler/*
• Copies Prometheus snapshots by first triggering a snapshot via the Prometheus API on the master node and then retrieving the generated snapshot.
• Additionally, log collection logic runs during multi-loader dry run to:
  • Validate that identified IPs are reachable.
  • Ensure SSH access and necessary permissions.
  • Execute log retrieval commands to detect potential errors.
  • Delete any collected logs after validation, as no experiments have been executed yet.

External Dependencies 🍀

• N/A

Breaking API Changes ⚠️

• N/A

[leokondrashov]

Looks nice. I left some comments.

[nosnelmil]

@leokondrashov as discussed, added the log consolidation logic in 0dc0950

[leokondrashov]

Looks great, thanks. Please fix couple minor comments

[nosnelmil]

Hi @cvetkovic, this PR extends the previously added multi-loader tool by introducing enhanced log collection capabilities. The new feature allows users to gather logs from the Activator and Autoscaler nodes, retrieve TOP metrics from all cluster nodes, and capture Prometheus snapshots. Users can also specify the exact metrics they want to collect using the newly introduced Metric field in the multi-loader configuration.

I would appreciate your review and if everything looks good, I will tidy up the commits and prepare for merging into main. Thank you!