Set timeout for k8s HTTP client

We had a request to an ingesters take 20m and block the reconiliation loop. We want to avoid that, so adding a timeout. We use this client for both k8s and StatefulSet requests.

__I assume the operations that the StatefulSet requests will always be fast and will never have to actually do a lot of work (like uploading blocks). Can the reviewers confirm this assumption?__

CHANGELOG.md

@@ -3,6 +3,7 @@
 ## main / unreleased
 
 * [ENHANCEMENT] New parameter log.format allows to set logging format to logfmt (default) or json (new). #184
+* [ENHANCEMENT] Add a 5 minute timeout to requests to Pods and to the Kubernetes control plane.
 
 ## v0.21.0
 

cmd/rollout-operator/main.go

@@ -276,6 +276,10 @@ func buildKubeConfig(apiURL, cfgFile string) (*rest.Config, error) {
 		if err != nil {
 			return nil, err
 		}
+		// Set a generous timeout.
+		// We use this client for various HTTP operations against the k8s API and against the StatefulSets.
+		// We want to not be stuck waiting forever on a TCP timeout, but also not interrupt any process the StatefulSets might eb doing.
+		config.Timeout = 5 * time.Minute
 		return config, nil
 	}