Add ability to enable Kubernetes feature gates and runtimeconfig in KubeVirt CI

[Barakmor1]

What this PR does / why we need it:
There is a need to test new capabilities, such as the ImageVolume POC and DRA.

This change allows enabling/disabling Kubernetes feature gates and runtime config in the KubeVirt CI . Users can now set the K8S_FEATURE_GATES environment variable before running make cluster-up to enable specific feature gates. For example:

export K8S_FEATURE_GATES="<FeatureGate1>=true,<FeatureGate2>=false"

Users can also set runtime configuration by setting
the K8S_API_RUNTIME_CONFIG environment variable before
running make cluster-up.

For example:

export K8S_API_RUNTIME_CONFIG="RuntimeConfig=true"

This modifies the KubeController, KubeAPI, and Kubelet configurations files, restarts the components, and waits for them to be ready with the feature gates applied.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

• Design: A design document was considered and is present (link) or not required
• PR: The PR description is expressive enough and will help future contributors
• Code: Write code that humans can understand and Keep it simple
• Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
• Upgrade: Impact of this change on upgrade flows was considered and addressed if required
• Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
• Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
• Community: Announcement to kubevirt-dev was considered

Release note:

Add ability to enable Kubernetes feature gates and runtime config in KubeVirt CI

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign rmohr for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Barakmor1]

/cc @alicefr @victortoso @jean-edouard @brianmcarey

[iholder101]

Thank you @Barakmor1! Looks awesome!
I think this PR is really important as it will help us test alpha/beta features a lot of time in advance which is a huge advantage in terms of development.

This is a relatively shallow review as I'm not an expert in this area.

p.s. If in the future we'll also be able to support #1264, combining these two capabilities could be absolutely great!

[jean-edouard]

/cc @alicefr @victortoso @jean-edouard @brianmcarey

This PR makes sense to me. It's just surprising and unfortunate that we need that much code just to add a couple feature gates to k8s.
Let's just try and keep the code as simple as possible.
Please ping me once every review comment has been addressed and I'll approve.

[iholder101]

/cc @alicefr @victortoso @jean-edouard @brianmcarey

This PR makes sense to me. It's just surprising and unfortunate that we need that much code just to add a couple feature gates to k8s. Let's just try and keep the code as simple as possible. Please ping me once every review comment has been addressed and I'll approve.

@Barakmor1 Maybe there's a way to reduce code and make things a bit cleaner.
@victortoso used the concept of config file drop-ins [1] as part of #1299 which can also be used here. Briefly, it allows having multiple config files that would be merged later by kubelet. So instead of changing the config, you can add a new config (e.g. 05-k8s-feature-gates.conf) with the following contents:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
- NodeSwap

WDYT?

[1] https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/#kubelet-conf-d

[Barakmor1]

@iholder101 @victortoso Please have another look

[Barakmor1]

/cc @alicefr @victortoso @jean-edouard @brianmcarey

This PR makes sense to me. It's just surprising and unfortunate that we need that much code just to add a couple feature gates to k8s. Let's just try and keep the code as simple as possible. Please ping me once every review comment has been addressed and I'll approve.

@Barakmor1 Maybe there's a way to reduce code and make things a bit cleaner. @victortoso used the concept of config file drop-ins [1] as part of #1299 which can also be used here. Briefly, it allows having multiple config files that would be merged later by kubelet. So instead of changing the config, you can add a new config (e.g. 05-k8s-feature-gates.conf) with the following contents:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
- NodeSwap

WDYT?

[1] https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/#kubelet-conf-d

Hey, unfortunately, I couldn’t find an easy way to update Kubernetes components with feature gates and flags after the cluster is already deployed.

Most of the work in the PR is about changing the configurations of the kube-apiserver, kube-controller, kube-scheduler, and kubelet, and then making sure these changes are applied correctly. The final step is to ensure the components are ready again before moving on with other kubevirt CI tasks.

Using the drop-in would only change how the kubelet configuration is updated, but we’d still need to restart kubelet and verify that kubelet is ready. So, I don’t think this would simplify the code.
Please let me know if I missed something or if you have another idea,and I’d be happy to try it out! :)

[victortoso]

Using the drop-in would only change how the kubelet configuration is updated, but we’d still need to restart kubelet and verify that kubelet is ready. So, I don’t think this would simplify the code.

Right, so the kubevirtci image is already created when we do something like K8S_FEATURE_GATES=DRA make cluster-up and we would need to restart kubelet after copying the config file, on each worker node.

Still, we do have some logic that does something like that, see configure_cpu_manager introduced in 2981e7e

I'm not sure what is the best way forward. I think that, k8s FG requirements can change between k8s versions and if we want to help developers try those with kubevirtci, we should probably have to handle custom k8s configs and manage to update the cluster's components with it.

Drop-in would be great, but afaict not all k8s components have it and even kubelet support is beta. What about using kubeadm config?

[Barakmor1]

Using the drop-in would only change how the kubelet configuration is updated, but we’d still need to restart kubelet and verify that kubelet is ready. So, I don’t think this would simplify the code.

Right, so the kubevirtci image is already created when we do something like K8S_FEATURE_GATES=DRA make cluster-up and we would need to restart kubelet after copying the config file, on each worker node.

Still, we do have some logic that does something like that, see configure_cpu_manager introduced in 2981e7e

Please take a look at the prepareCommandsForConfiguration function in cluster-provision/gocli/opts/k8scomponents/kubelet.go. This is where the kubelet restart is handled in this PR.

I'm not sure what is the best way forward. I think that, k8s FG requirements can change between k8s versions and if we want to help developers try those with kubevirtci, we should probably have to handle custom k8s configs and manage to update the cluster's components with it.

Drop-in would be great, but afaict not all k8s components have it and even kubelet support is beta. What about using kubeadm config?

Well, The way I see it, this PR adds the option to include useful flags for Kubernetes components: API server, kube-controller-manager, kube-scheduler, and kubelet. If we want to modify the way these flags are being added for components or add new ones for any of the components in the future, we can do so by updating prepareCommandsForConfiguration function in the respective component's file:

For kubelet: cluster-provision/gocli/opts/k8scomponents/kubelet.go

For the API server: cluster-provision/gocli/opts/k8scomponents/kube-api.go

For kube-controller-manager: cluster-provision/gocli/opts/k8scomponents/kubecontrollermanager.go

For kube-scheduler: cluster-provision/gocli/opts/k8scomponents/kube-scheduler.go

I'm not certain if this is the optimal way to add the flags, but I aimed to make it easy to modify or extend in the future.

[kubevirt-bot]

@Barakmor1: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
check-provision-k8s-1.31-s390x	8f915b2	link	true	/test check-provision-k8s-1.31-s390x
check-up-kind-sriov	8f915b2	link	false	/test check-up-kind-sriov
check-provision-k8s-1.31	8f915b2	link	true	/test check-provision-k8s-1.31

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.