Kamaji infinite CoreDNS Configmap Overwrite

[sachinphogat]

When we create control plane using TenantControlPlane, we cannot update coredns Corefile for adding extra domains as Kamaji Controller keeps overwriting it with base Corefile.

Ideally with kubeadm, one CoreDns configmap is created , it is not overwritten and migration is also supported when we are upgrading the cluster. We expect similar behaviour with Kamaji as well that it should only create configmap and during cluster upgrade should migrate coredns config if required rather than overwriting.

[prometherion]

Kamaji aims for a constant reconciliation to ensure it respects the desired state of those components and doesn't support customisation of the CoreDNS configmap: any change is a diff and it must be reconciled.

We have a discussion about it (#475) and it's open for external contributions or request for development, such as engaging with CLASTIX to get it sorted out.

Otherwise, you can manage CoreDNS externally, such as an add-on, with Project Sveltos, FluxCD, ArgoCD, or in the Cluster API domain, with ClusterSet.

[sachinphogat]

It should be fine to maintain state for resources which are managed by Kamaji, for eg deployment of control plane components, konnectivity etc but components like coredns, kubeproxy are owned at cluster level and should be reconciled once or upgraded or migrated with k8s upgrade only. Overwriting coredns configmaps is not done in any of other control plane providers, eg cluster api provider kubeadm or cluster api provider k3s etc

[prometherion]

It should be fine to maintain state for resources which are managed by Kamaji,

CoreDNS addon is defined in the Tenant Control Plane definition, thus managed by Kamaji as an addon.

components like coredns, kubeproxy are owned at cluster level and should be reconciled once or upgraded or migrated with k8s upgrade only

I disagree, those components are essentials for cluster operations, like Konnectivity Agent which resides on user space. If user breaks the CoreDNS configuration, Service Discovery will be broke affecting production grade of the cluster.

Overwriting coredns configmaps is not done in any of other control plane providers

We're doing better than the others, it seems.

---

As I said before, I would suggest you relying on an external component such as FluxCD or Project Sveltos with no drift detection mode, or contribute upstream by continuing the discussion aforementioned, or engage with CLASTIX for development.

[sachinphogat]

We surely can manage them outside but then it lacks the feature of upgradation of such components automatically with cluster upgrade.

Also kamaji just holds whether to enable coredns or no, it doesnot own the configuration of coredns.
For eg, if someone wants to modify the TTL inside coredns configmap, they have to remove coredns from kamaji and use it externally. Which could be easily avoided if kamaji doesnot overwrite the configuration done directly on cluster side. Also kamaji doesnot provide any option to provide coredns configuration, until then kamaji should not be overwriting.

[prometherion]

We surely can manage them outside but then it lacks the feature of upgradation of such components automatically with cluster upgrade

Project Sveltos is our preferred choice when dealing with complex upgrade and update processes.

Also kamaji just holds whether to enable coredns or no, it doesnot own the configuration of coredns.
For eg, if someone wants to modify the TTL inside coredns configmap, they have to remove coredns from kamaji and use it externally. Which could be easily avoided if kamaji doesnot overwrite the configuration done directly on cluster side. Also kamaji doesnot provide any option to provide coredns configuration, until then kamaji should not be overwriting.

It sounds like a feature request, happy to receive contributions or a commercial engagement for the development.