set kubelet_status_update_frequency to 120s

[poblahblahblah]

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

The default setting out of the box for this is 5 minutes. As far as I can tell this is what EKS, AKS, and ACK all use. On our metal clusters we were seeing a lot more GC activity causing memory contention and higher CPU usage. It was eventually traced to how often the Node Conditions were updating. You can see when we rolled out this change to this cluster in the screenshot below.

On a ~300 node cluster updating this setting to 120s brought Prometheus' Bytes allocated and freed per second from 1Gi down to ~450Mi and the CPU time from ~7 seconds down to ~3 seconds. It also brought GC cycles per minute from ~8 down to ~3.3.

I recognize that updating this setting is called out in the Large Deployment docs, but I think 10s is a bit low for most cases. Even on our small clusters this was causing a noticeable difference with Prometheus.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Updated kubelet_status_update_frequency from 10s to 120s. Useful for larger clusters.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: poblahblahblah
Once this PR has been reviewed and has the lgtm label, please assign cristicalin for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @poblahblahblah. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[VannTen]

I don't know if we'll switch the default (we should investigate why we diverged from upstream k8s on this), but that would need a action required in the release note I think, since it changes default

[tico88612]

I think a better way to do this would be to document the record rather than change the defaults.
I agree with @VannTen if you want to change the defaults.

[poblahblahblah]

edit: I re-read the comment I wrote and I could have worded it better. I would love the know the reasoning behind 10s being the default here.

[VannTen]

I'll dig in git next week to see why this was done this way. In the meantime, would you have at hand a good background documentation on that particular setting ? Also, if there is a scale concern which depends on the number of nodes, if we change it, we could consider making the default a function of the number of nodes (something like `groups['k8s_cluster'] | dict2items | length * some factor` )

[yankay]

/ok-to-test

[VannTen]

The default setting out of the box for this is 5 minutes

Are we talking about --node-status-update-frequency ? Because from what I can tell the default is 10s like in kubespray. https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

(In that case we're unlikely to change this, deviating from kubernetes defaults has an higher bar)

[VannTen]

c07d60b -> commit which introduced the parameter.