HIVE-2847: AWSPrivateLink: improve VPCEndpoint reconcile

[2uasimojo]

• Detect whether a previously-configured hub VPCEndpoint went bad (was rejected or deleted) and replace it. NOTE: If the new VPCEndpoint is in a different VPC than the original, we will also replace the HostedZone, leaking the old one.
• Produce a log message with the count of the total number of hub VPCEndpoints to facilitate leak detection. This is at Warn level if >1, Info otherwise; and contains a field with key numVPCEndpoints for your alerting needs.

[openshift-ci-robot]

@2uasimojo: This pull request references HIVE-2847 which is a valid jira issue.

In response to this:

• Detect whether a previously-configured hub VPCEndpoint went bad (was rejected or deleted) and replace it.
• Produce a metric counting the total number of hub VPCEndpoints to facilitate leak detection. Consumers could alert on something like (hive_hub_vpc_endpoints_total / number_of_healthy_privatelink_clusters) > 1).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[2uasimojo]

/hold

I should add UT.

/cc @dustman9000 @joshbranham

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 50.28%. Comparing base (4bc5b48) to head (04adc57).
Report is 4 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2671      +/-   ##
==========================================
+ Coverage   50.24%   50.28%   +0.04%     
==========================================
  Files         287      287              
  Lines       33971    33995      +24     
==========================================
+ Hits        17069    17096      +27     
+ Misses      15553    15550       -3     
  Partials     1349     1349              

Files with missing lines	Coverage Δ
...roller/awsprivatelink/awsprivatelink_controller.go	69.68% <100.00%> (+1.07%)	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[2uasimojo]

/assign @jstuever

[2uasimojo]

/hold cancel

[2uasimojo]

/hold

I did the metrics wrong.

[openshift-ci-robot]

@2uasimojo: This pull request references HIVE-2847 which is a valid jira issue.

In response to this:

• Detect whether a previously-configured hub VPCEndpoint went bad (was rejected or deleted) and replace it.
• Produce a log message with the count of the total number of hub VPCEndpoints to facilitate leak detection. This is at Warn level if >1, Info otherwise; and contains a field with key numVPCEndpoints for your alerting needs.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[2uasimojo]

/hold cancel

/assign @jstuever

[jstuever]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 2uasimojo, jstuever

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [2uasimojo,jstuever]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@2uasimojo: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.