Fix the issue of missing workqueue metrics in the Karmada controller

[chaosi-zju]

What type of PR is this?

/kind bug

What this PR does / why we need it:

remove the dependency on the k8s.io/component-base/metrics/prometheus/workqueue package to fix the issue of missing workqueue metrics in the Karmada controller.

Which issue(s) this PR fixes:

Fixes #5696

Special notes for your reviewer:

• previous metrics number 900+, now 2400+

/ # curl http://127.0.0.1:8080/metrics | grep -v '#' | wc -l
2454

Does this PR introduce a user-facing change?:

`karmada-controller-manager`: fix the issue of missing workqueue metrics in the Karmada controller

[karmada-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kevin-wangzefeng for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Project coverage is 48.17%. Comparing base (471d850) to head (46794dc).
Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
...sourceinterpreter/customized/webhook/customized.go	0.00%	2 Missing ⚠️
pkg/karmadactl/promote/promote.go	0.00%	1 Missing ⚠️
pkg/resourceinterpreter/interpreter.go	0.00%	1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5945      +/-   ##
==========================================
+ Coverage   48.15%   48.17%   +0.01%     
==========================================
  Files         664      664              
  Lines       54803    54798       -5     
==========================================
+ Hits        26393    26400       +7     
+ Misses      26693    26683      -10     
+ Partials     1717     1715       -2     

Flag	Coverage Δ
unittests	48.17% <0.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[chaosi-zju]

1. Why miss metrics

In Karmada controller, there are two workqueue metric registration, sigs.k8s.io/controller-runtime/pkg/metrics and k8s.io/component-base/metrics/prometheus/workqueue.

The simultaneous import of the two packages is not explicitly declared by us, we just want to use sigs.k8s.io/controller-runtime/pkg/metrics, while k8s.io/component-base/metrics/prometheus/workqueue is from the deep dependency.

They both register workqueue metric in init fucntion, but workqueue provider can only register once. So the import order becomes very important.

Now, you can check by go build --ldflags=--dumpdep github.com/karmada-io/karmada/cmd/controller-manager 2>&1 | grep inittask and find that k8s.io/component-base/metrics/prometheus/workqueue is imported earlier, so we set a unexpected workqueue provider which cause workqueue metrics missed.

karmada/vendor/k8s.io/component-base/metrics/prometheus/workqueue/metrics.go

Lines 104 to 109 in 471d850

	func init() {
	for _, m := range metrics {
	legacyregistry.MustRegister(m)
	}
	workqueue.SetProvider(prometheusMetricsProvider{})
	}

karmada/vendor/sigs.k8s.io/controller-runtime/pkg/metrics/workqueue.go

Lines 90 to 100 in 471d850

	func init() {
	Registry.MustRegister(depth)
	Registry.MustRegister(adds)
	Registry.MustRegister(latency)
	Registry.MustRegister(workDuration)
	Registry.MustRegister(unfinished)
	Registry.MustRegister(longestRunningProcessor)
	Registry.MustRegister(retries)
	workqueue.SetProvider(workqueueMetricsProvider{})
	}

karmada/vendor/k8s.io/client-go/util/workqueue/metrics.go

Lines 262 to 266 in 471d850

	// SetProvider sets the metrics provider for all subsequently created work
	// queues. Only the first call has an effect.
	func SetProvider(metricsProvider MetricsProvider) {
	globalMetricsFactory.setProvider(metricsProvider)
	}

karmada/vendor/k8s.io/client-go/util/workqueue/metrics.go

Lines 223 to 227 in 471d850

	func (f *queueMetricsFactory) setProvider(mp MetricsProvider) {
	f.onlyOnce.Do(func() {
	f.metricsProvider = mp
	})
	}

2. When introduced

In Karmada v1.9.9, we still use go v1.20, the Karmada controller didn't miss workqueue metrics:

➜  karmada git:(release-1.9) ✗ kh exec -it `kh get pods -n karmada-system | grep $comp | sed -n $i'p' | awk '{print $1}'` -n karmada-system -- sh
/ # curl http://127.0.0.1:8080/metrics | grep workqueue_work_duration_seconds_sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0workqueue_work_duration_seconds_sum{name="binding-eviction"} 0
workqueue_work_duration_seconds_sum{name="cluster"} 1.9943413949999997
workqueue_work_duration_seconds_sum{name="cluster-binding-eviction"} 0
workqueue_work_duration_seconds_sum{name="clusterPropagationPolicy reconciler"} 0
workqueue_work_duration_seconds_sum{name="clusterResourceBinding_status_controller"} 0
workqueue_work_duration_seconds_sum{name="clusterresourcebinding"} 0
workqueue_work_duration_seconds_sum{name="cronfederatedhpa"} 0
workqueue_work_duration_seconds_sum{name="dependencies resource detector"} 0
workqueue_work_duration_seconds_sum{name="endpointslice-collect"} 0
workqueue_work_duration_seconds_sum{name="federatedhpa"} 0
workqueue_work_duration_seconds_sum{name="federatedresourcequota"} 0
workqueue_work_duration_seconds_sum{name="hpa-replicas-syncer"} 0
workqueue_work_duration_seconds_sum{name="multiclusterservice"} 0
workqueue_work_duration_seconds_sum{name="namespace"} 0.005409679
100 95079    0 950workqueue_work_duration_seconds_sum{name="propagationPolicy reconciler"} 079
    0     0  14.2M      0workqueue_work_duration_seconds_sum{name="remedy-controller"} 0.394528854
 --:--:-- --:--:-- --:--workqueue_work_duration_seconds_sum{name="resource detector"} 0.35019135200000034
:-- 15.1M
workqueue_work_duration_seconds_sum{name="resourceBinding_status_controller"} 0
workqueue_work_duration_seconds_sum{name="resourcebinding"} 0
workqueue_work_duration_seconds_sum{name="scale ref worker"} 0
workqueue_work_duration_seconds_sum{name="service-export"} 0
workqueue_work_duration_seconds_sum{name="serviceimport"} 0
workqueue_work_duration_seconds_sum{name="work"} 0.9787631499999999
workqueue_work_duration_seconds_sum{name="work-status"} 0.32815969899999997

From Karmada v1.10.0, we use go v1.21, start introducing this problem, metrics missed:

➜  karmada git:(release-1.10) ✗ kh exec -it `kh get pods -n karmada-system | grep $comp | sed -n $i'p' | awk '{print $1}'` -n karmada-system -- sh
/ # curl http://127.0.0.1:8080/metrics | grep workqueue_work_duration_seconds_sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6474    0  6474    0     0  1794k      0 --:--:-- --:--:-- --:--:-- 2107k

3. Why old version fine

In go 1.20, we actively prioritize the import of package "sigs.k8s.io/controller-runtime/pkg/metrics" to ensure its init function called earlier. Refer to:

• clean up metrics for scheduler, descheduler and estimator #2899
• cleanup pkg import in main function #3012

But, as go 1.21 release a change, go 1.21 release note:

Package initialization order is now specified more precisely. The new algorithm is:

Sort all packages by import path.
Repeat until the list of packages is empty:
Find the first package in the list for which all imports are already initialized.
Initialize that package and remove it from the list.

We cannot explicitly specify the import order; the init fucntion of k8s.io/component-base/metrics/prometheus/workqueue will always be imported earlier.

[RainbowMango]

Actually, before #2999 we used exactly the webhookutil.NewDefaultServiceResolver(), we need to figure out why changed to apiserver.NewClusterIPServiceResolver(serviceLister).

[RainbowMango]

/hold
This essentially reverts #2999, and breaks the use case of running Karmada without coreDNS reported by @lxtywypc.