Fix VirtualMachineCRCErrors not stop firing #742

machadovilaca · 2023-11-28T13:21:17Z

What this PR does / why we need it:

The controller keeps reporting old metric values even after the VM is deleted. This PR updates the metric name and labels so that we can set up the metric value to 0, and no longer trigger the alert on VM deletion

Reduces VirtualMachineCRCErrors operator impact to none

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Release note:

Fix VirtualMachineCRCErrors not stop firing

openshift-ci · 2023-11-28T13:21:20Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

machadovilaca · 2023-11-28T13:21:53Z

/test all

kubevirt-bot · 2023-11-28T13:21:56Z

@machadovilaca: No presubmit jobs available for kubevirt/ssp-operator@main

In response to this:

/test all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

github-actions · 2023-11-29T13:31:54Z

/cc sradco

github-actions · 2023-11-29T13:37:52Z

/cc sradco

machadovilaca · 2023-11-29T13:42:17Z

/test e2e-functests

kubevirt-bot · 2023-11-29T13:42:19Z

@machadovilaca: No presubmit jobs available for kubevirt/ssp-operator@main

In response to this:

/test e2e-functests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

0xFelix

This PR changes quite a lot. Can you explain a little how the fix works and add the explanation also to the PR and commit descriptions?

tests/monitoring_test.go

github-actions · 2023-12-04T10:53:39Z

/cc sradco

github-actions · 2023-12-04T10:57:49Z

/cc sradco

machadovilaca · 2023-12-04T11:00:58Z

This PR changes quite a lot. Can you explain a little how the fix works and add the explanation also to the PR and commit descriptions?

@0xFelix The problem was that the controller kept reporting the metric for old VMs even after they were deleted. In the controller, we can understand when a VM is deleted, but at that moment we don't have access to the details present in the old labels (pv name, pv type, etc...). This PR changes the metric so that it only uses the VM name and namespace as labels, so that we can update its value, when a VM is deleted, to a no-error value. Alert expression is updated accordingly.

machadovilaca · 2024-01-02T18:16:27Z

/retest

github-actions · 2024-01-04T18:21:23Z

/cc sradco

github-actions · 2024-01-04T18:25:17Z

/cc sradco

github-actions · 2024-01-04T18:48:35Z

/cc sradco

0xFelix · 2024-01-08T09:34:10Z

controllers/vm_controller.go

@@ -58,7 +58,11 @@ func (r *VmReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Re
 	vm := kubevirtv1.VirtualMachine{}
 	if err := r.client.Get(ctx, req.NamespacedName, &vm); err != nil {
 		if errors.IsNotFound(err) {
-			// VM was deleted, so we can ignore it
+			// VM was deleted
+			vm.Name = req.Name


If the VM was deleted, why do we add it to the metrics? Also it looks error-prone to call metrics.SetVmWithVolume with nil-pointers, as IIUC this func does not check for nil pointers sufficiently.

the issue was that the controller kept exposing the metric that the VM, although already deleted, as the problematic volume configuration
this way when a VM is deleted, the controller no longer has a true value for the metric

I think the nil check in the function covers this case completely:

if pv == nil || pvc == nil { vmRbdVolume.WithLabelValues(vm.Name, vm.Namespace).Set(0) return }

ssp-operator/pkg/monitoring/metrics/rbd_metrics.go

Line 29 in 33b6ca7

func SetVmWithVolume(vm *kubevirtv1.VirtualMachine, pvc *k8sv1.PersistentVolumeClaim, pv *k8sv1.PersistentVolume) {

The function that is called looks different, than what you quoted?

I also changed that function on this PR:
https://github.com/kubevirt/ssp-operator/pull/742/files#diff-387c26f005e40fae497f3a5c01da1f877fc4eee90da11b7d8cf47512d8474b82R29-R31

Right, my mistake.

tests/monitoring_test.go

The controller keeps reporting old metric values even after the VM is deleted. This commit updates the metric name and labels so that we can set up the metric value to 0, and no longer trigger the alert on VM deletion Signed-off-by: João Vilaça <[email protected]>

Signed-off-by: João Vilaça <[email protected]>

github-actions · 2024-01-08T14:31:25Z

/cc sradco

sonarqubecloud · 2024-01-08T14:31:36Z

Quality Gate failed

Failed conditions

3.4% Duplication on New Code (required ≤ 3%)

See analysis details on SonarCloud

0xFelix

/approve

0xFelix · 2024-01-09T10:13:41Z

controllers/vm_controller.go

@@ -58,7 +58,11 @@ func (r *VmReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Re
 	vm := kubevirtv1.VirtualMachine{}
 	if err := r.client.Get(ctx, req.NamespacedName, &vm); err != nil {
 		if errors.IsNotFound(err) {
-			// VM was deleted, so we can ignore it
+			// VM was deleted
+			vm.Name = req.Name


Right, my mistake.

kubevirt-bot · 2024-01-09T10:14:07Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 0xFelix

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [0xFelix]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sradco · 2024-01-09T10:46:49Z

/lgtm

0xFelix · 2024-01-09T12:50:57Z

/retest

0xFelix · 2024-01-09T18:09:11Z

If you want this to be in 4.15 you need to cherry-pick it to release-v0.19.

machadovilaca · 2024-01-10T10:59:57Z

/cherry-pick release-v0.19

kubevirt-bot · 2024-01-10T11:00:38Z

@machadovilaca: new pull request created: #825

In response to this:

/cherry-pick release-v0.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

machadovilaca · 2024-03-04T16:05:58Z

/cherry-pick release-v0.18

kubevirt-bot · 2024-03-04T16:06:41Z

@machadovilaca: new pull request created: #906

In response to this:

/cherry-pick release-v0.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kubevirt-bot requested review from 0xFelix and lyarwood November 28, 2023 13:21

kubevirt-bot added the size/XS label Nov 28, 2023

machadovilaca force-pushed the fix-VirtualMachineCRCErrors-not-stop-firing branch from 0256308 to ea42bff Compare November 29, 2023 13:31

kubevirt-bot added size/M and removed size/XS labels Nov 29, 2023

kubevirt-bot requested a review from sradco November 29, 2023 13:31

machadovilaca force-pushed the fix-VirtualMachineCRCErrors-not-stop-firing branch from ea42bff to 58ad69e Compare November 29, 2023 13:37

0xFelix reviewed Nov 30, 2023

View reviewed changes

tests/monitoring_test.go Outdated Show resolved Hide resolved

machadovilaca force-pushed the fix-VirtualMachineCRCErrors-not-stop-firing branch from 58ad69e to 725d64c Compare December 4, 2023 10:53

machadovilaca force-pushed the fix-VirtualMachineCRCErrors-not-stop-firing branch from 725d64c to 2a7b2f4 Compare December 4, 2023 10:57

kubevirt-bot added size/L and removed size/M labels Dec 4, 2023

machadovilaca marked this pull request as ready for review December 4, 2023 11:01

kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 4, 2023

kubevirt-bot requested review from akrejcir and ksimon1 December 4, 2023 11:01

machadovilaca force-pushed the fix-VirtualMachineCRCErrors-not-stop-firing branch from 10b0b31 to 47709c4 Compare January 4, 2024 18:21

machadovilaca force-pushed the fix-VirtualMachineCRCErrors-not-stop-firing branch from 47709c4 to f74b983 Compare January 4, 2024 18:25

machadovilaca force-pushed the fix-VirtualMachineCRCErrors-not-stop-firing branch from f74b983 to e18adc2 Compare January 4, 2024 18:48

0xFelix reviewed Jan 8, 2024

View reviewed changes

machadovilaca added 2 commits January 8, 2024 14:31

Add offset to ginkgo e2e validations

7af6434

Signed-off-by: João Vilaça <[email protected]>

machadovilaca force-pushed the fix-VirtualMachineCRCErrors-not-stop-firing branch from e18adc2 to 7af6434 Compare January 8, 2024 14:31

0xFelix reviewed Jan 9, 2024

View reviewed changes

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 9, 2024

kubevirt-bot assigned sradco Jan 9, 2024

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Jan 9, 2024

kubevirt-bot merged commit 4f1a32a into kubevirt:main Jan 9, 2024
13 checks passed

machadovilaca deleted the fix-VirtualMachineCRCErrors-not-stop-firing branch January 9, 2024 14:29

kubevirt-bot mentioned this pull request Jan 10, 2024

[release-v0.19] Fix VirtualMachineCRCErrors not stop firing #825

Merged

kubevirt-bot mentioned this pull request Mar 4, 2024

[release-v0.18] Fix VirtualMachineCRCErrors not stop firing #906

Closed

machadovilaca mentioned this pull request Mar 4, 2024

[release-v0.18] Fix VirtualMachineCRCErrors not stop firing #907

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix VirtualMachineCRCErrors not stop firing #742

Fix VirtualMachineCRCErrors not stop firing #742

machadovilaca commented Nov 28, 2023 •

edited

Loading

openshift-ci bot commented Nov 28, 2023

machadovilaca commented Nov 28, 2023

kubevirt-bot commented Nov 28, 2023

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

machadovilaca commented Nov 29, 2023

kubevirt-bot commented Nov 29, 2023

0xFelix left a comment

github-actions bot commented Dec 4, 2023

github-actions bot commented Dec 4, 2023

machadovilaca commented Dec 4, 2023

machadovilaca commented Jan 2, 2024

github-actions bot commented Jan 4, 2024

github-actions bot commented Jan 4, 2024

github-actions bot commented Jan 4, 2024

0xFelix Jan 8, 2024

machadovilaca Jan 8, 2024

0xFelix Jan 8, 2024

machadovilaca Jan 8, 2024

0xFelix Jan 9, 2024

github-actions bot commented Jan 8, 2024

sonarqubecloud bot commented Jan 8, 2024

0xFelix left a comment

0xFelix Jan 9, 2024

kubevirt-bot commented Jan 9, 2024

sradco commented Jan 9, 2024

0xFelix commented Jan 9, 2024

0xFelix commented Jan 9, 2024

machadovilaca commented Jan 10, 2024

kubevirt-bot commented Jan 10, 2024

machadovilaca commented Mar 4, 2024

kubevirt-bot commented Mar 4, 2024

Fix VirtualMachineCRCErrors not stop firing #742

Fix VirtualMachineCRCErrors not stop firing #742

Conversation

machadovilaca commented Nov 28, 2023 • edited Loading

openshift-ci bot commented Nov 28, 2023

machadovilaca commented Nov 28, 2023

kubevirt-bot commented Nov 28, 2023

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

machadovilaca commented Nov 29, 2023

kubevirt-bot commented Nov 29, 2023

0xFelix left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 4, 2023

github-actions bot commented Dec 4, 2023

machadovilaca commented Dec 4, 2023

machadovilaca commented Jan 2, 2024

github-actions bot commented Jan 4, 2024

github-actions bot commented Jan 4, 2024

github-actions bot commented Jan 4, 2024

0xFelix Jan 8, 2024

Choose a reason for hiding this comment

machadovilaca Jan 8, 2024

Choose a reason for hiding this comment

0xFelix Jan 8, 2024

Choose a reason for hiding this comment

machadovilaca Jan 8, 2024

Choose a reason for hiding this comment

0xFelix Jan 9, 2024

Choose a reason for hiding this comment

github-actions bot commented Jan 8, 2024

sonarqubecloud bot commented Jan 8, 2024

Quality Gate failed

0xFelix left a comment

Choose a reason for hiding this comment

0xFelix Jan 9, 2024

Choose a reason for hiding this comment

kubevirt-bot commented Jan 9, 2024

sradco commented Jan 9, 2024

0xFelix commented Jan 9, 2024

0xFelix commented Jan 9, 2024

machadovilaca commented Jan 10, 2024

kubevirt-bot commented Jan 10, 2024

machadovilaca commented Mar 4, 2024

kubevirt-bot commented Mar 4, 2024

machadovilaca commented Nov 28, 2023 •

edited

Loading