Unset CephBlockPool mirroring if Mirroring spec is nil on SC

[vbnrh]

https://issues.redhat.com/browse/DFBUGS-1185

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vbnrh
Once this PR has been reviewed and has the lgtm label, please assign blaineexe for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rewantsoni]

/hold
This will cause problems with mirroring controller. As for provider mode, we will try to enable mirroring from the mirroring controller and the storageCluster controller will try to disable the mirroring.

[rewantsoni]

What we could do is,

if the storageCluster does not have provider mode annotation ( ocs.openshift.io/deployment-mode!= "provider")
  if storageCluster.Spec.Mirroring !=nil && storageCluster.Spec.Mirroring.Enabled
     enable mirroring and set the peer secrets
  else 
     disable mirroring

@umangachapagain @vbnrh WDYT?
This way, MCO could disable mirroring by setting the MirroringSpec as empty and the storageCluster controller would disable mirroring/use this field in non-provider mode.

And with converged when we move to use the mirroring controller to manage mirroring/secret exchange, we could remove enabling mirroring from the storageCluster controller (cephblockpool.go)

[rewantsoni]

There are other places in this file where we are enabling mirroring, please make sure that we follow the same in other places as well

[rewantsoni]

We don't have the annotation for other modes

Suggested change

	deploymentType, ok := storageCluster.Annotations["ocs.openshift.io/deployment-mode"]
	if !ok {
	return fmt.Errorf("deployment mode annotation not found")
	}
	// Since provider mode handles mirroring, we only need to handle for converged mode
	if deploymentType != "provider" {
	// Since provider mode handles mirroring, we only need to handle for internal mode
	if storageCluster.Annotations["ocs.openshift.io/deployment-mode"] != "provider" {

[vbnrh]

Annotation is guaranteed to be there in all cases ?

[rewantsoni]

No, it will be present only in provider mode, but if we query an empty map with a key that is not present it will return an empty value

[rewantsoni]

Were you able to verify it on a cluster?

[vbnrh]

Were you able to verify it on a cluster?

Not tested it but should work.

[rewantsoni]

It would be good if you could verify it so we don't miss any edge cases. Maybe add the annotation to the internal Mode cluster and check that storageCluster doesn't reconcile the mirroring field. Anyway, I will un-hold the PR.
/unhold

[leelavg]

@rewantsoni pls be aware that most if not all the traces of provider mode will be removed and this specific annotation will also go away starting in #2936 (4.19)