[Backport 7.61.x] [Bug Fix] fix backward compatibility issue in remote tagger dca server

[agent-platform-auto-pr]

Backport 1c2acb4 from #31711.

---

<!--

• Contributors are encouraged to read our CONTRIBUTING documentation.
• Both Contributor and Reviewer Checklists are available at https://datadoghq.dev/datadog-agent/guidelines/contributing/#pull-requests.
• The pull request:
  • Should only fix one issue or add one feature at a time.
  • Must update the test suite for the relevant functionality.
  • Should pass all status checks before being reviewed or merged.
• Commit titles should be prefixed with general area of pull request's change.
• Please fill the below sections if possible with relevant information or links.
  -->

What does this PR do?

Fixes backward compatibility issue in the remote tagger server.

The broken backward compatibility makes it impossible for cluster check runners to get tags using the remote tagger (except for the first runner that connects to the server). This happens if we use older versions of cluster check runners with newer versions of the cluster agent. The backward compatibility was broken in 7.60, but we are fixing it in 7.61 because this feature is disabled by default and has no customer impact since it is not exposed in helm-chart or the operator.

Motivation

Restore backward compatibility.

Describe how you validated your changes

<!--
Validate your changes before merge, ensuring that:

• Your PR is tested by static / unit / integrations / e2e tests
• Your PR description details which e2e tests cover your changes, if any
• The PR description contains details of how you validated your changes. If you validated changes manually and not through automated tests, add context on why automated tests did not fit your changes validation.

If you want additional validation by a second person, you can ask reviewers to do it. Describe how to set up an environment for manual tests in the PR description. Manual validation is expected to happen on every commit before merge.

Any manual validation step should then map to an automated test. Manual validation should not substitute automation, minus exceptions not supported by test tooling yet.
-->

Deploy the cluster agent with CLC runners enabled and using the remote tagger with at least 2 replicas of CLCs.
All CLC replicas should be able to get tags from the cluster tagger.

The CLC runner should have a version 7.59 or older, and the DCA should be using version 7.61 or newer.

Example:

datadog:
  kubelet:
    tlsVerify: false
  clusterChecks:
    enabled: true
  logLevel: DEBUG
  apiKeyExistingSecret: datadog-secret
  appKeyExistingSecret: datadog-secret
  envDict:
    DD_TAGS: "source:node-tags"
    DD_EXTRA_TAGS: "source:node-extra"
    DD_CLUSTER_CHECKS_EXTRA_TAGS: "source:node-cluster_extra" # Note: should never see these
    DD_ORCHESTRATOR_EXPLORER_EXTRA_TAGS: "source:node-orch_extra" # Note: should never see these

clusterChecksRunner:
  replicas: 3
  enabled: true
  image:
    tag: 7.59.0
  envDict:
    DD_CLC_RUNNER_REMOTE_TAGGER_ENABLED: "true"
clusterAgent:
  envDict:
    DD_TAGS: "source:dca-tags"
    DD_EXTRA_TAGS: "source:dca-extra"
    DD_CLUSTER_CHECKS_EXTRA_TAGS: "source:dca-cluster_extra"
    DD_ORCHESTRATOR_EXPLORER_EXTRA_TAGS: "source:dca-orch_extra"
  enabled: true
  replicas: 1

Once everything gets into running state, we should be able to see the same tags in all CLC runner taggers:

kubectl exec datadog-agent-clusterchecks-6bbb898c96-6tfl2 -- agent tagger-list

=== Entity internal://global-entity-id ===
== Source remote =
=Tags: [source:dca-cluster_extra source:dca-extra source:dca-orch_extra source:dca-tags]
===

Without this fix, only one runner will get the tags, and others would fail to get them.

Possible Drawbacks / Trade-offs

Customers who:

• manually enable remote tagger in CLC runners (by setting DD_CLC_RUNNER_REMOTE_TAGGER_ENABLED to true)
• AND upgrade from version 7.59 to 7.60
• AND have at least 2 replicas of clustercheckrunners

Will still be facing broken backward compatibility.

However, we choose not to fix this in 7.60 in order to avoid building a new release candidate and considering that enabling remote tagger in clc runners is disabled by default and not exposed publicly in helm chart or in operator, so it should not have any customer impact (unless the customer manually sets this option).

Additional Notes

<!--

• Anything else we should know when reviewing?
• Include benchmarking information here whenever possible.
• Include info about alternatives that were considered and why the proposed
  version was chosen.
  -->

[agent-platform-auto-pr]

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=50293875 --os-family=ubuntu

Note: This applies to commit 5e86b260

[adel121]

/merge

[dd-devflow]

Devflow running: /merge

View all feedbacks in Devflow UI.

---

2024-12-24 10:37:31 UTC ℹ️ MergeQueue: pull request added to the queue

The median merge time in 7.61.x is 33m.

---

2024-12-24 11:16:18 UTC ℹ️ MergeQueue: This merge request was merged