CRDs

[sashalarsoo]

What happened:

I installed karmada with a custom path. I joined my clusters to control plane. I can't create a propagationpolicy because of CRDs not found. The error :

kubectl apply -f propa.yaml 
error: resource mapping not found for name: "example-policy" namespace: "" from "propa.yaml": no matches for kind "PropagationPolicy" in version "policy.karmada.io/v1alpha1"
ensure CRDs are installed first

If i go to the install dir of karmada, find a crd folder. I try to apply on all this CRD folder :
~/karmada/crds$ kubectl apply -f . error: error validating "kustomization.yaml": error validating data: [apiVersion not set, kind not set]; if you choose to ignore these errors, turn validation off with --validate=false

[zhzhuang-zju]

@sashalarsoo What installation method are you using? helm, karmada-operator or karmadactl init?

[sashalarsoo]

Here is the command i used : kubectl karmada init --karmada-data karmada/ --karmada-pki karmada/

[chaosi-zju]

If i go to the install dir of karmada, find a crd folder. I try to apply on all this CRD folder :
~/karmada/crds$ kubectl apply -f . error: error validating "kustomization.yaml": error validating data: [apiVersion not set, kind not set]; if you choose to ignore these errors, turn validation off with --validate=false

not the root cause, but if you failed with this error message, your can try following command instead:

kubectl --context karmada-apiserver apply -k ~/karmada/crds --server-side=true

pay attention to -k and --server-side=true.

[zhzhuang-zju]

Is the kubeconfig and context for kubectl configured correctly?

[sashalarsoo]

Thanks for the command actually i got the following output : customresourcedefinition.apiextensions.k8s.io/overridepolicies.policy.karmada.io serverside-applied customresourcedefinition.apiextensions.k8s.io/propagationpolicies.policy.karmada.io serverside-applied customresourcedefinition.apiextensions.k8s.io/remedies.remedy.karmada.io serverside-applied customresourcedefinition.apiextensions.k8s.io/resourceinterpretercustomizations.config.karmada.io serverside-applied customresourcedefinition.apiextensions.k8s.io/resourceinterpreterwebhookconfigurations.config.karmada.io serverside-applied customresourcedefinition.apiextensions.k8s.io/serviceexports.multicluster.x-k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/serviceimports.multicluster.x-k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/workloadrebalancers.apps.karmada.io serverside-applied customresourcedefinition.apiextensions.k8s.io/works.work.karmada.io serverside-applied Error from server: failed to convert new object (/clusterresourcebindings.work.karmada.io; apiextensions.k8s.io/v1, Kind=CustomResourceDefinition) to proper version: unable to convert unstructured object to apiextensions.k8s.io/v1, Kind=CustomResourceDefinition: error decoding from json: illegal base64 data at input byte 0 Error from server: failed to convert new object (/resourcebindings.work.karmada.io; apiextensions.k8s.io/v1, Kind=CustomResourceDefinition) to proper version: unable to convert unstructured object to apiextensions.k8s.io/v1, Kind=CustomResourceDefinition: error decoding from json: illegal base64 data at input byte 0

I guess it partially worked because the apply on policy worked : kubectl apply -f propa.yaml propagationpolicy.policy.karmada.io/example-policy created

[sashalarsoo]

Quick question : How could i differenciate two target clusters ?

karmadactl get clusters --kubeconfig karmada/karmada-apiserver.config 
NAME            CLUSTER   VERSION   MODE   READY   AGE     ADOPTION
karmada-test    Karmada   v1.31.1   Push   True    3h19m   -
karmada-test2   Karmada   v1.31.1   Push   True    85m     -

The two members have the same "CLUSTER". However, there are my two targets clusters because my propagationpolicy worked and deployed nginx pods on the two clusters.

karmadactl get deployments --kubeconfig karmada/karmada-apiserver.config 
NAME    CLUSTER   READY   UP-TO-DATE   AVAILABLE   AGE     ADOPTION
nginx   Karmada   2/1     2            2           7m54s   -

I just would like to differenciate the two clusters in the output.

[zhzhuang-zju]

#5915 (comment)

You can refer to the usage of karmadactl get inside the link

[sashalarsoo]

Thanks a lot for the help !

[chaosi-zju]

partially worked

you still have two crd not installed successful, is it convenient for you to reinstall karmada?

I suggest you try: kubectl karmada init --karmada-data karmada/ --karmada-pki karmada/ --crds karmada/crds.tar.gz to reinstall it?

pay attention to set the value --crds to your local path of crds.tar.gz.

[sashalarsoo]

kubectl karmada init --karmada-data karmada/ --karmada-pki karmada/ --crds karmada/crds.tar.gz
I1209 12:27:17.644299   13234 deploy.go:266] kubeconfig file: , kubernetes: https://10.5.1.17:6443
W1209 12:27:18.371417   13234 node.go:52] The kubernetes cluster does not have a Master role.
I1209 12:27:18.371453   13234 node.go:60] Randomly select 3 Node IPs in the kubernetes cluster.
I1209 12:27:18.383924   13234 deploy.go:286] karmada apiserver ip: [10.5.1.110 10.5.1.123 10.5.1.67]
I1209 12:27:19.196329   13234 cert.go:254] Generate ca certificate success.
I1209 12:27:19.488244   13234 cert.go:254] Generate karmada certificate success.
I1209 12:27:19.981394   13234 cert.go:254] Generate apiserver certificate success.
I1209 12:27:20.256773   13234 cert.go:254] Generate front-proxy-ca certificate success.
I1209 12:27:21.668357   13234 cert.go:254] Generate front-proxy-client certificate success.
I1209 12:27:22.313117   13234 cert.go:254] Generate etcd-ca certificate success.
I1209 12:27:23.377902   13234 cert.go:254] Generate etcd-server certificate success.
I1209 12:27:24.546583   13234 cert.go:254] Generate etcd-client certificate success.
I1209 12:27:24.547103   13234 deploy.go:395] local crds file name: karmada/crds.tar.gz
error: prepare karmada failed.inValid crd tar, err: open karmada/crds.tar.gz: no such file or directory

[sashalarsoo]

I'm trying to download crds before

[sashalarsoo]

Thanks, it seems CRDs have been well installed !

[sashalarsoo]

I tried the following example : https://karmada.io/docs/userguide/cicd/working-with-flux/#helm-release-propagation.
I well installed flux on my clusters and tried to make the example work :

kubectl --kubeconfig karmada/karmada-apiserver.config get propagationpolicies
NAME             CONFLICT-RESOLUTION   PRIORITY   AGE
example-policy   Abort                 0          139m
helm-release     Abort                 0          32m
helm-repo        Abort                 0          32m

but my chart doesnt deploy :

kubectl --kubeconfig karmada/karmada-apiserver.config get helmreleases
NAME      AGE   READY   STATUS
podinfo   32m          

One of my slave clusters :

[sashalarsoo]

[chaosi-zju]

I am not much familiar with this example, so from your this picture, do you finally succeed?
If no, what is the current problem?

[sashalarsoo]

I succeed but the demo in the website is outdated. https://karmada.io/docs/userguide/cicd/working-with-flux/
When declaring Helm objects remove the beta in apiVersion.

[chaosi-zju]

I succeed but the demo in the website is outdated. https://karmada.io/docs/userguide/cicd/working-with-flux/ When declaring Helm objects remove the beta in apiVersion.

thank you very much for pointing out the error!

by the way, would you like to contribute a PR to correct the doc? If you are interested in it, you can refer to how to contribute docs.

[sashalarsoo]

[chaosi-zju]

hi @sashalarsoo, we usually do like this:

1. fork the repo to personal repo.
2. push the commit to personal repo.
3. submit a PR from personal repo to karmada repo

[sashalarsoo]

Thanks !

[sashalarsoo]

Hey i'v got a final question.
I would like to distribute a Indexed Job to clusters accross regions.
Example : I have a cluster in FRA region and AMS region. I want to split Index Job workload.
I have 800 jobs, i want the first cluster executes jobs with indexes from 0 to 399.
The second cluster must execute jobs from 400 to 799.

[chaosi-zju]

I have 800 jobs, i want the first cluster executes jobs with indexes from 0 to 399.
The second cluster must execute jobs from 400 to 799.

First, what does Index Job mean, can you give some reference?

how about mark these jobs with different label? like some jobs with a label match to policyA and propagate to FRA cluster, another jobs with b label match to policyB and propagate to AMS cluster?

[sashalarsoo]

Check this: https://kubernetes.io/blog/2021/04/19/introducing-indexed-jobs/
You can identify each job with the variable JOB_COMPLETION_INDEX. So we just have one yaml.
And we want to split the deployment of jobs in function of clusters.

[chaosi-zju]

Can index job define the starting index ?

If not, the two member clusters will all get job indexed from 0 to 399, just like:

...
status:
  completedIndexes: 0-399
  completionTime: "2024-12-11T03:54:24Z"

[sashalarsoo]

No we can't. So i think i want to pass env variable to a ressource in Karmada Propagationpolicy. Is it possible ?

[chaosi-zju]

No we can't. So i think i want to pass env variable to a ressource in Karmada Propagationpolicy. Is it possible ?

No

Does the OverwritePolicy or ResourceInterpreter suitable to you?

[sashalarsoo]

Perfect thanks !

[sashalarsoo]

I saw Karmada has a WorkloadRebalance ressource.
For example, i want 360 nodes divided in 4 clusters so with an equal weight.
If there are no nodes available in one cluster, is Karmada capable of just rebalance the nodes requests in the three other clusters ?

[sashalarsoo]

In case of rebalancing because not enough nodes are available to scale in the cluster, could i intercept a payload before rebalancing ? I'd like to know when rebalancing the target cluster etc to update a state so basically i just want before rebalancing to get all the infos about the rebalancing.

[chaosi-zju]

I saw Karmada has a WorkloadRebalance ressource.
For example, i want 360 nodes divided in 4 clusters so with an equal weight.
If there are no nodes available in one cluster, is Karmada capable of just rebalance the nodes requests in the three other clusters ?

do you mean, you use static weight propagation prolicy just like:

    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        staticWeightList:
          - targetCluster:
              clusterNames:
                - member1
            weight: 1
          - targetCluster:
              clusterNames:
                - member2
            weight: 1
          ......

you has a deployment propagated to these four clusters, but pods in member1 is pending due to lack of needed resources,

at this moment, you want to use WorkloadRebalance to rebalance the existing pending pods to other three normal clusters?

answer: no, as for static weight strategy, even if rescheduled, the new scheduling result will still contain member1?

Please tell me if it is the above scenario, and if so, we will discuss it further.

[sashalarsoo]

Yeah in fact my deployment is a job with a completions attribute. This attribute says hey if 50 completions launch 50 jobs. Each job is made to go and request for one node. Each node must contain one job maximum. So if i have 50 jobs i have to scale 50 nodes. Sometimes it fails because there is a lack of available computers in the data center (for big numbers). So in this case i need to ensure that unprovisioned computers are provisionned in another cluster.

[sashalarsoo]

Hello Karmada team.
I deployed a propagationpolicy with :
weightPreference:
dynamicWeight: AvailableReplicas

As you can see above, the scheduler failed to bind replicas to cluster because of a lack of ressources.
Before testing this policy, i was using :

 staticWeightList:
          - targetCluster:
              clusterNames:
                - member1
            weight: 1
          - targetCluster:
              clusterNames:
                - member2
            weight: 1 

With the same number of replicas, i had no problem because my clusters are scaling automatically.
My point is so the scheduler estimates a lack of ressources but scaling is enabled so my jobs cant be deployed with the availablereplicas policy.

[RainbowMango]

Hi @sashalarsoo Yes, currently when propagating applications with staticWeight, the scheduler doesn't take the available resources into account. (Note: This behavior might not be right and subject to change )

[sashalarsoo]

My error comes from this one :

It seems to take only available replicas in actual nodes. Does it take care about parallelism or not ?
With staticweightlist my 500 jobs have been launched maybe 250 by 250 in parallel but here the scheduler failed to bind the same number of jobs...

[sashalarsoo]

My app is scaling so when using availablereplicas the scheduler takes count only of actual capacity of actual nodes and is blocking my deployment but the deployment would be handled by the autoscaler.