feat: add rollout API spec #897

ryanzhang-oss · 2024-08-05T06:22:31Z

Description of your changes

An overall idea of rollout API

Fixes #

I have:

Run make reviewable to ensure this PR is ready for review.

How has this code been tested

Special notes for your reviewer

michaelawyu

Added some comments, PTAL 🙏

apis/placement/v1alpha1/stagerollout_types.go

apis/placement/v1beta1/clusterresourceplacement_types.go

apis/placement/v1alpha1/stagerollout_types.go

michaelawyu · 2024-08-06T00:35:58Z

apis/placement/v1alpha1/stagerollout_types.go

+// StageRolloutConditionType identifies a specific condition of the StageRollout.
+type StageRolloutConditionType string
+
+const (


Hi Ryan! Just for clarification, from the comments it seems that StageRolloutWaiting is just a special case of StageRollingOut being true?

apis/placement/v1alpha1/stagerollout_types.go

zhiying-lin · 2024-08-15T07:30:54Z

apis/placement/v1alpha1/stagerollout_types.go

+	// the common post rollout tasks. This setting will override the common post rollout tasks if there are
+	// duplicate type of tasks with different settings.
+	// +optional
+	PostRolloutTasks PostRolloutTask `json:"postRolloutTasks,omitempty"`


So we will run the PostRolloutTasks first then the common PostRolloutTasks?

removed the common one for now to simplify

zhiying-lin · 2024-08-15T08:21:56Z

apis/placement/v1alpha1/stagerollout_types.go

+}
+
+// StageRolloutConfig describes a single rollout stage group configuration.
+type StageRolloutConfig struct {


how do we rollout the selected clusters within a stage?

sequentially or concurrently?

apis/placement/v1alpha1/stagerollout_types.go

zhiying-lin · 2024-08-15T08:31:19Z

apis/placement/v1alpha1/stagerollout_types.go

+type PostRolloutTask struct {
+	// The name of the RolloutApprovalRequest we need to create to get the approval for the next stage.
+	// An RolloutApprovalRequest object will be created if this is not empty.
+	// +optional
+	ApprovalRequestName string `json:"approvalTask,omitempty"`
+
+	// The time to wait after all the clusters in the current stage complete the update
+	// before we move to the next stage.
+	// +optional
+	WaitTime *metav1.Duration `json:"waitTime,omitempty"`
+}


Suggested change

type PostRolloutTask struct {

// The name of the RolloutApprovalRequest we need to create to get the approval for the next stage.

// An RolloutApprovalRequest object will be created if this is not empty.

// +optional

ApprovalRequestName string `json:"approvalTask,omitempty"`

// The time to wait after all the clusters in the current stage complete the update

// before we move to the next stage.

// +optional

WaitTime *metav1.Duration `json:"waitTime,omitempty"`

}

type PostRolloutTask struct {

type TaskType // approval, wait

// The time to wait after all the clusters in the current stage complete the update

// before we move to the next stage.

// +optional

WaitTime *metav1.Duration `json:"waitTime,omitempty"`

}

PostRolloutTasks []PostRolloutTask

Then user can define their own sequence. Otherwise, it's hard to tell which one runs first.

My current idea is that they run at the same time. For example, we don't wait before or after the approval, the wait time is the minimum time needed between stages.

Also, I thought about the type pattern. The problem with that is each type requires a distinct set of parameters which pretty much makes the taskType field redundant.

zhiying-lin · 2024-08-15T08:33:00Z

apis/placement/v1alpha1/stagerollout_types.go

+	// There can only be one active rollout for each ClusterResourcePlacement,
+	// we will fail the rollout if there is already an active rollout.
+	// +required
+	ParentCRPName string `json:"parentCRPName"`


i'm thinking of the other way. CRP refers to the StageRollout name.
StageRollout will be a common rollout configurations for multiple CRP.

All the rollout status will be part of the CRP status, like today.

I am not sure if that would work as different CRPs can pick different clusters which means the stages are different between them.

I am actually thinking the other way around, one CRP can have multiple rollouts just like our AKS rollout. We have daily releases to canary only and one or two weekly releases going on simultaneously.

Do we need this field? This is auto created for a CRP. Its owner reference should contain the CRP. Just like pod. Do we need a deployment name field in a pod?

it's not auto created for a CRP. This CR is created by user to trigger an update run.

apis/placement/v1alpha1/stagerollout_types.go

apis/placement/v1beta1/clusterresourceplacement_types.go

circy9 · 2024-08-20T17:40:38Z

apis/placement/v1alpha1/stagerollout_types.go

+
+// StagedRolloutRun defines a stage by stage rollout run that is applied to
+// the clusters that the corresponding ClusterResourcePlacement selects.
+type StagedRolloutRun struct {


RollingUpdate -> RollingUpdateConfig in the current API.

Use StagedUpdate -> StagedUpdateRun, StagedUpdateStrategy in the new API.

circy9 · 2024-08-20T17:51:17Z

apis/placement/v1alpha1/stagerollout_types.go

+)
+
+// +genclient
+// +genclient:namespaced


Why is the resource namespaced if it is used by CRP which is not namespaced? How to decide which namespace a run should be in?

circy9 · 2024-08-20T17:54:55Z

apis/placement/v1alpha1/stagerollout_types.go

+// +kubebuilder:resource:scope="Namespaced",categories={fleet,fleet-placement}
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
+
+// StagedRolloutRun defines a stage by stage rollout run that is applied to


Who creates this resource?

I assume it is auto created whennever we need to roll out a change in CRP. If that's the case, CRP must contain StagedUpdateConfig (similar with RollingUpdateConfig), which includes at least one field, a reference to StagedUpdateStrategy (as strategy must be provide by customers, not auto generated by us).

circy9 · 2024-08-20T23:25:19Z

apis/placement/v1alpha1/stagerollout_types.go

+	// There can only be one active rollout for each ClusterResourcePlacement,
+	// we will fail the rollout if there is already an active rollout.
+	// +required
+	ParentCRPName string `json:"parentCRPName"`


Do we need this field? This is auto created for a CRP. Its owner reference should contain the CRP. Just like pod. Do we need a deployment name field in a pod?

circy9 · 2024-08-20T23:28:17Z

apis/placement/v1alpha1/stagerollout_types.go

+	// The name of the rollout strategy that specifies the stages and the sequence in which
+	// the application will be updated on the member clusters.
+	// +required
+	StagedRolloutStrategy string `json:"stagedRolloutStrategy"`


This should be the first field?

circy9 · 2024-08-21T00:20:44Z

apis/placement/v1alpha1/stagerollout_types.go

+//   - `ParentStageRollout` which points to its owner stage rollout.
+//   - `TargetStage` which is the name of the stage that this approval request is for.
+//   - `IsLatestRolloutApproval` which indicates whether this approval request is the latest one related to this rollout
+type RolloutApprovalRequest struct {


Which namespace should the rollout approve request be in? Will it be in the same one as rollout update run? How do we manage access control of approve requests vs runs vs strategies?

apis/placement/v1alpha1/stagedupdate_types.go

circy9 · 2024-08-22T23:39:33Z

apis/placement/v1alpha1/stagedupdate_types.go

+	// The clusters in each stage are ordered following its order to be rolled out.
+	// The updateRun will stop if the cluster stages change during the update.
+	// +required
+	ComputedStages [][]string `json:"currentStages"`


Use []StageStatus which contains each cluster's status. See this one: https://learn.microsoft.com/en-us/rest/api/fleet/update-runs/get?view=rest-fleet-2023-10-15&tabs=HTTP

circy9 · 2024-08-22T23:43:01Z

apis/placement/v1alpha1/stagedupdate_types.go

+	// Therefore, we need to keep track of the clusters that we have applied the resources to but no long is selected.
+	// We remove the application from these clusters after all the stages are completed and remove them from this list.
+	// +required
+	ToBeRevertedClusters []string `json:"toBeRevertedClusters"`


DeletionStageStatus StageStatus
contains a list of clusters whose work will be deleted ordered by the given sort label key.

circy9 · 2024-08-22T23:50:34Z

apis/placement/v1alpha1/stagedupdate_types.go

+	// The clusters in each stage are ordered following its order to be rolled out.
+	// The updateRun will stop if the cluster stages change during the update.
+	// +required
+	ComputedStages [][]string `json:"currentStages"`


We should add UpdateStatus at each level: cluster, stage, run.

https://learn.microsoft.com/en-us/rest/api/fleet/update-runs/get?view=rest-fleet-2023-10-15&tabs=HTTP#updatestatus

apis/placement/v1alpha1/stagedupdate_types.go

ryanzhang-oss · 2024-08-21T00:48:42Z

apis/placement/v1alpha1/stagerollout_types.go

+	// There can only be one active rollout for each ClusterResourcePlacement,
+	// we will fail the rollout if there is already an active rollout.
+	// +required
+	ParentCRPName string `json:"parentCRPName"`


it's not auto created for a CRP. This CR is created by user to trigger an update run.

ryanzhang-oss · 2024-08-22T23:07:54Z

apis/placement/v1alpha1/stagedupdate_types.go

+	// The clusters in each stage are ordered following its order to be rolled out.
+	// The updateRun will stop if the cluster stages change during the update.
+	// +required
+	ComputedStages [][]string `json:"currentStages"`


add cluster level detail

apis/placement/v1alpha1/stagedupdate_types.go

apis/placement/v1beta1/clusterresourceplacement_types.go

zhiying-lin · 2024-08-26T07:19:43Z

apis/placement/v1alpha1/stagedupdate_types.go

+
+	// The override snapshot index that this update run is based on.
+	// +required
+	OverrideSnapShotIndex string `json:"overrideSnapShotIndex"`


for the override, we don't support snapshot index because there will be always one snapshot per cro or ro. Here user needs to configure a list of ROs snapshot or CROs snapshot.

How do users find out which ROs and CROs apply to this given placement? Listing them all seems a very cumbersome task.

This has to go to the status field instead.

apis/placement/v1alpha1/stagedupdate_types.go

circy9 · 2024-08-29T00:35:11Z

apis/placement/v1alpha1/stagedupdate_types.go

+
+	// The override snapshot index that this update run is based on.
+	// +required
+	OverrideSnapShotIndex string `json:"overrideSnapShotIndex"`


How do users find out which ROs and CROs apply to this given placement? Listing them all seems a very cumbersome task.

apis/placement/v1alpha1/stagedupdate_types.go

circy9 · 2024-09-06T00:34:53Z

apis/placement/v1alpha1/stagedupdate_types.go

+	// Conditions is an array of current observed conditions for the specific type of post update task.
+	// Known conditions are "ApprovalRequestCreated", "WaitTimeElapsed", and "ApprovalRequestApproved".
+	// +kubebuilder:validation:Optional
+	Conditions []metav1.Condition `json:"conditions"`


Can you reuse UpdateStatus instead of Conditions? Maybe rename to StepStatus? Everything: a stage, a cluster, a after stage task is a step in the staged update run.

You also need to record start time and complete time of the step. This is especially useful for TimedWait so the user knows when the wait started and when it ended.

The convention in k8s is to use conditions to record information for each state (running, wait, completed). I think it's better to at least keep the overall state in condition instead of a single status. We can convert all the state back into conditions too but it's little bit too much.

How about this? We use conditions at the updaterun level, but use updatestatus for stages and below.

Better to have a start time and end time for these AfterStageTasks too. So it is better to use UpdateStatus instead conditions for AfterStageTasks.

The condition states (ApprovalRequestCreated, ApprovalRequestApproved, WaitTimeElapsed) in the afterStageTasks are different from the update status so it's hard to reuse it.

NotStarted -> (created approval request) -> waiting -> (approve request) -> completed
NotStarted -> (start wait time) -> waiting -> (wait time has passed) -> completed

UpdateRun is a tree of steps: a step can be a run, a stage, a cluster, a after-stage task.

It is better to have the same state model for each step so customers don't need to learn new state/condition model for each step.

That's why we should have the same state for the run and each after-stage task as well.

Fleet RP also have the wait time step and use the same state data model.

apis/placement/v1alpha1/stagedupdate_types.go

jwtty · 2024-09-06T05:42:49Z

apis/placement/v1alpha1/stagedupdate_types.go

+	// for this stage. There cannot be overlapping clusters between stages when the stagedUpdateRun is created.
+	// If the label selector is absent, all the selected clusters are selected.
+	// +kubebuilder:validation:Optional
+	LabelSelector *metav1.LabelSelector `json:"labelSelector"`


We are going to do an intersection between the clusters selected by the scheduler and the clusters selected here right?

apis/placement/v1alpha1/stagedupdate_types.go

jwtty · 2024-09-06T07:35:59Z

apis/placement/v1alpha1/stagedupdate_types.go

+//   - `ParentStagedUpdateRun` which points to its owner stage update.
+//   - `TargetStage` which is the name of the stage that this approval request is for.
+//   - `IsLatestUpdateRunApproval` which indicates whether this approval request is the latest one related to this update run.
+type StageApprovalRequest struct {


A little bit confused: how will this request get approved?

user needs to patch the "Approved" condition of the request as true

apis/placement/v1alpha1/stagedupdate_types.go

zhiying-lin · 2024-09-09T02:18:00Z

apis/placement/v1alpha1/stagedupdate_types.go

@@ -346,6 +368,12 @@ const (
 	// - "False" means the staged update run is not waiting between stages.
 	// - "Unknown" means it is unknown.
 	StagedUpdateRunConditionWaitingBetweenStage StagedUpdateRunConditionType = "WaitingBetweenStage"


do we need the "Running" and "WaitingBetweenStage" conditions? they could be "reason" when the completed is "false".

"Running", "WaitingBetweenStage" and "Completed" all these conditions are delivering some duplicate information.

With "running" and "waittingBetweenStage" conditions, we will get the information like last time the stage has completed. If we collapse them into one condition then we lost a bit of the history.

Do we really need both conditions and state? State serves the same purpose as conditions. State contains starttime and endtime.

RunStatus {
runState State
stages []StageStatus
deletionStage StageStatus
}

StageStatus {
stageName string
stageState State
clusters []ClusterStatus
afterStageTasks []TaskStatus
}

ClusterStatus {
clusterName string
clusterState State
}

TastkStatus {
taskType
taskState State
}

It's k8s convention in the status to use condition to show the state transition which is pretty much equivalent to the runState. I think using state is more efficient and acceptable as an internal field but we better to keep condition as the first layer fields status. If we really want to unify them then they all should be conditions like in CPR.

agree with Liqian about the detailed stage information. They are already part of the StageStatus.

For the conditions in the "StagedUpdateRunStatus", can we treat it as overall condition/status? so that user can query this condition to know about whether the updateRun has completed or not.

yes, the conditions in the "StagedUpdateRunStatus" are the overall condition/status of the updateRun.

apis/placement/v1alpha1/stagedupdate_types.go

circy9 · 2024-09-09T20:48:10Z

apis/placement/v1alpha1/stagedupdate_types.go

@@ -346,6 +368,12 @@ const (
 	// - "False" means the staged update run is not waiting between stages.
 	// - "Unknown" means it is unknown.
 	StagedUpdateRunConditionWaitingBetweenStage StagedUpdateRunConditionType = "WaitingBetweenStage"


Do we really need both conditions and state? State serves the same purpose as conditions. State contains starttime and endtime.

RunStatus {
runState State
stages []StageStatus
deletionStage StageStatus
}

StageStatus {
stageName string
stageState State
clusters []ClusterStatus
afterStageTasks []TaskStatus
}

ClusterStatus {
clusterName string
clusterState State
}

TastkStatus {
taskType
taskState State
}

circy9 · 2024-09-09T20:49:56Z

apis/placement/v1alpha1/stagedupdate_types.go

 type StagedUpdateRunSpec struct {
 	// A reference to the placement that this update run is applied to.
 	// There can be multiple active update runs for each placement but
 	// it's up to the devOps to make sure they don't conflict with each other.
 	// +kubebuilder:validation:Required
 	PlacementRef PlacementReference `json:"placementRef"`

-	// The resource snapshot index that this update run is based on.
+	// The snapshot index of the selected resources to be updated across clusters.


Could you give a concrete example on what you mean by the snaptshot index?

Something like:

e.g., for resource snapshots "xxx-123-1" and "xxx-123-2", "123" is the index.

I noticed that you used policysnaptshot name instead of index below. Is that because there are N resource snapshot for a single version of selected resources?

yes, for each resource snapshot index, there are N resource snapshots. The index is "123" in the example you gave. There is only 1 policy snapshot object for a snapshot of the CRP placement policy.

apis/placement/v1alpha1/stagedupdate_types.go

circy9 · 2024-09-10T00:07:24Z

apis/placement/v1alpha1/stagedupdate_types.go

+	// Conditions is an array of current observed conditions for the specific type of post update task.
+	// Known conditions are "ApprovalRequestCreated", "WaitTimeElapsed", and "ApprovalRequestApproved".
+	// +kubebuilder:validation:Optional
+	Conditions []metav1.Condition `json:"conditions"`


How about this? We use conditions at the updaterun level, but use updatestatus for stages and below.

Better to have a start time and end time for these AfterStageTasks too. So it is better to use UpdateStatus instead conditions for AfterStageTasks.

apis/placement/v1alpha1/stagedupdate_types.go

zhiying-lin · 2024-09-10T03:16:44Z

apis/placement/v1alpha1/stagedupdate_types.go

@@ -346,6 +368,12 @@ const (
 	// - "False" means the staged update run is not waiting between stages.
 	// - "Unknown" means it is unknown.
 	StagedUpdateRunConditionWaitingBetweenStage StagedUpdateRunConditionType = "WaitingBetweenStage"


agree with Liqian about the detailed stage information. They are already part of the StageStatus.

For the conditions in the "StagedUpdateRunStatus", can we treat it as overall condition/status? so that user can query this condition to know about whether the updateRun has completed or not.

apis/placement/v1alpha1/stagedupdate_types.go

circy9 · 2024-09-10T17:48:52Z

apis/placement/v1alpha1/stagedupdate_types.go

+	// Conditions is an array of current observed conditions for the specific type of post update task.
+	// Known conditions are "ApprovalRequestCreated", "WaitTimeElapsed", and "ApprovalRequestApproved".
+	// +kubebuilder:validation:Optional
+	Conditions []metav1.Condition `json:"conditions"`


NotStarted -> (created approval request) -> waiting -> (approve request) -> completed
NotStarted -> (start wait time) -> waiting -> (wait time has passed) -> completed

UpdateRun is a tree of steps: a step can be a run, a stage, a cluster, a after-stage task.

It is better to have the same state model for each step so customers don't need to learn new state/condition model for each step.

That's why we should have the same state for the run and each after-stage task as well.

circy9 · 2024-09-10T17:49:45Z

apis/placement/v1alpha1/stagedupdate_types.go

+	// Conditions is an array of current observed conditions for the specific type of post update task.
+	// Known conditions are "ApprovalRequestCreated", "WaitTimeElapsed", and "ApprovalRequestApproved".
+	// +kubebuilder:validation:Optional
+	Conditions []metav1.Condition `json:"conditions"`


Fleet RP also have the wait time step and use the same state data model.

circy9 · 2024-09-10T17:54:57Z

apis/placement/v1alpha1/stagedupdate_types.go

+//   - `ParentStagedUpdateRun` which points to its owner stage update.
+//   - `TargetStage` which is the name of the stage that this approval request is for.
+//   - `IsLatestUpdateRunApproval` which indicates whether this approval request is the latest one related to this update run.
+type StageApprovalRequest struct {


I would recommend to use ApprovalRequest instead of StageApprovalRequest.

Add TargetStep in the spec. A step can be a stage, a group or a cluster.

StepReference { RunName StageName (future) GroupName (future) ClusterName }

apis/placement/v1alpha1/stagedupdate_types.go

ryanzhang-oss marked this pull request as draft August 5, 2024 06:22

ryanzhang-oss force-pushed the rollout-poc branch from d65db0e to 195a9b0 Compare August 5, 2024 06:29

michaelawyu reviewed Aug 6, 2024

View reviewed changes

ryanzhang-oss commented Aug 6, 2024

View reviewed changes

ryanzhang-oss marked this pull request as ready for review August 14, 2024 18:09

ryanzhang-oss force-pushed the rollout-poc branch from 5cc24ae to 7fd558b Compare August 15, 2024 01:36

ryanzhang-oss changed the title ~~feat: rollout API~~ feat: add rollout API spec Aug 15, 2024

zhiying-lin reviewed Aug 15, 2024

View reviewed changes

ryanzhang-oss commented Aug 16, 2024

View reviewed changes

apis/placement/v1alpha1/stagerollout_types.go Outdated Show resolved Hide resolved

ryanzhang-oss commented Aug 19, 2024

View reviewed changes

apis/placement/v1alpha1/stagerollout_types.go Outdated Show resolved Hide resolved

apis/placement/v1alpha1/stagerollout_types.go Outdated Show resolved Hide resolved

ryanzhang-oss force-pushed the rollout-poc branch from 33331bd to 21783bf Compare August 19, 2024 22:37

circy9 reviewed Aug 21, 2024

View reviewed changes

ryanzhang-oss force-pushed the rollout-poc branch from 21783bf to 228f754 Compare August 22, 2024 05:38

circy9 reviewed Aug 22, 2024

View reviewed changes

circy9 reviewed Aug 23, 2024

View reviewed changes

apis/placement/v1alpha1/stagedupdate_types.go Show resolved Hide resolved

ryanzhang-oss commented Aug 23, 2024

View reviewed changes

zhiying-lin reviewed Aug 26, 2024

View reviewed changes

circy9 reviewed Sep 3, 2024

View reviewed changes

circy9 reviewed Sep 6, 2024

View reviewed changes

jwtty reviewed Sep 6, 2024

View reviewed changes

apis/placement/v1alpha1/stagedupdate_types.go Outdated Show resolved Hide resolved

jwtty reviewed Sep 6, 2024

View reviewed changes

apis/placement/v1alpha1/stagedupdate_types.go Outdated Show resolved Hide resolved

zhiying-lin reviewed Sep 6, 2024

View reviewed changes

jwtty reviewed Sep 6, 2024

View reviewed changes

zhiying-lin reviewed Sep 6, 2024

View reviewed changes

apis/placement/v1alpha1/stagedupdate_types.go Outdated Show resolved Hide resolved

zhiying-lin reviewed Sep 9, 2024

View reviewed changes

ryanzhang-oss force-pushed the rollout-poc branch from dc4e9b0 to 2563c04 Compare September 9, 2024 18:27

circy9 reviewed Sep 9, 2024

View reviewed changes

circy9 reviewed Sep 10, 2024

View reviewed changes

zhiying-lin reviewed Sep 10, 2024

View reviewed changes

circy9 reviewed Sep 10, 2024

View reviewed changes

apis/placement/v1alpha1/stagedupdate_types.go Outdated Show resolved Hide resolved

update run api

0c9519e

ryanzhang-oss force-pushed the rollout-poc branch from b4e840f to 0c9519e Compare September 10, 2024 20:43

replace status with conditions

29e5877

circy9 reviewed Sep 12, 2024

View reviewed changes

fix start end time

a1e2061

circy9 previously approved these changes Sep 13, 2024

View reviewed changes

apis/placement/v1alpha1/stagedupdate_types.go Outdated Show resolved Hide resolved

ryanzhang-oss dismissed circy9’s stale review via e718378 September 13, 2024 20:05

improve the api comments

cec70f3

ryanzhang-oss force-pushed the rollout-poc branch from e718378 to cec70f3 Compare September 13, 2024 20:44

circy9 approved these changes Sep 13, 2024

View reviewed changes

ryanzhang-oss merged commit 19ed98c into Azure:main Sep 13, 2024
12 checks passed

feat: add rollout API spec #897

feat: add rollout API spec #897

Conversation

ryanzhang-oss commented Aug 5, 2024 • edited Loading

Description of your changes

How has this code been tested

Special notes for your reviewer

michaelawyu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanzhang-oss Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jwtty Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanzhang-oss commented Aug 5, 2024 •

edited

Loading

ryanzhang-oss Sep 6, 2024 •

edited

Loading

jwtty Sep 6, 2024 •

edited

Loading