Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking (storage): support state cleaning on non-pk prefix watermark column #18802

Open
hzxa21 opened this issue Oct 8, 2024 · 2 comments
Open
Assignees
Milestone

Comments

@hzxa21
Copy link
Collaborator

hzxa21 commented Oct 8, 2024

We decided to adopt option 3 mentioned here

Copy link
Contributor

github-actions bot commented Dec 8, 2024

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean.
Don't worry if you think the issue is still valuable to continue in the future.
It's searchable and can be reopened when it's time. 😄

@Li0k
Copy link
Contributor

Li0k commented Dec 20, 2024

Disscuss wtih @hzxa21 and @st1page, i found some assumptions exist for non_pk_watermark

  1. the length of the watermark col is 1.
  2. the state table that needs to be cleaned, the watermrak_col exists in pk.
  3. the operator guarantees not to access data below the watermark, storage does not need to give this guarantee.

Therefore, the design on the storage side is as follows:

  1. state table holds watermark_col
  2. serialise watermark based on watermark_col instead of prefix(1)
  3. when writing to TableWaterMark, record that it is a non_pk_watermark
  4. modify the skip_watermark_iter implementation so that non_pk_watermark needs to solve for watermark_col before comparing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants