Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Spark][Version Checksum] Track histogram of file sizes in the checksum #3907

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

dhruvarya-db
Copy link
Collaborator

@dhruvarya-db dhruvarya-db commented Nov 27, 2024

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

Follow up for #3899.
Major changes:

  1. Makes it so that histograms of file sizes are tracked in the version checksum --- these are later used for validation. These are computed incrementally to avoid state reconstruction.
  2. Also, now that Version Checksum can be incrementally computed without triggering a full state reconstruction (see [Spark][Version Checksum] Incrementally compute allFiles in the checksum #3899, [Spark][Version Checksum] Incrementally compute VersionChecksum setTransactions and domainMetadata #3895), this PR enables writing the version checksum by default.

How was this patch tested?

Added FileSizeHistogramSuite and updated some existing suites.

Does this PR introduce any user-facing changes?

No

@dhruvarya-db dhruvarya-db force-pushed the checksum-incremental-filesizehistogram branch 7 times, most recently from cb14903 to 0d22f8f Compare December 5, 2024 17:07
vkorukanti pushed a commit that referenced this pull request Dec 6, 2024
…rom the Checksum during Snapshot construction (#3920)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->
Stacked over #3907. 
This PR makes the Checksum (if available) the source of truth for
Protocol, Metadata, ICT during snapshot construction. This helps us
avoid a Spark query and improves performance.

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->
Added some test cases to existing suites

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No
@dhruvarya-db dhruvarya-db force-pushed the checksum-incremental-filesizehistogram branch 3 times, most recently from 5216639 to 5d522ac Compare December 6, 2024 14:12
@dhruvarya-db dhruvarya-db force-pushed the checksum-incremental-filesizehistogram branch from 5d522ac to a25ab08 Compare December 12, 2024 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants