Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(deps): Bump apache-avro to 0.16.0 #13523

Merged
merged 3 commits into from
Nov 20, 2023
Merged

Conversation

xiangjinwu
Copy link
Contributor

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Before the bump, our fork was based on main branch on 2023-03-21.

After the bump, the fork is based on release-1.11.3 (crates.io 0.16.0).

Major API change: Schema::Record and Schema::Decimal are wrapped in RecordSchema and DecimalSchema.

Note this does not fully resolve #13505 (comment) yet.

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added test labels as necessary. See details.
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
  • All checks passed in ./risedev check (or alias, ./risedev c)
  • My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
  • My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

@xiangjinwu xiangjinwu requested a review from a team as a code owner November 20, 2023 10:23
@@ -15,7 +15,7 @@ normal = ["workspace-hack"]

[dependencies]
anyhow = "1"
apache-avro = { git = "https://github.com/risingwavelabs/avro", rev = "3c257c1f65e63a80ac44f1cffaf083cdaeab431a", features = [
apache-avro = { git = "https://github.com/risingwavelabs/avro", rev = "b251943586e7fe7dd52319fc84f8be47921434aa", features = [
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -204,6 +204,7 @@ mod tests {
let inner_shema_str = r#"{
"type": "record",
"name": "Value",
"namespace": "dbserver1.inventory.customers",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is as reasonable as select v from t being equivalent to select t.v from t. But I have not checked how we are handling namespaces in other places.

@@ -34,20 +34,19 @@
{
"name": "entrance_date",
"type": "int",
"logicalType": "date",
"default": null
"logicalType": "date"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default value for date field in a struct will be added back when we fully resolve the linked issue.

Copy link
Contributor

@StrikeW StrikeW left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, may let experts to have a look

Copy link
Collaborator

@hzxa21 hzxa21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

"thiserror",
"typed-builder",
"typed-builder 0.14.0",
"uuid",
"zerocopy",
]

[[package]]
name = "apache-avro"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have two apache-avro dependencies? Is this expected?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One is our fork used directly, another is indirect dependency introduced by icelake. This has been the case before this PR, and is expected. We also have multiple versions of some other crates.

Depending on two incompatible versions of the same crate does have some downsides:

  • Doubles building time, binary size as well as memory.
  • Types defined in one version cannot be passed as arguments to functions defined in another version.

Thus it is better avoided, but not that rare in reality.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More discussion on having 2 (and even possibly 3) versions of arrow (not avro): #13600 (comment)

@xiangjinwu xiangjinwu marked this pull request as draft November 20, 2023 13:28
@xiangjinwu

This comment was marked as resolved.

@xiangjinwu xiangjinwu marked this pull request as ready for review November 20, 2023 14:42
Copy link

codecov bot commented Nov 20, 2023

Codecov Report

Attention: 6 lines in your changes are missing coverage. Please review.

Comparison is base (16e4d2a) 68.39% compared to head (42cc97f) 68.46%.
Report is 8 commits behind head on main.

Files Patch % Lines
src/connector/src/parser/avro/parser.rs 55.55% 4 Missing ⚠️
src/connector/src/parser/avro/util.rs 33.33% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #13523      +/-   ##
==========================================
+ Coverage   68.39%   68.46%   +0.06%     
==========================================
  Files        1508     1508              
  Lines      259655   259657       +2     
==========================================
+ Hits       177598   177776     +178     
+ Misses      82057    81881     -176     
Flag Coverage Δ
rust 68.46% <68.42%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@xiangjinwu xiangjinwu added this pull request to the merge queue Nov 20, 2023
Merged via the queue into main with commit 8e5477a Nov 20, 2023
27 of 28 checks passed
@xiangjinwu xiangjinwu deleted the chore-deps-avro-bump-0.16.0 branch November 20, 2023 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug: avro schema with default value for date cannot be parsed
3 participants