fix(connector): file scan use correct path #19793
Conversation
The reviewer is automatically invited, will completed pr description after modifying Cargo.lock. |
wcy-fdu
force-pushed
the
wcy/fix_file_scan.pr
branch
from
d7bef25 to
a65ff03
Compare
|
Tess passed. |
…nto wcy/fix_file_scan.pr
| .iter() | ||
| .filter_map(|column| { | ||
| parquet_column_names | ||
| .iter() | ||
| .position(|&name| name == column.name) | ||
| .and_then(|pos| { | ||
| let arrow_data_type: &risingwave_common::array::arrow::arrow_schema_udf::DataType = converted_arrow_schema.field(pos).data_type(); | ||
| let rw_data_type: &risingwave_common::types::DataType = &column.data_type; | ||
|
|
||
| if is_parquet_schema_match_source_schema(arrow_data_type, rw_data_type) { | ||
| Some(pos) | ||
| } else { | ||
| None | ||
| } | ||
| }) | ||
| }) | ||
| .collect(); | ||
| Ok(valid_column_indices) | ||
| } | ||
| None => Ok(vec![]), | ||
| } | ||
| if is_parquet_schema_match_source_schema(arrow_data_type, rw_data_type) { | ||
| Some(pos) | ||
| } else { | ||
| None | ||
| } | ||
| }) | ||
| }) | ||
| .collect(); |
There was a problem hiding this comment.
Strange, how did these lines escape cargo fmt escape. I think it's already formatted.
I tried to add some test in CI, but found that currently TVF file scan only support reading file in s3, but not support reading file in minio, see definition of endpoint, it is hard-coded into s3 instead of letting users pass it. I think we can add more backend such as and then enhance ci test. This will take some time but this fix is a bit urgent, so I just test it locally and leave these in subsequent PRs. |
github-merge-queue
bot
removed this pull request from the merge queue due to no response for status checks
…anch release-2.1 (#19833)
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
#19398 bring a bug that file_scan use incorrect path(With s3://bucket_name/ as prefix), so this pr fix this bug.
Checklist
Documentation
Release note