feat(connector): add minio file scan type and enhance test #19950
Conversation
…nto wcy/file_scan_more_backend.pr
| .secret_access_key(&s3_secret_key) | ||
| .bucket(&bucket); | ||
| builder = match is_minio { | ||
| true => builder.endpoint(&format!("http://{}.127.0.0.1:9301", bucket)), |
There was a problem hiding this comment.
Should we allow users to provide s3 endpoint instead of hardcoding the endpoint?
There was a problem hiding this comment.
I think minio is only for test, so I hardcode it. IMO, we don't public file_scan(minio) to user, what do you think?
There was a problem hiding this comment.
Should we allow users to provide s3 endpoint instead of hardcoding the endpoint?
And this will require another parameter in file scan, which is a bit redudant. I prefer to provide users with only the file scan backend that is consistent with Cloud(s3, gcs, azblob).
There was a problem hiding this comment.
Can we reuse the s3 region parameter position to allow users to set the endpoint as well, i.e. (region or endpoint)?
There was a problem hiding this comment.
region is a required field. If we change region to endpoint, the user's endpoint must include region. Is this a bit strange?
There was a problem hiding this comment.
When testing minio, the region is custom, the endpoint is http://bucket/127.0.0.1:9301, they have no intersection.
There was a problem hiding this comment.
If an endpoint is specified, the region field is useless (Correct?). I mean file_scan('parquet', 's3', region_or_endpoint, ...) the region_or_endpoint parameter could be region or endpoint, we can detect it simply by http prefix.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Previously,
file_scanonly supported reading Parquet files on S3 and had no test coverage. This PR adds support forfile_scan(parquet, minio, ...)and includes file scan test in the file source tests of the main cron job. Since MinIO is S3-compatible, it requires nearly the same parameters as S3, with the only difference being the need for a special endpoint setting when building the operator. Therefore, this PR reuses the existing file scan node.Note: File scan from MinIO is currently for testing purposes only.
Later, we will also add file_scan(parquet, gcs), and there may be a new plan node for that; we'll address that in subsequent PRs.
Checklist
Documentation
Release note