feat(iceberg): support iceberg engine table (in local env)

[chenzl25]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

• Tracking: Iceberg engine table #19418
• Support ./risedev d iceberg-engine to set up a local environment to run the iceberg engine table. Use ./risedev d full directly.
• Support create/drop/select iceberg engine table. The ddl in this PR is not atomic, but it should be fine in the first version.
• The iceberg catalog is stored in our sql meta backend and the data is stored in S3 compatible cloud storage.
• This PR will retrieve meta backend connection info and S3 warehouse info from the environment variable for simplicity, but we will improve it by fetching this info from meta in a later PR.
• Make the iceberg sink s3 access key and secret key optional and add an enable_config_load field to load credentials from the default credential provided chain.
• No compaction in this PR.

Example

create table t(id int primary key, name varchar) engine = iceberg;
insert into t values(1, 'xxx');
select * from t;

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

[fuyufjh]

Yeah, this is a critical issue... especially if we create tables first, before source and sink. This is because table is self-contained, while create source employs a validation stage to check whether the upstream system really work, so it has a high chance to fail. Shall we create the source/sink before table?

[chenzl25]

There are some dependencies here. To create an iceberg source, we first need to have an iceberg table. To create an iceberg table we need to create an iceberg sink (with create_table_if_not_exists). To create an iceberg sink we need to create a hummock table first. Finally, we have this order hummock table -> iceberg sink -> iceberg source

[fuyufjh]

Here the source is passed as well, can it work? 🤔

What I am thinking is that, for a common table with connector, the corresponding source is supposed to generate changes that will be applied to the table. But here the Iceberg source is just for batch read, which I think is actually irrelevant/unconnected to the iceberg table internally.

[chenzl25]

The source here is something like kafka and postgres cdc connector instead of the iceberg source. For example

create table t (a int) with (connector = 'kafka' ...) engine = iceberg.

[xiangjinwu]

for Cargo.lock

[BugenZhao]

MetaBackend::Sql will be widely adopted since #19560. Shall we support it as well?

[chenzl25]

Oh, I previously thought it was deprecated. For iceberg jdbc right now, we need to know the underlying database implementation to choose the right driver.

[BugenZhao]

I think we can simply add a jdbc: prefix to the database URL? 🤣

[yezizp2012]

When getting the endpoint from meta, it has already been converted into a form with postgres:, mysql: as prefixes, including configure as MetaBackend::Sql. So MetaBackend::Sql should be unreachable.

Unrelated to this PR, 🤔 I think we'd better use sql config in risedev only for testing purpose. To support scenarios where user and password contain special characters, it's best to specify them separately through env in both production environment and cloud. That's the reason why a subdivided backend was introduced in #17530 .

[chenzl25]

If the underlying database is Oracle or SQL Server, I think that's acceptable. However, I still want to verify the underlying database type. For instance, SQLite is not a suitable catalog for Iceberg. Concurrent updates to SQLite by both the metadata service and Iceberg can easily cause SQLite to become unresponsive.