Tracking: schema changes of source | (table with connector)

[Rossil2012]

A relay for #11800.

• ALTER TABLE <name> FORMAT ... ENCODE ... ( ... )
• ALTER SOURCE <name> FORMAT ... ENCODE ... ( ... ): feat: implement ALTER SOURCE xx FORMAT xx ENCODE xx (...) #14057
• ALTER TABLE <name> REFRESH SCHEMA: feat: alter the schema registry of table with connector #15025
• ALTER SOURCE <name> REFRESH SCHEMA: feat: refresh the schema of sources with schema registry #15541

Note: A minor difference between ALTER SOURCE <name> REFRESH SCHEMA and ALTER TABLE <name> REFRESH SCHEMA is that the former rejects if columns are missing or changed, while the latter do this check only when the column is referenced by downstream tasks.

[Rossil2012]

Possible solution: using the semantics whether the source executor is reset.
#15025 (comment)

[Rossil2012]

cc @tabVersion @fuyufjh @st1page.

[xiangjinwu]

Panic: #16486

[fuyufjh]

Related discussion about watermark risingwavelabs/rfcs#85 (comment)

[fuyufjh]

Note down a requirement for ALTER SOURCE CONNECTOR today: without this, changing a single argument will need to recreate the source/table and re-consume the upstream.

Slack thread

[yuhao-su]

We probably also need the same for alter sink

[st1page]

We probably also need the same for alter sink

Can you explain more about it? Should the sink re-send all the historical data or just send it with the current progress?

[st1page]

Changed the pr description and will create a new PR for "alter connector" to make the issue clearer

[fuyufjh]

Note down a reminder: alter a secret that referenced by a connector is also kind of alter source i.e. it should be refreshed without rebuilding the SourceExecutor.

[fuyufjh]

We divided the cases into 2 categories: No Rebuild and Rebuild.

1. If a property change doesn’t need to rebuild the connector instance or state, then it can be done in a lightweight fashion, such as mutate it with a single barrier
2. If a property change need to rebuild the connector or state, such as change the Kafka consumer topic or change the connector type e.g. from File to Kafka, then we clean the previous state, drop the old actors and create new ones.

[xxchan]

For rebuild, there's another case: alter schema

	alter small param	alter connector	alter schema
non-shared source	no rebuild	rebuild (clean state)	no rebuild ⚠️ (We only have this now)
shared source	no rebuild	rebuild (clean state)	rebuild (keep state)
materialized source (table)	no rebuild	rebuild (clean state)	rebuild (keep state)

(More discussion on https://risingwave-labs.slack.com/archives/C069EV2Q5MK/p1726639203021329

[fuyufjh]

Just found a new issue addressing this. #18440 Let's move there for discussion.

[tabVersion]

close this one for duplicate with #18440