diff --git a/ingestion/modify-source-or-table-schemas.mdx b/ingestion/modify-source-or-table-schemas.mdx index a7e2256d..5751e34e 100644 --- a/ingestion/modify-source-or-table-schemas.mdx +++ b/ingestion/modify-source-or-table-schemas.mdx @@ -126,8 +126,6 @@ Refresh schema ALTER SOURCE src_user REFRESH SCHEMA; ``` -For more details about this example, see our [test file](https://github.com/risingwavelabs/risingwave/blob/994a2831088c9befc71721ed6f2f2d2e35c4d0a9/e2e%5Ftest/schema%5Fregistry/alter%5Fsr.slt). - ### Table Similarly, you can use the following statement to refresh the schema of a table with connectors. For more details, see [ALTER TABLE](/sql/commands/sql-alter-table#refresh-schema). diff --git a/sql/commands/sql-alter-connection.mdx b/sql/commands/sql-alter-connection.mdx index e2ea0077..4afbf374 100644 --- a/sql/commands/sql-alter-connection.mdx +++ b/sql/commands/sql-alter-connection.mdx @@ -10,7 +10,7 @@ ALTER CONNECTION connection_name alter_option; ``` -_`alteroption`_ depends on the operation you want to perform on the connection. For all supported clauses, see the sections below. +_`alter_option`_ depends on the operation you want to perform on the connection. For all supported clauses, see the sections below. ## Clause diff --git a/sql/commands/sql-alter-database.mdx b/sql/commands/sql-alter-database.mdx index 975ed039..4b0747f9 100644 --- a/sql/commands/sql-alter-database.mdx +++ b/sql/commands/sql-alter-database.mdx @@ -10,7 +10,7 @@ ALTER DATABASE database_name alter_option; ``` -_`alteroption`_ depends on the operation you want to perform on the database. For all supported clauses, see the sections below. +_`alter_option`_ depends on the operation you want to perform on the database. For all supported clauses, see the sections below. ## Clause ### `OWNER TO` diff --git a/sql/commands/sql-alter-function.mdx b/sql/commands/sql-alter-function.mdx index 42d1bc13..3bc6b285 100644 --- a/sql/commands/sql-alter-function.mdx +++ b/sql/commands/sql-alter-function.mdx @@ -10,7 +10,7 @@ ALTER FUNCTION function( argument_type [, ...] ) alter_option; ``` -_`alteroption`_ depends on the operation you want to perform on the function. For all supported clauses, see the sections below. +_`alter_option`_ depends on the operation you want to perform on the function. For all supported clauses, see the sections below. ## Clause diff --git a/sql/commands/sql-alter-index.mdx b/sql/commands/sql-alter-index.mdx index 8ca8aa1e..eb60add6 100644 --- a/sql/commands/sql-alter-index.mdx +++ b/sql/commands/sql-alter-index.mdx @@ -10,7 +10,7 @@ ALTER INDEX index_name alter_option; ``` -_`alteroption`_ depends on the operation you want to perform on the index. For all supported clauses, see the sections below. +_`alter_option`_ depends on the operation you want to perform on the index. For all supported clauses, see the sections below. ## Clause diff --git a/sql/commands/sql-alter-materialized-view.mdx b/sql/commands/sql-alter-materialized-view.mdx index 356c46dc..a5d6bdbe 100644 --- a/sql/commands/sql-alter-materialized-view.mdx +++ b/sql/commands/sql-alter-materialized-view.mdx @@ -16,7 +16,7 @@ ALTER MATERIALIZED VIEW materialized_view_name alter_option; ``` -_`alteroption`_ depends on the operation you want to perform on the materialized view. For all supported clauses, see the sections below. +_`alter_option`_ depends on the operation you want to perform on the materialized view. For all supported clauses, see the sections below. ## Clause diff --git a/sql/commands/sql-alter-schema.mdx b/sql/commands/sql-alter-schema.mdx index 410568bb..5aa2e402 100644 --- a/sql/commands/sql-alter-schema.mdx +++ b/sql/commands/sql-alter-schema.mdx @@ -10,7 +10,7 @@ ALTER SCHEMA current_schema_name alter_option; ``` -_`alteroption`_ depends on the operation you want to perform on the schema. For all supported clauses, see the sections below. +_`alter_option`_ depends on the operation you want to perform on the schema. For all supported clauses, see the sections below. ## Clause diff --git a/sql/commands/sql-alter-sink.mdx b/sql/commands/sql-alter-sink.mdx index 5bdbcef5..8a17db94 100644 --- a/sql/commands/sql-alter-sink.mdx +++ b/sql/commands/sql-alter-sink.mdx @@ -16,7 +16,7 @@ ALTER SINK sink_name alter_option; ``` -_`alteroption`_ depends on the operation you want to perform on the sink. For all supported clauses, see the sections below. +_`alter_option`_ depends on the operation you want to perform on the sink. For all supported clauses, see the sections below. ## Clause diff --git a/sql/commands/sql-alter-source.mdx b/sql/commands/sql-alter-source.mdx index 91d5c47e..2acf6055 100644 --- a/sql/commands/sql-alter-source.mdx +++ b/sql/commands/sql-alter-source.mdx @@ -10,7 +10,7 @@ ALTER SOURCE current_source_name alter_option; ``` -_`alteroption`_ depends on the operation you want to perform on the source. For all supported clauses, see the sections below. +_`alter_option`_ depends on the operation you want to perform on the source. For all supported clauses, see the sections below. ## Clause @@ -34,11 +34,46 @@ ALTER SOURCE src1 ``` -* To alter columns in a source created with a schema registry, see [FORMAT and ENCODE options](/sql/commands/sql-alter-source#format-and-encode-options). +* To alter columns in a source created with a schema registry, see [REFRESH SCHEMA](/sql/commands/sql-alter-source#refresh-schema). * You cannot add a primary key column to a source or table in RisingWave. To modify the primary key of a source or table, you need to recreate the table. * You cannot remove a column from a source in RisingWave. If you intend to remove a column from a source, you'll need to drop the source and create the source again. + +### `REFRESH SCHEMA` + +Fetch the latest schema from the schema registry and update the source schema. + +```sql +ALTER SOURCE source_name REFRESH SCHEMA; +``` + + +Currently when refreshing the schema registry of a source, it is not allowed to drop columns or change types. + + + +For example, assume we have a source as follows: + +```sql Create a source +CREATE SOURCE src_user WITH ( + connector = 'kafka', + topic = 'sr_pb_test', + properties.bootstrap.server = 'message_queue:29092', + scan.startup.mode = 'earliest' +) +FORMAT PLAIN ENCODE PROTOBUF( + schema.registry = 'http://message_queue:8081', + message = 'test.User' +); +``` + +Then we can refresh its schema with the following statement: + +```sql Refresh schema +ALTER SOURCE src_user REFRESH SCHEMA; +``` + ### `RENAME TO` ```sql @@ -91,77 +126,6 @@ ALTER SOURCE current_source_name ALTER SOURCE test_source SET SCHEMA test_schema; ``` -### `FORMAT and ENCODE options` - -At present, combined with the `ALTER SOURCE` command, you can refresh the schema registry of a source by refilling the FORMAT and ENCODE options. For more details about these options, see [FORMAT and ENCODE parameters](/ingestion/format-and-encode-parameters). - -```sql -ALTER SOURCE source_name FORMAT data_format ENCODE data_encode [ ( - message='message', - schema.location='location', ...) ]; -``` - -Here is an example. Let's assume the original FORMAT and ENCODE options are as follows: - -```sql --- Create a source. -CREATE SOURCE src_user WITH ( - connector = 'kafka', - topic = 'sr_pb_test', - properties.bootstrap.server = 'message_queue:29092', - scan.startup.mode = 'earliest' -) -FORMAT PLAIN ENCODE PROTOBUF( - schema.registry = 'http://message_queue:8081', - message = 'test.User'); -``` - -Then you can refresh the schema registry by the following command: - -```sql -ALTER SOURCE src_user FORMAT PLAIN ENCODE PROTOBUF( - schema.registry = 'http://message_queue:8081', - message = 'test.UserWithMoreFields' -); -``` - - -Currently, it is not supported to modify the `data_format` and `data_encode`. Furthermore, when refreshing the schema registry of a source, it is not allowed to drop columns or change types. - -Another way of refreshing the schema is using the [REFRESH SCHEMA clause](#refresh-schema). - - -### `REFRESH SCHEMA` - -This is another way of refreshing the schema of sources when the [FORMAT and ENCODE options](#format-and-encode-options) are not changed. - -```sql -ALTER SOURCE source_name REFRESH SCHEMA; -``` - -For example, assume we have a source as follows: - -```sql Create a source -CREATE SOURCE src_user WITH ( - connector = 'kafka', - topic = 'sr_pb_test', - properties.bootstrap.server = 'message_queue:29092', - scan.startup.mode = 'earliest' -) -FORMAT PLAIN ENCODE PROTOBUF( - schema.registry = 'http://message_queue:8081', - message = 'test.User' -); -``` - -Then we can refresh its schema with the following statement: - -```sql Refresh schema -ALTER SOURCE src_user REFRESH SCHEMA; -``` - -For more details about this example, see our [test file](https://github.com/risingwavelabs/risingwave/blob/994a2831088c9befc71721ed6f2f2d2e35c4d0a9/e2e%5Ftest/schema%5Fregistry/alter%5Fsr.slt). - ### `SET SOURCE_RATE_LIMIT` ```sql diff --git a/sql/commands/sql-create-source.mdx b/sql/commands/sql-create-source.mdx index 32e2f96c..022fe8b5 100644 --- a/sql/commands/sql-create-source.mdx +++ b/sql/commands/sql-create-source.mdx @@ -98,6 +98,12 @@ Shared source improves resource utilization and data consistency when working wi This feature is currently in public preview, meaning it is nearing the final product but may not yet be fully stable. If you encounter any issues or have feedback, please reach out to us via our [Slack channel](https://www.risingwave.com/slack). Your input is valuable in helping us improve this feature. For more details, see our [Public Preview Feature List](/changelog/product-lifecycle#features-in-the-public-preview-stage). + +Shared Kafka source is available since version 2.1. Other sources are unaffected. We plan to gradually upgrade other sources to the be shared as well in the future. + +`ALTER SOURCE [ADD COLUMN | REFRESH SCHEMA]` for shared source is available since version 2.2. + + ### Configure Shared source is enabled by default. You can also set the session variable `streaming_use_shared_source` to control whether to enable it. @@ -160,14 +166,6 @@ Tables offer other features that enhance their utility in data ingestion workflo - -**LIMITATION** - -Currently, shared source is only applicable to Kafka sources. Other sources are unaffected. We plan to gradually upgrade other sources to the be shared as well in the future. - -Shared sources do not support `ALTER SOURCE`. Use non-shared sources if you require this functionality. - - ## See also