-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add note for alter shared source #132
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,7 +10,7 @@ ALTER SOURCE current_source_name | |
alter_option; | ||
``` | ||
|
||
_`alteroption`_ depends on the operation you want to perform on the source. For all supported clauses, see the sections below. | ||
_`alter_option`_ depends on the operation you want to perform on the source. For all supported clauses, see the sections below. | ||
|
||
## Clause | ||
|
||
|
@@ -34,11 +34,46 @@ ALTER SOURCE src1 | |
``` | ||
|
||
<Note> | ||
* To alter columns in a source created with a schema registry, see [FORMAT and ENCODE options](/sql/commands/sql-alter-source#format-and-encode-options). | ||
* To alter columns in a source created with a schema registry, see [REFRESH SCHEMA](/sql/commands/sql-alter-source#refresh-schema). | ||
* You cannot add a primary key column to a source or table in RisingWave. To modify the primary key of a source or table, you need to recreate the table. | ||
* You cannot remove a column from a source in RisingWave. If you intend to remove a column from a source, you'll need to drop the source and create the source again. | ||
</Note> | ||
|
||
|
||
### `REFRESH SCHEMA` | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Moved up, since I think putting |
||
|
||
Fetch the latest schema from the schema registry and update the source schema. | ||
|
||
```sql | ||
ALTER SOURCE source_name REFRESH SCHEMA; | ||
``` | ||
|
||
<Note> | ||
Currently when refreshing the schema registry of a source, it is not allowed to drop columns or change types. | ||
</Note> | ||
|
||
|
||
For example, assume we have a source as follows: | ||
|
||
```sql Create a source | ||
CREATE SOURCE src_user WITH ( | ||
connector = 'kafka', | ||
topic = 'sr_pb_test', | ||
properties.bootstrap.server = 'message_queue:29092', | ||
scan.startup.mode = 'earliest' | ||
) | ||
FORMAT PLAIN ENCODE PROTOBUF( | ||
schema.registry = 'http://message_queue:8081', | ||
message = 'test.User' | ||
); | ||
``` | ||
|
||
Then we can refresh its schema with the following statement: | ||
|
||
```sql Refresh schema | ||
ALTER SOURCE src_user REFRESH SCHEMA; | ||
``` | ||
|
||
### `RENAME TO` | ||
|
||
```sql | ||
|
@@ -91,77 +126,6 @@ ALTER SOURCE current_source_name | |
ALTER SOURCE test_source SET SCHEMA test_schema; | ||
``` | ||
|
||
### `FORMAT and ENCODE options` | ||
|
||
At present, combined with the `ALTER SOURCE` command, you can refresh the schema registry of a source by refilling the FORMAT and ENCODE options. For more details about these options, see [FORMAT and ENCODE parameters](/ingestion/format-and-encode-parameters). | ||
|
||
Comment on lines
-94
to
-97
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Although cc @xiangjinwu There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agree on hiding this. |
||
```sql | ||
ALTER SOURCE source_name FORMAT data_format ENCODE data_encode [ ( | ||
message='message', | ||
schema.location='location', ...) ]; | ||
``` | ||
|
||
Here is an example. Let's assume the original FORMAT and ENCODE options are as follows: | ||
|
||
```sql | ||
-- Create a source. | ||
CREATE SOURCE src_user WITH ( | ||
connector = 'kafka', | ||
topic = 'sr_pb_test', | ||
properties.bootstrap.server = 'message_queue:29092', | ||
scan.startup.mode = 'earliest' | ||
) | ||
FORMAT PLAIN ENCODE PROTOBUF( | ||
schema.registry = 'http://message_queue:8081', | ||
message = 'test.User'); | ||
``` | ||
|
||
Then you can refresh the schema registry by the following command: | ||
|
||
```sql | ||
ALTER SOURCE src_user FORMAT PLAIN ENCODE PROTOBUF( | ||
schema.registry = 'http://message_queue:8081', | ||
message = 'test.UserWithMoreFields' | ||
); | ||
``` | ||
|
||
<Note> | ||
Currently, it is not supported to modify the `data_format` and `data_encode`. Furthermore, when refreshing the schema registry of a source, it is not allowed to drop columns or change types. | ||
|
||
Another way of refreshing the schema is using the [REFRESH SCHEMA clause](#refresh-schema). | ||
</Note> | ||
|
||
### `REFRESH SCHEMA` | ||
|
||
This is another way of refreshing the schema of sources when the [FORMAT and ENCODE options](#format-and-encode-options) are not changed. | ||
|
||
```sql | ||
ALTER SOURCE source_name REFRESH SCHEMA; | ||
``` | ||
|
||
For example, assume we have a source as follows: | ||
|
||
```sql Create a source | ||
CREATE SOURCE src_user WITH ( | ||
connector = 'kafka', | ||
topic = 'sr_pb_test', | ||
properties.bootstrap.server = 'message_queue:29092', | ||
scan.startup.mode = 'earliest' | ||
) | ||
FORMAT PLAIN ENCODE PROTOBUF( | ||
schema.registry = 'http://message_queue:8081', | ||
message = 'test.User' | ||
); | ||
``` | ||
|
||
Then we can refresh its schema with the following statement: | ||
|
||
```sql Refresh schema | ||
ALTER SOURCE src_user REFRESH SCHEMA; | ||
``` | ||
|
||
For more details about this example, see our [test file](https://github.com/risingwavelabs/risingwave/blob/994a2831088c9befc71721ed6f2f2d2e35c4d0a9/e2e%5Ftest/schema%5Fregistry/alter%5Fsr.slt). | ||
|
||
### `SET SOURCE_RATE_LIMIT` | ||
|
||
```sql | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -98,6 +98,12 @@ Shared source improves resource utilization and data consistency when working wi | |
This feature is currently in public preview, meaning it is nearing the final product but may not yet be fully stable. If you encounter any issues or have feedback, please reach out to us via our [Slack channel](https://www.risingwave.com/slack). Your input is valuable in helping us improve this feature. For more details, see our [Public Preview Feature List](/changelog/product-lifecycle#features-in-the-public-preview-stage). | ||
</Note> | ||
|
||
<Note> | ||
Shared Kafka source is available since version 2.1. Other sources are unaffected. We plan to gradually upgrade other sources to the be shared as well in the future. | ||
|
||
`ALTER SOURCE [ADD COLUMN | REFRESH SCHEMA]` for shared source is available since version 2.2. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. New feature added in 2.2. Add note here |
||
</Note> | ||
|
||
### Configure | ||
|
||
Shared source is enabled by default. You can also set the session variable `streaming_use_shared_source` to control whether to enable it. | ||
|
@@ -160,14 +166,6 @@ Tables offer other features that enhance their utility in data ingestion workflo | |
<img src="/images/table-with-connectors.png"/> | ||
</Frame> | ||
|
||
<Note> | ||
**LIMITATION** | ||
|
||
Currently, shared source is only applicable to Kafka sources. Other sources are unaffected. We plan to gradually upgrade other sources to the be shared as well in the future. | ||
|
||
Shared sources do not support `ALTER SOURCE`. Use non-shared sources if you require this functionality. | ||
</Note> | ||
|
||
## See also | ||
|
||
<CardGroup> | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think it's a good practice to refer to test file in doc, so removed.