Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add note for alter shared source #132

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions ingestion/modify-source-or-table-schemas.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -126,8 +126,6 @@ Refresh schema
ALTER SOURCE src_user REFRESH SCHEMA;
```

For more details about this example, see our [test file](https://github.com/risingwavelabs/risingwave/blob/994a2831088c9befc71721ed6f2f2d2e35c4d0a9/e2e%5Ftest/schema%5Fregistry/alter%5Fsr.slt).
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's a good practice to refer to test file in doc, so removed.


### Table

Similarly, you can use the following statement to refresh the schema of a table with connectors. For more details, see [ALTER TABLE](/sql/commands/sql-alter-table#refresh-schema).
Expand Down
2 changes: 1 addition & 1 deletion sql/commands/sql-alter-connection.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ ALTER CONNECTION connection_name
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the connection. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the connection. For all supported clauses, see the sections below.

## Clause

Expand Down
2 changes: 1 addition & 1 deletion sql/commands/sql-alter-database.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ ALTER DATABASE database_name
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the database. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the database. For all supported clauses, see the sections below.

## Clause
### `OWNER TO`
Expand Down
2 changes: 1 addition & 1 deletion sql/commands/sql-alter-function.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ ALTER FUNCTION function( argument_type [, ...] )
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the function. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the function. For all supported clauses, see the sections below.

## Clause

Expand Down
2 changes: 1 addition & 1 deletion sql/commands/sql-alter-index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ ALTER INDEX index_name
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the index. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the index. For all supported clauses, see the sections below.

## Clause

Expand Down
2 changes: 1 addition & 1 deletion sql/commands/sql-alter-materialized-view.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ ALTER MATERIALIZED VIEW materialized_view_name
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the materialized view. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the materialized view. For all supported clauses, see the sections below.

## Clause

Expand Down
2 changes: 1 addition & 1 deletion sql/commands/sql-alter-schema.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ ALTER SCHEMA current_schema_name
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the schema. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the schema. For all supported clauses, see the sections below.

## Clause

Expand Down
2 changes: 1 addition & 1 deletion sql/commands/sql-alter-sink.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ ALTER SINK sink_name
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the sink. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the sink. For all supported clauses, see the sections below.

## Clause

Expand Down
110 changes: 37 additions & 73 deletions sql/commands/sql-alter-source.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ ALTER SOURCE current_source_name
alter_option;
```

_`alteroption`_ depends on the operation you want to perform on the source. For all supported clauses, see the sections below.
_`alter_option`_ depends on the operation you want to perform on the source. For all supported clauses, see the sections below.

## Clause

Expand All @@ -34,11 +34,46 @@ ALTER SOURCE src1
```

<Note>
* To alter columns in a source created with a schema registry, see [FORMAT and ENCODE options](/sql/commands/sql-alter-source#format-and-encode-options).
* To alter columns in a source created with a schema registry, see [REFRESH SCHEMA](/sql/commands/sql-alter-source#refresh-schema).
* You cannot add a primary key column to a source or table in RisingWave. To modify the primary key of a source or table, you need to recreate the table.
* You cannot remove a column from a source in RisingWave. If you intend to remove a column from a source, you'll need to drop the source and create the source again.
</Note>


### `REFRESH SCHEMA`
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved up, since I think putting ADD COLUMN and REFRESH SCHEMA together is better.


Fetch the latest schema from the schema registry and update the source schema.

```sql
ALTER SOURCE source_name REFRESH SCHEMA;
```

<Note>
Currently when refreshing the schema registry of a source, it is not allowed to drop columns or change types.
</Note>


For example, assume we have a source as follows:

```sql Create a source
CREATE SOURCE src_user WITH (
connector = 'kafka',
topic = 'sr_pb_test',
properties.bootstrap.server = 'message_queue:29092',
scan.startup.mode = 'earliest'
)
FORMAT PLAIN ENCODE PROTOBUF(
schema.registry = 'http://message_queue:8081',
message = 'test.User'
);
```

Then we can refresh its schema with the following statement:

```sql Refresh schema
ALTER SOURCE src_user REFRESH SCHEMA;
```

### `RENAME TO`

```sql
Expand Down Expand Up @@ -91,77 +126,6 @@ ALTER SOURCE current_source_name
ALTER SOURCE test_source SET SCHEMA test_schema;
```

### `FORMAT and ENCODE options`

At present, combined with the `ALTER SOURCE` command, you can refresh the schema registry of a source by refilling the FORMAT and ENCODE options. For more details about these options, see [FORMAT and ENCODE parameters](/ingestion/format-and-encode-parameters).

Comment on lines -94 to -97
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although ALTER SOURCE ... FORAMT ENCODE ... is technically supported, I doubt it's usability and should prefer REFRESH SCHEMA instead. So removed from doc.

cc @xiangjinwu

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree on hiding this.

```sql
ALTER SOURCE source_name FORMAT data_format ENCODE data_encode [ (
message='message',
schema.location='location', ...) ];
```

Here is an example. Let's assume the original FORMAT and ENCODE options are as follows:

```sql
-- Create a source.
CREATE SOURCE src_user WITH (
connector = 'kafka',
topic = 'sr_pb_test',
properties.bootstrap.server = 'message_queue:29092',
scan.startup.mode = 'earliest'
)
FORMAT PLAIN ENCODE PROTOBUF(
schema.registry = 'http://message_queue:8081',
message = 'test.User');
```

Then you can refresh the schema registry by the following command:

```sql
ALTER SOURCE src_user FORMAT PLAIN ENCODE PROTOBUF(
schema.registry = 'http://message_queue:8081',
message = 'test.UserWithMoreFields'
);
```

<Note>
Currently, it is not supported to modify the `data_format` and `data_encode`. Furthermore, when refreshing the schema registry of a source, it is not allowed to drop columns or change types.

Another way of refreshing the schema is using the [REFRESH SCHEMA clause](#refresh-schema).
</Note>

### `REFRESH SCHEMA`

This is another way of refreshing the schema of sources when the [FORMAT and ENCODE options](#format-and-encode-options) are not changed.

```sql
ALTER SOURCE source_name REFRESH SCHEMA;
```

For example, assume we have a source as follows:

```sql Create a source
CREATE SOURCE src_user WITH (
connector = 'kafka',
topic = 'sr_pb_test',
properties.bootstrap.server = 'message_queue:29092',
scan.startup.mode = 'earliest'
)
FORMAT PLAIN ENCODE PROTOBUF(
schema.registry = 'http://message_queue:8081',
message = 'test.User'
);
```

Then we can refresh its schema with the following statement:

```sql Refresh schema
ALTER SOURCE src_user REFRESH SCHEMA;
```

For more details about this example, see our [test file](https://github.com/risingwavelabs/risingwave/blob/994a2831088c9befc71721ed6f2f2d2e35c4d0a9/e2e%5Ftest/schema%5Fregistry/alter%5Fsr.slt).

### `SET SOURCE_RATE_LIMIT`

```sql
Expand Down
14 changes: 6 additions & 8 deletions sql/commands/sql-create-source.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,12 @@ Shared source improves resource utilization and data consistency when working wi
This feature is currently in public preview, meaning it is nearing the final product but may not yet be fully stable. If you encounter any issues or have feedback, please reach out to us via our [Slack channel](https://www.risingwave.com/slack). Your input is valuable in helping us improve this feature. For more details, see our [Public Preview Feature List](/changelog/product-lifecycle#features-in-the-public-preview-stage).
</Note>

<Note>
Shared Kafka source is available since version 2.1. Other sources are unaffected. We plan to gradually upgrade other sources to the be shared as well in the future.

`ALTER SOURCE [ADD COLUMN | REFRESH SCHEMA]` for shared source is available since version 2.2.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New feature added in 2.2. Add note here

</Note>

### Configure

Shared source is enabled by default. You can also set the session variable `streaming_use_shared_source` to control whether to enable it.
Expand Down Expand Up @@ -160,14 +166,6 @@ Tables offer other features that enhance their utility in data ingestion workflo
<img src="/images/table-with-connectors.png"/>
</Frame>

<Note>
**LIMITATION**

Currently, shared source is only applicable to Kafka sources. Other sources are unaffected. We plan to gradually upgrade other sources to the be shared as well in the future.

Shared sources do not support `ALTER SOURCE`. Use non-shared sources if you require this functionality.
</Note>

## See also

<CardGroup>
Expand Down
Loading