-
Notifications
You must be signed in to change notification settings - Fork 96
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feature(kafka): design for testing scylla connectors
- Loading branch information
Showing
3 changed files
with
84 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
@startuml | ||
skinparam nodesep 30 | ||
skinparam ranksep 30 | ||
|
||
|
||
legend | ||
|= |= API | | ||
|<back:red> </back>| CQL | | ||
|<back:green> </back>| Kafka API | | ||
endlegend | ||
|
||
database "DB Nodes" { | ||
card [source table] | ||
} | ||
|
||
package "Loaders" { | ||
card [cassandra-stress] | ||
card [kafka-consumer] | ||
} | ||
|
||
package "Kafka" { | ||
card [kafka] | ||
card [kafka-connect] | ||
} | ||
|
||
|
||
[cassandra-stress] -[bold,#red]-> [source table]: **1)** c-s write\n | ||
[kafka-connect] -[bold,#red]-> [source table]: **2)** Kafka connector read CDC | ||
[kafka-consumer] -[bold,#green]-> [kafka]: **3)** reading from\nkafka for validation | ||
|
||
@enduml |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
# Kafka support in SCT | ||
|
||
## Requirements | ||
|
||
1) Be able to run a kafka setup, with a db cluster SCT created | ||
2) Have any of scylla connectors installed on it | ||
1) from a tarball (local file / URL) | ||
2) from confluent-hub (latest/specific version) | ||
3) Work locally with docker backend | ||
|
||
## Design | ||
|
||
![Design](./kafka-setup.png?raw=true ) | ||
|
||
### Local setup | ||
|
||
we should be working with a https://github.com/conduktor/kafka-stack-docker-compose | ||
and it would run where the test is running (on the SCT runner) | ||
for case it won't be enough, we should support running it on it's own machine | ||
|
||
### High load setup | ||
|
||
for high load setups we can go in few routes | ||
* **SaaS** - confluent cloud / AWS MSK / others | ||
- Pros: | ||
* All configuration and setup, and would match better how user would be using scylla connectors | ||
* AMS MSK, we are already working mostly in AWS, and familiar with operating its services | ||
- Cons: | ||
* Might be pricey | ||
* Connectivity to our test setup might be a bit more complex | ||
* on MSK we'll need to run your own kafka-connect instances | ||
* there are limitations to install custom connectors on SaaS | ||
|
||
* **VMs** - create and configured our own VMs and cluster | ||
- Pros: | ||
* We will have full control how it would be setup | ||
* it's fit the current model of SCT is built (or supposed to be) | ||
- Cons: | ||
* We will need to know all the different ways to configure and optimize kafka deployment | ||
* We don't want to be kafka "Experts" to that level... | ||
* We will need to keep it updated all the time | ||
* **K8S** - deploy using helm/operators onto k8s cluster | ||
-Pros: | ||
* We have all the code to support k8s setup on kind/EKS/GKE | ||
* Operation and setup into k8s should be relativity easy, and designed to scale | ||
* it would fit testing with scylla-operator | ||
- Cons: | ||
* manage setups with k8s isn't that easy, lots of "buttons" to play with | ||
|
||
## Open questions | ||
|
||
* how validation is going to be done ? with `kafka-console-consumer` ? | ||
with python code we'll need to write https://kafka-python.readthedocs.io/en/master/apidoc/KafkaConsumer.html ? |