Skip to content

Commit

Permalink
fix(gemini): Increase instance size and max retries
Browse files Browse the repository at this point in the history
Currently some of our gemini tests are very flakey failing with
read validations due to "missing rows" on the cluster side.
Since Gemini is random, some tests can be heavier than others.
Looking on failed tests, espcially with dirsuptive nemesis
one could easily notice that CPU utlization per core reaches to
100%. In this cases, some of the validations fail with 1-4 attempts
until there is a validation that fails all the 5 attempts, mostly due
to missing row or several rows on the test cluster. However, connecting
to the cluster afterwards, I could see all rows are available.

This change first increase the instances size from i4i.large to 2xlarge.
Second, it increases the retries from 5 to 10, same as we have in c-s.
  • Loading branch information
roydahan authored and soyacz committed Feb 9, 2024
1 parent 1a13504 commit fba1669
Show file tree
Hide file tree
Showing 4 changed files with 6 additions and 6 deletions.
2 changes: 1 addition & 1 deletion test-cases/gemini/gemini-1tb-10h.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ nemesis_seed: '041'

gemini_cmd: "gemini -d --duration 8h --warmup 2h -c 50 \
-m mixed -f --non-interactive --cql-features normal \
--max-mutation-retries 5 --max-mutation-retries-backoff 500ms \
--max-mutation-retries 10 --max-mutation-retries-backoff 500ms \
--async-objects-stabilization-attempts 5 --async-objects-stabilization-backoff 500ms \
--replication-strategy \"{'class': 'NetworkTopologyStrategy', 'replication_factor': '3'}\"
--oracle-replication-strategy \"{'class': 'NetworkTopologyStrategy', 'replication_factor': '1'}\" "
Expand Down
4 changes: 2 additions & 2 deletions test-cases/gemini/gemini-3h-with-nemesis.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ n_db_nodes: 3
n_test_oracle_db_nodes: 1
n_loaders: 1
n_monitor_nodes: 1
instance_type_db: 'i4i.large'
instance_type_db: 'i4i.2xlarge'

user_prefix: 'gemini-with-nemesis-3h-normal'

Expand All @@ -18,7 +18,7 @@ nemesis_seed: '032'
# the below cmd runs about 3 hours
gemini_cmd: "gemini -d --duration 3h --warmup 30m \
-c 50 -m mixed -f --non-interactive --cql-features normal \
--max-mutation-retries 5 --max-mutation-retries-backoff 500ms \
--max-mutation-retries 10 --max-mutation-retries-backoff 500ms \
--async-objects-stabilization-attempts 5 --async-objects-stabilization-backoff 500ms \
--replication-strategy \"{'class': 'NetworkTopologyStrategy', 'replication_factor': '3'}\" \
--oracle-replication-strategy \"{'class': 'NetworkTopologyStrategy', 'replication_factor': '1'}\""
Expand Down
4 changes: 2 additions & 2 deletions test-cases/gemini/gemini-3h-with-nondisruptive-nemesis.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ n_db_nodes: 3
n_test_oracle_db_nodes: 1
n_loaders: 1
n_monitor_nodes: 1
instance_type_db: 'i4i.large'
instance_type_db: 'i4i.xlarge'

user_prefix: 'gemini-basic-3h'

Expand All @@ -15,7 +15,7 @@ nemesis_interval: 5
# the below cmd runs about 3 hours
gemini_cmd: "gemini -d --duration 10800s --warmup 1800s -c 50 \
-m mixed -f --non-interactive --cql-features normal \
--max-mutation-retries 5 --max-mutation-retries-backoff 500ms \
--max-mutation-retries 10 --max-mutation-retries-backoff 500ms \
--async-objects-stabilization-attempts 5 --async-objects-stabilization-backoff 500ms \
--replication-strategy \"{'class': 'NetworkTopologyStrategy', 'replication_factor': '3'}\" \
--oracle-replication-strategy \"{'class': 'NetworkTopologyStrategy', 'replication_factor': '1'}\" "
Expand Down
2 changes: 1 addition & 1 deletion test-cases/gemini/gemini-8h-large-num-columns.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ gemini_cmd: "gemini -d --duration 7h --warmup 1h \
-c 10 -m mixed -f --non-interactive \
--cql-features normal --async-objects-stabilization-backoff 500ms \
--replication-strategy \"{'class': 'NetworkTopologyStrategy', 'replication_factor': '3'}\" \
--max-mutation-retries 5 --max-mutation-retries-backoff 500ms \
--max-mutation-retries 10 --max-mutation-retries-backoff 500ms \
--max-partition-keys 12 --min-partition-keys 8 \
--max-clustering-keys 20 --min-clustering-keys 12 \
--max-columns 100 --min-columns 80 --verbose"
Expand Down

0 comments on commit fba1669

Please sign in to comment.