AwsKmsEvent error: No such file or directory

[juliayakovlev]

AwsKmsEvent errors - not clear why it happens and is this problem of SCT or Scylla.
It tries to read /var/lib/scylla/data/mview/users-15e2cee0b49011ee9d88ca66254d3b15/me-6164-big-Data.db SStable - it is sstable of mview.users table and failed due to problem No such file or directory [/var/lib/scylla/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/me-7860-big-Filter.db that's system sstable:

< t:2024-01-16 19:02:49,408 f:sstable_utils.py l:122  c:SstableUtils         p:WARNING > Unexpected error reading sstable located at '/var/lib/scylla/data/mview/users-15e2cee0b49011ee9d88ca66254d3b15/me-6164-big-Data.db': DEBUG 2024-01-16 19:02:48,058 [shard 0] scylla-sstable - Successfully read scylla.yaml from user provided location of /etc/scylla/scylla.yaml
< t:2024-01-16 19:02:49,408 f:sstable_utils.py l:122  c:SstableUtils         p:WARNING > DEBUG 2024-01-16 19:02:48,058 [shard 0] scylla-sstable - Single schema source provided
< t:2024-01-16 19:02:49,408 f:sstable_utils.py l:122  c:SstableUtils         p:WARNING > error processing arguments: could not load schema via schema-tables: std::filesystem::__cxx11::filesystem_error (error system:2, filesystem error: stat failed: No such file or directory [/var/lib/scylla/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/me-7860-big-Filter.db])

< t:2024-01-16 19:02:49,479 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > 2024-01-16 19:02:49.439: (AwsKmsEvent Severity.ERROR) period_type=one-time event_id=0e83e119-165f-4327-83db-21b6ac868595: message=Failed to check the fact of encryption (KMS) for sstables
< t:2024-01-16 19:02:49,479 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > Traceback (most recent call last):
< t:2024-01-16 19:02:49,479 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR >   File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 4354, in _rotate_kms_key
< t:2024-01-16 19:02:49,479 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR >     assert all(encryption_result is True for encryption_result in encryption_results)
< t:2024-01-16 19:02:49,479 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > AssertionError

We need to understand what is the problem there, maybe need to improve the KMS validation

Issue description

• This issue is a regression.
• It is unknown if this issue is a regression.

Describe your issue in detail and steps it took to produce it.

Impact

Describe the impact this issue causes to the user.

How frequently does it reproduce?

Describe the frequency with how this issue can be reproduced.

Installation details

Kernel Version: 5.15.0-1051-aws
Scylla version (or git commit hash): 2023.1.4-20240116.bedad080681e with build-id c0c543b5c81473e26e5111d8f379a77b786bc450

Cluster size: 5 nodes (i4i.8xlarge)

Scylla Nodes used in this run:

• longevity-mv-si-4d-2023-1-db-node-ed5f163f-9 (54.160.192.144 | 10.12.11.64) (shards: 30)
• longevity-mv-si-4d-2023-1-db-node-ed5f163f-8 (54.146.255.212 | 10.12.8.106) (shards: 30)
• longevity-mv-si-4d-2023-1-db-node-ed5f163f-7 (54.89.193.100 | 10.12.11.183) (shards: 30)
• longevity-mv-si-4d-2023-1-db-node-ed5f163f-6 (54.175.114.83 | 10.12.10.234) (shards: 30)
• longevity-mv-si-4d-2023-1-db-node-ed5f163f-5 (34.229.79.250 | 10.12.10.89) (shards: 30)
• longevity-mv-si-4d-2023-1-db-node-ed5f163f-4 (54.235.5.222 | 10.12.10.72) (shards: 30)
• longevity-mv-si-4d-2023-1-db-node-ed5f163f-3 (54.242.31.181 | 10.12.8.55) (shards: 30)
• longevity-mv-si-4d-2023-1-db-node-ed5f163f-2 (54.210.129.143 | 10.12.8.101) (shards: 30)
• longevity-mv-si-4d-2023-1-db-node-ed5f163f-11 (34.229.96.234 | 10.12.8.168) (shards: 30)
• longevity-mv-si-4d-2023-1-db-node-ed5f163f-10 (3.80.50.180 | 10.12.8.48) (shards: 30)
• longevity-mv-si-4d-2023-1-db-node-ed5f163f-1 (54.196.229.76 | 10.12.11.67) (shards: 30)

OS / Image: ami-001477f22cadc6d9c (aws: undefined_region)

Test: longevity-mv-si-4days-test
Test id: ed5f163f-eb0e-4c72-a534-25cc345133a9
Test name: enterprise-2023.1/longevity/longevity-mv-si-4days-test
Test config file(s):

• longevity-mv-si-4days.yaml

Logs and commands

• Restore Monitor Stack command: $ hydra investigate show-monitor ed5f163f-eb0e-4c72-a534-25cc345133a9
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs ed5f163f-eb0e-4c72-a534-25cc345133a9

Logs:

• db-cluster-ed5f163f.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/ed5f163f-eb0e-4c72-a534-25cc345133a9/20240117_070650/db-cluster-ed5f163f.tar.gz
• sct-runner-events-ed5f163f.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/ed5f163f-eb0e-4c72-a534-25cc345133a9/20240117_070650/sct-runner-events-ed5f163f.tar.gz
• sct-ed5f163f.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/ed5f163f-eb0e-4c72-a534-25cc345133a9/20240117_070650/sct-ed5f163f.log.tar.gz
• loader-set-ed5f163f.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/ed5f163f-eb0e-4c72-a534-25cc345133a9/20240117_070650/loader-set-ed5f163f.tar.gz
• monitor-set-ed5f163f.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/ed5f163f-eb0e-4c72-a534-25cc345133a9/20240117_070650/monitor-set-ed5f163f.tar.gz

Jenkins job URL
Argus

[vponomaryov]

FIxed by the following PR:

• fix(kms): Detect sstable not found situation better #7140