Error: table - Memtable flush failed due to: std::filesystem::__cxx11::filesystem_error and coredump during second bootstrap after first was aborted #6865

aleksbykov · 2023-09-03T14:03:05Z

Issue description

This issue is a regression.
It is unknown if this issue is a regression.

node10 was bootstrapped, but bootstrap process was aborted with scylla restart

2023-09-02T07:18:37+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 0] compaction - [Compact system_schema.columns ef1409e0-4960-11ee-a854-787b2ccbbb9f] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/mc-21-big-Data.db:level=0]. 37kB to 31kB (~85% of original) in 28ms = 1MB/s. ~256 total partitions merged to 8.
2023-09-02T07:18:37+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 0] schema_tables - Schema version changed to ead1fd37-e51a-39fd-88c0-fb4e47b604fb
2023-09-02T07:18:37+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 0] raft_group_registry - marking Raft server 8a376b65-36c1-4cca-af00-b48c23f19b40 as alive for raft groups
2023-09-02T07:18:37+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10  !WARNING | scylla[5444]:  [shard 2] raft_group_registry - Raft server id 717dce78-3a3a-4dc0-abc0-968e62866ef4 cannot be translated to an IP address.
2023-09-02T07:18:37+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10  !WARNING | scylla[5444]:  [shard 4] raft_group_registry - Raft server id 7779ff5b-a5d9-4b6e-ab09-358b2457ed13 cannot be translated to an IP address.
2023-09-02T07:18:37+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 0] raft_group_registry - marking Raft server 9096d16a-571c-476c-9ab7-e6fadad1858b as alive for raft groups
2023-09-02T07:18:37+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 0] raft_group0 - setup_group0: waiting for peers to synchronize state...
2023-09-02T07:18:37+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 0] raft_group0_upgrade - wait_for_peers_to_enter_synchronize_state: 10.4.8.17 confirmed that they finished upgrade.
2023-09-02T07:18:37+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 0] raft_group0 - setup_group0: the cluster is ready to use Raft. Finishing.
2023-09-02T07:18:37+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 0] raft_group_registry - marking Raft server d9badaab-3e88-4501-a202-0eca1f1fc797 as alive for raft groups
2023-09-02T07:18:37+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 0] raft_group_registry - marking Raft server ffd441c9-fed0-4d58-a38c-679ee6c1038c as alive for raft groups
2023-09-02T07:18:37+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 0] raft_group_registry - marking Raft server 0c9e1268-a82b-402c-99c3-22f9a384649a as alive for raft groups
2023-09-02T07:18:37+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 1] compaction - [Compact system.scylla_local ef30e0b0-4960-11ee-9928-787c2ccbbb9f] Compacting [/var/lib/scylla/data/system/scylla_local-2972ec7ffb2038ddaac1d876f2e3fcbd/mc-8-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/system/scylla_local-2972ec7ffb2038ddaac1d876f2e3fcbd/mc-15-big-Data.db:level=0:origin=memtable]
2023-09-02T07:18:37+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 1] compaction - [Compact system.scylla_local ef30e0b0-4960-11ee-9928-787c2ccbbb9f] Compacted 2 sstables to [/var/lib/scylla/data/system/scylla_local-2972ec7ffb2038ddaac1d876f2e3fcbd/mc-22-big-Data.db:level=0]. 10kB to 5kB (~52% of original) in 4ms = 2MB/s. ~256 total partitions merged to 1.

2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10   !NOTICE | sudo[5522]: scyllaadm : PWD=/home/scyllaadm ; USER=root ; COMMAND=/usr/bin/systemctl stop scylla-server.service
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | sudo[5522]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | systemd[1]: scylla-housekeeping-daily.timer: Deactivated successfully.
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | systemd[1]: Stopped Run Scylla Housekeeping daily mode.
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | systemd[1]: scylla-housekeeping-restart.timer: Deactivated successfully.
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | systemd[1]: Stopped Run Scylla Housekeeping restart mode.
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | systemd[1]: Stopping Scylla JMX...
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | systemd[1]: scylla-jmx.service: Deactivated successfully.
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | systemd[1]: scylla-jmx.service: Unit process 5476 (select-java) remains running after unit stopped.
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | systemd[1]: scylla-jmx.service: Unit process 5477 (select-java) remains running after unit stopped.
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | systemd[1]: scylla-jmx.service: Unit process 5478 (java) remains running after unit stopped.
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | systemd[1]: scylla-jmx.service: Unit process 5479 (awk) remains running after unit stopped.
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | systemd[1]: Stopped Scylla JMX.
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | systemd[1]: Stopping Scylla Server...

2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 0] init - Shutting down sighup was successful
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 0] init - Shutting down configurables
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 0] init - Shutting down configurables was successful
2023-09-02T07:18:38+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[5444]:  [shard 0] init - Startup interrupted
2023-09-02T07:18:39+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10   !NOTICE | sudo[5524]: scyllaadm : PWD=/home/scyllaadm ; USER=root ; COMMAND=/usr/bin/apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 17723034C56D4B19

After that scylla is started and during next bootstrap next aborting happened and coredump

2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:  [shard 0] gossip - Node 10.4.8.169 will be removed from gossip at [2023-09-05 05:08:07]: (expire = 1693890487350178251, now = 1693639209006617884, diff = 251278 seconds)
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:  [shard 0] storage_service - Removing tokens {2288345973318160718, -7498528026347997444, 4723929905651532509, 1592780218825891478, -6470205015042574452, -8516384109707374349, -7324114398293506967, -822687537740371966, -1893833690224154813, -1428377621550993109, -6577316863666114095, 7308884488290193156, 5830981095323930057, -2176298339717880812, -1765428225695066947, 3688037165703824730, 7513724757469850199, 3188791257593368747, -3870190881463750327, -4286037078309036103, -8850303082660667917, 5786918923694138286, -9207221139069366836, 7651258902825123183, 6004451204021841395, -6026060008568624337, 343957021436505021, 1347904708588647217, 4536077647113901072, -6824885036674024543, -7167499939047200581, 7212277846808291131, -2658988524732515988, -1456236610201863524, 140350051029587643, 8607261301585605559, 8171501706703519573, -3941086831470601254, -3143149522070009435, -2096063393427963406, 9118231222129881528, 5663133409102365011, -3748031589507634833, 1944539972703663724, 3848554468651143617, -4136404847416001451, -8121977120708365190, 5596604214434435875, 939796323098585278, 7860021676072439942, 5012616076560514087, -2540364845309931377, 6778346401881958063, -8447067820231342451, 4338531917278429165, 806371254502828197, 8148028749355620271, 3609936682391808415, -2578917035439595483, -3737815723692048523, 3472439781136891810, 2324533475536376221, 245129596048509923, -4310491954469114637, -8355414500214610990, -725454704453828800, 1695924199582202233, 1024265047204323391, -7338584707321806815, 614661211118722718, -7316140860108558226, -2564336314823005093, 1664778785760398855, 7911579513348859823, -1706167942100916845, -6183860137175106495, -7339687264119450581, -4396112621440430963, -1100570194367256537, -229630919920447203, 1393012096606487201, 6353461188796901278, -8103364347730989842, -2859795605523648436, -4435062154988945459, 4348941137592319042, -6286903163935121720, -3559126652017914621, -9063395311311504698, -7120069754409231859, -5207756135585606636, -6705381322303252379, -5224763080739143150, 584886108746044052, 8190119036676856187, -4045231294033006973, -1326813829610612821, -7071143919458182274, -5648903499864540510, 68751395924472444, 831076585050755115, 4254612055633486504, -8286999961211364187, 6812622166984288529, 1595799804180536473, -7377818516242787186, 710310922103699813, 9034816396756506301, 5834748049491277624, -2378542182064671957, -4538565569338592989, 6300456305585153283, -4518136625163008363, -7894474940676943773, 538639989285429517, -6633514190986487220, -1391857722770144101, 2319618016651692500, 3101786548055014421, 1761015425799218086, 8847259065193638038, 5244938052921406275, -1560512564217023239, 218507857571212693, 6526811093304456394, -2771900920257509620, -2559145883537801087, 6435595169737524226, 3089545194172031311, -3507442410717313901, 8802279541500270127, -703889990238594310, -2074139110326863602, -2363895103684083867, 6878077637195655877, -2886718640462018825, 1774844946462314558, 3046971953326402966, 1339449303514518459, 3038516574022279644, -8239341950397820342, -2357512659322349121, 8579831061400210206, 6116265275472584124, 2610895041629864441, 8106305240079287196, -7724351284082644460, 2719639516735395905, 7324847698830994027, -205595442008871369, -1491345120144906030, 4069899575903387632, 1084098996889534707, -8498973949868748943, -6250788683174809576, 1258195935199721727, 1388512108191977643, 8873519271969639106, -3894801328999162346, -4002453664831747547, -468016608625823071, 1516054258211405014, -4481159345293144444, -570139719206721274, -2319863037863125897, 2301724798061152468, -5979049595844237137, 2777952232934629769, -7581560779178260370, -5980316689831588857, -6718409550417254643, -3805556600393157426, -3424299701568856469, 546713491468964955, -3174695753966379351, -4874860506728247476, 6493987991998639090, 6931080712212022446, -4632486439378799311, 27465119887904234, 993184668286238665, -1187884195510887347, 1980721695998160204, 3554028352797414118, -1344859821383396079, 2588093195353581681, -2107064537881765903, -8646871741283298238, -3041345525761426126, -3616032882122859376, -277901211030680849, -7667643614017101344, -3208735074973833611, -140564073071981985, 6945180605329668877, -1610542568738556793, -8424429843272143568, 1232543535476950055, -1525514197424714650, -7899758631464337016, -322724525541097540, -4737081380963763999, -3814808466140327530, -6510242274750604296, -3883460361806081, -4339132493697381219, -7378541087867758967, 8828885899579322971, 3754312057663110512, -6128427240558535027, 6300813138555441475, 1883726485784275553, 4706882200939381249, -899718853591787759, 4931462047321672229, 256059417717250578, -5880696813153881358, -1718022118832194067, -3941425361294528527, 4509963558416222577, -3577293054141097589, -1515988995672352641, -4524647528706823857, -3079483527478703316, 6410755854488817379, 6316870113261038038, -6683649517520254512, -1089460405718053123, 5954884438970558536, 1033307666949854534, 2594554934022026193, -7336202268171596908, 2469937819207299133, -4541387917514115393, 8191368315000449819, 6002201577037164079, -1219454923944941440, 631752647530082064, -1146479607231845202, 3498320054280231765, -6032531999404248277, 8390298117467966068, -2762901106638508126, 8481037822071926415, -7012925914471443875, -4542986488933923540, -7286153378531062866, -6792576880206881621, 4497633025281927736, 4527359763796887583, -4463613590128416165, -5635020300280120850, -3358446862168633295, -7976234691746091970, 3257523139181073525, -5560365777800207221} for 10.4.8.169
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:  [shard 0] storage_service - Node 10.4.9.120 state jump to normal
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:  [shard 0] storage_service - Node 10.4.11.41 state jump to normal
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10      !ERR | scylla[6536]:  [shard 6] table - failed to write sstable /var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/mc-48-big-Data.db: std::filesystem::__cxx11::filesystem_error (filesystem error: mkdir failed: No such file or directory [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/48.sstable])
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10      !ERR | scylla[6536]:  [shard 1] table - failed to write sstable /var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/mc-43-big-Data.db: std::filesystem::__cxx11::filesystem_error (filesystem error: mkdir failed: No such file or directory [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/43.sstable])
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10      !ERR | scylla[6536]:  [shard 6] table - Memtable flush failed due to: std::filesystem::__cxx11::filesystem_error (error system:2, filesystem error: mkdir failed: No such file or directory [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/48.sstable]). Aborting, at 0x5cc5e0e 0x5cc63d0 0x5cc66a8 0x1b4ddc6 0x12fa74a 0x58f676f 0x58f79a7 0x591aa73 0x58c6f5a /opt/scylladb/libreloc/libc.so.6+0x8c906 /opt/scylladb/libreloc/libc.so.6+0x11286f
   --------
   seastar::internal::coroutine_traits_base<void>::promise_type
   --------
   seastar::internal::coroutine_traits_base<void>::promise_type
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_0>(replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_0&&)::{lambda(auto:1&&)#1}, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_0>(replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_0&&)::{lambda(auto:1&&)#1}>(seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_0>(replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_0&&)::{lambda(auto:1&&)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_0>(auto:1&&)::{lambda(auto:1&&)#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<replica::memtable_list::flush()::$_0::operator()<replica::flush_permit>(replica::flush_permit) const::{lambda()#1}, false>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::finally_body<replica::memtable_list::flush()::$_0::operator()<replica::flush_permit>(replica::flush_permit) const::{lambda()#1}, false> >(seastar::future<void>::finally_body<replica::memtable_list::flush()::$_0::operator()<replica::flush_permit>(replica::flush_permit) const::{lambda()#1}, false>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::finally_body<replica::memtable_list::flush()::$_0::operator()<replica::flush_permit>(auto:1) const::{lambda()#1}, false>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::shared_future<>::shared_state::get_future(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda(seastar::future<void>&&)#1}, seastar::future<void>::then_wrapped_nrvo<void, seastar::shared_future<>::shared_state::get_future(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda(seastar::future<void>&&)#1}>(seastar::shared_future<>::shared_state::get_future(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda(seastar::future<void>&&)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::shared_future<>::shared_state::get_future(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda(seastar::future<void>&&)#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10      !ERR | scylla[6536]:  [shard 1] table - Memtable flush failed due to: std::filesystem::__cxx11::filesystem_error (error system:2, filesystem error: mkdir failed: No such file or directory [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/43.sstable]). Aborting, at 0x5cc5e0e 0x5cc63d0 0x5cc66a8 0x1b4ddc6 0x12fa74a 0x58f676f 0x58f79a7 0x591aa73 0x58c6f5a /opt/scylladb/libreloc/libc.so.6+0x8c906 /opt/scylladb/libreloc/libc.so.6+0x11286f
   --------
   seastar::internal::coroutine_traits_base<void>::promise_type
   --------
   seastar::internal::coroutine_traits_base<void>::promise_type
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_0>(replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_0&&)::{lambda(auto:1&&)#1}, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_0>(replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_0&&)::{lambda(auto:1&&)#1}>(seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_0>(replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_0&&)::{lambda(auto:1&&)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_0>(auto:1&&)::{lambda(auto:1&&)#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<replica::memtable_list::flush()::$_0::operator()<replica::flush_permit>(replica::flush_permit) const::{lambda()#1}, false>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::finally_body<replica::memtable_list::flush()::$_0::operator()<replica::flush_permit>(replica::flush_permit) const::{lambda()#1}, false> >(seastar::future<void>::finally_body<replica::memtable_list::flush()::$_0::operator()<replica::flush_permit>(replica::flush_permit) const::{lambda()#1}, false>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::finally_body<replica::memtable_list::flush()::$_0::operator()<replica::flush_permit>(auto:1) const::{lambda()#1}, false>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::shared_future<>::shared_state::get_future(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda(seastar::future<void>&&)#1}, seastar::future<void>::then_wrapped_nrvo<void, seastar::shared_future<>::shared_state::get_future(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda(seastar::future<void>&&)#1}>(seastar::shared_future<>::shared_state::get_future(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda(seastar::future<void>&&)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::shared_future<>::shared_state::get_future(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda(seastar::future<void>&&)#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]: Aborting on shard 6.
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]: Backtrace:
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:   0x58e5338
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:   0x591a4a2
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:   /opt/scylladb/libreloc/libc.so.6+0x3db6f
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:   /opt/scylladb/libreloc/libc.so.6+0x8e843
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:   /opt/scylladb/libreloc/libc.so.6+0x3dabd
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:   /opt/scylladb/libreloc/libc.so.6+0x2687e
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:   0x1b4ddf9
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:   0x12fa74a
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:   0x58f676f
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:   0x58f79a7
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:   0x591aa73
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:   0x58c6f5a
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:   /opt/scylladb/libreloc/libc.so.6+0x8c906
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6536]:   /opt/scylladb/libreloc/libc.so.6+0x11286f
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | systemd[1]: Created slice Slice /system/systemd-coredump.
2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | systemd[1]: Started Process Core Dump (PID 6741/UID 0).

After that scylla was restarted again and bootstrapped correctly

Impact

Describe the impact this issue causes to the user.

How frequently does it reproduce?

Describe the frequency with how this issue can be reproduced.

Installation details

Kernel Version: 5.15.0-1043-aws
Scylla version (or git commit hash): 5.4.0~dev-20230901.3bdbe620aa1e with build-id f44ecb6140c40cc466146304eb075258461d5dd7

Cluster size: 5 nodes (i4i.2xlarge)

Scylla Nodes used in this run:

longevity-parallel-topology-schema--db-node-7f42d685-9 (34.244.115.135 | 10.4.8.17) (shards: 7)
longevity-parallel-topology-schema--db-node-7f42d685-8 (34.246.181.172 | 10.4.11.41) (shards: 7)
longevity-parallel-topology-schema--db-node-7f42d685-7 (52.208.112.13 | 10.4.8.169) (shards: 7)
longevity-parallel-topology-schema--db-node-7f42d685-6 (54.74.165.246 | 10.4.9.31) (shards: 7)
longevity-parallel-topology-schema--db-node-7f42d685-5 (34.254.254.187 | 10.4.9.120) (shards: 7)
longevity-parallel-topology-schema--db-node-7f42d685-4 (3.253.103.223 | 10.4.8.40) (shards: 7)
longevity-parallel-topology-schema--db-node-7f42d685-3 (54.194.249.157 | 10.4.8.154) (shards: 7)
longevity-parallel-topology-schema--db-node-7f42d685-2 (34.241.213.118 | 10.4.11.110) (shards: 7)
longevity-parallel-topology-schema--db-node-7f42d685-18 (3.250.128.115 | 10.4.9.12) (shards: 7)
longevity-parallel-topology-schema--db-node-7f42d685-17 (3.249.73.58 | 10.4.10.139) (shards: 7)
longevity-parallel-topology-schema--db-node-7f42d685-16 (34.240.66.239 | 10.4.8.72) (shards: -1)
longevity-parallel-topology-schema--db-node-7f42d685-15 (3.249.126.137 | 10.4.10.243) (shards: 7)
longevity-parallel-topology-schema--db-node-7f42d685-14 (34.255.124.228 | 10.4.11.181) (shards: 7)
longevity-parallel-topology-schema--db-node-7f42d685-13 (52.208.27.182 | 10.4.10.96) (shards: 7)
longevity-parallel-topology-schema--db-node-7f42d685-12 (54.74.134.6 | 10.4.11.248) (shards: -1)
longevity-parallel-topology-schema--db-node-7f42d685-11 (54.78.231.0 | 10.4.8.126) (shards: 7)
longevity-parallel-topology-schema--db-node-7f42d685-10 (34.240.61.162 | 10.4.11.209) (shards: -1)
longevity-parallel-topology-schema--db-node-7f42d685-1 (3.253.86.212 | 10.4.8.150) (shards: 7)

OS / Image: ami-0dd654338b4cc2a70 (aws: undefined_region)

Test: longevity-schema-topology-changes-12h-test
Test id: 7f42d685-d644-4597-88ee-b71088f4d1c4
Test name: scylla-master/longevity/longevity-schema-topology-changes-12h-test
Test config file(s):

longevity-parallel-schema-changes-12h.yaml

Logs and commands

Restore Monitor Stack command: $ hydra investigate show-monitor 7f42d685-d644-4597-88ee-b71088f4d1c4
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs 7f42d685-d644-4597-88ee-b71088f4d1c4

Logs:

db-cluster-7f42d685.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7f42d685-d644-4597-88ee-b71088f4d1c4/20230903_041237/db-cluster-7f42d685.tar.gz
sct-runner-events-7f42d685.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7f42d685-d644-4597-88ee-b71088f4d1c4/20230903_041237/sct-runner-events-7f42d685.tar.gz
sct-7f42d685.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7f42d685-d644-4597-88ee-b71088f4d1c4/20230903_041237/sct-7f42d685.log.tar.gz
loader-set-7f42d685.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7f42d685-d644-4597-88ee-b71088f4d1c4/20230903_041237/loader-set-7f42d685.tar.gz
monitor-set-7f42d685.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7f42d685-d644-4597-88ee-b71088f4d1c4/20230903_041237/monitor-set-7f42d685.tar.gz

Jenkins job URL
Argus

The text was updated successfully, but these errors were encountered:

aleksbykov · 2023-09-03T14:03:28Z

2023-09-02 07:24:53.837 <2023-09-02 07:20:09.000>: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=eb780324-1c7f-4b28-9aeb-4afbb9873102 during_nemesis=BootstrapStreamingError node=Node longevity-parallel-topology-schema--db-node-7f42d685-10 [34.240.61.162 | 10.4.11.209] (seed: False)
corefile_url=https://storage.cloud.google.com/upload.scylladb.com/core.scylla.112.5ac4fbd29e794a1098f9309538996d99.6536.1693639209000000/core.scylla.112.5ac4fbd29e794a1098f9309538996d99.6536.1693639209000000.gz
backtrace= PID: 6536 (scylla)
UID: 112 (scylla)
GID: 118 (scylla)
Signal: 6 (ABRT)
Timestamp: Sat 2023-09-02 07:20:09 UTC (2min 9s ago)
Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 25 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-7 --lock-memory=1
Executable: /opt/scylladb/libexec/scylla
Control Group: /scylla.slice/scylla-server.slice/scylla-server.service
Unit: scylla-server.service
Slice: scylla-server.slice
Boot ID: 5ac4fbd29e794a1098f9309538996d99
Machine ID: a537b993337d49af947ce0c1ecde2f5b
Hostname: longevity-parallel-topology-schema--db-node-7f42d685-10
Storage: /var/lib/systemd/coredump/core.scylla.112.5ac4fbd29e794a1098f9309538996d99.6536.1693639209000000 (present)
Disk Size: 57.0G
Message: Process 6536 (scylla) of user 112 dumped core.

            Found module linux-vdso.so.1 with build-id: f53652994793115be6fe21e67be2a7459433e449
            Found module libcrypto.so.3 with build-id: d871cd7f304722b26629ffb37ed31a5f6258b14d
            Metadata for module libcrypto.so.3 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "openssl",
            	"version" : "3.0.9-2.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libssl.so.3 with build-id: 782d564b4727cdfd9eb0b5c18dc9d609cf653666
            Metadata for module libssl.so.3 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "openssl",
            	"version" : "3.0.9-2.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libzstd.so.1 with build-id: 95fc1c046547d542d2913ebc72290732112d8b9b
            Metadata for module libzstd.so.1 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "zstd",
            	"version" : "1.5.5-1.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module liblzma.so.5 with build-id: 71fbd52ad4c7efd194c0eca39ad53642e99d26b9
            Metadata for module liblzma.so.5 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "xz",
            	"version" : "5.4.1-1.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libcap.so.2 with build-id: 2fe8ebb5811811046e04d8f4c000696cb9b24e24
            Metadata for module libcap.so.2 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "libcap",
            	"version" : "2.48-6.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libicudata.so.72 with build-id: 64ea912ec6a442517d6e32c2bdfa629379739b45
            Metadata for module libicudata.so.72 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "icu",
            	"version" : "72.1-2.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libffi.so.8 with build-id: f5312719df7de0c6ae9d42ed18680d8016bb59d4
            Metadata for module libffi.so.8 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "libffi",
            	"version" : "3.4.4-2.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libunistring.so.2 with build-id: 364176cb9ffd75bd9dd1b2b4e042b63c6d19c769
            Found module ld.so with build-id: cca57d3930cdf4dbe40c899fa84de37e638e3b06
            Found module libc.so.6 with build-id: 245240a31888ad5c11bbc55b18e02d87388f59a9
            Found module libgcc_s.so.1 with build-id: 2a5300931e0d29bd90b6a8681c0131c3f4bdb53d
            Found module libstdc++.so.6 with build-id: d25eaf3596d9455fe9725f6e9cd1aa5433f31b92
            Found module libboost_unit_test_framework.so.1.78.0 with build-id: 6c1b0aff72db6de9e035cb7bc1d96be160ba975b
            Metadata for module libboost_unit_test_framework.so.1.78.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "boost",
            	"version" : "1.78.0-11.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libthrift-0.14.0.so with build-id: 7ec8f2a10108f6bf0e82ab70c7df8e255c53ffd5
            Metadata for module libthrift-0.14.0.so owned by FDO found: {
            	"type" : "rpm",
            	"name" : "thrift",
            	"version" : "0.14.0-13.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_low_level_hash.so.2206.0.0 with build-id: 54f02731e4aac190276a846427c1756f68624ca8
            Metadata for module libabsl_low_level_hash.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_bad_variant_access.so.2206.0.0 with build-id: aac271b1b6c0028affbd997fd6fb00580f1d802c
            Metadata for module libabsl_bad_variant_access.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_city.so.2206.0.0 with build-id: f15af062278721a0c450b5fbc5bbc5004cca1d66
            Metadata for module libabsl_city.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_hash.so.2206.0.0 with build-id: ded250f9194b4ee9ac86543f1a866840a3bd5c92
            Metadata for module libabsl_hash.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_time_zone.so.2206.0.0 with build-id: a05382267d52bb37dbd18d46434e39b941856f57
            Metadata for module libabsl_time_zone.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_log_severity.so.2206.0.0 with build-id: 32ad933bdf75d9d50f6611ceb9277891d49ba610
            Metadata for module libabsl_log_severity.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_raw_logging_internal.so.2206.0.0 with build-id: de1609c0ce2fc11fef180bb402d42a0667263fb3
            Metadata for module libabsl_raw_logging_internal.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_throw_delegate.so.2206.0.0 with build-id: b9e0846035e3d3c81367d561af32852bdd91de32
            Metadata for module libabsl_throw_delegate.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_int128.so.2206.0.0 with build-id: aef7256436e9507891e7c273b3cfb5afe6a3f961
            Metadata for module libabsl_int128.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_spinlock_wait.so.2206.0.0 with build-id: 4c9cc99cb1cbd0021fe2162e05d115183ae5216e
            Metadata for module libabsl_spinlock_wait.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_base.so.2206.0.0 with build-id: 6923bdbc6c7adca9e3cc630c587ca7bc569aea7c
            Metadata for module libabsl_base.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_strings_internal.so.2206.0.0 with build-id: 37882a7545d19410520b45db8bda13e1ae0f24fb
            Metadata for module libabsl_strings_internal.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_strings.so.2206.0.0 with build-id: ea08a07cb064096a6979c9b6784dffbe7f937d30
            Metadata for module libabsl_strings.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_civil_time.so.2206.0.0 with build-id: c0bf69065233de3b7aa4e9b38ce36c491321fbca
            Metadata for module libabsl_civil_time.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_time.so.2206.0.0 with build-id: b14a9ea5584875df161bd47297fe8c7d1904334d
            Metadata for module libabsl_time.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_malloc_internal.so.2206.0.0 with build-id: 2e9b98b4d58498633ce5ead7d192ade01f0b32b1
            Metadata for module libabsl_malloc_internal.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_demangle_internal.so.2206.0.0 with build-id: 751c512b9503a4f9e6481a92fcb1d31149fb4fb0
            Metadata for module libabsl_demangle_internal.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_debugging_internal.so.2206.0.0 with build-id: 60c2a1e174cafffc739f2317ff27af1889684ec2
            Metadata for module libabsl_debugging_internal.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_symbolize.so.2206.0.0 with build-id: 028562e3f9f0e5d78c9addca1b0270f4d4ffbd0b
            Metadata for module libabsl_symbolize.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_stacktrace.so.2206.0.0 with build-id: 4f532283ceb52eab32a4fbd2e9c2dfcd33798244
            Metadata for module libabsl_stacktrace.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_graphcycles_internal.so.2206.0.0 with build-id: d6501a5562045c64bd2766a4d0594c120690c36a
            Metadata for module libabsl_graphcycles_internal.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_synchronization.so.2206.0.0 with build-id: 1fded83eda5a15a6b176635d7cb82f3bbf2e77fa
            Metadata for module libabsl_synchronization.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_exponential_biased.so.2206.0.0 with build-id: f548148b21727aee3b652e8f4ae0b876093cca80
            Metadata for module libabsl_exponential_biased.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_hashtablez_sampler.so.2206.0.0 with build-id: 46c8178e24dfc52f2c6f23ce94a79aedbcc0ee0b
            Metadata for module libabsl_hashtablez_sampler.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_bad_optional_access.so.2206.0.0 with build-id: ce66b4e2a512e919c08550f16f196080cf1a925f
            Metadata for module libabsl_bad_optional_access.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libabsl_raw_hash_set.so.2206.0.0 with build-id: 658c5086d69b467a1fda3762f24cb0ef44b718a7
            Metadata for module libabsl_raw_hash_set.so.2206.0.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "abseil-cpp",
            	"version" : "20220623.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libsystemd.so.0 with build-id: 44d11574d5b9f1d883d5e0623330efe8a7f0fdb2
            Metadata for module libsystemd.so.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "systemd",
            	"version" : "253.7-1.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module liblua-5.4.so with build-id: edd5d6e6ba778bd017c12bd130d23d0f39ba6ab2
            Metadata for module liblua-5.4.so owned by FDO found: {
            	"type" : "rpm",
            	"name" : "lua",
            	"version" : "5.4.4-9.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libdeflate.so.0 with build-id: da0669fb51cad4d74cdcab5fbcde464592c07911
            Metadata for module libdeflate.so.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "libdeflate",
            	"version" : "1.9-6.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libxxhash.so.0 with build-id: 7941174ea5471182e7ffa380dff1a4056d104a45
            Metadata for module libxxhash.so.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "xxhash",
            	"version" : "0.8.2-1.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libicui18n.so.72 with build-id: 13e5dae11fa14e4ac67e847e96ff38e07b8470a7
            Metadata for module libicui18n.so.72 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "icu",
            	"version" : "72.1-2.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libicuuc.so.72 with build-id: f6dc975ad37df68318b229b945a1735f330fe363
            Metadata for module libicuuc.so.72 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "icu",
            	"version" : "72.1-2.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libboost_regex.so.1.78.0 with build-id: ecbb0bd3a76b91d4668e8cbdaef774b2b347121c
            Metadata for module libboost_regex.so.1.78.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "boost",
            	"version" : "1.78.0-11.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libboost_date_time.so.1.78.0 with build-id: 810789c70ea64e46db892a861c3d0f8e4ee9dc01
            Metadata for module libboost_date_time.so.1.78.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "boost",
            	"version" : "1.78.0-11.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libcrypt.so.2 with build-id: 1b328a05d4e5133c864e3cdf1eff45de72fc5733
            Metadata for module libcrypt.so.2 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "libxcrypt",
            	"version" : "4.4.36-1.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libjsoncpp.so.25 with build-id: e62d05e5ee3f6da3d3bd0fae2633c298d58cb43e
            Metadata for module libjsoncpp.so.25 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "jsoncpp",
            	"version" : "1.9.5-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libsnappy.so.1 with build-id: 26964a0b88eaa9c9260fe89937654a1a993138cf
            Metadata for module libsnappy.so.1 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "snappy",
            	"version" : "1.1.9-7.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libz.so.1 with build-id: d8f77557d166386619afc2932ae77119e6644575
            Metadata for module libz.so.1 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "zlib",
            	"version" : "1.2.13-3.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libyaml-cpp.so.0.7 with build-id: f570d4b78b416c86aed9044f9b0f9d479dcfd432
            Metadata for module libyaml-cpp.so.0.7 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "yaml-cpp",
            	"version" : "0.7.0-3.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libm.so.6 with build-id: 69a765cf3b65ad7e3446edfcd5c2b12e86f7589b
            Found module libhwloc.so.15 with build-id: a0ed39ba2a7bfeea4cbaa8fff2257e34bfbac01c
            Found module libp11-kit.so.0 with build-id: cc5d147fe844b09973c58e10a9a1ffc95e4a7523
            Metadata for module libp11-kit.so.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "p11-kit",
            	"version" : "0.25.0-1.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libidn2.so.0 with build-id: c25f9a2ef9b6fb843df0fc37c6c8903af04cbbe1
            Metadata for module libidn2.so.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "libidn2",
            	"version" : "2.3.4-2.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libtasn1.so.6 with build-id: c684c041d1e59673bb0406fe9ae108f6b6d64c6f
            Metadata for module libtasn1.so.6 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "libtasn1",
            	"version" : "4.19.0-2.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libhogweed.so.6 with build-id: d4becfdf402e462138154123f6c66e11c9afa11e
            Metadata for module libhogweed.so.6 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "nettle",
            	"version" : "3.8-3.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libnettle.so.8 with build-id: c750c0ee69d294221c16f0379c511cd6313671f8
            Metadata for module libnettle.so.8 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "nettle",
            	"version" : "3.8-3.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libunistring.so.5 with build-id: 586ab8815558450155f6e0285363811bbca61fef
            Metadata for module libunistring.so.5 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "libunistring",
            	"version" : "1.1-3.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libgmp.so.10 with build-id: 44b35c0168894f171067115c93846b4dc7cedcdc
            Metadata for module libgmp.so.10 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "gmp",
            	"version" : "6.2.1-4.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libgnutls.so.30 with build-id: 65f5be42fbfb612c1cebe50e59402d0b6b6f0753
            Metadata for module libgnutls.so.30 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "gnutls",
            	"version" : "3.8.0-2.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module liblz4.so.1 with build-id: c32727dc2cf56f54d7a9cfc6d8776bfb765a5d60
            Metadata for module liblz4.so.1 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "lz4",
            	"version" : "1.9.4-2.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libatomic.so.1 with build-id: d91a4408d57b1f3f458b6b8f43ea2c9a48631272
            Found module libnuma.so.1 with build-id: 46b869f140bc78106d923a9ce337eb4aabf513a6
            Metadata for module libnuma.so.1 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "numactl",
            	"version" : "2.0.16-2.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libsctp.so.1 with build-id: 63f4589f2de2c90e478723bf58edea02ad69ce2c
            Metadata for module libsctp.so.1 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "lksctp-tools",
            	"version" : "1.0.19-3.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libfmt.so.9 with build-id: d6a5c60746b90af0b091919a4a59857b3b0b5afc
            Metadata for module libfmt.so.9 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "fmt",
            	"version" : "9.1.0-2.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libcryptopp.so.8 with build-id: 7ffca42977be17e283a6a06996c54470babe89c2
            Found module libcares.so.2 with build-id: 8d2383158a726d417556f5c4b8b68940f9cc1e17
            Metadata for module libcares.so.2 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "c-ares",
            	"version" : "1.19.1-1.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libboost_system.so.1.78.0 with build-id: a29ba75cbae4feda7899a221c2631d5edcb5cbcf
            Metadata for module libboost_system.so.1.78.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "boost",
            	"version" : "1.78.0-11.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libboost_thread.so.1.78.0 with build-id: e822f4347e8baee1c7064ffdfd3ca3753f51db74
            Metadata for module libboost_thread.so.1.78.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "boost",
            	"version" : "1.78.0-11.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module libboost_program_options.so.1.78.0 with build-id: 731ced70fa56f31c78105a891f2d0601f5c28a11
            Metadata for module libboost_program_options.so.1.78.0 owned by FDO found: {
            	"type" : "rpm",
            	"name" : "boost",
            	"version" : "1.78.0-11.fc38",
            	"architecture" : "x86_64",
            	"osCpe" : "cpe:/o:fedoraproject:fedora:38"
            }
            
            Found module scylla with build-id: f44ecb6140c40cc466146304eb075258461d5dd7
            Stack trace of thread 6543:
            #0  0x00007fe7dcc98844 __pthread_kill_implementation (libc.so.6 + 0x8e844)
            scylladb/scylladb#1  0x00007fe7dcc47abe raise (libc.so.6 + 0x3dabe)
            scylladb/scylladb#2  0x00007fe7dcc3087f abort (libc.so.6 + 0x2687f)
            scylladb/scylladb#3  0x0000000001b4ddfa _ZZN7replica5table20seal_active_memtableERNS_16compaction_groupEONS_12flush_permitEENK3$_0clESt8functionIFN7seastar6futureIvEEvEE.resume (scylla + 0x194ddfa)
            scylladb/scylladb#4  0x00000000012fa74b _ZN7seastar8internal21coroutine_traits_baseIvE12promise_type15run_and_disposeEv (scylla + 0x10fa74b)
            scylladb/scylladb#5  0x00000000058f6770 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x56f6770)
            scylladb/scylladb#6  0x00000000058f79a8 _ZN7seastar7reactor6do_runEv (scylla + 0x56f79a8)
            scylladb/scylladb#7  0x000000000591aa74 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x571aa74)
            scylladb/scylladb#8  0x00000000058c6f5b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x56c6f5b)
            scylladb/scylladb#9  0x00007fe7dcc96907 start_thread (libc.so.6 + 0x8c907)
            scylladb/scylladb#10 0x00007fe7dcd1c870 __clone3 (libc.so.6 + 0x112870)
            
            Stack trace of thread 6550:
            #0  0x00007fe7dcd0b0fa read (libc.so.6 + 0x1010fa)
            scylladb/scylladb#1  0x000000000593d4f5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x573d4f5)
            scylladb/scylladb#2  0x000000000593d803 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x573d803)
            scylladb/scylladb#3  0x00000000058c6f5b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x56c6f5b)
            scylladb/scylladb#4  0x00007fe7dcc96907 start_thread (libc.so.6 + 0x8c907)
            scylladb/scylladb#5  0x00007fe7dcd1c870 __clone3 (libc.so.6 + 0x112870)
            
            Stack trace of thread 6547:
            #0  0x00007fe7dcd0b0fa read (libc.so.6 + 0x1010fa)
            scylladb/scylladb#1  0x000000000593d4f5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x573d4f5)
            scylladb/scylladb#2  0x000000000593d803 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x573d803)
            scylladb/scylladb#3  0x00000000058c6f5b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x56c6f5b)
            scylladb/scylladb#4  0x00007fe7dcc96907 start_thread (libc.so.6 + 0x8c907)
            scylladb/scylladb#5  0x00007fe7dcd1c870 __clone3 (libc.so.6 + 0x112870)
            
            Stack trace of thread 6549:
            #0  0x00007fe7dcd0b0fa read (libc.so.6 + 0x1010fa)
            scylladb/scylladb#1  0x000000000593d4f5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x573d4f5)
            scylladb/scylladb#2  0x000000000593d803 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x573d803)
            scylladb/scylladb#3  0x00000000058c6f5b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x56c6f5b)
            scylladb/scylladb#4  0x00007fe7dcc96907 start_thread (libc.so.6 + 0x8c907)
            scylladb/scylladb#5  0x00007fe7dcd1c870 __clone3 (libc.so.6 + 0x112870)
            
            Stack trace of thread 6548:
            #0  0x00007fe7dcd0b0fa read (libc.so.6 + 0x1010fa)
            scylladb/scylladb#1  0x000000000593d4f5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x573d4f5)
            scylladb/scylladb#2  0x000000000593d803 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x573d803)
            scylladb/scylladb#3  0x00000000058c6f5b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x56c6f5b)
            scylladb/scylladb#4  0x00007fe7dcc96907 start_thread (libc.so.6 + 0x8c907)
            scylladb/scylladb#5  0x00007fe7dcd1c870 __clone3 (libc.so.6 + 0x112870)
            
            Stack trace of thread 6546:
            #0  0x00007fe7dcd0b0fa read (libc.so.6 + 0x1010fa)
            scylladb/scylladb#1  0x000000000593d4f5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x573d4f5)
            scylladb/scylladb#2  0x000000000593d803 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x573d803)
            scylladb/scylladb#3  0x00000000058c6f5b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x56c6f5b)
            scylladb/scylladb#4  0x00007fe7dcc96907 start_thread (libc.so.6 + 0x8c907)
            scylladb/scylladb#5  0x00007fe7dcd1c870 __clone3 (libc.so.6 + 0x112870)
            
            Stack trace of thread 6545:
            #0  0x00007fe7dcd0b0fa read (libc.so.6 + 0x1010fa)
            scylladb/scylladb#1  0x000000000593d4f5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x573d4f5)
            scylladb/scylladb#2  0x000000000593d803 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x573d803)
            scylladb/scylladb#3  0x00000000058c6f5b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x56c6f5b)
            scylladb/scylladb#4  0x00007fe7dcc96907 start_thread (libc.so.6 + 0x8c907)
            scylladb/scylladb#5  0x00007fe7dcd1c870 __clone3 (libc.so.6 + 0x112870)
            
            Stack trace of thread 6544:
            #0  0x00007fe7dcd0b0fa read (libc.so.6 + 0x1010fa)
            scylladb/scylladb#1  0x000000000593d4f5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x573d4f5)
            scylladb/scylladb#2  0x000000000593d803 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x573d803)
            scylladb/scylladb#3  0x00000000058c6f5b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x56c6f5b)
            scylladb/scylladb#4  0x00007fe7dcc96907 start_thread (libc.so.6 + 0x8c907)
            scylladb/scylladb#5  0x00007fe7dcd1c870 __clone3 (libc.so.6 + 0x112870)
            
            Stack trace of thread 6541:
            #0  0x00007fe7dcd14b5d syscall (libc.so.6 + 0x10ab5d)
            scylladb/scylladb#1  0x0000000005943391 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x5743391)
            scylladb/scylladb#2  0x000000000593f103 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x573f103)
            scylladb/scylladb#3  0x000000000593f82d _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x573f82d)
            scylladb/scylladb#4  0x00000000058f7c9d _ZN7seastar7reactor6do_runEv (scylla + 0x56f7c9d)
            scylladb/scylladb#5  0x000000000591aa74 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x571aa74)
            scylladb/scylladb#6  0x00000000058c6f5b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x56c6f5b)
            scylladb/scylladb#7  0x00007fe7dcc96907 start_thread (libc.so.6 + 0x8c907)
            scylladb/scylladb#8  0x00007fe7dcd1c870 __clone3 (libc.so.6 + 0x112870)
            
            Stack trace of thread 6542:
            #0  0x00007fe7dcd14b5d syscall (libc.so.6 + 0x10ab5d)
            scylladb/scylladb#1  0x0000000005943391 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x5743391)
            scylladb/scylladb#2  0x000000000593f103 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x573f103)
            scylladb/scylladb#3  0x000000000593f82d _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x573f82d)
            scylladb/scylladb#4  0x00000000058f7c9d _ZN7seastar7reactor6do_runEv (scylla + 0x56f7c9d)
            scylladb/scylladb#5  0x000000000591aa74 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x571aa74)
            scylladb/scylladb#6  0x00000000058c6f5b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x56c6f5b)
            scylladb/scylladb#7  0x00007fe7dcc96907 start_thread (libc.so.6 + 0x8c907)
            scylladb/scylladb#8  0x00007fe7dcd1c870 __clone3 (libc.so.6 + 0x112870)
            
            Stack trace of thread 6537:
            #0  0x00007fe7dcc98844 __pthread_kill_implementation (libc.so.6 + 0x8e844)
            scylladb/scylladb#1  0x00007fe7dcc47abe raise (libc.so.6 + 0x3dabe)
            scylladb/scylladb#2  0x00007fe7dcc3090f abort (libc.so.6 + 0x2690f)
            scylladb/scylladb#3  0x0000000001b4ddfa _ZZN7replica5table20seal_active_memtableERNS_16compaction_groupEONS_12flush_permitEENK3$_0clESt8functionIFN7seastar6futureIvEEvEE.resume (scylla + 0x194ddfa)
            scylladb/scylladb#4  0x00000000012fa74b _ZN7seastar8internal21coroutine_traits_baseIvE12promise_type15run_and_disposeEv (scylla + 0x10fa74b)
            scylladb/scylladb#5  0x00000000058f6770 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x56f6770)
            scylladb/scylladb#6  0x00000000058f79a8 _ZN7seastar7reactor6do_runEv (scylla + 0x56f79a8)
            scylladb/scylladb#7  0x000000000591aa74 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x571aa74)
            scylladb/scylladb#8  0x00000000058c6f5b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x56c6f5b)
            scylladb/scylladb#9  0x00007fe7dcc96907 start_thread (libc.so.6 + 0x8c907)
            scylladb/scylladb#10 0x00007fe7dcd1c870 __clone3 (libc.so.6 + 0x112870)
            
            Stack trace of thread 6539:
            #0  0x00007fe7dcd14b5d syscall (libc.so.6 + 0x10ab5d)
            scylladb/scylladb#1  0x0000000005943391 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x5743391)
            scylladb/scylladb#2  0x000000000593f103 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x573f103)
            scylladb/scylladb#3  0x000000000593f82d _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x573f82d)
            scylladb/scylladb#4  0x00000000058f7c9d _ZN7seastar7reactor6do_runEv (scylla + 0x56f7c9d)
            scylladb/scylladb#5  0x000000000591aa74 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x571aa74)
            scylladb/scylladb#6  0x00000000058c6f5b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x56c6f5b)
            scylladb/scylladb#7  0x00007fe7dcc96907 start_thread (libc.so.6 + 0x8c907)
            scylladb/scylladb#8  0x00007fe7dcd1c870 __clone3 (libc.so.6 + 0x112870)
            
            Stack trace of thread 6540:
            #0  0x00007fe7dcd14b5d syscall (libc.so.6 + 0x10ab5d)
            scylladb/scylladb#1  0x0000000005943391 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x5743391)
            scylladb/scylladb#2  0x000000000593f103 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x573f103)
            scylladb/scylladb#3  0x000000000593f82d _ZN7seastar19reactor_backend_aio23wait_and_process_eventsEPK10__sigset_t (scylla + 0x573f82d)
            scylladb/scylladb#4  0x00000000058f7c9d _ZN7seastar7reactor6do_runEv (scylla + 0x56f7c9d)
            scylladb/scylladb#5  0x000000000591aa74 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x571aa74)
            scylladb/scylladb#6  0x00000000058c6f5b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x56c6f5b)
            scylladb/scylladb#7  0x00007fe7dcc96907 start_thread (libc.so.6 + 0x8c907)
            scylladb/scylladb#8  0x00007fe7dcd1c870 __clone3 (libc.so.6 + 0x112870)
            
            Stack trace of thread 6536:
            #0  0x0000000003d5afbd _ZNK7locator15simple_strategy27calculate_natural_endpointsERKN3dht5tokenERKNS_14token_metadataE (scylla + 0x3b5afbd)
            scylladb/scylladb#1  0x0000000003d01d22 _ZN7locator35calculate_effective_replication_mapEN7seastar10shared_ptrIKNS_29abstract_replication_strategyEEENS0_13lw_shared_ptrIKNS_14token_metadataEEE (scylla + 0x3b01d22)
            scylladb/scylladb#2  0x0000000003d0940a _ZN7locator33effective_replication_map_factory32create_effective_replication_mapEN7seastar10shared_ptrIKNS_29abstract_replication_strategyEEENS1_13lw_shared_ptrIKNS_14token_metadataEEE (scylla + 0x3b0940a)
            scylladb/scylladb#3  0x0000000003fa3a9c _ZN7service15storage_service22replicate_to_all_coresEN7seastar13lw_shared_ptrIN7locator14token_metadataEEE.resume (scylla + 0x3da3a9c)
            scylladb/scylladb#4  0x00000000012fa74b _ZN7seastar8internal21coroutine_traits_baseIvE12promise_type15run_and_disposeEv (scylla + 0x10fa74b)
            scylladb/scylladb#5  0x00000000058f6770 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x56f6770)
            scylladb/scylladb#6  0x00000000058f79a8 _ZN7seastar7reactor6do_runEv (scylla + 0x56f79a8)
            scylladb/scylladb#7  0x00000000058f6d1a _ZN7seastar7reactor3runEv (scylla + 0x56f6d1a)
            scylladb/scylladb#8  0x000000000589a298 _ZN7seastar12app_template14run_deprecatedEiPPcOSt8functionIFvvEE (scylla + 0x569a298)
            scylladb/scylladb#9  0x000000000589944d _ZN7seastar12app_template3runEiPPcOSt8functionIFNS_6futureIiEEvEE (scylla + 0x569944d)
            scylladb/scylladb#10 0x000000000127f200 _ZL11scylla_mainiPPc (scylla + 0x107f200)
            scylladb/scylladb#11 0x0000000001280c71 _ZNKSt8functionIFiiPPcEEclEiS1_ (scylla + 0x1080c71)
            scylladb/scylladb#12 0x000000000127d78f main (scylla + 0x107d78f)
            scylladb/scylladb#13 0x00007fe7dcc31b4a __libc_start_call_main (libc.so.6 + 0x27b4a)
            scylladb/scylladb#14 0x00007fe7dcc31c0b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x27c0b)
            scylladb/scylladb#15 0x000000000127b525 _start (scylla + 0x107b525)

download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.112.5ac4fbd29e794a1098f9309538996d99.6536.1693639209000000/core.scylla.112.5ac4fbd29e794a1098f9309538996d99.6536.1693639209000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.112.5ac4fbd29e794a1098f9309538996d99.6536.1693639209000000.gz

avikivity · 2023-09-03T14:49:04Z

@bhalevy looks like a problem with the sstable creation/deletion protocol

bhalevy · 2023-09-03T15:43:15Z

@bhalevy looks like a problem with the sstable creation/deletion protocol

The following is non-recoverable by design:

2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10      !ERR | scylla[6536]:  [shard 6] table - failed to write sstable /var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/mc-48-big-Data.db: std::filesystem::__cxx11::filesystem_error (filesystem error: mkdir failed: No such file or directory [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/48.sstable])

@aleksbykov Maybe some other process is clearing the node while scylla is (re)starting?

aleksbykov · 2023-09-03T16:15:48Z

I will check

aleksbykov · 2023-09-04T03:48:02Z

@bhalevy , didn't find anything what could clean node. In parallel on another node, was running next cql

ALTER TABLE  scylla_bench.test WITH tombstone_gc = {'mode': 'timeout'};

bhalevy · 2023-09-04T08:53:43Z

This happend on 2 nodes at different times:

bhalevy@lt db-cluster-7f42d685$ grep 'failed to write' */messages.log 
longevity-parallel-topology-schema--db-node-7f42d685-10/messages.log:2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10      !ERR | scylla[6536]:  [shard 6] table - failed to write sstable /var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/mc-48-big-Data.db: std::filesystem::__cxx11::filesystem_error (filesystem error: mkdir failed: No such file or directory [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/48.sstable])
longevity-parallel-topology-schema--db-node-7f42d685-10/messages.log:2023-09-02T07:20:09+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10      !ERR | scylla[6536]:  [shard 1] table - failed to write sstable /var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/mc-43-big-Data.db: std::filesystem::__cxx11::filesystem_error (filesystem error: mkdir failed: No such file or directory [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/43.sstable])

longevity-parallel-topology-schema--db-node-7f42d685-16/messages.log:2023-09-02T11:05:02+00:00 longevity-parallel-topology-schema--db-node-7f42d685-16      !ERR | scylla[6588]:  [shard 6] table - failed to write sstable /var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/mc-83-big-Data.db: std::filesystem::__cxx11::filesystem_error (filesystem error: mkdir failed: No such file or directory [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/83.sstable])

This is weird... The directory seems ok after restart, e.g. on node-10:

2023-09-02T07:22:28+00:00 longevity-parallel-topology-schema--db-node-7f42d685-10     !INFO | scylla[6982]:  [shard 4] compaction - [Compact system.peers 78ae7aa0-4961-11ee-8fdf-212b0846c2ff] Compacted 2 sstables to [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/mc-25-big-Data.db:level=0]. 23kB to 30kB (~129% of original) in 5ms = 4MB/s. ~256 total partitions merged to 2.

and on node-16:

2023-09-02T11:06:46+00:00 longevity-parallel-topology-schema--db-node-7f42d685-16     !INFO | scylla[6987]:  [shard 6] compaction - [Compact system.peers cea586a0-4980-11ee-a9df-c8d0628688cd] Compacted 2 sstables to [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/mc-27-big-Data.db:level=0]. 17kB to 11kB (~68% of original) in 5ms = 3MB/s. ~256 total partitions merged to 1.

bhalevy · 2023-09-04T09:07:34Z

@xemul please look into this.
Maybe something got broken with the introduction of the sstables storage layer.

aleksbykov · 2023-09-07T08:08:14Z

Issue reproduced with:

Installation details

Kernel Version: 5.15.0-1043-aws
Scylla version (or git commit hash): 5.4.0~dev-20230901.3bdbe620aa1e with build-id f44ecb6140c40cc466146304eb075258461d5dd7

Cluster size: 4 nodes (i3en.2xlarge)

Scylla Nodes used in this run:

longevity-twcs-48h-master-db-node-8c749234-9 (54.171.26.141 | 10.4.10.80) (shards: -1)
longevity-twcs-48h-master-db-node-8c749234-8 (3.249.144.192 | 10.4.11.28) (shards: 7)
longevity-twcs-48h-master-db-node-8c749234-7 (3.248.202.249 | 10.4.9.159) (shards: -1)
longevity-twcs-48h-master-db-node-8c749234-6 (3.249.171.168 | 10.4.9.97) (shards: 7)
longevity-twcs-48h-master-db-node-8c749234-5 (52.214.222.105 | 10.4.8.248) (shards: 7)
longevity-twcs-48h-master-db-node-8c749234-4 (54.78.226.233 | 10.4.8.124) (shards: 7)
longevity-twcs-48h-master-db-node-8c749234-3 (52.17.223.248 | 10.4.8.165) (shards: 7)
longevity-twcs-48h-master-db-node-8c749234-2 (34.245.90.3 | 10.4.10.220) (shards: 7)
longevity-twcs-48h-master-db-node-8c749234-10 (34.247.53.248 | 10.4.10.1) (shards: -1)
longevity-twcs-48h-master-db-node-8c749234-1 (3.248.203.238 | 10.4.10.85) (shards: 7)

OS / Image: ami-0dd654338b4cc2a70 (aws: undefined_region)

Test: longevity-twcs-48h-test
Test id: 8c749234-ddb2-436e-a1d9-bcebe225ac92
Test name: scylla-master/longevity/longevity-twcs-48h-test
Test config file(s):

longevity-twcs-48h.yaml

Logs and commands

Restore Monitor Stack command: $ hydra investigate show-monitor 8c749234-ddb2-436e-a1d9-bcebe225ac92
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs 8c749234-ddb2-436e-a1d9-bcebe225ac92

Logs:

db-cluster-8c749234.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/8c749234-ddb2-436e-a1d9-bcebe225ac92/20230903_035540/db-cluster-8c749234.tar.gz
sct-runner-events-8c749234.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/8c749234-ddb2-436e-a1d9-bcebe225ac92/20230903_035540/sct-runner-events-8c749234.tar.gz
sct-8c749234.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/8c749234-ddb2-436e-a1d9-bcebe225ac92/20230903_035540/sct-8c749234.log.tar.gz
loader-set-8c749234.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/8c749234-ddb2-436e-a1d9-bcebe225ac92/20230903_035540/loader-set-8c749234.tar.gz
monitor-set-8c749234.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/8c749234-ddb2-436e-a1d9-bcebe225ac92/20230903_035540/monitor-set-8c749234.tar.gz

Jenkins job URL
Argus

fruch · 2023-11-28T14:53:58Z

happened again (you can find the cordump full details in agrus)

2023-11-25 10:34:44.887 <2023-11-25 10:34:44.239>: (DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=8821a13a-bf62-49df-8518-5921699851b8 during_nemesis=CreateIndex,BootstrapStreamingError: type=FILESYSTEM_ERROR regex=filesystem_error line_number=2673 node=longevity-parallel-topology-schema--db-node-ec71f603-14
2023-11-25T10:34:44.239+00:00 longevity-parallel-topology-schema--db-node-ec71f603-14      !ERR | scylla[7021]:  [shard 0:main] table - Memtable flush failed due to: std::filesystem::__cxx11::filesystem_error (error system:2, filesystem error: mkdir failed: No such file or directory [/var/lib/scylla/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/3gbd_0tdv_5ch0g2dtvyb1g91ov9.sstable]). Aborting, at 0x604231e 0x60428e0 0x6042bb8 0x1c1adb6 0x1396b4a 0x5b3e25f 0x5b3f537 0x5b3e8a9 0x5adf977 0x5adeb2c 0x1319baa 0x131b600 0x131813c /opt/scylladb/libreloc/libc.so.6+0x27b89 /opt/scylladb/libreloc/libc.so.6+0x27c4a 0x1315b64

Installation details

Kernel Version: 5.15.0-1050-aws
Scylla version (or git commit hash): 5.5.0~dev-20231122.65e42e4166ce with build-id 44721ca3535e729f9dd7dee34b519bfc91b8c564

Cluster size: 5 nodes (i4i.2xlarge)

Scylla Nodes used in this run:

longevity-parallel-topology-schema--db-node-ec71f603-9 (34.249.99.31 | 10.4.8.191) (shards: 7)
longevity-parallel-topology-schema--db-node-ec71f603-8 (34.248.132.107 | 10.4.10.172) (shards: 7)
longevity-parallel-topology-schema--db-node-ec71f603-7 (3.254.94.27 | 10.4.8.236) (shards: 7)
longevity-parallel-topology-schema--db-node-ec71f603-6 (54.74.237.45 | 10.4.10.205) (shards: 7)
longevity-parallel-topology-schema--db-node-ec71f603-5 (3.249.107.73 | 10.4.8.228) (shards: 7)
longevity-parallel-topology-schema--db-node-ec71f603-4 (52.30.40.185 | 10.4.11.143) (shards: 7)
longevity-parallel-topology-schema--db-node-ec71f603-3 (3.253.81.188 | 10.4.8.184) (shards: 7)
longevity-parallel-topology-schema--db-node-ec71f603-2 (3.254.196.7 | 10.4.8.182) (shards: 7)
longevity-parallel-topology-schema--db-node-ec71f603-16 (34.246.162.192 | 10.4.10.4) (shards: -1)
longevity-parallel-topology-schema--db-node-ec71f603-15 (46.51.163.229 | 10.4.11.90) (shards: 7)
longevity-parallel-topology-schema--db-node-ec71f603-14 (3.250.127.6 | 10.4.11.196) (shards: -1)
longevity-parallel-topology-schema--db-node-ec71f603-13 (54.216.178.195 | 10.4.10.73) (shards: 7)
longevity-parallel-topology-schema--db-node-ec71f603-12 (34.244.253.224 | 10.4.9.140) (shards: -1)
longevity-parallel-topology-schema--db-node-ec71f603-11 (34.242.166.189 | 10.4.9.68) (shards: 7)
longevity-parallel-topology-schema--db-node-ec71f603-10 (54.78.185.17 | 10.4.9.76) (shards: 7)
longevity-parallel-topology-schema--db-node-ec71f603-1 (52.18.250.144 | 10.4.10.82) (shards: 7)

OS / Image: ami-0a28a97891191910e (aws: undefined_region)

Test: longevity-schema-topology-changes-12h-test
Test id: ec71f603-026b-4d4c-91d1-0f14a3045454
Test name: scylla-master/longevity/longevity-schema-topology-changes-12h-test
Test config file(s):

longevity-parallel-schema-changes-12h.yaml

Logs and commands

Restore Monitor Stack command: $ hydra investigate show-monitor ec71f603-026b-4d4c-91d1-0f14a3045454
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs ec71f603-026b-4d4c-91d1-0f14a3045454

Logs:

db-cluster-ec71f603.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/ec71f603-026b-4d4c-91d1-0f14a3045454/20231125_122029/db-cluster-ec71f603.tar.gz
sct-runner-events-ec71f603.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/ec71f603-026b-4d4c-91d1-0f14a3045454/20231125_122029/sct-runner-events-ec71f603.tar.gz
sct-ec71f603.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/ec71f603-026b-4d4c-91d1-0f14a3045454/20231125_122029/sct-ec71f603.log.tar.gz
loader-set-ec71f603.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/ec71f603-026b-4d4c-91d1-0f14a3045454/20231125_122029/loader-set-ec71f603.tar.gz
monitor-set-ec71f603.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/ec71f603-026b-4d4c-91d1-0f14a3045454/20231125_122029/monitor-set-ec71f603.tar.gz

Jenkins job URL
Argus

fruch · 2023-11-28T14:54:57Z

@xemul did you ever looked at this one ?

bhalevy · 2023-11-28T16:36:19Z

@fruch it looks like the bug is in SCT:

sct-ec71f603.log:

< t:2023-11-25 10:34:17,311 f:nemesis.py      l:4937 c:sdcm.nemesis         p:DEBUG > sdcm.nemesis.SisyphusMonkey: Clean old scylla data and restart scylla service
< t:2023-11-25 10:34:17,311 f:cluster.py      l:1811 c:sdcm.cluster_aws     p:DEBUG > Node longevity-parallel-topology-schema--db-node-ec71f603-14 [3.250.127.6 | 10.4.11.196] (seed: False): Clean all files from scylla data dirs
< t:2023-11-25 10:34:17,311 f:remote_base.py  l:521  c:RemoteLibSSH2CmdRunner p:DEBUG > Running command "sudo rm -rf /var/lib/scylla/data/*"...
...
< t:2023-11-25 10:34:44,884 f:db_log_reader.py l:114  c:sdcm.db_log_reader   p:DEBUG > 2023-11-25T10:34:44.239+00:00 longevity-parallel-topology-schema--db-node-ec71f603-14      !ERR | scylla[7021]:  [shard 0:main] table - Memtable flush failed due to: std::filesystem::__cxx11::filesystem_error (error system:2, filesystem error: mkdir failed: No such file or directory [/var/lib/scylla/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/3gbd_0tdv_5ch0g2dtvyb1g91ov9.sstable]). Aborting, at 0x604231e 0x60428e0 0x6042bb8 0x1c1adb6 0x1396b4a 0x5b3e25f 0x5b3f537 0x5b3e8a9 0x5adf977 0x5adeb2c 0x1319baa 0x131b600 0x131813c /opt/scylladb/libreloc/libc.so.6+0x27b89 /opt/scylladb/libreloc/libc.so.6+0x27c4a 0x1315b64

This is in disrupt_bootstrap_streaming_error

        if not new_node.db_up():
            try:
                if self.target_node.raft.get_diff_group0_token_ring_members():
                    self.target_node.raft.clean_group0_garbage(raise_exception=True)
                    self.log.debug("Clean old scylla data and restart scylla service")
                    new_node.clean_scylla_data()

bhalevy · 2023-11-28T16:36:58Z

@mykaul can you please move this issue to the scylla-cluster-tests repo?

fruch · 2023-11-28T16:54:50Z

@aleksbykov @temichus, can you look into this one ?

aleksbykov · 2023-11-29T09:03:40Z

This could happened, because scylla was restarted after stopping with scylla-manager client. In PR #6804 it should be fixed. There scylla is stopped before any next operation.

bhalevy · 2023-12-18T08:05:41Z

Hit again in enterprise-2024.1/SCT_Enterprise_Features/FIPS/longevity-100gb-4h-fips-test#5
With 2024.1.0~rc2-20231217.f57117d9cfe3

fruch · 2023-12-18T08:14:19Z

Hit again in enterprise-2024.1/SCT_Enterprise_Features/FIPS/longevity-100gb-4h-fips-test#5 With 2024.1.0~rc2-20231217.f57117d9cfe3

@aleksbykov

please take a look, #6804 is in, but still we have issues

aleksbykov · 2023-12-18T12:24:21Z

@fruch , we got coredump, which was the cause of sct error. I will prepare a patch for sct.

Some times ParallelObject, abort_action wait timeout could race, and when they happened at same moment, restart scylla-service by scylla-manager in node_setup method could raise issue scylladb#6865 Set correct timeouts for operations and fix minor mistypes

aleksbykov · 2023-12-25T10:53:16Z

PR: #7004

fruch · 2024-01-01T20:29:37Z

fixed by #7004

mykaul assigned bhalevy Sep 3, 2023

mykaul added the master/triage label Sep 3, 2023

bhalevy assigned xemul Sep 4, 2023

bhalevy removed the master/triage label Sep 4, 2023

mykaul transferred this issue from scylladb/scylladb Nov 28, 2023

github-actions bot assigned aleksbykov Nov 28, 2023

bhalevy unassigned xemul and bhalevy Nov 28, 2023

fruch closed this as completed Jan 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: table - Memtable flush failed due to: std::filesystem::__cxx11::filesystem_error and coredump during second bootstrap after first was aborted #6865

Error: table - Memtable flush failed due to: std::filesystem::__cxx11::filesystem_error and coredump during second bootstrap after first was aborted #6865

aleksbykov commented Sep 3, 2023

Logs:

aleksbykov commented Sep 3, 2023

avikivity commented Sep 3, 2023

bhalevy commented Sep 3, 2023

aleksbykov commented Sep 3, 2023

aleksbykov commented Sep 4, 2023

bhalevy commented Sep 4, 2023

bhalevy commented Sep 4, 2023

aleksbykov commented Sep 7, 2023

Logs:

fruch commented Nov 28, 2023

Logs:

fruch commented Nov 28, 2023

bhalevy commented Nov 28, 2023

bhalevy commented Nov 28, 2023

fruch commented Nov 28, 2023

aleksbykov commented Nov 29, 2023

bhalevy commented Dec 18, 2023

fruch commented Dec 18, 2023

aleksbykov commented Dec 18, 2023

aleksbykov commented Dec 25, 2023

fruch commented Jan 1, 2024

Error: table - Memtable flush failed due to: std::filesystem::__cxx11::filesystem_error and coredump during second bootstrap after first was aborted #6865

Error: table - Memtable flush failed due to: std::filesystem::__cxx11::filesystem_error and coredump during second bootstrap after first was aborted #6865

Comments

aleksbykov commented Sep 3, 2023

Issue description

Impact

How frequently does it reproduce?

Installation details

Logs:

aleksbykov commented Sep 3, 2023

avikivity commented Sep 3, 2023

bhalevy commented Sep 3, 2023

aleksbykov commented Sep 3, 2023

aleksbykov commented Sep 4, 2023

bhalevy commented Sep 4, 2023

bhalevy commented Sep 4, 2023

aleksbykov commented Sep 7, 2023

Installation details

Logs:

fruch commented Nov 28, 2023

Installation details

Logs:

fruch commented Nov 28, 2023

bhalevy commented Nov 28, 2023

bhalevy commented Nov 28, 2023

fruch commented Nov 28, 2023

aleksbykov commented Nov 29, 2023

bhalevy commented Dec 18, 2023

fruch commented Dec 18, 2023

aleksbykov commented Dec 18, 2023

aleksbykov commented Dec 25, 2023

fruch commented Jan 1, 2024