Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug(storage): panics when file cache write buffer exceeds region size #19945

Closed
MrCroxx opened this issue Dec 26, 2024 · 1 comment
Closed

bug(storage): panics when file cache write buffer exceeds region size #19945

MrCroxx opened this issue Dec 26, 2024 · 1 comment
Assignees
@MrCroxx
Labels
type/bug Something isn't working
Milestone
release-2.3

Comments

@MrCroxx
Copy link
Contributor

MrCroxx commented Dec 26, 2024 •
edited
Loading

Describe the bug

If there is a single cache entry (meta / data block) whose size exceeds the file cache region size (file capacity), foyer will panic with the following logs.

This is caused by a bug in foyer (version < 0.11).

FYI: https://docs.rs/foyer-storage/0.9.3/src/foyer_storage/large/flusher.rs.html#403-420

There are some ways to fix/mitigate the bug:

  1. Set larger file_capacity for the file cache. (mitigate)
  2. Upgrade to RisingWave version >= 2.1 (fix)

Error message/log

thread 'foyer.meta.runtime' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/device/direct_fs.rs:160:9:
offset (0) + aligned (109027328) = total (109027328) <= region size (67108864)
stack backtrace:
   0:     0x5805bb724199 - std::backtrace_rs::backtrace::libunwind::trace::hb2aae96f7a63ad27
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1:     0x5805bb724199 - std::backtrace_rs::backtrace::trace_unsynchronized::h21f79cd140dff600
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x5805bb724199 - std::sys_common::backtrace::_print_fmt::hc8ebdc1b64ae52ee
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/sys_common/backtrace.rs:68:5
   3:     0x5805bb724199 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h7ba491a968dfae1d
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x5805aef0553b - core::fmt::rt::Argument::fmt::he322040f856802e1
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/fmt/rt.rs:165:63
   5:     0x5805aef0553b - core::fmt::write::hb4f7da2a314ef133
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/fmt/mod.rs:1168:21
   6:     0x5805bb71ce1f - std::io::Write::write_fmt::h9e79d11af9b4686b
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/io/mod.rs:1835:15
   7:     0x5805bb723f6e - std::sys_common::backtrace::_print::h9276d72631ea2ebd
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x5805bb723f6e - std::sys_common::backtrace::print::h301edb80f46c58f8
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x5805bb7252d9 - std::panicking::default_hook::{{closure}}::h08c68fc043d12d43
  10:     0x5805bb725085 - std::panicking::default_hook::hdf65558cafad5039
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:298:9
  11:     0x5805b8b846ab - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hdc9eda93008da126
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/alloc/src/boxed.rs:2077:9
  12:     0x5805b8b846ab - risingwave_rt::panic_hook::set_panic_hook::{{closure}}::h40b35e75a058830c
                               at /risingwave/src/utils/runtime/src/panic_hook.rs:25:9
  13:     0x5805b8b846ab - std::panicking::update_hook::{{closure}}::h68c4945031a81df0
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:233:47
  14:     0x5805bb72591b - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h69f90a497fd344b0
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/alloc/src/boxed.rs:2077:9
  15:     0x5805bb72591b - std::panicking::rust_panic_with_hook::h9576f09a7a058c94
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:799:13
  16:     0x5805bb7256c4 - std::panicking::begin_panic_handler::{{closure}}::hd8f56243f630c203
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:664:13
  17:     0x5805bb724659 - std::sys_common::backtrace::__rust_end_short_backtrace::h5bed6f770a09acf8
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/sys_common/backtrace.rs:171:18
  18:     0x5805bb725407 - rust_begin_unwind
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:652:5
  19:     0x5805aef01a93 - core::panicking::panic_fmt::h088f9f84f0369284
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/panicking.rs:72:14
  20:     0x5805b2a3704a - <foyer_storage::device::direct_fs::DirectFsDevice as foyer_storage::device::Device>::write::{{closure}}::{{closure}}::hd027d3ffc6ebabcc
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/device/direct_fs.rs:160:9
  21:     0x5805b2a3704a - <minitrace::future::InSpan<T> as core::future::future::Future>::poll::h93fd5b0a49d7e5c2
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/minitrace-0.6.7/src/future.rs:125:19
  22:     0x5805b2a3704a - <foyer_storage::device::direct_fs::DirectFsDevice as foyer_storage::device::Device>::write::{{closure}}::h641d96d46d4a4c00
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/device/direct_fs.rs:153:91
  23:     0x5805b2a3704a - foyer_storage::device::monitor::Monitored<D>::write::{{closure}}::{{closure}}::h766e6644e1188c43
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/device/monitor.rs:108:58
  24:     0x5805b2a3704a - <minitrace::future::InSpan<T> as core::future::future::Future>::poll::hd20f4d894e079f50
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/minitrace-0.6.7/src/future.rs:125:19
  25:     0x5805b2d3821c - foyer_storage::device::monitor::Monitored<D>::write::{{closure}}::ha10b685939f91bc7
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/device/monitor.rs:101:87
  26:     0x5805b2d3821c - <foyer_storage::device::monitor::Monitored<D> as foyer_storage::device::Device>::write::{{closure}}::h5c528bd20c10a373
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/device/monitor.rs:168:41
  27:     0x5805b2d3821c - foyer_storage::region::Region<D>::write::{{closure}}::hacf3e7c66f73a257
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/region.rs:92:49
  28:     0x5805b2d2ec86 - foyer_storage::large::flusher::Runner<K,V,S,D>::commit::{{closure}}::{{closure}}::{{closure}}::hdde703ede8bee135
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/large/flusher.rs:550:73
  29:     0x5805b2d2ec86 - <F as futures_core::future::TryFuture>::try_poll::h688d2e599e249d24
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
  30:     0x5805b2d2ec86 - <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll::h40c15360cce59e3e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_future/into_future.rs:34:9
  31:     0x5805b2d2863b - <F as futures_core::future::TryFuture>::try_poll::ha3cb528550b4d8e1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
  32:     0x5805b2d2863b - <futures_util::future::try_maybe_done::TryMaybeDone<Fut> as core::future::future::Future>::poll::h087456788ff0b31c
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_maybe_done.rs:79:57
  33:     0x5805b2d2863b - <F as futures_core::future::TryFuture>::try_poll::he3c3582ad2f8067a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
  34:     0x5805b2d2863b - <futures_util::future::try_join_all::TryJoinAll<F> as core::future::future::Future>::poll::hb78726d40b56efd2
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_join_all.rs:163:32
  35:     0x5805b2d2863b - <F as futures_core::future::TryFuture>::try_poll::h60e1b26fcd1fd65a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
  36:     0x5805b2d2863b - <futures_util::future::try_maybe_done::TryMaybeDone<Fut> as core::future::future::Future>::poll::h52c043a0c00259b5
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_maybe_done.rs:79:57
  37:     0x5805b2d2863b - <futures_util::future::try_join::TryJoin<Fut1,Fut2> as core::future::future::Future>::poll::hca620dd278963720
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_join.rs:72:29
  38:     0x5805b28b161b - foyer_storage::large::flusher::Runner<K,V,S,D>::commit::{{closure}}::h34e41c670f7ae68f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/large/flusher.rs:595:53
  39:     0x5805b28b161b - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::ha76e80b923ca613f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  40:     0x5805b2907552 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hf647fdf85c48e860
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/core.rs:328:17
  41:     0x5805b2907552 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h1d713b24aaadf9bd
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/loom/std/unsafe_cell.rs:16:9
  42:     0x5805b2907552 - tokio::runtime::task::core::Core<T,S>::poll::h693cf411a4f4b88e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/core.rs:317:30
  43:     0x5805b2921aff - tokio::runtime::task::harness::poll_future::{{closure}}::h42f008a4ffe8f683
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:485:19
  44:     0x5805b2921aff - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h98d99a00562a52b7
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/panic/unwind_safe.rs:272:9
  45:     0x5805b2921aff - std::panicking::try::do_call::h735840178f7ed098
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:559:40
  46:     0x5805b2921aff - std::panicking::try::hc34af7bbb0479483
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:523:19
  47:     0x5805b2921aff - std::panic::catch_unwind::hb91f735fe53752a6
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panic.rs:149:14
  48:     0x5805b2921aff - tokio::runtime::task::harness::poll_future::h01b241699dee01bd
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:473:18
  49:     0x5805b2921aff - tokio::runtime::task::harness::Harness<T,S>::poll_inner::he86eea8bc69eeb0b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:208:27
  50:     0x5805b2921aff - tokio::runtime::task::harness::Harness<T,S>::poll::h70bd050b31a85e73
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:153:15
  51:     0x5805bb7ad84b - tokio::runtime::task::raw::RawTask::poll::h626249af0491f4e3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/raw.rs:201:18
  52:     0x5805bb7ad84b - tokio::runtime::task::LocalNotified<S>::run::hb8addeee3a3c54c0
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/mod.rs:427:9
  53:     0x5805bb7ad84b - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h1c841ab6748b3bcf
2024-12-26T12:51:03.499129141Z  WARN risingwave_connector::parser: failed to parse non-pk column, padding with `NULL` error=Undefined field `pinned_tweet_id` at `` split_id="shardId-000000000106" offset="49657343697896857946042645181555228388015869701002364578" column="pinned_tweet_id" suppressed_count=21111
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:648:22
  54:     0x5805bb7ad84b - tokio::runtime::coop::with_budget::h0bbe3908baf78f4a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/coop.rs:107:5
  55:     0x5805bb7ad84b - tokio::runtime::coop::budget::ha88aed2bb84c970d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/coop.rs:73:5
  56:     0x5805bb7ad84b - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h3832fe87202eebc2
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:584:9
  57:     0x5805bb7a9785 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h27a10854a9d63bfa
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:535:24
  58:     0x5805bb7a9785 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::hf07c3ad4fe0cf498
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:500:21
  59:     0x5805bb7a9785 - tokio::runtime::context::scoped::Scoped<T>::set::h75da236f9d897da1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/context/scoped.rs:40:9
  60:     0x5805bb7ecc58 - tokio::runtime::context::set_scheduler::{{closure}}::h59b22c1a323eb811
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/context.rs:180:26
  61:     0x5805bb7ecc58 - std::thread::local::LocalKey<T>::try_with::hc20591cd664efd9d
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/thread/local.rs:283:12
  62:     0x5805bb7ecc58 - std::thread::local::LocalKey<T>::with::hd228c34d1db99eda
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/thread/local.rs:260:9
  63:     0x5805bb7ecc58 - tokio::runtime::context::set_scheduler::he6413f92dcfbb709
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/context.rs:180:17
  64:     0x5805bb7ecc58 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h90145e52818cfa51
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:495:9
  65:     0x5805bb7ecc58 - tokio::runtime::context::runtime::enter_runtime::h6e91b28ba394eb3e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/context/runtime.rs:65:16
  66:     0x5805bb7ad55c - tokio::runtime::scheduler::multi_thread::worker::run::he443bfcf26770424
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:487:5
  67:     0x5805bb7e3f11 - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::h673489d56cab43a8
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:455:45
  68:     0x5805bb7e3f11 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::hf52c40ec5bbb3240
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/blocking/task.rs:42:21
  69:     0x5805bb7d42af - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h4f8a96a8215e6853
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  70:     0x5805bb7e2776 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h231abc877973020e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/core.rs:328:17
  71:     0x5805bb7e2776 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h7d95a0441d1292fe
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/loom/std/unsafe_cell.rs:16:9
  72:     0x5805bb7e2776 - tokio::runtime::task::core::Core<T,S>::poll::hdd5e41a0f32ed5a0
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/core.rs:317:30
  73:     0x5805bb7c0372 - tokio::runtime::task::harness::poll_future::{{closure}}::h7cb22b0f9a0bc4a3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:485:19
  74:     0x5805bb7c0372 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hfed28fbab9b0a03e
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/panic/unwind_safe.rs:272:9
  75:     0x5805bb7c0372 - std::panicking::try::do_call::hb65cc09bf2bd4d7b
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:559:40
  76:     0x5805bb7c0372 - std::panicking::try::h7db0e518a9b8a97d
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:523:19
  77:     0x5805bb7c0372 - std::panic::catch_unwind::hbdd2d16e43372a1b
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panic.rs:149:14
  78:     0x5805bb7c0372 - tokio::runtime::task::harness::poll_future::hd81255e7b4d02094
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:473:18
  79:     0x5805bb7c0372 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::hc2f4bd3db966b047
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:208:27
  80:     0x5805bb7c0372 - tokio::runtime::task::harness::Harness<T,S>::poll::h4b451fd3e734c254
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:153:15
  81:     0x5805bb7b4192 - tokio::runtime::task::raw::RawTask::poll::h626249af0491f4e3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/raw.rs:201:18
  82:     0x5805bb7b4192 - tokio::runtime::task::UnownedTask<S>::run::h98beb39b303a29c5
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/mod.rs:464:9
  83:     0x5805bb7b4192 - tokio::runtime::blocking::pool::Task::run::hac573af663f13975
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/blocking/pool.rs:159:9
  84:     0x5805bb7b4192 - tokio::runtime::blocking::pool::Inner::run::had6d86a4109477f7
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/blocking/pool.rs:513:17
  85:     0x5805bb7bd500 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::hf78130f6419d7891
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/blocking/pool.rs:471:13
  86:     0x5805bb7bd500 - std::sys_common::backtrace::__rust_begin_short_backtrace::hbf1dc341a32fb6c8
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/sys_common/backtrace.rs:155:18
thread 'foyer.meta.runtime' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/device/direct_fs.rs:160:9:
offset (0) + aligned (72335360) = total (72335360) <= region size (67108864)
  87:     0x5805bb7c9a6a - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hebce08b4b59837b0
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/thread/mod.rs:542:17
  88:     0x5805bb7c9a6a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hd490dda97349681d
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/panic/unwind_safe.rs:272:9
  89:     0x5805bb7c9a6a - std::panicking::try::do_call::h577f074763d50b1d
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:559:40
  90:     0x5805bb7c9a6a - std::panicking::try::h83c2307555e8761b
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:523:19
  91:     0x5805bb7c9a6a - std::panic::catch_unwind::h102860fca5fc3899
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panic.rs:149:14
  92:     0x5805bb7c9a6a - std::thread::Builder::spawn_unchecked_::{{closure}}::hd27a3f18384072df
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/thread/mod.rs:541:30
  93:     0x5805bb7c9a6a - core::ops::function::FnOnce::call_once{{vtable.shim}}::h0a939909032743c8
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/ops/function.rs:250:5
  94:     0x5805bb72bb9b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hd9d8a12a4cec8653
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/alloc/src/boxed.rs:2063:9
  95:     0x5805bb72bb9b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc87e6ebe32d18c90
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/alloc/src/boxed.rs:2063:9
  96:     0x5805bb72bb9b - std::sys::pal::unix::thread::Thread::new::thread_start::h68df5163d04f9130
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/sys/pal/unix/thread.rs:108:17
  97:     0x7cf433d21a94 - start_thread
                               at ./nptl/pthread_create.c:447:8
  98:     0x7cf433daec3c - __GI___clone3
                               at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
  99:                0x0 - <unknown>
stack backtrace:
   0:     0x5805bb724199 - std::backtrace_rs::backtrace::libunwind::trace::hb2aae96f7a63ad27
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1:     0x5805bb724199 - std::backtrace_rs::backtrace::trace_unsynchronized::h21f79cd140dff600
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x5805bb724199 - std::sys_common::backtrace::_print_fmt::hc8ebdc1b64ae52ee
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/sys_common/backtrace.rs:68:5
   3:     0x5805bb724199 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h7ba491a968dfae1d
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x5805aef0553b - core::fmt::rt::Argument::fmt::he322040f856802e1
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/fmt/rt.rs:165:63
   5:     0x5805aef0553b - core::fmt::write::hb4f7da2a314ef133
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/fmt/mod.rs:1168:21
   6:     0x5805bb71ce1f - std::io::Write::write_fmt::h9e79d11af9b4686b
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/io/mod.rs:1835:15
   7:     0x5805bb723f6e - std::sys_common::backtrace::_print::h9276d72631ea2ebd
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x5805bb723f6e - std::sys_common::backtrace::print::h301edb80f46c58f8
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x5805bb7252d9 - std::panicking::default_hook::{{closure}}::h08c68fc043d12d43
  10:     0x5805bb725085 - std::panicking::default_hook::hdf65558cafad5039
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:298:9
  11:     0x5805b8b846ab - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hdc9eda93008da126
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/alloc/src/boxed.rs:2077:9
  12:     0x5805b8b846ab - risingwave_rt::panic_hook::set_panic_hook::{{closure}}::h40b35e75a058830c
                               at /risingwave/src/utils/runtime/src/panic_hook.rs:25:9
  13:     0x5805b8b846ab - std::panicking::update_hook::{{closure}}::h68c4945031a81df0
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:233:47
  14:     0x5805bb72591b - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h69f90a497fd344b0
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/alloc/src/boxed.rs:2077:9
  15:     0x5805bb72591b - std::panicking::rust_panic_with_hook::h9576f09a7a058c94
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:799:13
  16:     0x5805bb7256c4 - std::panicking::begin_panic_handler::{{closure}}::hd8f56243f630c203
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:664:13
  17:     0x5805bb724659 - std::sys_common::backtrace::__rust_end_short_backtrace::h5bed6f770a09acf8
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/sys_common/backtrace.rs:171:18
  18:     0x5805bb725407 - rust_begin_unwind
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:652:5
  19:     0x5805aef01a93 - core::panicking::panic_fmt::h088f9f84f0369284
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/panicking.rs:72:14
  20:     0x5805b2a3704a - <foyer_storage::device::direct_fs::DirectFsDevice as foyer_storage::device::Device>::write::{{closure}}::{{closure}}::hd027d3ffc6ebabcc
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/device/direct_fs.rs:160:9
  21:     0x5805b2a3704a - <minitrace::future::InSpan<T> as core::future::future::Future>::poll::h93fd5b0a49d7e5c2
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/minitrace-0.6.7/src/future.rs:125:19
  22:     0x5805b2a3704a - <foyer_storage::device::direct_fs::DirectFsDevice as foyer_storage::device::Device>::write::{{closure}}::h641d96d46d4a4c00
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/device/direct_fs.rs:153:91
  23:     0x5805b2a3704a - foyer_storage::device::monitor::Monitored<D>::write::{{closure}}::{{closure}}::h766e6644e1188c43
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/device/monitor.rs:108:58
  24:     0x5805b2a3704a - <minitrace::future::InSpan<T> as core::future::future::Future>::poll::hd20f4d894e079f50
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/minitrace-0.6.7/src/future.rs:125:19
  25:     0x5805b2d3821c - foyer_storage::device::monitor::Monitored<D>::write::{{closure}}::ha10b685939f91bc7
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/device/monitor.rs:101:87
  26:     0x5805b2d3821c - <foyer_storage::device::monitor::Monitored<D> as foyer_storage::device::Device>::write::{{closure}}::h5c528bd20c10a373
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/device/monitor.rs:168:41
  27:     0x5805b2d3821c - foyer_storage::region::Region<D>::write::{{closure}}::hacf3e7c66f73a257
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/region.rs:92:49
  28:     0x5805b2d2ec86 - foyer_storage::large::flusher::Runner<K,V,S,D>::commit::{{closure}}::{{closure}}::{{closure}}::hdde703ede8bee135
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/large/flusher.rs:550:73
  29:     0x5805b2d2ec86 - <F as futures_core::future::TryFuture>::try_poll::h688d2e599e249d24
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
  30:     0x5805b2d2ec86 - <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll::h40c15360cce59e3e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_future/into_future.rs:34:9
  31:     0x5805b2d2863b - <F as futures_core::future::TryFuture>::try_poll::ha3cb528550b4d8e1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
  32:     0x5805b2d2863b - <futures_util::future::try_maybe_done::TryMaybeDone<Fut> as core::future::future::Future>::poll::h087456788ff0b31c
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_maybe_done.rs:79:57
  33:     0x5805b2d2863b - <F as futures_core::future::TryFuture>::try_poll::he3c3582ad2f8067a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
  34:     0x5805b2d2863b - <futures_util::future::try_join_all::TryJoinAll<F> as core::future::future::Future>::poll::hb78726d40b56efd2
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_join_all.rs:163:32
  35:     0x5805b2d2863b - <F as futures_core::future::TryFuture>::try_poll::h60e1b26fcd1fd65a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
  36:     0x5805b2d2863b - <futures_util::future::try_maybe_done::TryMaybeDone<Fut> as core::future::future::Future>::poll::h52c043a0c00259b5
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_maybe_done.rs:79:57
  37:     0x5805b2d2863b - <futures_util::future::try_join::TryJoin<Fut1,Fut2> as core::future::future::Future>::poll::hca620dd278963720
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_join.rs:72:29
  38:     0x5805b28b161b - foyer_storage::large::flusher::Runner<K,V,S,D>::commit::{{closure}}::h34e41c670f7ae68f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/foyer-storage-0.9.3/src/large/flusher.rs:595:53
  39:     0x5805b28b161b - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::ha76e80b923ca613f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  40:     0x5805b2907552 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hf647fdf85c48e860
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/core.rs:328:17
  41:     0x5805b2907552 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h1d713b24aaadf9bd
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/loom/std/unsafe_cell.rs:16:9
  42:     0x5805b2907552 - tokio::runtime::task::core::Core<T,S>::poll::h693cf411a4f4b88e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/core.rs:317:30
  43:     0x5805b2921aff - tokio::runtime::task::harness::poll_future::{{closure}}::h42f008a4ffe8f683
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:485:19
  44:     0x5805b2921aff - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h98d99a00562a52b7
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/panic/unwind_safe.rs:272:9
  45:     0x5805b2921aff - std::panicking::try::do_call::h735840178f7ed098
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:559:40
  46:     0x5805b2921aff - std::panicking::try::hc34af7bbb0479483
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:523:19
  47:     0x5805b2921aff - std::panic::catch_unwind::hb91f735fe53752a6
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panic.rs:149:14
  48:     0x5805b2921aff - tokio::runtime::task::harness::poll_future::h01b241699dee01bd
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:473:18
  49:     0x5805b2921aff - tokio::runtime::task::harness::Harness<T,S>::poll_inner::he86eea8bc69eeb0b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:208:27
  50:     0x5805b2921aff - tokio::runtime::task::harness::Harness<T,S>::poll::h70bd050b31a85e73
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:153:15
  51:     0x5805bb7ad84b - tokio::runtime::task::raw::RawTask::poll::h626249af0491f4e3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/raw.rs:201:18
  52:     0x5805bb7ad84b - tokio::runtime::task::LocalNotified<S>::run::hb8addeee3a3c54c0
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/mod.rs:427:9
  53:     0x5805bb7ad84b - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h1c841ab6748b3bcf
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:648:22
  54:     0x5805bb7ad84b - tokio::runtime::coop::with_budget::h0bbe3908baf78f4a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/coop.rs:107:5
  55:     0x5805bb7ad84b - tokio::runtime::coop::budget::ha88aed2bb84c970d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/coop.rs:73:5
  56:     0x5805bb7ad84b - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h3832fe87202eebc2
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:584:9
  57:     0x5805bb7a9785 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h27a10854a9d63bfa
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:535:24
  58:     0x5805bb7a9785 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::hf07c3ad4fe0cf498
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:500:21
  59:     0x5805bb7a9785 - tokio::runtime::context::scoped::Scoped<T>::set::h75da236f9d897da1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/context/scoped.rs:40:9
  60:     0x5805bb7ecc58 - tokio::runtime::context::set_scheduler::{{closure}}::h59b22c1a323eb811
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/context.rs:180:26
  61:     0x5805bb7ecc58 - std::thread::local::LocalKey<T>::try_with::hc20591cd664efd9d
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/thread/local.rs:283:12
  62:     0x5805bb7ecc58 - std::thread::local::LocalKey<T>::with::hd228c34d1db99eda
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/thread/local.rs:260:9
  63:     0x5805bb7ecc58 - tokio::runtime::context::set_scheduler::he6413f92dcfbb709
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/context.rs:180:17
  64:     0x5805bb7ecc58 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h90145e52818cfa51
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:495:9
  65:     0x5805bb7ecc58 - tokio::runtime::context::runtime::enter_runtime::h6e91b28ba394eb3e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/context/runtime.rs:65:16
  66:     0x5805bb7ad55c - tokio::runtime::scheduler::multi_thread::worker::run::he443bfcf26770424
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:487:5
  67:     0x5805bb7e3f11 - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::h673489d56cab43a8
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:455:45
  68:     0x5805bb7e3f11 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::hf52c40ec5bbb3240
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/blocking/task.rs:42:21
  69:     0x5805bb7d42af - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h4f8a96a8215e6853
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  70:     0x5805bb7e2776 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h231abc877973020e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/core.rs:328:17
  71:     0x5805bb7e2776 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h7d95a0441d1292fe
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/loom/std/unsafe_cell.rs:16:9
  72:     0x5805bb7e2776 - tokio::runtime::task::core::Core<T,S>::poll::hdd5e41a0f32ed5a0
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/core.rs:317:30
  73:     0x5805bb7c0372 - tokio::runtime::task::harness::poll_future::{{closure}}::h7cb22b0f9a0bc4a3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:485:19
  74:     0x5805bb7c0372 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hfed28fbab9b0a03e
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/panic/unwind_safe.rs:272:9
  75:     0x5805bb7c0372 - std::panicking::try::do_call::hb65cc09bf2bd4d7b
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:559:40
  76:     0x5805bb7c0372 - std::panicking::try::h7db0e518a9b8a97d
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:523:19
  77:     0x5805bb7c0372 - std::panic::catch_unwind::hbdd2d16e43372a1b
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panic.rs:149:14
  78:     0x5805bb7c0372 - tokio::runtime::task::harness::poll_future::hd81255e7b4d02094
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:473:18
  79:     0x5805bb7c0372 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::hc2f4bd3db966b047
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:208:27
  80:     0x5805bb7c0372 - tokio::runtime::task::harness::Harness<T,S>::poll::h4b451fd3e734c254
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:153:15
  81:     0x5805bb7b4192 - tokio::runtime::task::raw::RawTask::poll::h626249af0491f4e3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/raw.rs:201:18
  82:     0x5805bb7b4192 - tokio::runtime::task::UnownedTask<S>::run::h98beb39b303a29c5
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/mod.rs:464:9
  83:     0x5805bb7b4192 - tokio::runtime::blocking::pool::Task::run::hac573af663f13975
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/blocking/pool.rs:159:9
  84:     0x5805bb7b4192 - tokio::runtime::blocking::pool::Inner::run::had6d86a4109477f7
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/blocking/pool.rs:513:17
  85:     0x5805bb7bd500 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::hf78130f6419d7891
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/blocking/pool.rs:471:13
  86:     0x5805bb7bd500 - std::sys_common::backtrace::__rust_begin_short_backtrace::hbf1dc341a32fb6c8
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/sys_common/backtrace.rs:155:18
  87:     0x5805bb7c9a6a - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hebce08b4b59837b0
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/thread/mod.rs:542:17
  88:     0x5805bb7c9a6a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hd490dda97349681d
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/panic/unwind_safe.rs:272:9
  89:     0x5805bb7c9a6a - std::panicking::try::do_call::h577f074763d50b1d
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:559:40
  90:     0x5805bb7c9a6a - std::panicking::try::h83c2307555e8761b
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:523:19
  91:     0x5805bb7c9a6a - std::panic::catch_unwind::h102860fca5fc3899
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panic.rs:149:14
  92:     0x5805bb7c9a6a - std::thread::Builder::spawn_unchecked_::{{closure}}::hd27a3f18384072df
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/thread/mod.rs:541:30
  93:     0x5805bb7c9a6a - core::ops::function::FnOnce::call_once{{vtable.shim}}::h0a939909032743c8
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/ops/function.rs:250:5
  94:     0x5805bb72bb9b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hd9d8a12a4cec8653
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/alloc/src/boxed.rs:2063:9
  95:     0x5805bb72bb9b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc87e6ebe32d18c90
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/alloc/src/boxed.rs:2063:9
  96:     0x5805bb72bb9b - std::sys::pal::unix::thread::Thread::new::thread_start::h68df5163d04f9130
                               at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/sys/pal/unix/thread.rs:108:17
  97:     0x7cf433d21a94 - start_thread
                               at ./nptl/pthread_create.c:447:8
  98:     0x7cf433daec3c - __GI___clone3
                               at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
  99:                0x0 - <unknown>

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

RW version < 2.1

Additional context

No response
@MrCroxx MrCroxx added the type/bug Something isn't working label Dec 26, 2024
@MrCroxx MrCroxx self-assigned this Dec 26, 2024
@MrCroxx MrCroxx closed this as completed Dec 26, 2024
@github-actions github-actions bot added this to the release-2.3 milestone Dec 26, 2024
@MrCroxx
Copy link
Contributor Author

MrCroxx commented Dec 26, 2024

Thanks @zwang28 for reporting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MrCroxx MrCroxx

Labels
type/bug Something isn't working
Projects
None yet
Milestone
release-2.3
Development

No branches or pull requests
1 participant
@MrCroxx