Vector Panic

[tastyfrankfurt]

Vector will not start, have deleted the disk queues from /var/lib/vector but still get the following error with FULL trace enabled.

Aug 29 23:50:00 test.example.local vector[5933]: thread 'vector-worker' panicked at 'Tried to ack beyond read offset', /project/lib/vector-core/buffers/src/disk/leveldb_buffer/reader.rs:184:13
Aug 29 23:50:00 test.example.local vector[5933]: stack backtrace:
Aug 29 23:50:00 test.example.local vector[5933]: 0: 0x55ef46fd0340 - std::backtrace_rs::backtrace::libunwind::trace::h34055254b57d8e79
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
Aug 29 23:50:00 test.example.local vector[5933]: 1: 0x55ef46fd0340 - std::backtrace_rs::backtrace::trace_unsynchronized::h8f1e3fbd9afff6ec
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
Aug 29 23:50:00 test.example.local vector[5933]: 2: 0x55ef46fd0340 - std::sys_common::backtrace::_print_fmt::h3a99a796b770c360
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys_common/backtrace.rs:67:5
Aug 29 23:50:00 test.example.local vector[5933]: 3: 0x55ef46fd0340 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h32d1f94a80615d18
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys_common/backtrace.rs:46:22
Aug 29 23:50:00 test.example.local vector[5933]: 4: 0x55ef4703cb3c - core::fmt::write::h306731c068f7162c
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/fmt/mod.rs:1110:17
Aug 29 23:50:00 test.example.local vector[5933]: 5: 0x55ef46fc1d05 - std::io::Write::write_fmt::hd2fa90334eee2a21
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/io/mod.rs:1588:15
Aug 29 23:50:00 test.example.local vector[5933]: 6: 0x55ef46fd41cb - std::sys_common::backtrace::_print::h5abaa2601a852287
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys_common/backtrace.rs:49:5
Aug 29 23:50:00 test.example.local vector[5933]: 7: 0x55ef46fd41cb - std::sys_common::backtrace::print::h8d81445442bb638f
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys_common/backtrace.rs:36:9
Aug 29 23:50:00 test.example.local vector[5933]: 8: 0x55ef46fd41cb - std::panicking::default_hook::{{closure}}::hcfe804496a9fa747
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:208:50
Aug 29 23:50:00 test.example.local vector[5933]: 9: 0x55ef46fd3ca1 - std::panicking::default_hook::hbea8e3ccf2ba8901
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:225:9
Aug 29 23:50:00 test.example.local vector[5933]: 10: 0x55ef46fd49d4 - std::panicking::rust_panic_with_hook::h7ee9e1a2d0f8975a
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:622:17
Aug 29 23:50:00 test.example.local vector[5933]: 11: 0x55ef436205e4 - std::panicking::begin_panic::{{closure}}::hc4f15b317d0889cb
Aug 29 23:50:00 test.example.local vector[5933]: 12: 0x55ef435fc80c - std::sys_common::backtrace::__rust_end_short_backtrace::h8123e621a7044df9
Aug 29 23:50:00 test.example.local vector[5933]: 13: 0x55ef431b37cc - std::panicking::begin_panic::hda4466d9cb99be3f
Aug 29 23:50:00 test.example.local vector[5933]: 14: 0x55ef43b9a509 - buffers::disk::leveldb_buffer::reader::Reader::flush::hd8550b8fbb50016b
Aug 29 23:50:00 test.example.local vector[5933]: 15: 0x55ef433256c9 - <buffers::disk::leveldb_buffer::reader::Reader as futures_core::stream::Stream>::poll_next::h7223edf30de07c65
Aug 29 23:50:00 test.example.local vector[5933]: 16: 0x55ef44719598 - <vector::utilization::Utilization as futures_core::stream::Stream>::poll_next::ha94dfacc6a4b8e39
Aug 29 23:50:00 test.example.local vector[5933]: 17: 0x55ef432cdf63 - <stream_cancel::combinator::TakeUntilIf<S,F> as futures_core::stream::Stream>::poll_next::he18f0ed82d4d75b2
Aug 29 23:50:00 test.example.local vector[5933]: 18: 0x55ef44425c19 - <futures_util::stream::stream::fuse::Fuse as futures_core::stream::Stream>::poll_next::h64dee8bfda165581
Aug 29 23:50:00 test.example.local vector[5933]: 19: 0x55ef44425a6b - <futures_util::stream::stream::fuse::Fuse as futures_core::stream::Stream>::poll_next::h29aef6395e30c98d
Aug 29 23:50:00 test.example.local vector[5933]: 20: 0x55ef4431a8c3 - <core::future::from_generator::GenFuture as core::future::future::Future>::poll::hd5493fb48a04400d
Aug 29 23:50:00 test.example.local vector[5933]: 21: 0x55ef442a6d9c - <core::future::from_generator::GenFuture as core::future::future::Future>::poll::ha493d247a22d2675
Aug 29 23:50:00 test.example.local vector[5933]: 22: 0x55ef43a1b378 - tokio::runtime::task::harness::poll_future::h8d174f5c8dbaebad
Aug 29 23:50:00 test.example.local vector[5933]: 23: 0x55ef439d4aa4 - tokio::runtime::task::raw::poll::h95648d99a4de7c69
Aug 29 23:50:00 test.example.local vector[5933]: 24: 0x55ef46782f4f - tokio::runtime::thread_pool::worker::Context::run_task::h5aeb6dc5f7237987
Aug 29 23:50:00 test.example.local vector[5933]: 25: 0x55ef46782502 - tokio::runtime::thread_pool::worker::run::hb1eb5812bff9a631
Aug 29 23:50:00 test.example.local vector[5933]: 26: 0x55ef439d42f8 - tokio::runtime::task::raw::poll::h8eb845e9130b4063
Aug 29 23:50:00 test.example.local vector[5933]: 27: 0x55ef4674558a - std::sys_common::backtrace::__rust_begin_short_backtrace::h6374674f497847c1
Aug 29 23:50:00 test.example.local vector[5933]: 28: 0x55ef4674cb7e - core::ops::function::FnOnce::call_once{{vtable.shim}}::h42364c10fed52b6a
Aug 29 23:50:00 test.example.local vector[5933]: 29: 0x55ef46fe0ed7 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::h7ece6cfefaff1005
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/alloc/src/boxed.rs:1575:9
Aug 29 23:50:00 test.example.local vector[5933]: 30: 0x55ef46fe0ed7 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::hb8b48e55c21f193e
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/alloc/src/boxed.rs:1575:9
Aug 29 23:50:00 test.example.local vector[5933]: 31: 0x55ef46fe0ed7 - std::sys::unix::thread::Thread::new::thread_start::h8c7c4450dba62914
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys/unix/thread.rs:71:17
Aug 29 23:50:00 test.example.local vector[5933]: 32: 0x7f09cd537609 - start_thread
Aug 29 23:50:00 test.example.local vector[5933]: 33: 0x7f09cd307293 - clone
Aug 29 23:50:00 test.example.local vector[5933]: 34: 0x0 -
Aug 29 23:50:00 test.example.local vector[5933]: thread 'vector-worker' panicked at 'Tried to ack beyond read offset', /project/lib/vector-core/buffers/src/disk/leveldb_buffer/reader.rs:184:13
Aug 29 23:50:00 test.example.local vector[5933]: stack backtrace:
Aug 29 23:50:00 test.example.local vector[5933]: 0: 0x55ef46fd0340 - std::backtrace_rs::backtrace::libunwind::trace::h34055254b57d8e79
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
Aug 29 23:50:00 test.example.local vector[5933]: 1: 0x55ef46fd0340 - std::backtrace_rs::backtrace::trace_unsynchronized::h8f1e3fbd9afff6ec
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
Aug 29 23:50:00 test.example.local vector[5933]: 2: 0x55ef46fd0340 - std::sys_common::backtrace::_print_fmt::h3a99a796b770c360
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys_common/backtrace.rs:67:5
Aug 29 23:50:00 test.example.local vector[5933]: 3: 0x55ef46fd0340 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h32d1f94a80615d18
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys_common/backtrace.rs:46:22
Aug 29 23:50:00 test.example.local vector[5933]: 4: 0x55ef4703cb3c - core::fmt::write::h306731c068f7162c
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/fmt/mod.rs:1110:17
Aug 29 23:50:00 test.example.local vector[5933]: 5: 0x55ef46fc1d05 - std::io::Write::write_fmt::hd2fa90334eee2a21
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/io/mod.rs:1588:15
Aug 29 23:50:00 test.example.local vector[5933]: 6: 0x55ef46fd41cb - std::sys_common::backtrace::_print::h5abaa2601a852287
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys_common/backtrace.rs:49:5
Aug 29 23:50:00 test.example.local vector[5933]: 7: 0x55ef46fd41cb - std::sys_common::backtrace::print::h8d81445442bb638f
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys_common/backtrace.rs:36:9
Aug 29 23:50:00 test.example.local vector[5933]: 8: 0x55ef46fd41cb - std::panicking::default_hook::{{closure}}::hcfe804496a9fa747
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:208:50
Aug 29 23:50:00 test.example.local vector[5933]: 9: 0x55ef46fd3ca1 - std::panicking::default_hook::hbea8e3ccf2ba8901
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:225:9
Aug 29 23:50:00 test.example.local vector[5933]: 10: 0x55ef46fd49d4 - std::panicking::rust_panic_with_hook::h7ee9e1a2d0f8975a
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:622:17
Aug 29 23:50:00 test.example.local vector[5933]: 11: 0x55ef436205e4 - std::panicking::begin_panic::{{closure}}::hc4f15b317d0889cb
Aug 29 23:50:00 test.example.local vector[5933]: 12: 0x55ef435fc80c - std::sys_common::backtrace::__rust_end_short_backtrace::h8123e621a7044df9
Aug 29 23:50:00 test.example.local vector[5933]: 13: 0x55ef431b37cc - std::panicking::begin_panic::hda4466d9cb99be3f
Aug 29 23:50:00 test.example.local vector[5933]: 14: 0x55ef43b9a509 - buffers::disk::leveldb_buffer::reader::Reader::flush::hd8550b8fbb50016b
Aug 29 23:50:00 test.example.local vector[5933]: 15: 0x55ef437d6d47 - core::ptr::drop_in_place<buffers::disk::leveldb_buffer::reader::Reader<vector_core::event::Event>>::h25a4f4383fb30338
Aug 29 23:50:00 test.example.local vector[5933]: 16: 0x55ef437858ea - core::ptr::drop_in_placevector::utilization::Utilization::he26fff44ab499616
Aug 29 23:50:00 test.example.local vector[5933]: 17: 0x55ef43738bd4 - core::ptr::drop_in_place<core::pin::Pin<alloc::boxed::Box<core::pin::Pin<alloc::boxed::Box<dyn futures_core::stream::Stream+Item = vect>
Aug 29 23:50:00 test.example.local vector[5933]: 18: 0x55ef442a7673 - <core::future::from_generator::GenFuture as core::future::future::Future>::poll::ha493d247a22d2675
Aug 29 23:50:00 test.example.local vector[5933]: 19: 0x55ef43a1b378 - tokio::runtime::task::harness::poll_future::h8d174f5c8dbaebad
Aug 29 23:50:00 test.example.local vector[5933]: 20: 0x55ef439d4aa4 - tokio::runtime::task::raw::poll::h95648d99a4de7c69
Aug 29 23:50:00 test.example.local vector[5933]: 21: 0x55ef46782f4f - tokio::runtime::thread_pool::worker::Context::run_task::h5aeb6dc5f7237987
Aug 29 23:50:00 test.example.local vector[5933]: 22: 0x55ef46782502 - tokio::runtime::thread_pool::worker::run::hb1eb5812bff9a631
Aug 29 23:50:00 test.example.local vector[5933]: 23: 0x55ef439d42f8 - tokio::runtime::task::raw::poll::h8eb845e9130b4063
Aug 29 23:50:00 test.example.local vector[5933]: 24: 0x55ef4674558a - std::sys_common::backtrace::__rust_begin_short_backtrace::h6374674f497847c1
Aug 29 23:50:00 test.example.local vector[5933]: 25: 0x55ef4674cb7e - core::ops::function::FnOnce::call_once{{vtable.shim}}::h42364c10fed52b6a
Aug 29 23:50:00 test.example.local vector[5933]: 26: 0x55ef46fe0ed7 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::h7ece6cfefaff1005
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/alloc/src/boxed.rs:1575:9
Aug 29 23:50:00 test.example.local vector[5933]: 27: 0x55ef46fe0ed7 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::hb8b48e55c21f193e
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/alloc/src/boxed.rs:1575:9
Aug 29 23:50:00 test.example.local vector[5933]: 28: 0x55ef46fe0ed7 - std::sys::unix::thread::Thread::new::thread_start::h8c7c4450dba62914
Aug 29 23:50:00 test.example.local vector[5933]: at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys/unix/thread.rs:71:17
Aug 29 23:50:00 test.example.local vector[5933]: 29: 0x7f09cd537609 - start_thread
Aug 29 23:50:00 test.example.local vector[5933]: 30: 0x7f09cd307293 - clone
Aug 29 23:50:00 test.example.local vector[5933]: 31: 0x0 -
Aug 29 23:50:00 test.example.local vector[5933]: thread panicked while panicking. aborting.
Aug 29 23:50:00 test.example.local systemd-coredump[5956]: Process 5933 (vector) of user 1001 dumped core.
Aug 29 23:50:00 test.example.local systemd[1]: vector.service: Main process exited, code=killed, status=4/ILL
Aug 29 23:50:00 test.example.local systemd[1]: vector.service: Failed with result 'signal'.

[jszwedko]

Hey @tastyfrankfurt ,

Apologies for the inconvenience, this was a bug released in 0.16.0, but fixed in 0.16.1 (we noted this in the release notes). Would you mind trying 0.16.1?

Thanks!

[tastyfrankfurt]

Thanks :-) confirmed