error i have got while running vector

[muneesMohammed]

While running vector in Kubernetes, everything went smoothly, but a few days later, it began to report these errors, and I had no idea what they were. If anyone can help, that would be deeply appreciated.

source{component_kind="source" component_id=syslog component_type=syslog component_name=syslog}: vector::source thread 'vector-worker' panicked at 'assertion failed: min <= max', /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/cmp.rs:860:9 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace thread 'vector-worker' panicked at 'called Result::unwrap() on an Err value: PoisonError { .. }'

[davidhuie-dd]

@muneesMohammed thanks for bringing this up. I created #16103 for the team to look into this.

[davidhuie-dd]

Hi @muneesMohammed, could you try to trigger the problem with RUST_BACKTRACE=1?

Then, if you could provide the info we request for issues, that would help us out: https://github.com/vectordotdev/vector/issues/new?assignees=&labels=type%3A+bug&template=bug.yml

[christophemorio]

Hello,

I'm having same issue (AWS EKS, aarch64 node)
workarounds found:

• switch from image 0.27.0-distroless-libc to 0.27.0-distroless-static
• increase CPU limit from 1000m to 2000m

Also tried to force the number of threads, but this does not solve

      env:
        - name: VECTOR_THREADS
          value: "1"

[christophemorio]

I'm able to reproduce the issue when cpu limit is 1000m, this sounds related to Filebeat input.

Full stacktrace

2023-02-02T14:37:31.744032Z  INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-02-02T14:37:31.744216Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-02-02T14:37:31.744275Z  INFO vector::app: Loading configs. paths=["/etc/vector"]
2023-02-02T14:37:31.745564Z  WARN vector::config::source: Enabling `acknowledgements` on sources themselves is deprecated in favor of enabling them in the sink configuration, and will be removed in a future version. component_name="my_input"
2023-02-02T14:37:31.745692Z  INFO vector::topology::running: Running healthchecks.
2023-02-02T14:37:31.745833Z  INFO vector::topology::builder: Healthcheck passed.
2023-02-02T14:37:31.745874Z  INFO vector: Vector has started. debug="false" version="0.27.0" arch="x86_64" revision="5623d1e 2023-01-18"
2023-02-02T14:37:31.745897Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-02-02T14:37:31.746042Z  INFO source{component_kind="source" component_id=my_input component_type=logstash component_name=my_input}: vector::sources::util::net::tcp: Listening. addr=0.0.0.0:5044
thread 'vector-worker' panicked at 'assertion failed: min <= max', /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/cmp.rs:860:9
stack backtrace:
   0:     0x561dbb330bde - std::backtrace_rs::backtrace::libunwind::trace::he615646ea344481f
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x561dbb330bde - std::backtrace_rs::backtrace::trace_unsynchronized::h6ea8eaac68705b9c
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x561dbb330bde - std::sys_common::backtrace::_print_fmt::h7ac486a935ce0bf7
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x561dbb330bde - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h1b5a095d3db2e28f
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x561dba9b58de - core::fmt::write::h445545b92224a1cd
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/fmt/mod.rs:1209:17
   5:     0x561dbb2fc364 - std::io::Write::write_fmt::h55a43474c6520b00
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/io/mod.rs:1682:15
   6:     0x561dbb33233f - std::sys_common::backtrace::_print::h65d20526fdb736b0
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:47:5
   7:     0x561dbb33233f - std::sys_common::backtrace::print::h6555fbe12a1cc41b
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:34:9
   8:     0x561dbb331e3d - std::panicking::default_hook::{{closure}}::hbdf58083140e7ac6
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:267:22
   9:     0x561dbb3330f6 - std::panicking::default_hook::haef8271c56b74d85
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:286:9
  10:     0x561dbb3330f6 - std::panicking::rust_panic_with_hook::hfd45b6b6c12d9fa5
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:688:13
  11:     0x561dbb332a53 - std::panicking::begin_panic_handler::{{closure}}::hf591e8609a75bd4b
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:577:13
  12:     0x561dbb3329ec - std::sys_common::backtrace::__rust_end_short_backtrace::h81899558795e4ff7
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:137:18
  13:     0x561dbb3329c1 - rust_begin_unwind
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:575:5
  14:     0x561dba3ae252 - core::panicking::panic_fmt::h4235fa9b4675b332
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:65:14
  15:     0x561dba3ae34c - core::panicking::panic::h9ced3cf2f605ba6a
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:115:5
  16:     0x561dbd335a66 - core::ptr::drop_in_place<vector::sources::util::net::tcp::request_limiter::RequestLimiterPermit>::h84d9610340606322
  17:     0x561dbd690c6d - vector::sources::util::net::tcp::handle_stream::{{closure}}::h4ecc377796bd57d0
  18:     0x561dbd68efc5 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::ha3bc7d55fdaeb8aa
  19:     0x561dbd68de6a - tokio::runtime::task::raw::poll::h0cc5754b9f175a89
  20:     0x561dbb374a64 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h2063adb645b203ed
  21:     0x561dbb37be95 - tokio::runtime::task::raw::poll::he7bffe6f0c4c727e
  22:     0x561dbb35e009 - std::sys_common::backtrace::__rust_begin_short_backtrace::hfb0ab063688f3613
  23:     0x561dbb35dc8d - core::ops::function::FnOnce::call_once{{vtable.shim}}::h713fadf6047e137d
  24:     0x561dbb335533 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h4273f95ec44459b3
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/alloc/src/boxed.rs:1987:9
  25:     0x561dbb335533 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h70f28fa4ddc269e5
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/alloc/src/boxed.rs:1987:9
  26:     0x561dbb335533 - std::sys::unix::thread::Thread::new::thread_start::h85a9c16b988e2bd0
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys/unix/thread.rs:108:17
  27:     0x7f292344cea7 - start_thread
  28:     0x7f2923220a2f - clone
  29:                0x0 - <unknown>

How to reproduce on Kubernetes:
https://gist.github.com/christophemorio/256d8d2177da9e3f52834a85d162a2e6

[jszwedko]

Thanks for the reproduction! Do you mind opening a bug report for this: https://github.com/vectordotdev/vector/issues/new?assignees=&labels=type%3A+bug&template=bug.yml ?

[christophemorio]

Sure, this is the new bug report #16273.