Support load shedding for the file source
#18863
Labels
source: file
Anything `file` source related
type: feature
A value-adding code addition that introduce new functionality.
Comments
neuronull
added
source: file
syedriko
added a commit
to syedriko/vector
that referenced
this issue
Oct 21, 2023
* enhancement(kubernetes_logs source) For the file and kubernetes_logs sources, introduced a new configuration variable, rotate_wait_ms, of type Duration, defaulting to practical infinity. Out of the box, this default effectively turns this feature off. rotate_wait_ms determines for how long vector keeps trying to read from a log file that has been deleted (most likely due to log rotation, hence the name of the variable). Once that time span has expired, vector stops reading from and closes the file descriptor of the deleted file, thus allowing the OS to reclaim the storage space occupied by the file. This behavior is similar to that of Fluentd's tail plugin: https://docs.fluentd.org/input/tail#rotate_wait There is also a new metric introduced - vector_file_deleted_given_up_total, which is a counter. Addesses issue (vectordotdev#18863)
syedriko
added a commit
to syedriko/vector
that referenced
this issue
Nov 12, 2023
* enhancement(kubernetes_logs source) For the file and kubernetes_logs sources, introduced a new configuration variable, rotate_wait_ms, of type Duration, defaulting to practical infinity. Out of the box, this default effectively turns this feature off. rotate_wait_ms determines for how long vector keeps trying to read from a log file that has been deleted (most likely due to log rotation, hence the name of the variable). Once that time span has expired, vector stops reading from and closes the file descriptor of the deleted file, thus allowing the OS to reclaim the storage space occupied by the file. This behavior is similar to that of Fluentd's tail plugin: https://docs.fluentd.org/input/tail#rotate_wait There is also a new metric introduced - vector_file_deleted_given_up_total, which is a counter. Addesses issue (vectordotdev#18863)
syedriko
added a commit
to syedriko/vector
that referenced
this issue
Jan 7, 2024
* enhancement(kubernetes_logs source) For the file and kubernetes_logs sources, introduced a new configuration variable, rotate_wait_ms, of type Duration, defaulting to practical infinity. Out of the box, this default effectively turns this feature off. rotate_wait_ms determines for how long vector keeps trying to read from a log file that has been deleted (most likely due to log rotation, hence the name of the variable). Once that time span has expired, vector stops reading from and closes the file descriptor of the deleted file, thus allowing the OS to reclaim the storage space occupied by the file. This behavior is similar to that of Fluentd's tail plugin: https://docs.fluentd.org/input/tail#rotate_wait There is also a new metric introduced - vector_file_deleted_given_up_total, which is a counter. Addesses issue (vectordotdev#18863)
syedriko
added a commit
to syedriko/vector
that referenced
this issue
Jan 17, 2024
* enhancement(kubernetes_logs source) For the file and kubernetes_logs sources, introduced a new configuration variable, rotate_wait_ms, of type Duration, defaulting to practical infinity. Out of the box, this default effectively turns this feature off. rotate_wait_ms determines for how long vector keeps trying to read from a log file that has been deleted (most likely due to log rotation, hence the name of the variable). Once that time span has expired, vector stops reading from and closes the file descriptor of the deleted file, thus allowing the OS to reclaim the storage space occupied by the file. This behavior is similar to that of Fluentd's tail plugin: https://docs.fluentd.org/input/tail#rotate_wait There is also a new metric introduced - vector_file_deleted_given_up_total, which is a counter. Addesses issue (vectordotdev#18863)
syedriko
added a commit
to syedriko/vector
that referenced
this issue
Jan 17, 2024
* enhancement(kubernetes_logs source) For the file and kubernetes_logs sources, introduced a new configuration variable, rotate_wait_ms, of type Duration, defaulting to practical infinity. Out of the box, this default effectively turns this feature off. rotate_wait_ms determines for how long vector keeps trying to read from a log file that has been deleted (most likely due to log rotation, hence the name of the variable). Once that time span has expired, vector stops reading from and closes the file descriptor of the deleted file, thus allowing the OS to reclaim the storage space occupied by the file. This behavior is similar to that of Fluentd's tail plugin: https://docs.fluentd.org/input/tail#rotate_wait There is also a new metric introduced - vector_file_deleted_given_up_total, which is a counter. Addesses issue (vectordotdev#18863)
syedriko
added a commit
to syedriko/vector
that referenced
this issue
Jan 17, 2024
* enhancement(kubernetes_logs source) For the file and kubernetes_logs sources, introduced a new configuration variable, rotate_wait_ms, of type Duration, defaulting to practical infinity. Out of the box, this default effectively turns this feature off. rotate_wait_ms determines for how long vector keeps trying to read from a log file that has been deleted (most likely due to log rotation, hence the name of the variable). Once that time span has expired, vector stops reading from and closes the file descriptor of the deleted file, thus allowing the OS to reclaim the storage space occupied by the file. This behavior is similar to that of Fluentd's tail plugin: https://docs.fluentd.org/input/tail#rotate_wait There is also a new metric introduced - vector_file_deleted_given_up_total, which is a counter. Addesses issue (vectordotdev#18863)
syedriko
added a commit
to syedriko/vector
that referenced
this issue
Jan 22, 2024
* enhancement(kubernetes_logs source) For the file and kubernetes_logs sources, introduced a new configuration variable, rotate_wait_ms, of type Duration, defaulting to practical infinity. Out of the box, this default effectively turns this feature off. rotate_wait_ms determines for how long vector keeps trying to read from a log file that has been deleted (most likely due to log rotation, hence the name of the variable). Once that time span has expired, vector stops reading from and closes the file descriptor of the deleted file, thus allowing the OS to reclaim the storage space occupied by the file. This behavior is similar to that of Fluentd's tail plugin: https://docs.fluentd.org/input/tail#rotate_wait There is also a new metric introduced - vector_file_deleted_given_up_total, which is a counter. Addesses issue (vectordotdev#18863)
syedriko
added a commit
to syedriko/vector
that referenced
this issue
Feb 11, 2024
* enhancement(kubernetes_logs source) For the file and kubernetes_logs sources, introduced a new configuration variable, rotate_wait_secs, defaulting to practical infinity. Out of the box, this default effectively turns this feature off. rotate_wait_secs determines for how long Vector keeps trying to read from a log file that has been deleted (most likely due to log rotation, hence the name of the variable). Once that time span has expired, Vector stops reading from and closes the file descriptor of the deleted file, thus allowing the OS to reclaim the storage space occupied by the file. This behavior is similar to that of Fluentd's tail plugin: https://docs.fluentd.org/input/tail#rotate_wait Addesses issue (vectordotdev#18863)
syedriko
added a commit
to syedriko/vector
that referenced
this issue
Feb 13, 2024
* enhancement(kubernetes_logs source) For the file and kubernetes_logs sources, introduced a new configuration variable, rotate_wait_secs, defaulting to practical infinity. Out of the box, this default effectively turns this feature off. rotate_wait_secs determines for how long Vector keeps trying to read from a log file that has been deleted (most likely due to log rotation, hence the name of the variable). Once that time span has expired, Vector stops reading from and closes the file descriptor of the deleted file, thus allowing the OS to reclaim the storage space occupied by the file. This behavior is similar to that of Fluentd's tail plugin: https://docs.fluentd.org/input/tail#rotate_wait Addesses issue (vectordotdev#18863)
syedriko
added a commit
to syedriko/vector
that referenced
this issue
Feb 13, 2024
* enhancement(kubernetes_logs source) For the file and kubernetes_logs sources, introduced a new configuration variable, rotate_wait_secs, defaulting to practical infinity. Out of the box, this default effectively turns this feature off. rotate_wait_secs determines for how long Vector keeps trying to read from a log file that has been deleted (most likely due to log rotation, hence the name of the variable). Once that time span has expired, Vector stops reading from and closes the file descriptor of the deleted file, thus allowing the OS to reclaim the storage space occupied by the file. This behavior is similar to that of Fluentd's tail plugin: https://docs.fluentd.org/input/tail#rotate_wait Addresses issue (vectordotdev#18863) Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
github-merge-queue bot
pushed a commit
that referenced
this issue
Feb 14, 2024
…config option (#18904) * enhancement(file source) * enhancement(kubernetes_logs source) For the file and kubernetes_logs sources, introduced a new configuration variable, rotate_wait_secs, defaulting to practical infinity. Out of the box, this default effectively turns this feature off. rotate_wait_secs determines for how long Vector keeps trying to read from a log file that has been deleted (most likely due to log rotation, hence the name of the variable). Once that time span has expired, Vector stops reading from and closes the file descriptor of the deleted file, thus allowing the OS to reclaim the storage space occupied by the file. This behavior is similar to that of Fluentd's tail plugin: https://docs.fluentd.org/input/tail#rotate_wait Addresses issue (#18863) Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
AndrooTheChen
pushed a commit
to discord/vector
that referenced
this issue
Sep 23, 2024
…config option (vectordotdev#18904) * enhancement(file source) * enhancement(kubernetes_logs source) For the file and kubernetes_logs sources, introduced a new configuration variable, rotate_wait_secs, defaulting to practical infinity. Out of the box, this default effectively turns this feature off. rotate_wait_secs determines for how long Vector keeps trying to read from a log file that has been deleted (most likely due to log rotation, hence the name of the variable). Once that time span has expired, Vector stops reading from and closes the file descriptor of the deleted file, thus allowing the OS to reclaim the storage space occupied by the file. This behavior is similar to that of Fluentd's tail plugin: https://docs.fluentd.org/input/tail#rotate_wait Addresses issue (vectordotdev#18863) Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
As discussed in #11742 , this issue tracks the feature request to have a config option to essentially flag similar to Fluend's
rotate_waitthat is opt-in and would mean Vector doesn't read to EOF.See also: #18784
The text was updated successfully, but these errors were encountered: