Feature request: sleep() function for inserting delay into processing chain

[johnhtodd]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

We have some possible uses for inserting delays into output for testing purposes. There does not seem to be a "sleep()" function. I think that should be fairly easy, and may be useful in some circumstances. I know it is a seemingly contradictory function in a system designed for speed, but sometimes there is a need to take a nap.

Attempted Solutions

I could create some really awful artificial loops or reading events or other stuff.. but... ugh.

Proposal

I would like to see a sleep() function that would take as its only input the number of milliseconds of non-action before continuing.

References

No response

Version

0.44

[jszwedko]

@johnhtodd Do you think the reduce transform might be able to meet your needs? You should be able to configure to flush on an interval via flush_period_ms with starts_with: true and ends_with: false to only use the flush mechanism to emit on the interval.

Also, could you elaborate a bit more on your use-case for introducing a delay in Vector processing? Is it to simulate a real running config in some load test environment? Or something else?

[johnhtodd]

I don't think the "reduce" mechanism can work effectively, or at least it would require pretty significant spaghetti-coding of the config to make it work.

There are two instances that I need this method:

• creating delays in individual outputs for test circumstances when I'm using Vector as a "jig" for testing other downstream components where I am trying to simulate random intervals between events that are actually being read in from a file.

• for preventing race conditions where I have several parallel events happening that may be better served by waiting a few milliseconds for some condition to be better set. As an example: I am now using the as-yet-to-be-included cache enrichment model. I am then performing dns lookups to fill the cache. If I get a new event, I need to wait about 300ms to allow the cache fill path to have a chance to complete before I send the same event to my primary logging path to compare against the cache data.

It is the latter case where it becomes very difficult to see how I'd use "reduce". It is also very non-obvious that 'reduce' would be used as an intentional delay, and I suspect it would be rare or never that other implementers would arrive at that method without it being suggested by someone who is extensively experienced with Vector internals. "sleep" or "wait" functions, while unusual, are exceptionally handy and exist in most other processing systems and language models.

[pront]

Another idea is to introduce a delay transform (with a good usage warning on top) to introduce artificial delays at certain points in the topology. But let me know if that also doesn't address all your needs. We could consider a sleep function in VRL as a contribution if it is a must, again with a big warning and explanation on top.