Support fields/VRL for setting sample rate

[NeilJed]

I asked this in another question but experiemented to get the answer.

Currently with the sample filter, it only accepts an int as the input for the sample rate. It would be really good if we could set this via a field or VRL snippet.

The use case I have is that my event messages contains a field denoting the service that the log belongs to. I'm using this as the key field for hashing. However as event volume varies a lot, a fixed sample rate doesn't fit my use case.

What I would like to do is be able to set the sample rate based on a condition, in this case the value of the service field or even a event metadata value (if the sample rate is pre-caclulated and added in a previous transform).

For example:

[transforms.my_sampler]
type = "sample"
inputs = [ "upstream_source"]
rate = '''
int(get!({"serviceA": 1, "serviceB": 1, "serviceC": 1000}, [.service_name])) ?? 3
'''

I'm not a Rust programmer but I noticed that you have a condition input that supports VRL that supports a boolean response. Could one be added for Int to support this?

Another idea would be to have a secondary parameter rate_calculated for this value so that either can be used where one has priority.

Currently the way I've had to do it is use a route condition to match the service name and send it to numerous samplers pre-configured for set rates.

[jszwedko]

That would be nifty. Do you want to open a feature request?

[NeilJed]

Yeah that would be good. Is there a procedure for that or do I open an issue ticket?

[jszwedko]

You can use this issue template: https://github.com/vectordotdev/vector/issues/new?assignees=&labels=type%3A+feature&projects=&template=feature.yml