Ability to use metadata fields that do not end up in final sink

[hartfordfive]

I'm trying to figure out if there's a way to set metadata fields that can be used within the context of the VRL remap execution and within the path of the file sink although I can't seem to determine if this is actually possible or not. To compare to an existing similar feature, Logstash has the notion of metadata via a special @metadata field (see docs). Anything within this object can be used within the execution context of the pipeline but it will not end up in the final event that's sent to a given output destination.

I appreciate any help or guidance with this. Thanks!

[pront]

Hi @hartfordfive,

a way to set metadata fields that can be used within the context of the VRL remap execution

Yes, there is. See this playground example.

If you are using log namespacing you will already see some fields under %vector. You can see details here.

Let me know if you have any follow up questions.

[jszwedko]

There is also encoding.exclude_fields that can be used to exclude specific fields when encoding, but log namespacing is the longer term solution here (encoding.exclude_fields will probably be deprecated once log namespacing is stabilized).

[hartfordfive]

Great, this was exactly what I was looking for. Thank you for the help!