Caching feature

[logical-and]

Hey there, thank you for the plugin. Only one thing is missing in plugin to be perfect - is caching.
Add please option to cache value (refreshInterval) in case of successful response. Here is already implemented, so you can just copy-paste (https://github.com/angel9484/logstash-filter-lookup)

[gandalfb]

Hi, thank you for your proposal.
I can only point out my opinion on this, where local caching is maybe not desirable.

I.e. within a scaled environment with multiple logstash instances, different caching timeout settings could lead to inconsistent data. Worst case within the timeframe of .

For me the best approach here seems to have such caching on a central instance (the REST service itself or a reverse -- microservice -- proxy cache) to achieve consistent event processing.
With http keepalive the overhead and delay seems to be minimal.

The lookup filter (if I understand it correctly) loads a complete data set and refreshes this every x.
From my point of view, this is a different approach. As the rest filter queries a REST service and enriches the processed event, without having to load a set of data in advance.
So a "last recently used" cache (https://github.com/SamSaffron/lru_redux) is what it should be, I guess.
Then you go down the road of multi-threading and parallel processing, which drives complexity...

But long story short:
Of course with an option to disable the functionality, you are highly welcome to create a Pull Request.

[logical-and]

Even 5 sec of caching saves CPU - for example if you have a bunch of records are coming which do the same queries with the same arguments on their processing.
I wish I could create a PR, but my Ruby knowledges are too low :) Maybe if someone would add some caching library (or store it in memore), I could add cache with ttl by copying from that library I gave in topic. :)

[gandalfb]

Hm, well, I wonder how to easily/efficiently identify a http request... It might consist of an url, body or method to build up and use a lru cache.

[logical-and]

Hey @gandalfb, basicly request can be counted as unique if it returns the same result. This consists of:

• url
• method
• body

As result by doing two the same request you'll guaranteed receive the same response (if server state, env settings, and other remove stuff keeps the same)

[emrahtoy]

@logical-and cache also can be implemented on serverside ( where the api lives ).
@gandalfb Of course every request going to cost a connection which also costs cpu time so a local cache could be so desired and at that point i agree with @logical-and 👍