Possibility adding a remote configuration database for stream mode

[yusufsyaifudin]

Hi @Jeffail @mihaitodor

Currently, Benthos has Stream mode where we can add, replace or delete the active stream when running. It is good when we want to change the stream input, processor or output when Benthos already running on server without making any restart on server.

As I look into the code, currently it only support in memory to save the stream label and it's configuration here https://github.com/benthosdev/benthos/blob/v4.11.0/internal/stream/manager/type.go#L184-L205

Interestingly, if we use stream mode using file watcher config, I found that Benthos made attempt to listen the file changes using SubscribeStreamChanges that initiated in by the function readConfig which return the *config.Reader

Is it possible to make config.Reader returned by readConfig above using interface instead of struct?

By doing that, we can implement any Reader that implements the interface, and we probably can implement the Reader using etcd watcher, then we can have a database as Benthos source of truth. In addition, if we have a database server like etcd, we can deploy Benthos as many times and they will share the same configuration.

When this beneficial? It will useful when we use, for example, Kafka (consumer) as input where the maximum consumer per group is up to number of topic partition. If we use stream mode from directory, we need to configure each server to have the same configuration. If we use stream mode from REST API that only support in-memory, we need to call each Benthos deployment (which is most case will not be easy because we need to add Ingress in each Deployment if we deploy using Kubernetes).

If we add some capability in Benthos to listen from database server such as etcd, we only need to PUT in one Benthos server then etcd will notify the other Benthos deployment that use the same etcd.

Also, if some Benthos server restarting (due to server error), if we use Rest API stream mode, all configuration will be gone because Benthos is stateless. By adding the database configuration server, we can start to make Benthos stateful.

-- PS --
I don't know if it is as simple as making the Reader to interface for now, and then start to implementing another Reader backend (etcd or another distributed config database).