Auto generate env config from yaml

[jackmac92]

Looking to deploy with docker, and env var configuration seems easiest to manage for that. But I'd like to still write my config in yml, and have CI generate the correct env vars.

Since the env vars largely appear to be the same structure as the yaml, the below jq script gets me most of the way

cat benthos-conf.yml| yq | jq '. as $root |\n path(..) | . as $path |\n $root | getpath($path) as $value |\n select($value | scalars) |\n "\([$path[]] | join("_") | ascii_upcase)=\($value)"\n ' -r

But I noticed some env vars do not "map" from their yaml definitions, e.g. buffer: options are much different.

Any thoughts about supporting the above jq script, and ensuring the env var configuration can be generated by mapping yaml defs to env vars?

e.g.

input:
  http_server:
    path: /post
    ws_path: /post/ws
    allowed_verbs:
      - POST
    timeout: 5s
    rate_limit: ""

generates

INPUT_HTTP_SERVER_PATH=/post
INPUT_HTTP_SERVER_WS_PATH=/post/ws
INPUT_HTTP_SERVER_ALLOWED_VERBS_0=POST
INPUT_HTTP_SERVER_TIMEOUT=5s
INPUT_HTTP_SERVER_RATE_LIMIT=

(I know slight issue with array values getting suffixes, will find a fix for that if this could be supported)

[Jeffail]

Hey @jackmac92, the environment variable generated config is deprecated now as the naming conventions are hard to translate (as you're seeing), the new way going forwards is to use the -s flag: --set "metrics.type=prometheus", -s "metrics.type=prometheus" set a field (identified by a dot path) in the main configuration file, e.g. "metrics.type=prometheus".

The spec for dots paths is here: https://www.benthos.dev/docs/configuration/field_paths

The main advantage of using this flag is that the delimiter is explicit, and the same key can be repeated if required which allows for configuring array values like this:

benthos \
  -s "input.amqp_0_9.urls.-=amqp://127.0.0.1:5672/" \
  -s "input.amqp_0_9.urls.-=amqp://127.0.0.2:5672/"

I haven't got a tool yet to translate a config into a series of flags but it should be much easier. However, I'd also say that if your config is large enough to need such a tool then I would definitely suggest deploying it the proper way.