Secrets in commented-out lines in vector.toml cause config validation to fail

[chradcliffe]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When attempting to run a TOML config that contains a commented-out block containing a reference to a secret, vector fails to run properly because it thinks that secrets needs to be loaded (even though this is in an inactive section of the config). So running vector like:

vector -c vector.toml --require-healthy true

with the attached config causes it to fail; however, if you remove the commented-out part or replace "SECRET" with something else, it starts successfully.

This is probably due to the regex that is used to check for which secrets backends needs to be available. The regex doesn't take into consideration that a line of the config could be commented out.

I can probably find time to open a PR to fix the regex by excluding text that comes after a # along with a test if that helps.

Configuration

[sources.system_logs]
type = "journald"
since_now = true

[sinks.debug_output]
type = "console"
inputs = ["system_logs"]
encoding.codec = "json"
encoding.json.pretty = true

# [sinks.datadog_output]
# type = "datadog_logs"
# default_api_key = "SECRET[datadog_token_fetcher.api_key]"
# inputs = ["system_logs"]

Version

vector 0.43.0 (x86_64-unknown-linux-gnu 88e017f 2024-12-03 22:44:54.502541095)

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Thanks @chradcliffe . This is very related to #797 but I think the separate issue still make sense since they could be addressed separately. The issue is that secrets and environment variable interpolation happens before any configuration parsing. Fixing that is unfortunately a deeper issue than it might seem on the surface since configuration, without the interpolation, may not parse correctly (the simplest example is foo: ${FOO} where foo is expected to be a number).

[jszwedko]

Ah, I see your suggested solution of updating the regex to look for comment characters. That could be a reasonable stop-gap. It'd need to handle both TOML and YAML comments (respectively).

[jszwedko]

One potential wrinkle that pops to mind is YAML and TOML string literals. For example:

foo: |
  # ${FOO}

There I think # character doesn't start a comment.