[tempo-distributed]: Load external MinIO credentials from secrets

[peske]

To set external MinIO credentials in the current values file, you should do something like:

storage:
  trace:
    backend: s3
    s3:
      access_key: 'grafana-tempo'
      secret_key: 'supersecret'
      bucket: 'tempo-traces'
      endpoint: 'tempo-minio:9000'
      insecure: true

It would be much better if there's an option to load S3 access_key and especially secret_key from Kubernetes Secret.

[peske]

As a temporary solution I've came up with some gymnastic, to at least avoid risk of access keys ending up in source code, with the following steps:

Create secret:

kubectl -n tempo create secret generic minio-access-key \
  --from-literal=access_key_id="XXXXX" \
  --from-literal=access_key_secret="YYYYY"

In values file:

storage:
  trace:
    backend: s3
    s3:
      bucket: "grafana-tempo"
      endpoint: "http://minio-cluster.minio.svc.cluster.local:9000"
      insecure: true
      access_key: "${MINIO_ACCESS_KEY_ID}"
      secret_key: "${MINIO_SECRET_KEY}"

Then to deploy:

# Load credentials from secret:
export MINIO_ACCESS_KEY_ID=$(kubectl -n tempo get secret minio-access-key \
  -o jsonpath="{.data.access_key_id}" | base64 --decode)
export MINIO_SECRET_KEY=$(kubectl -n tempo get secret minio-access-key \
  -o jsonpath="{.data.access_key_secret}" | base64 --decode)

# Deploy:
envsubst < values.yaml | helm upgrade -i tempo \
  grafana/tempo-distributed --version 1.28.0 --namespace tempo -f -

# Remove access keys from memory:
unset MINIO_ACCESS_KEY_ID
unset MINIO_SECRET_KEY

But then again, the access keys are visible in config map tempo-config...

[peske]

I've just learned that Tempo config supports environment variables, so there's a better solution. First create a secret that contains the environment variables:

kubectl -n tempo create secret generic minio-access-key \
  --from-literal=MINIO_ACCESS_KEY_ID="XXXXX" \
  --from-literal=MINIO_SECRET_KEY="YYYYY"

Than configure storage in values file in the same way as in my previous comment:

storage:
  trace:
    backend: s3
    s3:
      bucket: "grafana-tempo"
      endpoint: "http://minio-cluster.minio.svc.cluster.local:9000"
      insecure: true
      access_key: "${MINIO_ACCESS_KEY_ID}"
      secret_key: "${MINIO_SECRET_KEY}"

Finally, also in the values file, ensure that ingester, metricsGenerator, compactor, querier and queryFrontend have the following set:

  extraArgs:
    - "-config.expand-env=true"
  extraEnvFrom:
    - secretRef:
        name: minio-access-key

For example, compactor should look something like:

compactor:
  replicas: 2
  extraArgs:
    - "-config.expand-env=true"
  extraEnvFrom:
    - secretRef:
        name: minio-access-key

So yes, it is possible to configure MinIO credentials in secret, but there's a lot of repetition in this approach. I still think that it should be better to improve the chart to provide a simpler way out-of-the-box.