Standalone agent on Mac won't reload changed output credentials

[jen-huang]

• Version: 8.14.0 BC 1
• Operating System: Mac, Apple M2 Pro
• Steps to Reproduce:

1. Start a stack with elastic-package stack up --version 8.14.0-SNAPSHOT
2. Download elastic-agent-8.14.0-darwin-aarch64.tar.gz from 8.14.0 BC1 builds
3. Create a simple agent policy in Kibana w/ just system integration, download it and modify it to suit running in standalone, copy it to the directory as elastic-agent.yml and give it incorrect ES credentials like this:

outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://127.0.0.1:9200'
    ssl.ca_trusted_fingerprint: REDACTED
    preset: balanced
    username: elastic
    password: wrongpassword

4. Run sudo ./elastic-agent install, don't enroll into fleet
5. Run sudo elastic-agent inspect, verify that it is using wrongpassword for ES output
6. Run sudo elastic-agent logs, verify that it is not connecting to ES:

{"log.level":"info","@timestamp":"2024-04-26T19:45:21.971Z","message":"Attempting to reconnect to backoff(elasticsearch(https://127.0.0.1:9200)) with 15 reconnect attempt(s)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":139,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-26T19:45:21.977Z","message":"'ca_trusted_fingerprint' set, looking for matching fingerprints","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":179,"file.name":"tlscommon/tls_config.go","function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.trustRootCA"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"tls","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-26T19:45:21.977Z","message":"CA certificate matching 'ca_trusted_fingerprint' found, adding it to 'certificate_authorities'","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"ecs.version":"1.6.0","log.logger":"tls","log.origin":{"file.line":199,"file.name":"tlscommon/tls_config.go","function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.trustRootCA"},"service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-04-26T19:45:32.372Z","message":"Failed to connect to backoff(elasticsearch(https://127.0.0.1:9200)): 401 Unauthorized: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

7. Edit elastic-agent.yml to fix the ES credentials:

outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://127.0.0.1:9200'
    ssl.ca_trusted_fingerprint: REDACTED
    preset: balanced
    username: elastic
    password: changeme

8. After waiting a little bit (I gave it a lot of time while writing this issue 😉), run sudo elastic-agent inspect again and see that it still shows using wrongpassword
9. Run sudo elastic-agent logs logs and see that logs still show ES connection issues

[jen-huang]

For fun, I uninstalled and reinstalled elastic agent and then the correct ES credentials were being used. Verified that data was being sent after that.

[ycombinator]

Edit elastic-agent.yml to fix the ES credentials.

Just to clarify, which elastic-agent.yml file did you edit? The one in the directory you extracted after downloading Agent but before running sudo ./elastic-agent install or the one in the directory where Elastic Agent is installed (/Library/Elastic/Agent/ by default on MacOS).

[jen-huang]

The one in the directory you extracted after downloading Agent but before running sudo ./elastic-agent install

This one!

[ycombinator]

The one in the directory you extracted after downloading Agent but before running sudo ./elastic-agent install

This one!

Thanks. That's expected behavior then.

Essentially, when Agent installs itself, it copies the elastic-agent.yml file from the directory where sudo ./elastic-agent install is run from to the Agent's installation location for that OS (which, for MacOS, is /Library/Elastic/Agent/ by default). So modifying the original elastic-agent.yml is not expected to have any effect. After installation, the config that Agent looks at is the one in the installation location.

Could you try modifying the elastic-agent.yml file in the installation location in your scenario, restart Agent, and see if the changes are picked up?

[jen-huang]

TIL :) Thanks, will re-test. Sorry for the false alarm!

[jen-huang]

Modifying the correct file at /Library/Elastic/Agent/elastic-agent.yml is indeed very helpful and works as expected. Closing!