Windows Agent gets unhealthy on adding Windows Integration when installed with --unprivileged flag,

[amolnater-qasource]

Kibana Build details:

VERSION: 8.14.0 BC2
BUILD: 73626
COMMIT: bcf6960778ae270d0894a8aab07f10197ee9b97f

Preconditions:

1. 8.14.0-BC2 Kibana cloud environment should be available.
2. Agent should be installed with unprivileged flag.

Steps to reproduce:

1. Add Windows integration to the agent policy.
2. Observe under Agents tab Windows agent gets unhealthy.
3. Observe windows.perfmon dataset is not generated too.

Expected Result:
Windows Agent should remain healthy on adding Windows Integration when installed with --unprivileged flag,

What's working fine:

• Windows Agent remains healthy on adding Windows Integration when installed without --unprivileged flag,

Diagnostics:
elastic-agent-diagnostics-2024-05-02T09-00-03Z-00.zip

Screen Recording:

Agents.-.Fleet.-.Elastic.-.Google.Chrome.2024-05-02.14-28-45.mp4

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Secondary review for this ticket is Done

[blakerouse]

Permission error because the elastic-agent-user cannot read those metrics.

Error is:

        units:
            input-windows/metrics-default-windows/metrics-windows-90101c17-86f7-46b9-9247-a1c36041c3e7:
                message: '1 error: initialization of reader failed: failed to expand counter (query=''\Process(*)\% Processor Time''): Unable to read the counter and/or explain text from the specified computer.'
                state: 4

More permission errors in the logs:

{"log.level":"error","@timestamp":"2024-05-02T08:55:00.490Z","message":"Error fetching data for metricset system.diskio: disk io counters: cannot open new key in the registry in order to enable the performance counters: Access is denied.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":256,"file.name":"module/wrapper.go","function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-05-02T08:55:10.487Z","message":"Error fetching data for metricset system.diskio: disk io counters: cannot open new key in the registry in order to enable the performance counters: Access is denied.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":256,"file.name":"module/wrapper.go","function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-05-02T08:55:20.500Z","message":"Error fetching data for metricset system.diskio: disk io counters: cannot open new key in the registry in order to enable the performance counters: Access is denied.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":256,"file.name":"module/wrapper.go","function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2024-05-02T08:56:30.924Z","message":"Error fetching data for metricset windows.service: OpenProcess failed for pid=2768: Access is denied.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"windows/metrics-default","type":"windows/metrics"},"log":{"source":"windows/metrics-default"},"log.origin":{"file.line":256,"file.name":"module/wrapper.go","function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

This is all expected in unprivileged mode. If you want those metrics the elastic-agent-user needs to be added to the correct groups.

[cmacknz]

We should use this issue to determine which group fixes this so we can explicitly document the fix for this error.

[ycombinator]

We should use this issue to determine which group fixes this so we can explicitly document the fix for this error.

@blakerouse could you figure this out, perhaps with help from @leehinman, and then document it as part of #4705? I'm trying to use that issue as a single place to capture all prerequisites required for successfully running Agent in unprivileged mode. Thanks!