[Diagnostic] Folder and file naming length

[pierrehilbert]

Issue

On windows, unpacking diagnostics can hit the max path allowed length of 255 characters.
This problem is happening because folders and files name are long especially when we are requesting tracers.
To solve this problem, one solution could be to avoid to use stream-id in both the folder name and the file name.

Definition of done

• Max length into diagnostic archived are matching the Windows limitation
• Tests are in place to ensure that this behavior will remain correct

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

This problem is happening because folders and files name are long especially when we are requesting tracers.

The original report of this was specific to the httpjson request tracer, which is not something Elastic Agent generates. It is a feature of the httpjson input. The simplest solution is to limit the file name generated for request traces at the source.

https://github.com/elastic/beats/blob/6080d9b7b3c5cc81782e06336f66a4ffc507eb03/x-pack/filebeat/docs/inputs/input-httpjson.asciidoc#L633-L642

[jlind23]

@pierrehilbert @cmacknz Looks like this should have been transferred to another team according @fearful-symmetry, could you please update this?

[andrewkroh]

Background

Today there are three inputs that support this tracing feature - http_endpoint, httpjson and cel. They use the Filebeat input ID as part of their filename. Filebeat constructs the input ID by taking the configured ID, which is set by Fleet to the stream ID, and appending ¹ the source name (e.g. the URL for cel/httpjson). An example Filebeat input ID would be slack.audit-01611180-5328-11ee-88c5-f510aa5a574a::https://api.slack.com/audit/v1/logs.

The integrations are then configured ² to write the request tracer logs into logs/httpjson/http-request-trace-*.ndjson where * is a sanitized version of the Filebeat input ID. This results in a long filename like

logs/httpjson/http-request-trace-httpjson-slack.audit-01611180-5328-11ee-88c5-f510aa5a574a_https_api.slack.com_audit_v1_logs.ndjson

I should also note that these files can be rotated and they will then contain the date like logs/httpjson/http-request-trace-httpjson-slack.audit-01611180-5328-11ee-88c5-f510aa5a574a_https_api.slack.com_audit_v1_logs-2023-11-25T13-10-48.578.ndjson.

Proposed Fix

Change Filebeat

• Add a new IDWithoutName string field to the v2/Context.
• Update the request tracer code that expands the * to use IDWithoutName instead of ID.

Change Fleet

Or if we had access to the stream ID directly in the Fleet handlebar templates (depends on elastic/kibana#121310), then we could avoid needing to change Filebeat, and we could directly configure each integration to use a pattern like

{{#if enable_request_tracer}}
request.tracer.filename: "../../logs/{{ stream_id }}/trace.ndjson"
{{/if}}

Footnotes

1. https://github.com/elastic/beats/blob/78dc6649400610e9f908af7c90f35819f061e4e0/filebeat/input/v2/input-cursor/input.go#L119 ↩

2. https://github.com/elastic/integrations/blob/9ad8d5cd46338e8665297cdda75d6cb10cc58d68/packages/slack/data_stream/audit/agent/stream/httpjson.yml.hbs#L4 ↩

[aleksmaus]

Reopening the issue since the change was reverted today.
elastic/beats#40980

Will follow up with further investigation.

[aleksmaus]

The revert was reverted back. The team found the root cause of the problem in the different place.
Closing it now.