[Flaky Test]: TestBeatsServerless/TestSetupDashboards – error importing Kibana dashboards

[rdner]

Failing test case

TestBeatsServerless/TestSetupDashboards

Error message

error importing Kibana dashboards

Build

https://buildkite.com/elastic/elastic-agent/builds/8118#018eae17-30fe-418a-aafa-74560b090baf

OS

Linux

Stacktrace and notes

>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): Test:       	TestBeatsServerless/TestSetupDashboards
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): beats_serverless_test.go:215: got response from dashboard setup: Loading dashboards (Kibana must be running and reachable)
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): Exiting: error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /tmp/TestBeatsServerless20915224/001/metricbeat-8.14.0-SNAPSHOT-linux-x86_64/kibana: Failed to import dashboard: Failed to load directory /tmp/TestBeatsServerless20915224/001/metricbeat-8.14.0-SNAPSHOT-linux-x86_64/kibana/7:
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): error loading /tmp/TestBeatsServerless20915224/001/metricbeat-8.14.0-SNAPSHOT-linux-x86_64/kibana/7/dashboard/a3f4d080-c7d3-11ec-891a-5f50c7522a3a.json: error dashboard asset: returned 500 to import file: An internal server error occurred. Check Kibana server logs for details.. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred. Check Kibana server logs for details."}
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): beats_serverless_test.go:216:
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): Error Trace:	/home/ubuntu/agent/testing/integration/beats_serverless_test.go:216
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): Error:      	Should be true
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): Test:       	TestBeatsServerless/TestSetupDashboards
>>> (linux-amd64-ubuntu-2204-default) Test output (sudo) (stdout): === FAIL: testing/integration TestBeatsServerless (321.08s)

First failure was around
2024-04-05 12:32:42 UTC

Serverless project:
stack_id: 8140-SNAPSHOT, deployment_id: a009bf2e816f42cd88e45d45d15c833b

This is the dashboard that causes the error https://github.com/elastic/beats/blob/main/x-pack/metricbeat/module/oracle/_meta/kibana/7/dashboard/a3f4d080-c7d3-11ec-891a-5f50c7522a3a.json

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[rdner]

Another failure https://buildkite.com/elastic/elastic-agent/builds/8165#018eb9f8-47cd-4cf1-8c68-d481385740c7

2024-04-07 19:32:41 UTC
Deployment becfa6cde22f4920b9d47c3acf73b534

[rdner]

The investigation led to some failed requests from Kibana to Elasticsearch. Will still observe for a couple of weeks, if there is no another occurrence, we can close this as a temporary connectivity issue.

[rdner]

Seems to be a temporary issue, no re-occurrence in weeks.