Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add config reload debouncer #8439

Closed
wants to merge 7 commits into from
Closed

add config reload debouncer #8439

wants to merge 7 commits into from

Conversation

stuartnelson3
Copy link
Contributor

@stuartnelson3 stuartnelson3 commented Jun 22, 2022 •
edited
Loading

prevent multiple reloads in rapid succession.

to be tested in cloud, an attempt at seeing if this fixes the bug setting the output registry (#8383)

how to test

  1. Start the stack in cloud
  2. In the admin console, enable logs and metrics. Send them to this deployment.
  3. Send events to the apm-server (grab the secret token from the apm integration, but do not update the integration!)
  4. Navigate to stack monitoring; observe that metrics appear under "Output Events Rate"

@stuartnelson3
add config reload debouncer
f3c2cc8 
prevent multiple reloads in rapid succession.
@mergify
Copy link
Contributor

mergify bot commented Jun 22, 2022

This pull request does not have a backport label. Could you fix it @stuartnelson3? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-7.x is the label to automatically backport to the 7.x branch.
  • backport-7./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Jun 22, 2022
@apmmachine
Copy link
Contributor

apmmachine commented Jun 22, 2022 •
edited
Loading

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-06-27T19:28:53.502+0000

  • Duration: 52 min 25 sec

Test stats 🧪

Test Results
Failed 1
Passed 4110
Skipped 13
Total 4124

Test errors 1

Expand to view the tests failures

Build and Test / System and Environment Tests / TestFleetIntegrationMonitoring – github.com/elastic/apm-server/systemtest
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestFleetIntegrationMonitoring
2022/06/27 19:55:22 Building image elastic-agent-systemtest:8.4.0-4fe26f2a-SNAPSHOT (amd64) from docker.elastic.co/beats/elastic-agent:8.4.0-4fe26f2a-SNAPSHOT...
2022/06/27 19:55:23 Built image elastic-agent-systemtest:8.4.0-4fe26f2a-SNAPSHOT (amd64)
2022/06/27 19:55:25 Starting container id: 9ec0c0399f26 image: elastic-agent-systemtest:8.4.0-4fe26f2a-SNAPSHOT
2022/06/27 19:55:25 Waiting for container id 9ec0c0399f26 image: elastic-agent-systemtest:8.4.0-4fe26f2a-SNAPSHOT
2022/06/27 19:55:30 Container is ready id: 9ec0c0399f26 image: elastic-agent-systemtest:8.4.0-4fe26f2a-SNAPSHOT
    fleet_test.go:88: 
        	Error Trace:	fleet_test.go:88
        	Error:      	Not equal: 
        	            	expected: map[string]interface {}{"output":map[string]interface {}{"events":map[string]interface {}{"acked":15, "active":0, "batches":1, "failed":0, "toomany":0, "total":15}, "type":"elasticsearch", "write":map[string]interface {}{}}, "pipeline":map[string]interface {}{"events":map[string]interface {}{"total":15}}}
        	            	actual  : map[string]interface {}{"output":map[string]interface {}{"events":map[string]interface {}{"acked":0, "active":0, "batches":0, "failed":0, "toomany":0, "total":0}, "type":"elasticsearch", "write":map[string]interface {}{}}, "pipeline":map[string]interface {}{"events":map[string]interface {}{"total":0}}}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -3,8 +3,8 @@
        	            	   (string) (len=6) "events": (map[string]interface {}) (len=6) {
        	            	-   (string) (len=5) "acked": (float64) 15,
        	            	+   (string) (len=5) "acked": (float64) 0,
        	            	    (string) (len=6) "active": (float64) 0,
        	            	-   (string) (len=7) "batches": (float64) 1,
        	            	+   (string) (len=7) "batches": (float64) 0,
        	            	    (string) (len=6) "failed": (float64) 0,
        	            	    (string) (len=7) "toomany": (float64) 0,
        	            	-   (string) (len=5) "total": (float64) 15
        	            	+   (string) (len=5) "total": (float64) 0
        	            	   },
        	            	@@ -16,3 +16,3 @@
        	            	   (string) (len=6) "events": (map[string]interface {}) (len=1) {
        	            	-   (string) (len=5) "total": (float64) 15
        	            	+   (string) (len=5) "total": (float64) 0
        	            	   }
        	Test:       	TestFleetIntegrationMonitoring
    fleet_test.go:107: 
        	Error Trace:	fleet_test.go:107
        	Error:      	Not equal: 
        	            	expected: map[string]interface {}{"elasticsearch":map[string]interface {}{"bulk_requests":map[string]interface {}{"available":10, "completed":1}}}
        	            	actual  : map[string]interface {}{"elasticsearch":map[string]interface {}{"bulk_requests":map[string]interface {}{"available":10, "completed":0}}}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,3 +4,3 @@
        	            	    (string) (len=9) "available": (float64) 10,
        	            	-   (string) (len=9) "completed": (float64) 1
        	            	+   (string) (len=9) "completed": (float64) 0
        	            	   }
        	Test:       	TestFleetIntegrationMonitoring
    fleet_test.go:175: elastic-agent logs: {"log.level":"info","@timestamp":"2022-06-27T19:55:26.353Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":471},"message":"Starting enrollment to URL: https://fleet-server:8220/","ecs.version":"1.6.0"}
        Successfully enrolled the Elastic Agent.
        {"log.level":"info","@timestamp":"2022-06-27T19:55:27.138Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":271},"message":"Elastic Agent might not be running; unable to trigger restart","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:27.283Z","log.origin":{"file.name":"cmd/run.go","file.line":184},"message":"APM instrumentation disabled","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:27.284Z","log.origin":{"file.name":"application/application.go","file.line":78},"message":"Detecting execution mode","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:27.311Z","log.origin":{"file.name":"application/application.go","file.line":102},"message":"Agent is managed by Fleet","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:27.311Z","log.origin":{"file.name":"capabilities/capabilities.go","file.line":59},"message":"capabilities file not found in /usr/share/elastic-agent/state/capabilities.yml","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:27.911Z","log.logger":"composable.providers.docker","log.origin":{"file.name":"docker/docker.go","file.line":44},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:27.912Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":68},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:27.912Z","log.origin":{"file.name":"application/managed_mode.go","file.line":315},"message":"Agent is starting","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:27.912Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":70},"message":"Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/elastic-agent.sock)","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:29.313Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is 4i8Y2rWC","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:29.313Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 2 step(s)","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:29.475Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for apm-server.8.4.0-SNAPSHOT","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:29.830Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-27T19:55:29Z - message: Application: apm-server--8.4.0-SNAPSHOT[3d6abe2f-54ad-40df-b385-3452db42927a]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:30.128Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for filebeat.8.4.0-SNAPSHOT","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:30.567Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-27T19:55:30Z - message: Application: filebeat--8.4.0-SNAPSHOT--36643631373035623733363936343635[3d6abe2f-54ad-40df-b385-3452db42927a]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:30.894Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-27T19:55:30Z - message: Application: apm-server--8.4.0-SNAPSHOT[3d6abe2f-54ad-40df-b385-3452db42927a]: State changed to CONFIG: Updating configuration - type: 'STATE' - sub_type: 'CONFIG'","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:31.137Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for metricbeat.8.4.0-SNAPSHOT","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:31.489Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-27T19:55:31Z - message: Application: metricbeat--8.4.0-SNAPSHOT--36643631373035623733363936343635[3d6abe2f-54ad-40df-b385-3452db42927a]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:31.492Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:31.905Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-27T19:55:31Z - message: Application: filebeat--8.4.0-SNAPSHOT--36643631373035623733363936343635[3d6abe2f-54ad-40df-b385-3452db42927a]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:32.770Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-27T19:55:32Z - message: Application: metricbeat--8.4.0-SNAPSHOT--36643631373035623733363936343635[3d6abe2f-54ad-40df-b385-3452db42927a]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-06-27T19:55:36.404Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-27T19:55:36Z - message: Application: apm-server--8.4.0-SNAPSHOT[3d6abe2f-54ad-40df-b385-3452db42927a]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
    fleet_test.go:177: apm-server logs:
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.870Z","log.origin":{"file.name":"instance/beat.go","file.line":710},"message":"Beat ID: 4d14ab63-e358-4ac1-914e-f72083b1283c","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.871Z","log.origin":{"file.name":"instance/beat.go","file.line":727},"message":"Set gc percentage to: 100","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.871Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":63},"message":"Starting stats endpoint","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.871Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1076},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"/usr/share/elastic-agent/data/elastic-agent-e043f3/install/apm-server-8.4.0-SNAPSHOT-linux-x86_64","data":"/usr/share/elastic-agent/state/data/run/default/apm-server--8.4.0-SNAPSHOT","home":"/usr/share/elastic-agent/data/elastic-agent-e043f3/install/apm-server-8.4.0-SNAPSHOT-linux-x86_64","logs":"/usr/share/elastic-agent/data/elastic-agent-e043f3/install/apm-server-8.4.0-SNAPSHOT-linux-x86_64/logs"},"type":"apm-server","uuid":"4d14ab63-e358-4ac1-914e-f72083b1283c"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.872Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1085},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"unknown","libbeat":"8.4.0","time":"0001-01-01T00:00:00.000Z","version":"8.4.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.872Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1088},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.17.10"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.872Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":65},"message":"Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/default/apm-server/apm-server.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/default/apm-server/apm-server.sock)","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.872Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1092},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-06-27T19:39:49Z","containerized":true,"name":"9ec0c0399f26","ip":["127.0.0.1/8","172.18.0.8/16"],"kernel_version":"5.4.0-1080-gcp","mac":["02:42:ac:12:00:08"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.4 LTS (Focal Fossa)","major":20,"minor":4,"patch":4,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.873Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1121},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/elastic-agent/data/elastic-agent-e043f3/install/apm-server-8.4.0-SNAPSHOT-linux-x86_64","exe":"/usr/share/elastic-agent/data/elastic-agent-e043f3/install/apm-server-8.4.0-SNAPSHOT-linux-x86_64/apm-server","name":"apm-server","pid":27,"ppid":7,"seccomp":{"mode":"filter","no_new_privs":false},"start_time":"2022-06-27T19:55:29.380Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.873Z","log.origin":{"file.name":"instance/beat.go","file.line":293},"message":"Setup Beat: apm-server; Version: 8.4.0","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.874Z","log.origin":{"file.name":"instance/beat.go","file.line":322},"message":"Output is configured through Central Management","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.877Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: 9ec0c0399f26","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.878Z","log.origin":{"file.name":"instance/beat.go","file.line":470},"message":"apm-server start running.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-06-27T19:55:29.878Z","log.logger":"cfgwarn","log.origin":{"file.name":"management/manager.go","file.line":116},"message":"BETA: Fleet management is enabled","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.878Z","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":117},"message":"Starting fleet management service","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.879Z","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":125},"message":"Ready to receive configuration","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:29.880Z","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":1138},"message":"maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.394Z","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":162},"message":"Status change to Configuring: Updating configuration","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.395Z","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":287},"message":"Applying settings for inputs","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-06-27T19:55:30.396Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.899Z","log.logger":"config","log.origin":{"file.name":"config/rum.go","file.line":86},"message":"Unable to determine sourcemap storage, sourcemaps will not be applied","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.901Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":144},"message":"Listening on: [::]:8200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.901Z","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":287},"message":"Applying settings for output","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.901Z","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":62},"message":"creating transaction metrics aggregation with config: {Interval:1m0s MaxTransactionGroups:10000 HDRHistogramSignificantFigures:2}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.908Z","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":77},"message":"creating service destinations aggregation with config: {Interval:1m0s MaxGroups:10000}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.908Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path / added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.908Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /config/v1/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.908Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /config/v1/rum/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.908Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v2/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.908Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v3/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.908Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v2/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.908Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v2/profile added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.908Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /v1/traces added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.908Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /v1/metrics added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.909Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /v1/logs added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.909Z","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":233},"message":"Starting apm-server [unknown built 0001-01-01 00:00:00 +0000 UTC]. Hit CTRL-C to stop it.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.909Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":88},"message":"RUM endpoints enabled!","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-06-27T19:55:30.909Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":91},"message":"CORS related setting `apm-server.rum.allow_origins` allows all origins. Consider more restrictive setting for production use.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.909Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":106},"message":"SSL disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.927Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"Go-http-client/1.1","source.address":"172.18.0.1","http.request.body.bytes":0,"http.request.id":"62c0d773-a685-45eb-8c3b-be7261f497fd","event.duration":194582,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.936Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"apm-agent-go/2.0.0","source.address":"10.11.12.13","http.request.body.bytes":0,"http.request.id":"76d4ebff-9c2e-4a97-8c03-21899122e367","event.duration":152943,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.941Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/config/v1/agents?service.name=systemtest_test","http.request.method":"GET","user_agent.original":"apm-agent-go/2.0.0","source.address":"10.11.12.13","http.request.body.bytes":0,"http.request.id":"f77fd232-c067-4bb0-b584-93b165c29571","event.duration":103240,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:30.946Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-go/2.0.0","source.address":"10.11.12.13","http.request.id":"9d7d88a3-cb7f-4b95-b294-f7cdfbc6c5ff","event.duration":3660211,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.403Z","log.logger":"config","log.origin":{"file.name":"config/rum.go","file.line":89},"message":"Falling back to elasticsearch output for sourcemap storage","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.403Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":144},"message":"Listening on: [::]:8200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.406Z","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":62},"message":"creating transaction metrics aggregation with config: {Interval:1m0s MaxTransactionGroups:10000 HDRHistogramSignificantFigures:2}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.407Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":112},"message":"Stop listening on: :8200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.409Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":40},"message":"blocking ingestion until all preconditions are satisfied","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.425Z","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":77},"message":"creating service destinations aggregation with config: {Interval:1m0s MaxGroups:10000}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.437Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path / added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.437Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /config/v1/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.437Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /config/v1/rum/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.437Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v2/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.437Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v3/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.437Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v2/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.437Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v2/profile added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.437Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /v1/traces added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.437Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /v1/metrics added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.437Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /v1/logs added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.438Z","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":233},"message":"Starting apm-server [unknown built 0001-01-01 00:00:00 +0000 UTC]. Hit CTRL-C to stop it.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.438Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":88},"message":"RUM endpoints enabled!","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-06-27T19:55:31.438Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":91},"message":"CORS related setting `apm-server.rum.allow_origins` allows all origins. Consider more restrictive setting for production use.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.438Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":106},"message":"SSL disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:31.700Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"apm-agent-go/2.0.0 (metricbeat)","source.address":"127.0.0.1","http.request.body.bytes":0,"http.request.id":"af4eaa65-600a-42f0-83ea-7801aa3bae40","event.duration":126054,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:36.164Z","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":249},"message":"Server stopped","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:36.165Z","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":190},"message":"service destinations aggregation stopped","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:36.165Z","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":190},"message":"transaction metrics aggregation stopped","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-06-27T19:55:36.165Z","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":425},"message":"context canceled","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:36.165Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: http://elasticsearch:9200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:36.166Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(elasticsearch(http://elasticsearch:9200))","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:36.178Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.4.0-SNAPSHOT","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-27T19:55:36.181Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(elasticsearch(http://elasticsearch:9200)) established","service.name":"apm-server","ecs.version":"1.6.0"}
--- FAIL: TestFleetIntegrationMonitoring (26.02s)

Steps errors 1

Expand to view the steps failures

Run Linux tests
  • Took 15 min 29 sec . View more details here
  • Description: ./.ci/scripts/linux-test.sh

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /hey-apm : Run the hey-apm benchmark.

  • /package : Generate and publish the docker images.

  • /test windows : Build & tests on Windows.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@stuartnelson3 stuartnelson3 marked this pull request as ready for review June 23, 2022 13:33
@stuartnelson3 stuartnelson3 requested a review from a team June 23, 2022 13:33
@stuartnelson3
Add comments explaining the debouncer
be8600d 
Provide some context as to how the debouncer is
working within the code.
@stuartnelson3
Merge branch 'main' into debounce-config-reload
db3d3b2
@apmmachine
Copy link
Contributor

apmmachine commented Jun 23, 2022 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (42/42) 💚
Files 91.96% (183/199) 👍
Classes 93.45% (428/458) 👍 0.014
Methods 89.18% (1088/1220) 👍 0.1
Lines 76.853% (13364/17389) 👍 0.088
Conditionals 100.0% (0/0) 💚

axw
axw reviewed Jun 24, 2022
View reviewed changes
Copy link
Member

@axw axw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did this fix the issues seen in Cloud? If so, why, and can we add a test to cover whatever the problem was? The reload stuff is already complicated, I'm wary of making it even more so.

@stuartnelson3
merge loop() into trigger()
9cd7a12 
start the debounce() method on the first call to
trigger(), using an internal buffered channel to
indicate that fn is currently being debounced.
marclop
marclop reviewed Jun 24, 2022
View reviewed changes
beater/beater.go
// Update the internal state to show we're currently debouncing
// fn execution and start the debounce method.
go func() {
d.debounce(ctx)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we need to send the result of debounce back to res? Otherwise the error won't be propagated and line 308 will never happen either.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that, when we send res on line 308, that will be added to the slice of channels that will have the error sent to them. I think this is being verified in the test.

However, maybe the code would be more readable if we send on this channel, and move d.triggerc <- res under default?

@stuartnelson3
Copy link
Contributor Author

After building apm-server locally and packaging that into an elastic-agent container which is then run on cloud, it appears that the issue isn't related to the actual code within the apm-server, but potentially something around the compilation in CI. The locally built apm-server binary did not have the same bug as is seen with the BC created by CI, so this fix is likely unnecessary. Potentially in the future it might be nice to reduce the number of reloads if both an update to the config and the output config appear in rapid succession.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marclop marclop marclop left review comments

@axw axw axw left review comments

Assignees
No one assigned
Labels
backport-skip Skip notification from the automated backport with mergify
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@stuartnelson3 @apmmachine @axw @marclop