MM-54986: Disable app and/or agent profiling

[agarciamontoro]

Summary

Motivated for the huge amount of memory consumed by profiling 50 agents in the ceiling tests, we decided to allow the user to selectively disable that type of profiling. This PR solves that by making it optional to profile both the app and the agent nodes.

Note that the approach to test the generated yaml is a bit esoteric, but parsing the generated yaml seemed even more convoluted than this. I'm open to changing it if your eyes bleed a lot when reading the test, though. Ended up using go-yaml to generate the config, much easier and cleaner.

Ticket Link

https://mattermost.atlassian.net/browse/MM-54986

[streamer45]

One concern is that without tweaking the template we'd still be registering an application but with no targets. I am wondering if that could potentially get confusing or even cause issues.

[isacikgoz]

LGTM, feel free to ask for another review if what Claudio mentioned causes any issues.

[agarciamontoro]

Ended up using go-yaml not to parse the generated string, but to generate the yaml from scratch. I'm testing it right now in a deployment to make sure that the generated config is valid for Pyroscope.

[agarciamontoro]

Tested this with 2 app nodes, 2 agents, and all combinations of the settings, and all work. If both are false, we just don't get anything:

• EnableAppProfiling: true
• EnableAgentProfiling: true

log-level: debug
no-self-profiling: true
scrape-configs:
    - job-name: pryoscope
      scheme: http
      scrape-interval: 60s
      enabled-profiles: [cpu, mem, goroutines]
      static-configs:
        - application: mattermost
          spy-name: gospy
          targets: ['app-0:8067', 'app-1:8067']
        - application: agentsab
          spy-name: gospy
          targets: ['agent-0:4000', 'agent-1:4000']

• EnableAppProfiling: false
• EnableAgentProfiling: true

log-level: debug
no-self-profiling: true
scrape-configs:
    - job-name: pryoscope
      scheme: http
      scrape-interval: 60s
      enabled-profiles: [cpu, mem, goroutines]
      static-configs:
        - application: agents
          spy-name: gospy
          targets: ['agent-0:4000', 'agent-1:4000']

• EnableAppProfiling: true
• EnableAgentProfiling: false

log-level: debug
no-self-profiling: true
scrape-configs:
    - job-name: pryoscope
      scheme: http
      scrape-interval: 60s
      enabled-profiles: [cpu, mem, goroutines]
      static-configs:
        - application: mattermost
          spy-name: gospy
          targets: ['app-0:8067', 'app-1:8067']

• EnableAppProfiling: false
• EnableAgentProfiling: false

log-level: debug
no-self-profiling: true
scrape-configs:
    - job-name: pryoscope
      scheme: http
      scrape-interval: 60s
      enabled-profiles: [cpu, mem, goroutines]

[streamer45]

Looks so much better, I appreciate the time taken to rework this 🥇

[agnivade]

Nice one