prometheus_scrape source: add compatibility with HTTP Service Discovery

[jhbigler]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Normally, prometheus discovers services with a local file - however it can also use an http-based service discovery, which returns services and labels in JSON format. It seems like vector's prometheus_scrape source expects the entire service definition in its config file - it would be nice if it had the ability to use the HTTP SD.

My use case: Zeek's telemetry framework recommends using its service discovery endpoint (it can collect all of the metrics from all nodes, but that is costly on Zeek). Because zeek can be using many workers with many endpoints to collect metrics from, it's hard to know ahead of time all of the endpoints to collect metrics from. As such, I'd like to use a HTTP SD.

Attempted Solutions

No response

Proposal

Perhaps a new parameter like http_service_discovery_endpoint?

References

No response

Version

No response

[jszwedko]

Hey @jhbigler ! I think this is a duplicate of #2303 so I'll close this one, but feel free to follow along or add additional details to the other.