Prometheus scrape does not support retry or retry-after

[st-omarkhalid]

Use Cases

We have vector-agent running in our production environment and noticed the logs have timeout errors like below.

component_type=prometheus_scrape}:http: vector::internal_events::http_client: Sending HTTP request. uri=http://<ip>:8080/metrics method=GET version=HTTP/1.1 headers={"accept": "text/plain", "user-agent": "Vector/0.35.0 (x86_64-unknown-linux-gnu)", "accept-encoding": "identity"} body=[empty]

vector::internal_events::http_client_source: HTTP request processing error. url=http://<ip>:8080/metrics error="Timeout error: request exceeded 30s" error_type="request_failed" stage="receiving" internal_log_rate_limit=true

It appears the prometheus_scrape does not have a retry option which could be helpful so we don't have to wait for the next scrape run.

Attempted Solutions

No response

Proposal

No response

References

No response

Version

0.35.0

[st-omarkhalid]

Config

data_dir: /var/lib/vector
api:
  enabled: true
  address: 0.0.0.0:8686
  playground: true
sources:
  host_metric_source:
    type: host_metrics
    scrape_interval_secs: 120
  internal_metric_source:
    type: internal_metrics
    scrape_interval_secs: 120
  node:
    type: prometheus_scrape
    endpoints:
    - http://<ip>:9100/metrics
    scrape_interval_secs: 120
    scrape_timeout_secs: 30
sinks:
  vector_sink:
    type: vector
    inputs:
    - tiering.tier1
    address: <url>
    tls:
      alpn_protocols:
      - h2
      enabled: true
    buffer:
      type: disk
      when_full: drop_newest
      max_size: 10737418240
    batch:
      max_bytes: 2500000
      timeout_secs: 10
    healthcheck: true
    compression: true
    headers:
      auth-version: "1"
transforms:
  remap_node:
    type: remap
    inputs:
    - node
    source: |-
      .tags.nodename = "node-<ip>"
      .tags.service_name = "nodeExporter"
  tiering:
    type: route
    inputs:
    - remap_*
    route:
      tier1: .name == .name
    reroute_unmatched: false