how to control amount of memory used by vector?

[yjagdale]

I tried controlling the batch at the sink level but still the amount of memory is not getting controller,

[pront]

Hi @yjagdale, this is a great question. Are you referring to a process wide limit or per component? Regardless, here is some information that you might find useful.

systemd service manager

If your Vector instance runs as a systemd service (Linux), you can configure resource limits in the service file which usually found at /etc/systemd/system/vector.service:

[Service]
MemoryLimit=1G

Configure in-memory buffers

Of course this is not really a process (or even a component) wide limit, but might be helpful:

    buffer:
    - type: memory
      max_events: 1000
      when_full: drop_newest

Tip

For monitoring memory usage you can use vector top, https://vector.dev/docs/reference/cli/#top.

[yjagdale]

When I set the memory limit, Then the service is going out of memory( I am seeing this at kubernetes pod level).
I am using kafka as input then few log to metrics and output to prometheus.

[jszwedko]

I think you may be interested in this feature request: #8792.

There is no way, at the moment, to globally limit Vector memory use in a way that Vector will apply back-pressure rather than just OOM. There are some knobs that can be turned like buffer limits mentioned above. You mentioned Kafka as the source so you may also be interested in some of the options mentioned in https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md