statsd metric overflow

[tkrafael]

Hi,
We're troubleshooting an app in production.
Our app does:

1. Read from S3 (using sqs for event)
2. Do some processing on input and generate a new root
3. Send output to SQS (batch mode, 64 concurrent requests)
   For metrics, we're using telegraf's statsd with all default configuration. Telegraf is running as side car in same ecs task

From time to time we get an error level=warning msg="[STATSD] 814 packets lost (overflow)" @service=benthos
After digging the problem further, we found that nearly 50% of metrics were lost. We're handing 1M messages, the target system receives 1M requests but benthos only shows 424k requests (this number varies).

Can someone provide some advice on it?

[Jeffail]

Hey @tkrafael, you might need to tune the flush period so that it clears its memory more frequently as I'd imagine the client simply cannot keep up with the volume of traffic, and rather than block the overal flow of traffic is leans on the side of dropping observability data. It's happening within the statsd client library we're using so I can't say for absolute sure what's happening with investigating it deeper. Converting to discussion as per #2026