Revisit default TBS storage size limit sampling.tail.storage_limit and storage limit handling

[carsonip]

sampling.tail.storage_limit has a default of 3GB which is usually insufficient for high throughput use cases. A storage limit too low will cause TBS to be bypassed and other downstream effects. Reconsider the default in 9.0 release.

[carsonip]

I have 2 ideas for the new default.

• to allow infinite disk usage
• to detect available disk space (on startup / periodically) and set the limit dynamically

The argument for either of them is that e.g. if you are running a mysql server, you expect it to use storage space but you don't expect yourself updating a "storage limit" every time it gets hit.

The argument for a detection is to avoid causing a node to be unhealthy in any environments.

[simitt]

@carsonip the solution to detect available disk space would be more elegant. We would need to ensure that it works on all supported OS systems + docker and have a sensible fallback if disk size cannot be detected.
Do you have capacity to look into this?

[carsonip]

Do you have capacity to look into this?

Yes, it is in scope of this task. I'll write down some possible solutions soon.