How does compression work?

[pmajocha]

Hi, I'd like to know more about the compression option for streams. We've got a situation, where du -h on the stream in a node returns 100 GB , while the actual stream takes up 626 GB when listed with nats stream ls. I don't mind it showing a value like that, but when we approached a memory limit, it takes into account that displayed through nats stream ls rather than the actual size that it takes returned by du -h. In conclusion the compression doesn't seem to save us any space, as after reaching the memory limit, nats refuses to work untill messages are dropped. We would like to keep disks as slim as possible with as long max-age setting as possible thus our interest in compression. How does it work under the hood?

[derekcollison]

Compression works per block under filestore but we do report raw usage and that is what is used for limits.

[wronan]

Do you mean that the physical size of the available storage (being the most basic and unavoidable limit) also applies to the raw message size? Till now I assumed that this applies only to the configurable limits (like MaxBytes).

[ripienaar]

Yes, like we keep saying, only the message bytes are considered not bytes on disk.

[ripienaar]

You've confirmed for yourself how it works, that's how it works :)

[wronan]

You've talked about NATS comparing limits to uncompressed message size - to be frank I haven't thought for a second that besides configurable limits you also mean hard limit of the physical storage.
I have to say that to me such policy sounds very peculiar but it is what it is.

Gentleman, everything seems clear now, thank you very much for your time and sorry that we've kept you engaged in this thread for so long. It just reflects how odd this behavior seems to us.
Anyway, thanks again for you patience.

[ripienaar]

No problem, happy to help clarify, I agree it seems weird that disk usage limits would be pre-compression. Something to ponder about for sure.