Cannot remove corrupted consumer

[b04112106]

Observed behavior

I'm using nats server with jetstream and have server consumers. Sometimes the meta.inf of some consumers might broken for many reasons like the loss of power. After that, I saw

Error unmarshalling consumer metafile "/root/nats/jetstream/$G/streams/<stream_name>/obs/<random_consumer_name>/meta.inf": unexpected end of JSON input

Expected behavior

And I cannot delete it in with the connection to nats server in golang because the consumer is not considered as a valid consumer though. All consumers I have are ephemeral consumers. I wonder if I could just delete the folders to avoid this problem.

Server and client version

NATs Server version: 2.10.5
Golang Version: 1.31.0

Host environment

No response

Steps to reproduce

Use default nats-server image and start a server. After initialize a jetStream and a pull consumer, you could modify the meta.inf file and make it an incomplete json file. Then you could not delete the consumer any more for you won't track the consumer by the nats.JetStreamContext in golang.

[neilalexander]

If it's a single server installation (not a cluster) then yes, shutting down the server and deleting the broken consumers from the obs folder should be enough to get rid of them, although we do consider this to be "last resort".

If you are expecting frequent power outages though, you should consider upgrading to the latest version and setting sync: always in your server config to ensure more aggressive flushing of consumer and stream data to disk.

2.10.23 and 2.10.24 also have a number of fixes to improve the recovery of a server at startup.

[b04112106]

Hi @neilalexander ! Yes, it's a single server installation. I just wonder if there is any side effect or inconsistency of data between the server and folder. If this operation is safe and won't cause any problem, I might try this solution.

May I know where to find the release note of version 2.10.23 and 2.10.24? The latest one I can find in this page https://docs.nats.io/release-notes/whats_new is the release note of 2.10.0.

[souravagrawal]

Hi @b04112106 you can get all changes for every version here here https://github.com/nats-io/nats-server/releases