Bucket Remove Event for Watchers

[amaster507]

Proposed change

Alert watchers that the bucket they are watching is being removed with a shutdown notification.

Use case

When using the CLI I received the error:

Unexpected NATS error from server <server>: nats: stream not found: recreating ordered consumer.

I did this by the following:

1. nats kv add my-kv
2. nats kv watch my-kv
3. Assynchronously nats kv rm my-kv

I did this testing theories as I learn NATS, but I wanted to add that although it may not be likely for key value buckets to be removed, the API allows for it, and the handling of it with watchers is error prone.

Furthermore, I proved that if you were to re-add the bucket before the error is reported in the CLI, and add/update/remove keys these do not post events to the original watcher. The API does not allow watching buckets that do not exist, which makes logical sense, but there is no safe closure of watchers to buckets that get removed.

With the error that was presented, without knowing better a client might try to re-watch the bucket, but if they knew the bucket was removed, then they would know to not try to re-watch.

Contribution

maybe, I could try, but not very experienced with golang.