Restart snapshotter gracefully even if there are running containers

[luodw]

recently, I have tried containerd lazy load feature. stargz-snapshotter use fusefs to fetch file data on demand, and in fuse userspace handler, if file is cached locally, it will return directly, if not cached locally, it will fetch from docker registry.

my question is, if stargz-snapshotter restart abnormally or when upgrade, the fuse connections will break, so container process read will failed. Is there some good practice?

[ktock]

@luodw Thanks for the question! Though we have graceful shutdown on SIGINT (#26), recovery on abnormal shutdown / support for service restart are in progress (#134). Very welcome for contribution.

[luodw]

@luodw Thanks for the question! Though we have graceful shutdown on SIGINT (#26), recovery on abnormal shutdown / support for service restart are in progress (#134). Very welcome for contribution.

Thanks for your reply, I got it.

[ktock]

@luodw Can you check if the master version (contains the patch #134) fixes this issue?

[luodw]

@luodw Can you check if the master version (contains the patch #134) fixes this issue?

I hava tried the latest master branch (containes the patch #134 ), but when I 'kill -9 ', and restart right now, the container still has err

The follow steps reproduce the issue

1. ctr-remote images rpull docker.io/stargz/golang:1.12.9-esgz
2. ctr-remote run --rm -t --snapshotter=stargz docker.io/stargz/golang:1.12.9-esgz test /bin/bash
3. kill -9 and restart right now
4. run some commands in container

[ktock]

Currently, you need to re-run containers too. And I agree with that the snapshotter needs to be able to gracefully restart even if there are running containers.

[luodw]

Currently, you need to re-run containers too. And I agree with that the snapshotter needs to be able to gracefully restart even if there are running containers.

Ok，I also think the ideal usage is when snapshotter restarts, the running containers can still run normally.

[amrmahdi]

@ktock can you describe what is required to do an update/restart to the snapshotter in a running cluster for instance? How do you do that today?

[ktock]

Currently, we need to kill all containers running on that node before restarting this snapshotter and re-deploy these containers after the snapshotter restarts.

One of the idea to solve this issue is spawning the FUSE server as a separated process instead of goroutine as done today.