rpull pulls all layers?

[nicktming]

containerd: v1.4.3
stargz-snapshotter: v0.3.0
$ ctr-remote images rpull --plain-http myip:5000/myimage:v0.1
fetching sha256:d737a6fd... application/vnd.docker.distribution.manifest.v2+json
fetching sha256:8c4e826b... application/vnd.docker.container.image.v1+json
fetching sha256:0703b966... application/vnd.docker.image.rootfs.diff.tar.gzip
fetching sha256:e966595c... application/vnd.docker.image.rootfs.diff.tar.gzip
fetching sha256:3b36b637... application/vnd.docker.image.rootfs.diff.tar.gzip

is it normal case? it pulls the tar.gzip besides image manifest/config files. and it is pending here, rpull takes a long time.

[ktock]

Have you done registry configuration on the snapshotter's side?

https://github.com/containerd/stargz-snapshotter/blob/v0.3.0/docs/overview.md#registry-mirrors-and-insecure-connection

something like (/etc/containerd-stargz-grpc/config.toml):

[[resolver.host."myip:5000".mirrors]]
host = "myip:5000"
insecure = true

If it doesn't work, please provide the debug log of containerd-stargz-grpc.

[nicktming]

Have you done registry configuration on the snapshotter's side?

https://github.com/containerd/stargz-snapshotter/blob/v0.3.0/docs/overview.md#registry-mirrors-and-insecure-connection

something like (/etc/containerd-stargz-grpc/config.toml):

[[resolver.host."myip:5000".mirrors]]
host = "myip:5000"
insecure = true

If it doesn't work, please provide the debug log of containerd-stargz-grpc.

it still pulls the tar.gzip layers. by the way, i forget to hide my ip address, would you mind hiding it in your comment?

[ktock]

@nicktming Could you provide the log output from containerd-stargz-grpc?

[nicktming]

@nicktming Could you provide the log output from containerd-stargz-grpc?

there is no output
my containerd config

# Plug stargz snapshotter into containerd
# Containerd recognizes stargz snapshotter through specified socket address.
# The specified address below is the default which stargz snapshotter listen to.
[proxy_plugins]
  [proxy_plugins.stargz]
    type = "snapshot"
    address = "/run/containerd-stargz-grpc/containerd-stargz-grpc.sock"

# Use stargz snapshotter through CRI
[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "stargz"
  disable_snapshot_annotations = false

i already covert my ubuntu image to stargz-format image and pushed to my private registry. but ctr-remote images rpull will pull the tar.gzip layer, as https://github.com/containerd/stargz-snapshotter

# ctr-remote image rm --sync registry2:5000/ubuntu:18.04
# ctr-remote images rpull --plain-http registry2:5000/ubuntu:18.04
fetching sha256:728332a6... application/vnd.docker.distribution.manifest.v2+json
fetching sha256:80026893... application/vnd.docker.container.image.v1+json
# ctr-remote run --rm -t --snapshotter=stargz registry2:5000/ubuntu:18.04 test /bin/bash
root@8dab301bd68d:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

it only pulls the manifest config.

[nicktming]

@ktock i tested the docker.io/stargz/golang:1.12.9-esgz

fetching sha256:e31d9d69... application/vnd.docker.distribution.manifest.v2+json
fetching sha256:7a26855a... application/vnd.docker.container.image.v1+json
fetching sha256:7b640c31... application/vnd.docker.image.rootfs.diff.tar.gzip
fetching sha256:1a2751b9... application/vnd.docker.image.rootfs.diff.tar.gzip
fetching sha256:cc61f9f9... application/vnd.docker.image.rootfs.diff.tar.gzip
fetching sha256:191d7ea8... application/vnd.docker.image.rootfs.diff.tar.gzip
fetching sha256:18c99e16... application/vnd.docker.image.rootfs.diff.tar.gzip
fetching sha256:8ea059e7... application/vnd.docker.image.rootfs.diff.tar.gzip
fetching sha256:3d16b66d... application/vnd.docker.image.rootfs.diff.tar.gzip

it pulls all the layers, is it the normal case?

logs of containerd, containerd-stargz does not output log.

ERRO[2021-02-02T14:38:14.794557043+08:00] (*service).Write failed                       error="rpc error: code = Canceled desc = context canceled" expected="sha256:cc61f9f96a6c0f62d035a81a44ee375f8b2e801179e04be279e1d70e2aca42c6" ref="layer-sha256:cc61f9f96a6c0f62d035a81a44ee375f8b2e801179e04be279e1d70e2aca42c6" total=10048621
ERRO[2021-02-02T14:38:14.794557113+08:00] (*service).Write failed                       error="rpc error: code = Canceled desc = context canceled" expected="sha256:7b640c317fbd89408a7be207ec14dd8565ebd40e9f3b6fdffd181f92a119295c" ref="layer-sha256:7b640c317fbd89408a7be207ec14dd8565ebd40e9f3b6fdffd181f92a119295c" total=52954604
ERRO[2021-02-02T14:38:14.794600664+08:00] (*service).Write failed                       error="rpc error: code = Canceled desc = context canceled" expected="sha256:8ea059e7807349882824ba1b38b90f2ea7bcd4210542bd35c92a0688fa6abbf2" ref="layer-sha256:8ea059e7807349882824ba1b38b90f2ea7bcd4210542bd35c92a0688fa6abbf2" total=54373319
ERRO[2021-02-02T14:38:14.795267667+08:00] (*service).Write failed                       error="rpc error: code = Canceled desc = context canceled" expected="sha256:18c99e163b830603226bcbeecd01bcf10718996db5eff7c43c4580c40773b101" ref="layer-sha256:18c99e163b830603226bcbeecd01bcf10718996db5eff7c43c4580c40773b101" total=131378290
ERRO[2021-02-02T14:38:14.795267687+08:00] (*service).Write failed                       error="rpc error: code = Canceled desc = context canceled" expected="sha256:191d7ea8704e513c526fb7724475ff0ab45449239a979d7202a362c99a28022a" ref="layer-sha256:191d7ea8704e513c526fb7724475ff0ab45449239a979d7202a362c99a28022a" total=70372300

[ktock]

@nicktming

there is no output

Please add --log-level=debug to containerd-stargz-grpc and paste the log output.

it pulls all the layers, is it the normal case?

no.

Check also fusermount is installed on your host.

[nicktming]

@nicktming

there is no output

Please add --log-level=debug to containerd-stargz-grpc and paste the log output.

it pulls all the layers, is it the normal case?
--- already added, still no logs

no.

Check also fusermount is installed on your host.

---

already installed.
[root@VM-0-6-centos ~]# which fuser
/usr/sbin/fuser
[root@VM-0-6-centos ~]# which fusermount
/usr/bin/fusermount

[ktock]

@nicktming Doesn't containerd-stargz-grpc provide any log even with --log-level=debug flag?

[nicktming]

@nicktming Doesn't containerd-stargz-grpc provide any log even with --log-level=debug flag?

it has logs now, logs like that:
failed to request: context canceled: failed to resolve","level":"debug",

[ktock]

@nicktming Please provide the complete log line. It's hard to make sure where is the root cause of that error.

[nicktming]

@nicktming Please provide the complete log line. It's hard to make sure where is the root cause of that error.

containerd logs

INFO[2021-02-02T17:15:39.912527373+08:00] StopPodSandbox for "01c5ba45767d4740688ddc50796afb9504c2101b2bdd4fe9e9a79a82de4ec241" 
ERRO[2021-02-02T17:15:39.914353327+08:00] StopPodSandbox for "01c5ba45767d4740688ddc50796afb9504c2101b2bdd4fe9e9a79a82de4ec241" failed  error="failed to destroy network for sandbox \"01c5ba45767d4740688ddc50796afb9504c2101b2bdd4fe9e9a79a82de4ec241\": invalid version \"\": the version is empty"
INFO[2021-02-02T17:15:39.914565152+08:00] StopPodSandbox for "1283bc3c639971d5b6257f893e287ff02d6a98567711dc3168abd757e7d1148f" 
ERRO[2021-02-02T17:15:39.916446931+08:00] StopPodSandbox for "1283bc3c639971d5b6257f893e287ff02d6a98567711dc3168abd757e7d1148f" failed  error="failed to destroy network for sandbox \"1283bc3c639971d5b6257f893e287ff02d6a98567711dc3168abd757e7d1148f\": invalid version \"\": the version is empty"
INFO[2021-02-02T17:15:39.916648867+08:00] StopPodSandbox for "2bb915d797896b8f7a3151a6b0b2187b2664b9c4c77180d2af334a4ba1c1c349" 
ERRO[2021-02-02T17:15:39.918794437+08:00] StopPodSandbox for "2bb915d797896b8f7a3151a6b0b2187b2664b9c4c77180d2af334a4ba1c1c349" failed  error="failed to destroy network for sandbox \"2bb915d797896b8f7a3151a6b0b2187b2664b9c4c77180d2af334a4ba1c1c349\": invalid version \"\": the version is empty"
INFO[2021-02-02T17:15:39.919055945+08:00] StopPodSandbox for "3cb633e1b7909a958839dcd6ffbd924280d474010fdb1fc5df78a1d95b498933" 
ERRO[2021-02-02T17:15:39.921034623+08:00] StopPodSandbox for "3cb633e1b7909a958839dcd6ffbd924280d474010fdb1fc5df78a1d95b498933" failed  error="failed to destroy network for sandbox \"3cb633e1b7909a958839dcd6ffbd924280d474010fdb1fc5df78a1d95b498933\": invalid version \"\": the version is empty"

containerd-stargz logs

{"error":"cannot resolve layer: failed to redirect (host \"ip:5000\", ref:\"ip:5000/myimage", digest:\"sha256:2ddd81e5692f21714879195f97d10cf39da3d448c014a0c66727914f20886656\"): failed to request: tls: first record does not look like a TLS handshake: failed to resolve","level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/31/fs","msg":"failed to resolve source","src":"ip:5000/myimage/sha256:2ddd81e5692f21714879195f97d10cf39da3d448c014a0c66727914f20886656","time":"2021-02-02T17:22:03.989493832+08:00"}

{"error":"failed to resolve layer \"sha256:4651026ea2e25b063e55e6543aa8c7a86ddac5e3fb895364986191e43f007178\" from \"ip:5000/myimage\": failed to resolve layer: failed to resolve the source: cannot resolve layer: failed to redirect (host \"ip:5000\", ref:\"ip:5000/myimage\", digest:\"sha256:4651026ea2e25b063e55e6543aa8c7a86ddac5e3fb895364986191e43f007178\"): failed to request: tls: first record does not look like a TLS handshake: failed to resolve: failed to resolve target","level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/33/fs","msg":"failed to resolve layer","time":"2021-02-02T17:48:02.294873885+08:00"}
{"error":null,"key":"default/28/extract-277323557-2I5Q sha256:3a2381ba2e8d730c5205d303af9cb77b50d7135b7dde9441cb70f3d10d5a4c2d","level":"debug","msg":"failed to prepare remote snapshot","parent":"","remote-snapshot-prepared":"false","time":"2021-02-02T17:48:02.294899673+08:00"}

{"error":"cannot resolve layer: failed to redirect (host \"ip:5000\", ref:\"ip:5000/myimage\", digest:\"sha256:a8bb704b3737af43af9314f7fc628e1218703fbdde08cb42681c679efe03aee4\"): failed to request: context canceled: failed to resolve","level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/29/fs","msg":"failed to resolve source","src":"ip:5000/myimage/sha256:a8bb704b3737af43af9314f7fc628e1218703fbdde08cb42681c679efe03aee4","time":"2021-02-02T16:35:28.350105129+08:00"}
{"error":"cannot resolve layer: failed to redirect (host \"ip:5000\", ref:\"ip:5000/myimage\", digest:\"sha256:30721a2fe2615f34937b2aa463b8f741102bf450e76578a07e4fd9bc9e865ff3\"): failed to request: context canceled: failed to resolve","level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/29/fs","msg":"failed to resolve source","src":"ip:5000/myimage/sha256:30721a2fe2615f34937b2aa463b8f741102bf450e76578a07e4fd9bc9e865ff3","time":"2021-02-02T16:35:28.350127541+08:00"}

[ktock]

@nicktming

{"error":"cannot resolve layer: failed to redirect (host "ip:5000", ref:"ip:5000/myimage", digest:"sha256:2ddd81e5692f21714879195f97d10cf39da3d448c014a0c66727914f20886656"): failed to request: tls: first record does not look like a TLS handshake: failed to resolve","level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/31/fs","msg":"failed to resolve source","src":"ip:5000/myimage/sha256:2ddd81e5692f21714879195f97d10cf39da3d448c014a0c66727914f20886656","time":"2021-02-02T17:22:03.989493832+08:00"}

this log says that you are trying to pull the image from your non-HTTPS (i.e. plain-http) registry ip:5000 but containerd-stargz-grpc doesn't recognize it as non-HTTPS registry and this tries to connect to your registry using HTTPS (then it fails).

As mentioned in #254 (comment), please make sure that your containerd-stargz-grpc configuration (tipically /etc/containerd-stargz-grpc/config.toml) contains the following registry configuration and containerd-stargz-grpc recognizes that config file correctly ( /etc/containerd-stargz-grpc/config.toml is read by default but if you store this config file somewhere else, specify the location using containerd-stargz-grpc's --config option).

https://github.com/containerd/stargz-snapshotter/blob/v0.3.0/docs/overview.md#registry-mirrors-and-insecure-connection

[[resolver.host."<ip>:5000".mirrors]]
host = "<ip>:5000"
insecure = true

(replace <ip> with your true registry host name or IP address)

[nicktming]

@ktock what is the stability of stargz-snapshotter in the range of different cluster nodes? such as 100+ nodes, 500+nodes, 1000+ nodes, 5000+ nodes . is there any docs to provide this kind of information?

[ktock]

Though there is no survey about stability, this is one of the topics we'll work on this year.

[nicktming]

@ktock ok, thx. Looking forward to the survey.

[ktock]

@nicktming
Of course, any contribution around stability is always welcome :)
BTW how is this issue going?

[nicktming]

@ktock it worked with your suggestions, but it failed some times after that with the same command, so i got a little confused.

i will get into the details to find what happens if having free times .

[mc256]

TL;DR
You may need to update containerd to a newer version.

---

It works for me a while ago. But today I found that rpull pulls all the layers. And I confirm it by running in the Docker container provided in /script/demo, it works. But it just does not work on my host machine. The host machine I am using is running Ubuntu 20.04. I got containerd from apt install, and containerd -v gives me

containerd github.com/containerd/containerd 1.3.3-0ubuntu2.3 

But then I check the Dockerfile, it uses CONTAINERD_VERSION=v1.5.0-beta.4, therefore I compile a newer version of containerd and it works!!!

[ktock]

@nicktming How is this issue going?