assertion error on digest type

[yarikoptic]

while trying to figure out

• pyout might be dragging us behind on large dandisets #1549
  on drogon, got in

2024-12-12T09:24:00-0500 [ERROR   ] dandi 3434539:140629314469952 Caught while downloading sub-I48/ses-SPIM/micr/sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr:                                           Traceback (most recent call last):                                                                                                                                                                                                
  File "/home/yoh/proj/dandi/dandi-cli/dandi/download.py", line 397, in _download_generator_guard                                                                                                                                     yield from generator                                                                                                                                                                                                          
  File "/home/yoh/proj/dandi/dandi-cli/dandi/download.py", line 1043, in _download_zarr                                                                                                                                               zarr_checksum = asset.get_digest().value                                                                                                                                                                                                          ^^^^^^^^^^^^^^^^^^                                                                                                                                                                                              File "/home/yoh/proj/dandi/dandi-cli/dandi/dandiapi.py", line 1513, in get_digest                                                                                                                                                   return Digest(algorithm=algorithm, value=self.get_raw_digest(algorithm))                                                                                                                                                                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                         File "/home/yoh/proj/dandi/dandi-cli/dandi/dandiapi.py", line 1499, in get_raw_digest                                                                                                                                               assert isinstance(digest, str)                                                                                                                                                                                                
AssertionError  

and of cause we do not know what type it is ...

[yarikoptic]

happened agin

2024-12-16T23:12:30-0500 [ERROR   ] dandi 893263:139627788764864 Caught while downloading sub-I48/ses-SPIM/micr/sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr:
Traceback (most recent call last):
  File "/mnt/backup/tmp/yoh/venv/dev/lib/python3.11/site-packages/dandi/download.py", line 421, in _download_generator_guard
    yield from generator
  File "/mnt/backup/tmp/yoh/venv/dev/lib/python3.11/site-packages/dandi/download.py", line 1065, in _download_zarr
    zarr_checksum = asset.get_digest().value
                    ^^^^^^^^^^^^^^^^^^
  File "/mnt/backup/tmp/yoh/venv/dev/lib/python3.11/site-packages/dandi/dandiapi.py", line 1513, in get_digest
    return Digest(algorithm=algorithm, value=self.get_raw_digest(algorithm))
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/backup/tmp/yoh/venv/dev/lib/python3.11/site-packages/dandi/dandiapi.py", line 1499, in get_raw_digest
    assert isinstance(digest, str)
AssertionError

need to do type checking etc.

[yarikoptic]

we encounter zarr for which we have size and checksum 0

❯ curl -X 'GET' --silent 'https://api.dandiarchive.org/api/dandisets/000026/versions/draft/assets/?path=sub-I48%2Fses-SPIM%2Fmicr%2Fsub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr&metadata=true' -H 'accept: application/json' -H 'X-CSRFTOKEN: wTUZx1u5sMThqoLPhpAxD6TubffnRiFNTWXBxjXPRokYFa3kNB71LWKnwuJndJcA' | jq '.results[0] | ( .size, .zarr, .metadata.digest )'
0
"7933bc72-ec3f-4043-8076-d7aa8a70e4f7"
{
  "dandi:dandi-zarr-checksum": null
}

although files are uploaded

$ aws s3 ls s3://dandiarchive/zarr/7933bc72-ec3f-4043-8076-d7aa8a70e4f7/
                           PRE 0/
                           PRE 1/
                           PRE 2/
                           PRE 3/
                           PRE 4/
2024-06-14 07:42:52       2824 .zattrs
2024-04-14 14:50:05         24 .zgroup

and that zarr is still pending!

❯ curl --silent -X 'GET' 'https://api.dandiarchive.org/api/zarr/7933bc72-ec3f-4043-8076-d7aa8a70e4f7/' -H 'accept: application/json' -H 'X-CSRFTOKEN: wTUZx1u5sMThqoLPhpAxD6TubffnRiFNTWXBxjXPRokYFa3kNB71LWKnwuJndJcA' | jq .
{
  "name": "sub-I48/ses-SPIM/micr/sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr",
  "dandiset": "000026",
  "zarr_id": "7933bc72-ec3f-4043-8076-d7aa8a70e4f7",
  "status": "Pending",
  "checksum": null,
  "file_count": 0,
  "size": 0
}

so we are back in the effects of various issues with zarrs in the dandi-archive:

• see Content Size (self.size) remaining at 0 for certain zarr-related assets dandi-archive#2067 (comment) and there on

[yarikoptic]

ok, so in this case we have downloaded some incomplete version of zarr, its checksum is null etc. We should at least issue a warning of some kind that upload was never finalized and they fetched potentially an incomplete copy . The zarr in question has already 27G in size

odd, interrupted and restarted with refresh completed without error

(dandisets-2) dandi@drogon:/mnt/backup/dandi$ dandi download -J 2:20 dandi://dandi/000026/sub-I48/ses-SPIM/micr/sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr
PATH                                                                  SIZE    DONE     DONE% CHECKSUM STATUS        MESSAGE
sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr 0 Bytes 1.3 GB      0%          error         FileExistsError
Summary:                                                                      1.3 GB                  1 error       1 FileExistsError
                                                                              0.00%
2024-12-19 23:05:02,256 [    INFO] Logs saved in /home/dandi/.local/state/dandi-cli/log/2024.12.20-04.04.40Z-2706158.log
Error: Encountered 1 error while downloading.
(dandisets-2) dandi@drogon:/mnt/backup/dandi$ dandi download -J 2:20 --existing refresh dandi://dandi/000026/sub-I48/ses-SPIM/micr/sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr
PATH                                                                  SIZE    DONE    DONE% CHECKSUM STATUS                 MESSAGE
sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr 0 Bytes 27.9 GB    0%          done                   15209 done, 1872 skipped
Summary:                                                                      27.9 GB                1 done                 1 some done, some skipped
                                                                              0.00%
2024-12-19 23:17:32,942 [    INFO] Logs saved in /home/dandi/.local/state/dandi-cli/log/2024.12.20-04.05.15Z-2706190.log

so we might not be checking upon refresh???
...

[yarikoptic]

repeated afresh, indeed -- no error if refresh after failed process

(dandisets-2) dandi@drogon:/mnt/backup/dandi/tmp$ dandi download -J 2:20 dandi://dandi/000026/sub-I48/ses-SPIM/micr/sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr
PATH                                                                  SIZE    DONE     DONE% CHECKSUM STATUS                 MESSAGE         
sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr 0 Bytes 27.9 GB     0%          error                  AssertionError  
Summary:                                                                      27.9 GB                 1 error                1 AssertionError
                                                                              0.00%                                                          
2024-12-19 23:37:09,310 [    INFO] Logs saved in /home/dandi/.local/state/dandi-cli/log/2024.12.20-04.24.21Z-2720891.log
Error: Encountered 1 error while downloading.
(dandisets-2) dandi@drogon:/mnt/backup/dandi/tmp$ dandi download -J 2:20 --existing refresh dandi://dandi/000026/sub-I48/ses-SPIM/micr/sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr
PATH                                                                  SIZE    DONE    DONE% CHECKSUM STATUS    MESSAGE       
sub-I48_ses-SPIM_sample-BrocaAreaS03_stain-Somatostatin_SPIM.ome.zarr 0 Bytes                        skipped   17082 skipped 
Summary:                                                                      0 Bytes                1 skipped 1 some skipped
                                                                              100.00%                                        
2024-12-19 23:45:18,033 [    INFO] Logs saved in /home/dandi/.local/state/dandi-cli/log/2024.12.20-04.42.45Z-2732233.log