Provide default options for chunking of datasets (issue #635)

[ehennestad]

Fix #635

Motivation

Provide better chunking options for files in cloud storage

How to test the behavior?

To be determined

Checklist

• Have you ensured the PR description clearly describes the problem and solutions?
• Have you checked to ensure that there aren't other open or previously closed Pull Requests for the same change?
• If this PR fixes an issue, is the first line of the PR description fix #XX where XX is the issue number?

[ehennestad]

Some open questions:

• Should each possible property / dataset name be present in the configuration (chunk_params.json)? Or should there be one default that applies to all datasets, and overwrites would happen on specific datasets?
• Should each dataset be a candidate for chunking, or only datasets like "data", "timestamps". Any others?
• Should chunking be specified for datasets that are smaller than the chunk size? I.e chunkSize == maxSize?
• Should chunk_dimensions be specified for each set of dimension options, similar to how it is done in the nwb schema?

I.e. https://github.com/NeurodataWithoutBorders/nwb-schema/blob/473fcc41e871288767cfb37d83315cca7469b9d1/core/nwb.base.yaml#L100-L110

dims:
    - - num_times
    - - num_times
      - num_DIM2
    - - num_times
      - num_DIM2
      - num_DIM3
    - - num_times
      - num_DIM2
      - num_DIM3
      - num_DIM4

@bendichter

[ehennestad]

Current implementation for schema/definition:

    "default_dataset_configuration": {
        "layout": "chunked",
        "target_chunk_size": {
            "value": 10000000,
            "unit": "bytes"
        },
        "chunk_dimensions": [null],
        "compression": {
            "algorithm": "gzip",
            "level": 6,
            "parameters": {},
            "prefilters": ["shuffle"]
        }
    },
    specific_dataset_overrides...

Open questions:
@bendichter
Should chunk_dimensions be specified for each set of dimension options, similar to how it is done in the nwb schema?

dims:
    - - num_times
    - - num_times
      - num_DIM2
    - - num_times
      - num_DIM2
      - num_DIM3
    - - num_times
      - num_DIM2
      - num_DIM3
      - num_DIM4

What would be the syntax?
Example (json):

"chunk_dimensions": [ [null], [null, 32], [null, 32, max] ]

[ehennestad]

Need to determine what to do with nested data types. For example: A RoiResponseSeries can be part of a Fluorescence or a DfOverF group. Should the spec support defining configuration for nested neurodata types dependent on were they are located, i.e:

"Fluorescence": {
        "RoiResponseSeries": {
            "data": {
                "chunk_dimensions": [null, 16]}
        }
    },
    "DfOverF": {
        "RoiResponseSeries": {
            "data": {
                "chunk_dimensions": [null, 32]}
        }
    }

[codecov]

Codecov Report

Attention: Patch coverage is 96.07143% with 11 lines in your changes missing coverage. Please review.

Project coverage is 95.34%. Comparing base (3d252da) to head (8f1e806).

Files with missing lines	Patch %	Lines
+schemes/listDatasetsOfNeurodataType.m	81.48%	5 Missing ⚠️
...+config/+internal/applyCustomMatNWBPropertyNames.m	91.66%	3 Missing ⚠️
+io/+config/+internal/computeChunkSizeFromConfig.m	98.21%	1 Missing ⚠️
+io/+config/+internal/configureDataPipeFromData.m	95.65%	1 Missing ⚠️
+io/+config/applyDatasetConfiguration.m	97.43%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #636      +/-   ##
==========================================
+ Coverage   95.29%   95.34%   +0.04%     
==========================================
  Files         139      153      +14     
  Lines        5332     5603     +271     
==========================================
+ Hits         5081     5342     +261     
- Misses        251      261      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.