Clodius crashes when converting bedgraph into hitile files

[MartinPippel]

Describe the bug
command:

sort -k1,1V -k2,2n -k3,3n  hifiasm-scaffolded-default_telomer.bedgraph | \
clodius aggregate bedgraph --no-header --chromsizes-filename test.sizes -o hifiasm-scaffolded-default_telomer.hitile -

crash report

Traceback (most recent call last):
  File "/opt/conda/bin/clodius", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/clodius/cli/aggregate.py", line 1408, in bedgraph
    _bedgraph(
  File "/opt/conda/lib/python3.12/site-packages/clodius/cli/aggregate.py", line 869, in _bedgraph
    d.attrs["chrom-names"] = chrom_order
    ~~~~~~~^^^^^^^^^^^^^^^
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/opt/conda/lib/python3.12/site-packages/h5py/_hl/attrs.py", line 104, in __setitem__
    self.create(name, data=value)
  File "/opt/conda/lib/python3.12/site-packages/h5py/_hl/attrs.py", line 202, in create
    attr = h5a.create(self._id, name, htype, space)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5a.pyx", line 50, in h5py.h5a.create
OSError: Unable to synchronously create attribute (object header message is too large)

To Reproduce
Steps to reproduce the behavior:
This seems to happen when the number of contigs is very large (>>1000 contigs). But when reducing the contigs another problem occurs - and that happens kind of randomly:

Traceback (most recent call last):
  File "/cfs/klemming/scratch/p/pippel/prog/conda_envs/clodius/bin/clodius", line 11, in <module>
    sys.exit(cli())
  File "/cfs/klemming/scratch/p/pippel/prog/conda_envs/clodius/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/cfs/klemming/scratch/p/pippel/prog/conda_envs/clodius/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/cfs/klemming/scratch/p/pippel/prog/conda_envs/clodius/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/cfs/klemming/scratch/p/pippel/prog/conda_envs/clodius/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/cfs/klemming/scratch/p/pippel/prog/conda_envs/clodius/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/cfs/klemming/scratch/p/pippel/prog/conda_envs/clodius/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/cfs/klemming/scratch/p/pippel/prog/conda_envs/clodius/lib/python3.7/site-packages/clodius/cli/aggregate.py", line 1400, in bedgraph
    zoom_step,
  File "/cfs/klemming/scratch/p/pippel/prog/conda_envs/clodius/lib/python3.7/site-packages/clodius/cli/aggregate.py", line 1023, in _bedgraph
    dsets[curr_zoom][curr_pos : curr_pos + chunk_size] = curr_chunk
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/cfs/klemming/scratch/p/pippel/prog/conda_envs/clodius/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 980, in __setitem__
    mspace = h5s.create_simple(selection.expand_shape(mshape))
  File "/cfs/klemming/scratch/p/pippel/prog/conda_envs/clodius/lib/python3.7/site-packages/h5py/_hl/selections.py", line 264, in expand_shape
    raise TypeError("Can't broadcast %s -> %s" % (source_shape, self.array_shape))  # array shape
TypeError: Can't broadcast (9852211,) -> (8672655,)

Solution

• do not convert bedgraph to hitile file, that seems to be problematic (see Clodius aggregate with custom assembly: TypeError: Can't broadcast higlass/clodius#96)
• instead convert bedgraph to bigwig, which can also be ingested into higlass