[SPIKE] Persisted Job Queue or Retry Logic

[frrist]

Problem Statement:

When aggregating pieces or submitting aggregations, failures may occur due to software-related issues or unexpected storage node restarts.

Expected Behavior:

Jobs in the "job queue" must be retried until they succeed. If a job fails, it should either:

• Be retrieved from its respective store and reprocessed, or
• Persist in the queue (e.g., saved to disk) to ensure it remains available for retrying.

At present when jobs fail, or the node restarts the contents of the queue is lost.

Ideas

One approach may be to implement or adopt a job queuing library with persistence. (e.g. https://github.com/maragudk/goqite)
Another option could be to persist jobs to a store (we already have one) in a pending state, and ensure they are completed across restarts and failures.

[travis]

@hannahhoward needs to review