refactor: queue-based block production + better separation of node's mutable state

[itegulov]

What 💻

Closes #501

Apologies for an insane diff once again - one change required another so I ended up refactoring more stuff than originally intended. Vast part of the diff is just moving from std::sync::RwLock to tokio::sync::RwLock (was needed due to lifetime shenanigans) and hence adding async/await in a lot of places that did not require it before. I will try to give a brief overview of functional changes below:

• Moved (most of) mutable node state into a separate module inner. Essentially trying to restrict writeable access to as little entrypoints as I could (see inner rustdoc for more info)
• Introduced a new layer purely for blockchain state: inner/blockchain.rs that has reader/writer structs. BlockchainWriter is held exclusively by InMemoryNodeInner and is inaccessible from outside inner module. All API endpoints were refactored to rely on BlockchainReader for their queries, thus removing the need to lock the entirety of InMemoryNodeInner.
• Refactored time module into TimeWriter and TimeReader. Former is owned by InMemoryNodeInner and is inaccessible from outside of inner module. Latter can still be used to read current time in API endpoints.
• BlockProducer was renamed to NodeExecutor and is the sole place that can seal blocks. Works via mpsc command queue that ensures no contentions between time lock and block production. Note: some more improvements can still be made to relax its lock holding but this at least removes the requirement to hold time lock for the entire block production time. Can also seal multiple blocks "atomically" (term used pretty loosely here but hopefully conveys the meaning).
• BlockSealer was adapted accordingly and is now a separate background process that pushes commands to BlockProducer

Why ✋

Hopefully better code quality + less lock contention across the board

[popzxc]

Overall I like what I see, but tbh it feels much more complicated and low-level than it intuitively has to be.

Left a few preliminary comments.

[popzxc]

Can't we express BlockchainReader and BlockchainWriter as two traits? Actually, probably BlockchainWriter is not even needed -- you can expose only Box<dyn BlockchainReader> outside of the crate, and rework Blockchain as pub(crate) struct Blockchain { inner: Arc<RwLock<Blockchain>> }.

With that, hopefully, you will be able to expose all the required mutability methods on the structure itself so that you don't have to leak guards in the interface.

[itegulov]

Done, I had to move some seemingly API-specific logic into Blockchain (to avoid exposing generic inspect_* methods that are not object safe). But IMHO worked out pretty okay.

I had to wrap ReadBlockchain and ReadTime into Arc<dyn _> instead of Box<dyn _> just so they would be cloneable. Do you think it is worth bothering with DynClone to avoid this?

[popzxc]

You can add new_cloned method to the trait and then do impl Clone for Box<dyn ReadBlockchain> { fn clone(&self) -> Self { self.new_cloned() }