Thoughts on using ZMQ for indexer syncing

[HelloKashif]

Prob a question for @patrick-ogrady

Any thoughts on using the bitcoin zmq pubsub to sync the indexer instead of the current polling based method?
This could greatly reduce both the sync time and load on the btc rpc as we can just use the messages to populate the indexer db.

I see that zmq was disabled in the bitcoind build so you might have already investigated this.
If you have already considered I'd love to know some thoughts on this.

[sidhujag]

This makes sense, blockbook also uses this to sync and calls RPC only when details are needed or specific lookups are required like for custom data in a transaction on other chains:

https://github.com/trezor/blockbook/blob/master/bchain/mq.go

It also uses golang so design patterns are easily portable to rosetta.

In recent bitcoin core theres also a nonce or an index that will let subscribers know if a message has been missed as well.

You can sync blocks, transactions and mempool this way.

[patrick-ogrady]

Sorry for the delay on responding here, @HelloKashif. IMO the syncing bottleneck is exclusively related to storing indexed data to disk. Until we make that like 50x faster, I don't think this migration will really improve anything related to sync performance (in fact, the pre-store mechanism in added in #53 typically gets 50-100 blocks ahead of the sequencer).

That being said, I think ZMQ would be the best mechanism for monitoring mempool coins for implementing mempool support on /account/coins.

[HelloKashif]

@patrick-ogrady understood. Yeah I actually started an issue to add mempool support for coins. #54

Thanks