Node: Channel writes without blocking

[bruce-riley]

Overview
The purpose of this PR is to address instances of writes to go channels that may block unexpectedly. I went through the guardian code, identified unbuffered channel writes, and made a determination if blocking is appropriate or not.

Conclusion
This review of the guardian code did not find any instances where a channel write would block the critical path. It did detect lesser cases that are addressed.

Implementation
In the cases where blocking is appropriate, I tagged the line with //nolint:channelcheck // <reason_why_blocking_is_appropriate>. That way when the linter is updated with this plugin, these lines will be correctly ignored..

In some of the cases where blocking is not appropriate, I changed it to use the common.WriteToChannelWithoutBlocking function instead. This does the write with a select and pegs a metric if the write would block.

In some cases (the processor, for example), a write that would block is handled in a way more consistent with the existing code.

Change in Behavior
One significant change in behavior introduced by this PR is that previously, the channel used by all the watchers to post observations to the processor (msgC) was unbuffered, meaning that if more than one watcher posted an event at roughly the same time, it could block. This doesn't seem like desirable behavior now that we have so many watchers, so I changed msgC to be buffered. This change is open for discussion!

[bruce-riley]

I started out using the output of the analyze.go tool being developed by @johnsaigle which identifies writes to channels that may block (among other things). This was a helpful starting point. However, it turns out that this tool was missing some instances (I notified the author) so I ended up searching in vsc for "<-" (note the spaces on either side of the arrow) and limiting it to *.go files. This also includes writes that already have selects, which needed to be weeded out.

[johnsaigle]

After looking into the recent buffered channel PRs as well as the EVM watcher recently, I think that making msgC non-blocking seems like a good idea. 👍🏻