Skip to content

DCL MainNet Deployment

Jump to bottom
Andrey Kononykhin edited this page Feb 10, 2022 · 38 revisions

Summary

This is WIP

DCL is in a better position regarding DDoS protection comparing to public Cosmos networks.

DCL is a permissioned network consisting of quite a limited set of trusted or semi-trusted nodes (validators and observers), and we don't require to make all the nodes public to anyone in the world (a company may make its nodes accessible to that company applications only). But Cosmos public networks are permisionless meaning that it may have any number of nodes, and that nodes need to be public for anyone in the world.

Moreover, DCL nodes do not compete for proposing the blocks, they don't play to "game of stake". There is no tokenomics in the permissioned DCL network (at least so far). So, if one node dies or unavailable for some time, this is not a catastrophe. A Node Admin can fix and repair it. But a crashed/non-available node can be a problem in a staking-based network (Cosmos), as the node can not propose new blocks, and may lost the "clients" (delegators) and their tokens.

In other words, DCL is more collaborative, while staking-based networks (like Cosmos) usually consist of competitive entities.

Options for Validators

All options assume that the validator node is not public and accepts incoming connections from trusted validators and observers only (see Options for network protection)

  • Option 1: Cloud, no HSM
    • Option 1A: no Sentry, private keys and secrets at the Validator machine
    • Option 1B: with Sentry, private keys and secrets at the Validator machine
    • Option 1C: no Sentry, private keys and secrets are not at Validator machine (tmkms, HashiCorp Vault)
    • Option 1D: with Sentry, private keys and secrets are not at Validator machine (tmkms, HashiCorp Vault)
  • Option 2: Physical machine, HSM, with Sentries

Recommended Options for DCL 1.0

Option 2 (Physical machine, HSM, with Sentries) or Option 1B (Cloud, no HSM, with Sentry, private keys and secrets at the Validator machine). DCL Deployment

Why use Sentries:

  • Harder to DDoS the real Validator node (in case malicious Validators present)
  • Hides real Validator node's IP, so harder to attack a real validator
  • Can support HSM and Validators at physical machines w/o access to Internet (if not from beginning, then HSM support can be added in future)
  • Public Sentries are essentially Observers, so no need for more Observers
  • Can potentially auto-scale Sentry nodes (create new Sentries when attack is detected)

Why use separate KMS for Validator Keys:

  • Security best practice: do not keep secrets at Validator machine, so that if Validator is compromised, secrets are not accessed
  • In particular, helps to prevent double-signing by Validators (see https://kb.certus.one/hsm.html#double-signing)
  • Please note though, that double signing is not that critical for DCL comparing to permissionless proof-of-stake networks (Cosmos). In DCL nodes don't have any tokens and don't manage public reputation and clients. So, if a node tries to double sign, it will be just slashed (removed from the network). Later on Node Admins and Trustees can investigate what was the reason.

Why use HSM for Validator Keys:

  • The most secure key management
  • Not that critical for DCL comparing to permissionless proof-of-stake networks (Cosmos), see the previous Item.

Options for the network protection (Trust Link between Private Sentries)

https://kb.certus.one/peers.html#private-nodes

  • Option 1: no IPSec/VPN, just whitelist/blacklist via firewall rules
    • pros:
      • Seems enough and quite easy to do since
        • We can expect/assume that all IPs are static
        • We don't need encryption at IP level, as auth encryption will be done on Tendermint P2P level in any case
          • Done in for example link, Sections 6.6 and 6.7
      • no additional cost (cloud providers)
      • no additional resources (gateways)
    • cons:
      • there might be some concerns in Tendermint P2P auth encryption (one may don't trust Tendermint's implementation)
  • Option 2: IPSec site-to-site VPN (Cloud providers)
    • pros:
      • managed VPN gateway resources (highly available)
      • IPSec - old, trusted technology
      • IPSec encryption best practices (certificates, their rotation, key exchange IKEv2)
      • mature service (access control, configuration automation)
    • cons:
      • p2p cross-cloud connections will require VPN-to-VPN configuration which is not a straightforward thing even if a documentation is good e.g. Google to AWS
        • A single connection routine with additional resources (gateways), even if they are managed for HA
        • Plus multiple VPN connections per a cloud (gateway)
        • Plus additional firewall rules
        • Certificates management
        • Additional costs:
          • E.g. AWS pricing:
            • Site-to-Site VPN connection fee (per hour)
            • Data transfer fee (per Gb)
            • (optionally) accelerated connection and data transfer fees
  • Option 3: WireGuard
    • pros:
      • Encryption
      • easy to install (part of Linux, FreeBSD, Android kernels, Windows ...) and configure, in short:
        • each peer
          • installs the package
          • creates net interface (ip link add...) assign an address and with a mask (ip address add...)
          • creates pub/priv pair of keys (wg genkey and wg pubkey)
        • peers share the pubkeys along with endpoint and private (local) IPs assigned to the created interfaces
        • each peer:
          • for each (other) peer create a record (pubkey, private IP, endpoint IP) in the configuration file and applies using command wg setconf
          • makes the net interface up (ip link set up)
          • configures nodes to send to the private IPs (wireguard will route encrypted packets that to corresponded public endpoints)
      • fast (e.g. vs OpenVPN, some IPSec comparison)
      • Real p2p (full mesh)
      • additional protection:
      • No additional cost
      • not a big codebase (~4000 lines) so security audit more real than for other protocols
    • cons:
      • Young technology
        • but Linus Torvalds accepted it (with a preference to OpenVPN and IPSec) and other known good developers work around the technology (e.g. tailscale project)
      • known (self-realized) trade-offs (but likely no stoppers)
    • comparison to IPSec and OpenVPN
  • Option 4: P2P VPN
    • Mentioned as an option in https://docs.tendermint.com/master/spec/p2p/node.html#validator-node for validators that trust each other (actually our DCL case)
    • pros:
      • May handle IP changes better (?)
      • no additional cost and resources
    • cons:
      • one more VPN protocol: had some vulnerabilities in the past, not often releases, likely no deep security audit in the past
      • requires additional SW that we need to trust (e.g. tinc)
      • May be more tricky for configuration, especially in heterogeneous environment (different cloud providers etc.)
      • Additional layer of encryption can be beneficial if there are concerns in Tendermint P2P auth encryption

Options for Nodes Discovery

  • Persistent peers between all Validators (or private Sentries if Validator is behind a Sentry Node)
    • This is how our current TestNet is deployed
    • May need to maintain and update the list of peers
  • One or multiple Seed nodes that all nodes use for discovery. The node can be managed by CSA for example.
    • All nodes have to trust and rely on that seed node
  • Every Validator starts up its own Seed Node

Options for Account Keys

Details

Goals

  1. DDoS Protection
  2. Private Key and secrets security
  3. Trusted relationship (can trust query results, no MITM)
  4. Health and monitoring
  5. Stability and performance
  6. High Availability and scalability

1. DDoS Protection

1.1 General notes

  • there are numerous types of attacks: e.g. on different OSI levels, long-lived, highly distributed (some references: wiki, cloudflare.com ) ...
  • cloud providers accomplish well with layer 3 and 4 attacks mitigation and provides a service for other more sophisticated attacks
    • e.g. AWS:
      • AWS Shield: is active by default (at no additional charge) for all users and mitigates layer 3 and 4 attacks
      • for some fee additional protection (AWS Shield Advanced) is provided to help with more sophisticated attacks including layer 7 attacks
      • a list of additional techniques will help to make that even better
    • Google Cloud:
  • firewall rules is a good way but only a part and not sufficient to mitigate all possible (known) types of attacks
  • the same for embedded CosmosSDK/Tendermint protection logic: it allows to prevent mostly application level attacks only
    • Only valid txns are broadcasted to other nodes
    • Read requests are not broadcasted to other nodes
    • Tendermint/Cosmos TPS is quite high
    • Need to attack a lot of ONs
    • Possible to not allow random ONs to be connected to your ON
  • Cosmos SDK targets of attacks:
    • p2p connections
    • client connections (client service)
  • Conclusions (mitigation directions):
    • (in case of cloud) don't ignore cloud provider anti-DDoS services and consider non-free ones as well
    • separate client and p2p channels (so p2p would continue to work even if all client services are down) and protect with different set of techniques
    • hide validators from the public as the most important part of the system, so even if all public relays are down (are being restarted to back online) validators are still healthy and doesn't require recovery

1.2 Recommendations

2. Private Keys and secrets security

3. Trusted relationship

  • [MUST] gRPC/REST over HTTPS (not HTTP)
  • [MUST] Tendermint RPC over HTTPS (not HTTP)
  • [MUST] Clients connect to trusted Observer nodes only. If there is no trusted Observer to connect to, clients should use Tendermint RPC queries and verify proofs via light client
    • There is support for Light Client Proxy Node, so that clients can run a Proxy node, send all RPC queries to that Proxy, and the Proxy will verify the proofs automatically.

3.1 TLS support

  • TLS 1.3 is supported by Tendermint RPC and CLI client (it uses the endpoint)
    • Note. there is no way to config CLI to work with self-signed certificate (e.g. for testing purposes)
  • TLS is not supported by gRPC/REST endpoints (see details)

Thus, we don't have full TLS support and need here a proxy with SSL termination. Options:

  • Option 1:
    • reverse proxy on each DCL edge node (e.g. nginx)
    • pros:
      • most secure: proxy communicates with a node directly (localhost), unencrypted traffic between proxy and the backend is not visible
    • cons:
      • more securing efforts: more secrets to manage on each machine
      • additional configuration routine (proxy installation and configuration)
      • additional point of failure, node would become unresponsive even if validator process is healthy
  • Option 2:
    • dedicated node with a reverse proxy
    • pros:
      • no noticeable ones
    • cons:
      • the same as for Option 1
      • plus proxy - DCL node unencrypted connection requires additional securing efforts
        • but likely not so much in cloud VPC environment
  • Option 3:
    • cloud resource (either load balancer or a reverse proxy)
    • pros:
      • certificate management is easy and secure (e.g. AWS ACM)
      • quite straightforward configuration in one place
        • no need to add certificate on each server
        • no need to install and configure proxy server on each node as well
      • unencrypted communication with DCL node might be considered as secure enough (as cloud provider declares)
    • cons:
      • additional cost (TODO more details, likely not so much)
    • sub-options:
      • AWS: docs
        • A reverse proxy accepts a request from a client, forwards it to a server that can fulfill it, and returns the server’s response to the client. It can:
          • Increased security: malicious clients cannot access backend directly to exploit any vulnerabilities
          • Increased scalability and flexibility: you are free to change the configuration of your backend infrastructure
          • web acceleration – reducing the time to return the response
            • Compressing server responses before returning them to the client (reduces bandwidth)
            • SSL termination/SSL offloading: decrypting incoming requests and encrypting server responses to free up resources on backend (e.g. in nginx)
            • Caching – cache on proxy decreases response time to the client and reduces the load on the backend
        • A load balancer distributes incoming client requests among a group of servers, in each case returning the response from the selected server to the appropriate client. In can:
          • distribute the workload in a way that makes the best use of each server’s capacity
          • prevents overload on any server
          • (as result) provide the fastest possible response to the client
          • enhance the user experience by** reducing the number of error responses** the client sees
          • provides session persistence: sending all requests from a particular client to the same server
        • (additionally) Application Load Balancer it only supports static targets (fixed IP address), no dynamic targets (domain name) but it might be ok for our case

4. Health and monitoring

  • [SHOULD] Monitor performance: prometheus
  • [SHOULD] Monitor logs: ELK stack

4.1 Performance (metrics) monitoring

  • what
    • server metrics:
      • CPU usage
      • memory usage
      • disk utilization
      • network performance
      • IO performance
    • application metrics:
      • metrics endpoints exposed by cosmos-sdk
        • blocks:
          • good ones / missed / missed rate (%)
          • time perspective: all time / 1w / 1d / 1h
          • current block height:
            • sentries
            • validators
        • validator / KMS performance
          • sync latency
          • sign latency
          • signatures per minute
        • number of peers
    • ??? RPC endpoints (to provide supplemental data regarding the networks)
  • how
    • tools:
    • notes:
      • HA setup should be considered
      • monitor (Prometheus) should be monitored itself (e.g. by cloud service)
      • alerts for:
        • downtime (e.g. more than 1 min)
        • ...
  • TODO
    • explore more metrics to consider
    • explore and define metrics thresholds
    • work on tools options

4.2 Logs monitoring

  • what
    • application logs
    • system logs
    • authentication logs
  • how:
    • tools:
    • notes:
      • logs are pushed to a high-availability queuing service
      • log processor (Fluentd or Logstash) pops, parses, tokenizes, indexes and stores in a search engine (e.g. Elasticsearch)
      • alerts may be triggered on specific phrases (e.g. CONSENSUS FAILURE or Disk Full)
      • debugging facilities
  • TODO
    • explore and prepare a map: phrase-event
    • list of events for alerts
    • list of events for debugging
    • work on tools options

5. Stability and performance

  • [MUST] Recommended config
    • disable PEX for private nodes
    • adjust timeouts
  • [SHOULD] State-Sync for new Nodes
  • [SHOULD] Seed Nodes for peer discovery??

6. High Availability and scalability

  • [SHOULD] Multiple Observers (Sentries)
  • [SHOULD] Load Balancers for Observers (Public Sentries)

References

Clone this wiki locally