Merge branch 'master' into improve_scripts

Author: electrocucaracha

.github/workflows/verify.yml

@@ -46,7 +46,7 @@ jobs:
               - 'go.mod'
               - 'go.sum'
             src:
-              - '!((**.md)|(**/*.md)|(.github/**))'
+              - '!((**.md)|(**/*.md)|(.github/**)|(docs/**)|(vue/**))'
             docs:
               - '**.md'
               - '**/*.md'

README-DEV.md

@@ -130,7 +130,7 @@ Have a look at [How To](docs/how-to.md) and [transactions](docs/transactions.md)
 
 Start a local pool as described above.
 
-Every node exposes a REST API at `http://<node-host>:1317` (see <https://docs.cosmos.network/master/core/grpc_rest.html>).
+Every node exposes a REST API at `http://<node-host>:1317` (see <https://docs.cosmos.network/v0.44/core/grpc_rest.html>).
 
 Have a look at [transactions](docs/transactions.md) for a full list of REST endpoints.
 

README.md

@@ -45,16 +45,17 @@ In order to send write transactions to the ledger you need:
 - A network of Tendermint-based validator nodes (Validators and Observers) maintaining the ledger.
 - Every validator node (`dcld` binary) runs DC Ledger application code (based on Cosmos SDK) implementing the use cases.
 - See the proposed deployment in [deployment](docs/deployment.png) and [deployment-detailed](docs/deployment-detailed.png).
+- See recommended design for DCL MainNet deployment on AWS in [aws deployment](./docs/deployment-design-aws.md)
 
 ### Node Types
 
 - **Full Node**: contains a full replication of data (ledger, state, etc.):
   - **Validator Node (VN)**: a full node participating in consensus protocol (ordering transactions).
   - **Sentry Node:** a full nodes that doesn't participate in consensus and wraps the validator node representing it for the rest of the network
     as one of the ways for DDoS protection.
-    - **Private Sentry Node:** connected to other Validators or Sentry nodes only; should not be accessed by clients.
-    - **Public Sentry Node:** clients and other nodes can access it; basically the same as an Observer node.
-  - **Observer Node (ON):** a full node that doesn't participate in consensus. Should be used to receive read/write requests from the clients. Technically can be a Public Sentry node.
+    - **Private Sentry Node:** a full node to connect other Validator or Sentry nodes only; should not be accessed by clients.
+    - **Public Sentry Node:** a full node to connect other external full nodes (possibly observer nodes).
+  - **Observer Node (ON):** a full node that doesn't participate in consensus. Should be used to receive read/write requests from the clients.
 - **Light Client Proxy Node**: doesn't contain a full replication of data. Can be used as a proxy to untrusted Full nodes for single-value query requests sent via CLI or Tendermint RPC.
   It will verify all state proofs automatically.
 - **Seed Node**: provides a list of peers which a node can connect to.
@@ -64,6 +65,7 @@ See
 - [Deployment](docs/deployment.png)
 - [Deployment-detailed](docs/deployment-detailed.png).
 - [Deployment Recommendations](https://github.com/zigbee-alliance/distributed-compliance-ledger/wiki/DCL-MainNet-Deployment)
+- [Deployment Recommendations for AWS](./docs/deployment-design-aws.md)
 - <https://docs.tendermint.com/master/nodes/validators.html>
 - [Run Light Client Proxy](docs/running-light-client-proxy.md)
 
@@ -121,7 +123,7 @@ See [Run Light Client Proxy](docs/running-light-client-proxy.md) for details how
 
 - **There are no state proofs in REST, so REST queries should be sent to trusted Validator or Observer nodes only.**
 - OpenAPI specification: <https://zigbee-alliance.github.io/distributed-compliance-ledger/>.
-- Any running node exposes a REST API at port `1317`. See <https://docs.cosmos.network/master/core/grpc_rest.html>.
+- Any running node exposes a REST API at port `1317`. See <https://docs.cosmos.network/v0.44/core/grpc_rest.html>.
 - See [transactions](docs/transactions.md) for a full list of endpoints.
 - REST HTTP(S) queries can be directly used for read requests.
   See [How to read from the Ledger](docs/transactions.md#how-to-read-from-the-ledger).
@@ -132,14 +134,14 @@ See [Run Light Client Proxy](docs/running-light-client-proxy.md) for details how
 ### gRPC
 
 - **There are no state proofs in gRPC, so gRPC queries should be sent to trusted Validator or Observer nodes only.**
-- Any running node exposes a REST API at port `9090`. See <https://docs.cosmos.network/master/core/grpc_rest.html>.
+- Any running node exposes a REST API at port `9090`. See <https://docs.cosmos.network/v0.44/core/grpc_rest.html>.
 - A client code can be generated for all popular languages from the proto files [proto](proto), see <https://grpc.io/docs/languages/>.
 - The generated client code can be used for read and write requests, i.e. generation and signing of transactions
   See [How to read from the Ledger](docs/transactions.md#how-to-read-from-the-ledger) and [How to write to the Ledger](docs/transactions.md#how-to-write-to-the-ledger) for details.
 
 ### Tendermint RPC and Light Client
 
-- Tendermint RPC is exposed by every running node  at port `26657`. See <https://docs.cosmos.network/master/core/grpc_rest.html#tendermint-rpc>.
+- Tendermint RPC is exposed by every running node  at port `26657`. See <https://docs.cosmos.network/v0.44/core/grpc_rest.html#tendermint-rpc>.
 - Tendermint RPC supports state proofs. Tendermint's Light Client library can be used to verify the state proofs.
     So, if Light Client API is used, then it's possible to communicate with non-trusted nodes.
 - Please note, that multi-value queries don't have state proofs support and should be sent to trusted nodes only.
@@ -194,7 +196,7 @@ See [Run local pool](README-DEV.md#run-local-pool) section in [README-DEV.md](RE
 
 ### Deploy a persistent pool of nodes
 
-A recommended way for deployment and client connection: [diagram](docs/deployment.png) and [diagram-detailed](docs/deployment-detailed.png).
+A recommended way for deployment and client connection: [diagram](docs/deployment.png), [diagram-detailed](docs/deployment-detailed.png) and [diagram-aws](docs/deployment-aws.png).
 
 One can either deploy its own network of validator nodes or join one of the persistent DC Ledger Networks.
 
@@ -237,6 +239,7 @@ See [Pool Upgrade](docs/pool-upgrade.md) and [Pool Upgrade How To](docs/pool-upg
 - [Deployment Pattern](docs/deployment.png)
 - [Deployment Pattern Detailed](docs/deployment-detailed.png)
 - [Deployment Recommendations](https://github.com/zigbee-alliance/distributed-compliance-ledger/wiki/DCL-MainNet-Deployment)
+- [Deployment Recommendations for AWS](./docs/deployment-design-aws.md)
 - [Running Node in a new network](docs/running-node.md)
   - [Running Genesis Validator Node](docs/advanced/running-genesis-node.md)
   - [Running Validator Node](docs/advanced/running-validator-node.md)

docs/deployment-aws.png

@@ -0,0 +1,210 @@
+# Recommended design for DCL MainNet deployment on AWS
+## AWS deployment diagram
+![AWS deployment diagram](./deployment-aws.png)
+
+## Recommended IaC structure and frameworks
+
+- [Ansible](https://www.ansible.com) - provision of the following node types:
+    - `Genesis Validator` - Validator Node created at the beginning of a network
+    - `Non-genesis Validator` - Validator Node joined a network after a significant time period
+    - `Private Sentry` - Full Node to connect other(external) Validator Nodes ([Sentry Node Architecture](https://forum.cosmos.network/t/sentry-node-architecture-overview/454))
+    - `Public Sentry` - Full Node to connect other(external) Full Nodes
+    - `Observer` - Full Node for serving gRPC / REST / RPC clients
+    - `Seed` - Full Node for sharing IP addresses of `Public Sentry` Nodes ([Seed Node](https://docs.tendermint.com/master/nodes/#seed-nodes))
+
+    > **_Note:_** Most of the nodes should enable `state sync` to avoid catching up with a network from scratch. Refer to [running-node-in-existing-netwrok.md](./running-node-in-existing-network.md) for details.
+
+- [Terraform](https://www.terraform.io) - deploy an AWS infrastructure from one or more of the following modules:
+    - Validator - `Validator` node instance
+    - Private Sentries - Cluster of `Private Sentry` node instances
+    - Public Sentries - Cluster of `Public Sentry` node instances with a collocated `Seed` node
+    - Observers - Cluster of `Observer` node instances
+    - Load Balancers - AWS Network Load Balancers for load balancing between `Observer` clusters
+  
+
+## Node specific AWS and DCL configurations
+
+### Validator Node:
+- Tendermint:
+    - [config.toml]
+        - [p2p]
+            - `pex` = false
+            - `persistent_peers` = [`Private Sentry` nodes with private IPs]
+            - `addr_book_strict` = false
+        - [statesync] (only for `Non-genesis Validator` nodes)
+            - `enable` = true
+            - `rpc_servers` = [existing `Genesis Validator` / `Sentry` nodes' RPC endpoints]
+            - `trust_height` = trust-height
+            - `trust_hash` = trust-hash
+    - [app.toml]
+        - [state-sync]
+            - `snapshot-interval` = snapshot-interval
+            - `snapshot-keep-recent` = snapshot-keep-recent
+
+- AWS:
+    - Instance type = EC2 instance
+    - Network:
+        - Private IPv4 = IPv4 address
+        - Public IPv4 = not assigned
+    - Security:
+        - inbound:
+            - allow `Tendermint p2p` port from `Private Sentry` Nodes' VPC CIDR
+            - allow `RPC` port from `Private Sentry` Nodes' VPC CIDR
+        - outbound:
+            - all
+
+
+### Private Sentry Node: 
+- Tendermint:
+    - [config.toml]
+        - [p2p]
+            - `pex` = true
+            - `persistent_peers` = [`Validator` node with private IP + other orgs' validator/sentry nodes with public IPs] 
+            - `private_peer_ids` = [`Validator` node id]
+            - `unconditional_peers` = [`Validator` node id]
+            - `addr_book_strict` = false
+        - [statesync]
+            - `enable` = true
+            - `rpc_servers` = [`Validator` node's RPC endpoint]
+            - `trust_height` = trust-height
+            - `trust_hash` = trust-hash
+    - [app.toml]
+        - [state-sync]
+            - `snapshot-interval` = snapshot-interval
+            - `snapshot-keep-recent` = snapshot-keep-recent
+- AWS:
+    - Instance type = EC2 instance
+    - Network:
+        - Private IPv4 = IPv4 address
+        - Public IPv4 = Elastic IP
+    - Security:
+        - inbound:
+            - allow `Tendermint p2p` port for whitelist IPs
+            - allow `RPC` port from `Observer` Nodes' VPC CIDR
+            - allow `RPC` port from `Public Sentry` Nodes' VPC CIDR
+        - outbound:
+            - all
+
+### Observer Node: 
+- Tendermint:
+    - [config.toml]
+        - [p2p]
+            - `pex` = true
+            - `persistent_peers` = [`Private Sentry` nodes with private IPs]
+            - `addr_book_strict` = false
+        - [statesync]
+            - `enable` = true
+            - `rpc_servers` = [`Private Sentry` nodes' RPC endpoints]
+            - `trust_height` = trust-height
+            - `trust_hash` = trust-hash
+    - [app.toml]
+        - [api]
+            - `enable` = true
+- AWS:
+    - Instance type = EC2 instance
+    - Network:
+        - Private IPv4 = IPv4 address
+        - Public IPv4 = not assigned
+    - Security:
+        - inbound:
+            - allow gRPC / REST / RPC ports from the same VPC CIDR
+        - outbound:
+            - all
+
+### Public Sentry Node: 
+- Tendermint:
+    - [config.toml]
+        - [p2p]
+            - `pex` = true
+            - `persistent_peers` = [`Private Sentry` nodes with private IPs]
+        - [statesync]
+            - `enable` = true
+            - `rpc_servers` = [`Private Sentry` nodes' RPC endpoints]
+            - `trust_height` = trust-height
+            - `trust_hash` = trust-hash
+    - [app.toml]
+        - [state-sync]
+            - `snapshot-interval` = snapshot-interval
+            - `snapshot-keep-recent` = snapshot-keep-recent
+- AWS:
+    - Instance type = EC2 instance
+    - Network:
+        - Private IPv4 = IPv4 address
+        - Public IPv4 = Elastic IP
+    - Security:
+        - inbound:
+            - allow `Tendermint p2p` port from anywhere
+            - allow `Tendermint RPC` port from anywhere
+        - outbound:
+            - all
+
+
+### Seed Node: 
+- Tendermint:
+    - [config.toml]
+        - [p2p]
+            - `pex` = true
+            - `seed_mode` = true
+            - `persistent_peers` = [`Public Sentry` nodes with public IP]
+        - [statesync]
+            - `enable` = true
+            - `rpc_servers` = [`Private Sentry` nodes' RPC endpoints]
+            - `trust_height` = trust-height
+            - `trust_hash` = trust-hash
+- AWS:
+    - Instance type = EC2 instance
+    - Network:
+        - Private IPv4 = IPv4 address
+        - Public IPv4 = Elastic IP
+        - Public DNS = optional
+    - Security:
+        - inbound:
+            - allow `Tendermint p2p` port from everywhere
+        - outbound:
+            - all
+
+### Load Balancer: 
+- AWS:
+    - Instance type = Elastic Network Load Balancer
+    - Availability Zones = [availability zones of observer nodes from the same region]
+    - Network:
+        - Private IPv4 = IPv4 address
+        - Public IPv4 = not assigned
+        - Public DNS = assigned by AWS
+
+    - Target groups:
+        - gRPC
+            - Registered targets = [observer nodes from all availability zones in the same region]
+            - Attributes:
+                - `Preserve client IP addresses` = disabled
+            - Health checks:
+                - protocol = TCP
+        - REST
+            - Registered targets = [observer nodes from all availability zones in the same region]
+            - Attributes:
+                - `Preserve client IP addresses` = disabled
+            - Health checks:
+                - protocol = TCP
+        - RPC
+            - Registered targets = [observer nodes from all availability zones the same region]
+            - Attributes:
+                - `Preserve client IP addresses` = disabled
+            - Health checks:
+                - protocol = TCP
+
+    - Listeners:
+        - gRPC
+            - Protocol = TLS
+            - Forward to = gRPC target group
+            - Security policy = ELBSecurityPolicy-TLS13-1-2-2021-06
+            - Default SSL/TLS certificate = CA signed TLS certificate
+        - REST
+            - Protocol = TLS
+            - Forward to = REST target group
+            - Security policy = ELBSecurityPolicy-TLS13-1-2-2021-06
+            - Default SSL/TLS certificate = CA signed TLS certificate
+        - RPC 
+            - Protocol = TLS
+            - Forward to = RPC target group
+            - Security policy = ELBSecurityPolicy-TLS13-1-2-2021-06
+            - Default SSL/TLS certificate = CA signed TLS certificate

docs/deployment-design-aws.md

@@ -42,9 +42,9 @@ The following ones makes sense to track:
 
 ### Cosmos SDK metrics
 
-Starting from `v0.40.0` Cosmos SDK provides [telemetry](https://docs.cosmos.network/master/core/telemetry.html) package as a server-side support for application performance and behavior explorations.
+Starting from `v0.40.0` Cosmos SDK provides [telemetry](https://docs.cosmos.network/v0.44/core/telemetry.html) package as a server-side support for application performance and behavior explorations.
 
-The following [metrics](https://docs.cosmos.network/master/core/telemetry.html#supported-metrics) make sense to track:
+The following [metrics](https://docs.cosmos.network/v0.44/core/telemetry.html#supported-metrics) make sense to track:
 
 - `tx_count`: Total number of txs processed via DeliverTx (tx)
 - `tx_successful`: Total number of successful txs processed via DeliverTx  (tx)
@@ -67,7 +67,7 @@ Notes:
 
 - For the moment it's not clear enough what production setup will look like, in particular:
   - number of validators
-  - type of external endpoints, options are [Cosmos SDK / Tendermint endpoints](https://docs.cosmos.network/master/core/grpc_rest.html)
+  - type of external endpoints, options are [Cosmos SDK / Tendermint endpoints](https://docs.cosmos.network/v0.44/core/grpc_rest.html)
   - type and number of proxies for validator-validator and client-validator connections
 
 Current assumptions for production:
@@ -94,7 +94,7 @@ Current assumptions for production:
 
 **ToDo** define which client endpoints are considered in production
 
-As long as CosmosSDK (Tendermint) provides multiple client [endpoints](https://docs.cosmos.network/master/core/grpc_rest.html) makes sense to benchmark all of them (separately and in a combination), in particular:
+As long as CosmosSDK (Tendermint) provides multiple client [endpoints](https://docs.cosmos.network/v0.44/core/grpc_rest.html) makes sense to benchmark all of them (separately and in a combination), in particular:
 
 - http RPC
 - websocket RPC
@@ -114,7 +114,7 @@ As long as CosmosSDK (Tendermint) provides multiple client [endpoints](https://d
 
 ## Load Generation Framework
 
-As long as DCledger based on Cosmos SDK and Tendermint which provide standard HTTP/websocket RPC and REST  [endpoints](https://docs.cosmos.network/master/core/grpc_rest.html) to perform both read & write txns generic production ready tools like [jMeter](https://jmeter.apache.org/), [Locust](https://locust.io/), [K6](https://k6.io/) may be used.
+As long as DCledger based on Cosmos SDK and Tendermint which provide standard HTTP/websocket RPC and REST  [endpoints](https://docs.cosmos.network/v0.44/core/grpc_rest.html) to perform both read & write txns generic production ready tools like [jMeter](https://jmeter.apache.org/), [Locust](https://locust.io/), [K6](https://k6.io/) may be used.
 
 [Locust](https://locust.io/) looks like the most easy-to-go option: