Skip to content

Commit 0e88b5b

Browse files
bingyuyapbingyuyap
committed
historical_uptime: README
Signed-off-by: bingyuyap <bingyu.yap.21@gmail.com>
1 parent 3f8e4ee commit 0e88b5b

File tree

3 files changed

+129
-2
lines changed

3 files changed

+129
-2
lines changed

‎fly/cmd/historical_uptime/.env.sample

+1-1
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ ETH_RPC_URL=https://rpc.ankr.com/eth
66
CORE_BRIDGE_ADDR=0x98f3c9e6E3fAce36bAAd05FE09d375Ef1464288B
77
PROM_REMOTE_URL=https://prometheus_remote_url/api/v1/write
88
GCP_PROJECT_ID=gcp-project-id
9-
GCP_CREDENTIALS_FILE=/path/to/application_default_credentials.json
9+
GCP_CREDENTIALS_FILE=/path/to/credentials.json
1010
BIGTABLE_INSTANCE_ID=bigtable-instance-id
1111

1212
# Bigtable emulator if you are using local emulator

‎fly/cmd/historical_uptime/README.md

+127
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
# Historical Uptime Service
2+
3+
This service tracks the historical uptime of Wormhole guardians by listening to heartbeats and observations from the Wormhole network.
4+
5+
## Prerequisites
6+
7+
Before running the service, ensure you have the following:
8+
9+
1. Go installed (version 1.20)
10+
2. Prometheus remote write URL for metrics
11+
3. Google Cloud Platform (GCP) project with BigTable instance
12+
13+
For development environment, make sure you have the following:
14+
15+
1. Local Prometheus instance
16+
2. Local BigTable emulator
17+
18+
To set up BigTable emulator locally, you can follow the tutorial [here](https://cloud.google.com/bigtable/docs/emulator) or execute this command below with Docker running:
19+
20+
```
21+
docker run -p 127.0.0.1:8086:8086 --rm -ti google/cloud-sdk gcloud beta emulators bigtable start --host-port=0.0.0.0:8086
22+
```
23+
24+
## BigTable
25+
26+
1. Set up a BigTable instance. You can follow the tutorials found [online](https://cloud.google.com/bigtable/docs/creating-instance).
27+
28+
2. Then set up the tables in the instance you created. Tutorials can be found [here](https://cloud.google.com/bigtable/docs/managing-tables#gcloud).
29+
30+
There are 3 tables and the column families that you need for this project.
31+
32+
| Table Name | Column Families |
33+
| --------------------------------- | ----------------- |
34+
| `historical_uptime_messages` | `messageData` |
35+
| `historical_uptime_observations` | `observationData` |
36+
| `historical_uptime_message_index` | `indexData` |
37+
38+
For example, if you are using `gcloud` to create the `historical_uptime_messages` table:
39+
40+
```bash
41+
gcloud bigtable instances tables create historical_uptime_messages \
42+
--instance=bigtable-instance-id \
43+
--project=gcp-project-id \
44+
--column-families=messageData
45+
```
46+
47+
3. Lastly, remember to configure a service account in IAM to allow access control to this BigTable instance. Details can be found [here](https://cloud.google.com/bigtable/docs/access-control).
48+
49+
Keys exported should be stored in `/path/to/credentials.json` and the path is required in `.env` later.
50+
51+
## Configuration
52+
53+
The service can be configured using environment variables. Create a `.env` file in the `cmd/historical_uptime` directory with the following variables:
54+
55+
```
56+
P2P_NETWORK_ID=/wormhole/mainnet/2
57+
P2P_PORT=8999
58+
NODE_KEY_PATH=/tmp/node.key
59+
LOG_LEVEL=info
60+
ETH_RPC_URL=https://rpc.ankr.com/eth
61+
CORE_BRIDGE_ADDR=0x98f3c9e6E3fAce36bAAd05FE09d375Ef1464288B
62+
PROM_REMOTE_URL=https://prometheus_remote_url/api/v1/write
63+
GCP_PROJECT_ID=gcp-project-id
64+
GCP_CREDENTIALS_FILE=/path/to/credentials.json
65+
BIGTABLE_INSTANCE_ID=bigtable-instance-id
66+
USE_BIGTABLE_EMULATOR=false
67+
BIGTABLE_EMULATOR_HOST=
68+
```
69+
70+
For development environment, update add the 2 environment variables below:
71+
72+
```
73+
# USE_BIGTABLE_EMULATOR configures the BigTable library to use local emulator instead
74+
USE_BIGTABLE_EMULATOR=true
75+
76+
# BIGTABLE_EMULATOR_HOST defines where the emulator is hosted
77+
BIGTABLE_EMULATOR_HOST=localhost:8086
78+
```
79+
80+
## Running the Service
81+
82+
To run the service, execute the following commands:
83+
84+
```bash
85+
cd fly/cmd/historical_uptime
86+
go run main.go
87+
```
88+
89+
## Metrics
90+
91+
The Historical Uptime Service exports the following metrics to Prometheus:
92+
93+
| Metric Name | Description | Labels |
94+
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ |
95+
| `guardian_observations_total` | The number of observations done by each guardian on each chain. This tracks if guardians are doing any observations at any time. | `guardian`, `chain_name` |
96+
| `guardian_missed_observations_total` | The number of observations missed by each guardian. This tracks if the guardians are doing observations late or not doing them at all. | `guardian`, `chain_name` |
97+
| `guardian_chain_height` | The current blockheight for each chain on each guardian. | `guardian`, `chain_name` |
98+
| `guardian_chain_height_differences` | The current difference between the blockheight of each guardian for each chain and the maximum blockheight of the chain across all guardians. This tracks if a guardian is behind on syncing their nodes for each blockchain. | `guardian`, `chain_name` |
99+
| `guardian_heartbeats` | The number of heartbeats sent by a guardian. A decrease in this means that a guardian has been reset. | `guardian` |
100+
101+
These metrics can be accessed via the Prometheus remote write URL configured in the `.env` file.
102+
103+
> **Note:** The `guardian_missed_observations_total` metric has a 30-hour delay to account for potential governor delays and network delays. To learn more, please refer to the [monorepo](https://github.com/wormhole-foundation/wormhole/blob/main/node/pkg/processor/cleanup.go#L60-L63).
104+
105+
## Testing
106+
107+
The Historical Uptime Monitor includes a suite of tests to ensure its functionality and reliability. To run the test, follow these steps:
108+
109+
1. Make sure you have all the prerequisites installed and the development environment set up as described in the [Prerequisites](##Prerequisites) section.
110+
111+
2. Navigate to the `fly` directory:
112+
113+
```bash
114+
cd fly
115+
```
116+
117+
3. Run the tests using the following commands:
118+
119+
```bash
120+
# for BigTable specific functionalities
121+
go test -v pkg/bigtable/message_test.go pkg/bigtable/message.go pkg/bigtable/message_index.go pkg/bigtable/test_setup.go pkg/bigtable/observation.go
122+
123+
# for higher level historical_uptime functionalities
124+
go test -v pkg/historical_uptime/process_observation_test.go pkg/historical_uptime/process_observation.go
125+
```
126+
127+
> **Note:** Please make sure that the Bigtable emulator is running as these tests depend on it.

‎fly/pkg/historical_uptime/process_observation_test.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ func TestMain(m *testing.M) {
9292
m.Run()
9393
}
9494

95-
// go test -v pkg/bigtable/message_test.go pkg/bigtable/message.go pkg/bigtable/message_index.go pkg/bigtable/test_setup.go pkg/bigtable/observation.go
95+
// go test -v pkg/historical_uptime/process_observation_test.go pkg/historical_uptime/process_observation.go
9696
func TestProcessObservation(t *testing.T) {
9797
expectedLastObservedAt := time.Now()
9898

0 commit comments

Comments
 (0)