feat(simulator): add hardhat network and fix JsonRpcService import (#676)

* feat: add hardhat node and web3 package

- Add Hardhat node service with Docker configuration
- Configure hardhat.config.js with local network settings
- Add 20 test accounts with 10,000 ETH each
- Add web3 Python package to backend requirements
- Update .gitignore for Hardhat artifacts and cache

* feat: add contract and try script

* feat: setup contract compilation and artifact handling

- Add contract compilation setup with hardhat
- Configure project structure for smart contracts
- Fix file path for GhostContract.json artifact
- Add necessary dependencies for contract compilation
- Setup directory structure for contracts and artifacts

* feat: creating hardhat transactions in consensus mechanism

- Moved web3 python package into backend requirements file
- Added access to compiled hardhat contract in jsonrpc service
- Added database migration file so that a transaction has the ghost contract address of hardhat network
- When a genlayer contract is deployed then a hardhat contract is deployed, both are linked in the CurrentState table
- When a genlayer write method is executed then the new transaction gets the hardhat contract from the CurrentState table
- When a genlayer transaction changes from status then a rollup transaction is created on the hardhat network
- todo: remove rollup transaction table, put hardhat port in env, link genlayer account to hardhat account, check for out of gas, remove prints

* feat: put hardhat port in env

* feat: remove rollup transactions database table

* feat: free transactions on hardhat, one hardhat account

* test: add hardhat test with code from the transaction_processor

* fix: resolve pre-commit error

* test: moved test to integration tests, upgrade web3 version in requirements as it gave an import error

* feat(hardhat): add genlayer-consensus contracts and setup compilation

- Add genlayer-consensus smart contracts to hardhat/contracts directory
- Update hardhat.config.js to enable new code generator (viaIR: true)
- Add @openzeppelin/contracts and @openzeppelin/contracts-upgradeable dependencies
- Configure hardhat Docker container for contract compilation
- Verify hardhat node functionality in genlayer-studio

This commit sets up the smart contract development environment with the
necessary dependencies and configurations to compile and deploy
genlayer-consensus contracts.

* feat: deleted amm_adaptive.py

- Deleted amm_adaptive.py because is not using the new syntax

I will upload the new amm_adaptive.py whenevir will be ready

* fix: contract interaction

- Fixed contract deployment state updates for frontend synchronization
- Improved contract method interactions and state management
- Ensured proper state updates after contract value changes

* fix: pre-commit errors solved

* fix: solved black pre-commit issue

* fix(migrations): resolve multiple head revisions issue

- Identified and merged multiple migration heads
- Ensured database schema consistency
- Updated migration scripts to prevent future conflicts

This change resolves the issue with multiple Alembic head revisions, ensuring smooth database migrations and consistent schema state.

* fix: restore JsonRpcService import removed in last PR

Restores the JsonRpcService import that was accidentally removed in the last PR.
This fixes the build error where JsonRpcService was undefined.

---------

Co-authored-by: kstroobants <stroobants.kristof@hotmail.com>
Co-authored-by: Cristiam Da Silva <cristiam86@gmail.com>

Author: 3 people

.dockerignore

@@ -27,3 +27,12 @@ frontend/vite.config.ts.timestamp-*
 # Byte-compiled / optimized / DLL files
 **/__pycache__/
 **/*.py[cod]
+
+# Hardhat files
+hardhat/cache/
+hardhat/artifacts/
+hardhat/node_modules/
+hardhat/coverage/
+hardhat/.env
+hardhat/coverage.json
+hardhat/typechain-types/

.env.example

@@ -51,6 +51,12 @@ VITE_PROXY_JSON_RPC_SERVER_URL = 'http://jsonrpc:4000'
 VITE_PROXY_WS_SERVER_URL       = 'ws://jsonrpc:4000'
 VITE_IS_HOSTED                 = 'false'
 
+FRONTEND_BUILD_TARGET = 'final' # change to 'dev' to run in dev mode
+
+# Hardhat port
+HARDHAT_URL = 'http://hardhat'
+HARDHATPORT = '8545'
+HARDHAT_PRIVATE_KEY = '0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80'
 # LLM Providers Configuration
 # If you want to use OpenAI LLMs, add your key here
 OPENAIKEY               = '<add_your_openai_api_key_here>'

.github/workflows/backend_integration_tests_pr.yml

@@ -140,3 +140,28 @@ jobs:
     uses: ./.github/workflows/load-test-oha.yml
     with:
       oha-version: "v1.4.5"
+
+  hardhat-test:
+    needs: triggers
+    if: ${{ needs.triggers.outputs.is_pull_request_opened == 'true' || needs.triggers.outputs.is_pull_request_review_approved == 'true' || needs.triggers.outputs.is_pull_request_labeled_with_run_tests == 'true' }}
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Cache Docker layers
+        uses: actions/cache@v4
+        with:
+          path: /tmp/.buildx-cache
+          key: ${{ runner.os }}-buildx-${{ github.sha }}
+          restore-keys: |
+            ${{ runner.os }}-buildx-
+
+      - name: Run Docker Compose
+        run: docker compose -f tests/hardhat/docker-compose.yml --project-directory . up tests --build --force-recreate --always-recreate-deps
+

.gitignore

@@ -147,8 +147,17 @@ consensus/nodes/nodes.json
 # npm
 node_modules/
 package-lock.json
-package.json
+/package.json
 
 # Nginx TLS certificates and keys
 nginx/ssl/*.key
 nginx/ssl/*.crt
+
+# Hardhat files
+hardhat/cache
+hardhat/artifacts
+hardhat/.openzeppelin
+hardhat/coverage
+hardhat/coverage.json
+hardhat/typechain
+hardhat/typechain-types

backend/consensus/base.py

@@ -420,6 +420,7 @@ def finalize_transaction(
                     "data": {
                         "state": leader_receipt["contract_state"],
                         "code": transaction.data["contract_code"],
+                        "ghost_contract_address": transaction.ghost_contract_address,
                     },
                 }
                 leaders_contract_snapshot.register_contract(new_contract)

backend/database_handler/contract_snapshot.py

@@ -26,10 +26,14 @@ def __init__(self, contract_address: str | None, session: Session):
             self.contract_data = contract_account.data
             self.contract_code = self.contract_data["code"]
             self.encoded_state = self.contract_data["state"]
+            self.ghost_contract_address = (
+                self.contract_data["ghost_contract_address"]
+                if "ghost_contract_address" in self.contract_data
+                else None
+            )
 
     def _load_contract_account(self) -> CurrentState:
         """Load and return the current state of the contract from the database."""
-
         result = (
             self.session.query(CurrentState)
             .filter(CurrentState.id == self.contract_address)

backend/database_handler/migration/versions/cb34b6b353ed_add_ghost_contract_address_column.py

@@ -0,0 +1,30 @@
+"""add ghost contract address column
+
+Revision ID: cb34b6b353ed
+Revises: 0d9538be0318
+Create Date: 2024-11-28 14:00:00
+
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = "cb34b6b353ed"
+down_revision: Union[str, None] = "37196a51038e"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "transactions",
+        sa.Column("ghost_contract_address", sa.String(length=255), nullable=True),
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("transactions", "ghost_contract_address")

backend/database_handler/models.py

@@ -94,6 +94,7 @@ class Transactions(Base):
     r: Mapped[Optional[int]] = mapped_column(Integer)
     s: Mapped[Optional[int]] = mapped_column(Integer)
     v: Mapped[Optional[int]] = mapped_column(Integer)
+    ghost_contract_address: Mapped[Optional[str]] = mapped_column(String(255))
 
     # Relationship for triggered transactions
     triggered_by_hash: Mapped[Optional[str]] = mapped_column(
@@ -117,38 +118,6 @@ class Transactions(Base):
     timestamp_accepted: Mapped[Optional[int]] = mapped_column(BigInteger, default=None)
 
 
-class RollupTransactions(Base):
-    __tablename__ = "rollup_transactions"
-    __table_args__ = (
-        PrimaryKeyConstraint("transaction_hash", name="rollup_transactions_pkey"),
-    )
-
-    transaction_hash: Mapped[str] = mapped_column(
-        String(66), primary_key=True, unique=True
-    )
-    from_: Mapped[str] = mapped_column(
-        String(255),
-    )
-    to_: Mapped[Optional[dict]] = mapped_column(
-        String(255),
-    )
-    gas: Mapped[int] = mapped_column(
-        Integer,
-    )
-    gas_price: Mapped[int] = mapped_column(
-        Integer,
-    )
-    value: Mapped[Optional[int]] = mapped_column(
-        Integer,
-    )
-    input: Mapped[str] = mapped_column(
-        Text,
-    )
-    nonce: Mapped[int] = mapped_column(
-        BigInteger,
-    )
-
-
 class Validators(Base):
     __tablename__ = "validators"
     __table_args__ = (

backend/database_handler/transactions_processor.py

@@ -2,7 +2,7 @@
 from enum import Enum
 import rlp
 
-from .models import Transactions, RollupTransactions
+from .models import Transactions
 from sqlalchemy.orm import Session
 from sqlalchemy import or_, and_
 
@@ -11,6 +11,10 @@
 import json
 import base64
 import time
+from backend.domain.types import TransactionType
+from web3 import Web3
+from backend.database_handler.contract_snapshot import ContractSnapshot
+import os
 
 
 class TransactionAddressFilter(Enum):
@@ -26,6 +30,12 @@ def __init__(
     ):
         self.session = session
 
+        # Connect to Hardhat Network
+        port = os.environ.get("HARDHAT_PORT")
+        url = os.environ.get("HARDHAT_URL")
+        hardhat_url = f"{url}:{port}"
+        self.web3 = Web3(Web3.HTTPProvider(hardhat_url))
+
     @staticmethod
     def _parse_transaction_data(transaction_data: Transactions) -> dict:
         return {
@@ -49,6 +59,7 @@ def _parse_transaction_data(transaction_data: Transactions) -> dict:
                 transaction.hash
                 for transaction in transaction_data.triggered_transactions
             ],
+            "ghost_contract_address": transaction_data.ghost_contract_address,
             "appealed": transaction_data.appealed,
             "timestamp_accepted": transaction_data.timestamp_accepted,
         }
@@ -130,6 +141,64 @@ def insert_transaction(
             from_address, to_address, data, value, type, nonce
         )
 
+        if type == TransactionType.DEPLOY_CONTRACT.value:
+            # Hardhat account
+            account = self.web3.eth.accounts[0]
+            private_key = os.environ.get("HARDHAT_PRIVATE_KEY")
+
+            # Ghost contract
+            # Read contract ABI and bytecode from compiled contract
+            contract_file = os.path.join(
+                os.getcwd(),
+                "hardhat/artifacts/contracts/GhostContract.sol/GhostContract.json",
+            )
+
+            with open(contract_file, "r") as f:
+                contract_json = json.loads(f.read())
+                abi = contract_json["abi"]
+                bytecode = contract_json["bytecode"]
+
+            # Create the contract instance
+            contract = self.web3.eth.contract(abi=abi, bytecode=bytecode)
+
+            # Build the transaction
+            gas_estimate = self.web3.eth.estimate_gas(
+                contract.constructor().build_transaction(
+                    {
+                        "from": account,
+                        "nonce": self.web3.eth.get_transaction_count(account),
+                        "gasPrice": 0,
+                    }
+                )
+            )
+            transaction = contract.constructor().build_transaction(
+                {
+                    "from": account,
+                    "nonce": self.web3.eth.get_transaction_count(account),
+                    "gas": gas_estimate,
+                    "gasPrice": 0,
+                }
+            )
+
+            # Sign the transaction
+            signed_tx = self.web3.eth.account.sign_transaction(
+                transaction, private_key=private_key
+            )
+
+            # Send the transaction
+            tx_hash = self.web3.eth.send_raw_transaction(signed_tx.raw_transaction)
+
+            # Wait for the transaction receipt
+            receipt = self.web3.eth.wait_for_transaction_receipt(tx_hash)
+            ghost_contract_address = receipt.contractAddress
+
+        elif type == TransactionType.RUN_CONTRACT.value:
+            genlayer_contract_address = to_address
+            contract_snapshot = ContractSnapshot(
+                genlayer_contract_address, self.session
+            )
+            ghost_contract_address = contract_snapshot.ghost_contract_address
+
         new_transaction = Transactions(
             hash=transaction_hash,
             from_address=from_address,
@@ -152,6 +221,7 @@ def insert_transaction(
                 if triggered_by_hash
                 else None
             ),
+            ghost_contract_address=ghost_contract_address,
             appealed=False,
             timestamp_accepted=None,
         )
@@ -195,29 +265,44 @@ def create_rollup_transaction(self, transaction_hash: str):
         transaction = (
             self.session.query(Transactions).filter_by(hash=transaction_hash).one()
         )
-        rollup_input_data = self._transaction_data_to_str(
+        rollup_input_data = json.dumps(
             self._parse_transaction_data(transaction)
+        ).encode("utf-8")
+
+        # Hardhat transaction
+        account = self.web3.eth.accounts[0]
+        private_key = os.environ.get("HARDHAT_PRIVATE_KEY")
+
+        gas_estimate = self.web3.eth.estimate_gas(
+            {
+                "from": account,
+                "to": transaction.ghost_contract_address,
+                "value": transaction.value,
+                "data": rollup_input_data,
+            }
         )
-        rollup_nonce = int(time.time() * 1000)
-        rollup_transaction_hash = self._generate_transaction_hash(
-            transaction.from_address,
-            transaction.to_address,
-            rollup_input_data,
-            transaction.value,
-            0,
-            rollup_nonce,
-        )
-        rollup_transaction_record = RollupTransactions(
-            transaction_hash=rollup_transaction_hash,
-            from_=transaction.from_address,
-            to_=transaction.to_address,
-            gas=0,
-            gas_price=0,
-            value=transaction.value,
-            input=rollup_input_data,
-            nonce=rollup_nonce,
+
+        transaction = {
+            "from": account,
+            "to": transaction.ghost_contract_address,
+            "value": transaction.value,
+            "data": rollup_input_data,
+            "nonce": self.web3.eth.get_transaction_count(account),
+            "gas": gas_estimate,
+            "gasPrice": 0,
+        }
+
+        # Sign and send the transaction
+        signed_tx = self.web3.eth.account.sign_transaction(
+            transaction, private_key=private_key
         )
-        self.session.add(rollup_transaction_record)
+        tx_hash = self.web3.eth.send_raw_transaction(signed_tx.raw_transaction)
+
+        # Wait for transaction to be actually mined and get the receipt
+        receipt = self.web3.eth.wait_for_transaction_receipt(tx_hash)
+
+        # Get full transaction details including input data
+        transaction = self.web3.eth.get_transaction(tx_hash)
 
     def get_transaction_count(self, address: str) -> int:
         count = (

backend/domain/types.py

@@ -79,6 +79,7 @@ class Transaction:
     leader_only: bool = (
         False  # Flag to indicate if this transaction should be processed only by the leader. Used for fast and cheap execution of transactions.
     )
+    ghost_contract_address: str | None = None
     appealed: bool = False
     timestamp_accepted: int | None = None
 
@@ -99,6 +100,7 @@ def to_dict(self):
             "s": self.s,
             "v": self.v,
             "leader_only": self.leader_only,
+            "ghost_contract_address": self.ghost_contract_address,
             "appealed": self.appealed,
             "timestamp_accepted": self.timestamp_accepted,
         }
@@ -121,6 +123,7 @@ def transaction_from_dict(input: dict) -> Transaction:
         s=input.get("s"),
         v=input.get("v"),
         leader_only=input.get("leader_only", False),
+        ghost_contract_address=input.get("ghost_contract_address"),
         appealed=input.get("appealed"),
         timestamp_accepted=input.get("timestamp_accepted"),
     )

backend/protocol_rpc/requirements.txt

@@ -21,3 +21,4 @@ Flask-SQLAlchemy==3.1.1
 jsf==0.11.2
 jsonschema==4.23.0
 loguru==0.7.2
+web3==7.5.0