feat(hardhat): add rollup consensus service (#713)

* feat: add hardhat node and web3 package

- Add Hardhat node service with Docker configuration
- Configure hardhat.config.js with local network settings
- Add 20 test accounts with 10,000 ETH each
- Add web3 Python package to backend requirements
- Update .gitignore for Hardhat artifacts and cache

* feat: add contract and try script

* feat: setup contract compilation and artifact handling

- Add contract compilation setup with hardhat
- Configure project structure for smart contracts
- Fix file path for GhostContract.json artifact
- Add necessary dependencies for contract compilation
- Setup directory structure for contracts and artifacts

* feat: creating hardhat transactions in consensus mechanism

- Moved web3 python package into backend requirements file
- Added access to compiled hardhat contract in jsonrpc service
- Added database migration file so that a transaction has the ghost contract address of hardhat network
- When a genlayer contract is deployed then a hardhat contract is deployed, both are linked in the CurrentState table
- When a genlayer write method is executed then the new transaction gets the hardhat contract from the CurrentState table
- When a genlayer transaction changes from status then a rollup transaction is created on the hardhat network
- todo: remove rollup transaction table, put hardhat port in env, link genlayer account to hardhat account, check for out of gas, remove prints

* feat: put hardhat port in env

* feat: remove rollup transactions database table

* feat: free transactions on hardhat, one hardhat account

* test: add hardhat test with code from the transaction_processor

* fix: resolve pre-commit error

* test: moved test to integration tests, upgrade web3 version in requirements as it gave an import error

* feat(hardhat): add genlayer-consensus contracts and setup compilation

- Add genlayer-consensus smart contracts to hardhat/contracts directory
- Update hardhat.config.js to enable new code generator (viaIR: true)
- Add @openzeppelin/contracts and @openzeppelin/contracts-upgradeable dependencies
- Configure hardhat Docker container for contract compilation
- Verify hardhat node functionality in genlayer-studio

This commit sets up the smart contract development environment with the
necessary dependencies and configurations to compile and deploy
genlayer-consensus contracts.

* feat: deleted amm_adaptive.py

- Deleted amm_adaptive.py because is not using the new syntax

I will upload the new amm_adaptive.py whenevir will be ready

* fix: contract interaction

- Fixed contract deployment state updates for frontend synchronization
- Improved contract method interactions and state management
- Ensured proper state updates after contract value changes

* fix: pre-commit errors solved

* fix: solved black pre-commit issue

* feat(hardhat): add deployment persistence to hardhat node

- Add volume mappings for deployments, artifacts and cache
- Implement manual deployment file saving
- Update hardhat configuration for proper deployment paths

This change ensures that deployed contracts persist between container
restarts by saving deployment data to mounted volumes.

Note: Block persistence still requires a different solution (Ganache/Geth/Besu)
as Hardhat Network doesn't support state persistence.

Related issue: #669

* refactor(hardhat): integrate start script into Dockerfile

- Removed start.sh script
- Incorporated start script directly into Dockerfile for streamlined execution

* fix(db): remove duplicate ghost_contract_address assignment

Removes redundant assignment of ghost_contract_address in transaction_from_dict
function to improve code clarity and prevent potential inconsistencies

* fix(security): address SonarCloud warnings and improve test coverage

- Ensure non-root user is used in Dockerfile to enhance security
- Adjust permissions to avoid overly permissive access (777) on /app directories
- Add tests to cover deployment scripts, ensuring proper functionality and coverage

* commit: Add contract initialization and consensus service integration

- Updated deploy_contracts.js to handle contract initialization after deployment
- Added initial ConsensusService integration into server.py
- Enhanced Dockerfile.hardhat with proper permissions and chainId handling
- Started implementation of consensus_service.py with basic contract interactions

Note: consensus_service.py still needs additional work for complete contract initialization and interaction methods.

* feat(deploy): integrate contract deployment and backend service

Integrate complete contract deployment persistence in Hardhat and create ConsensusService for contract interaction.

- Add deployFixture to Hardhat deployment script to handle contract initialization
- Create ConsensusService class to interact with deployed contracts from backend
- Add deploy_fixture method to ConsensusService as fallback initialization option
- Ensure consistent contract initialization between Hardhat and backend
- Load contract artifacts and addresses in ConsensusService from deployment files

The deployFixture now handles all contract initialization during deployment, while
ConsensusService provides a Python interface to interact with the deployed contracts.
The deploy_fixture method in ConsensusService mirrors the Hardhat initialization
logic but is kept as a fallback option.

* feat(hardhat): implement contract deployment flow and deployment backup system

- Add consensus contract deployment flow with proper initialization sequence
- Implement contract backup system for deployment persistence
- Add deployment verification to prevent unnecessary redeployments
- Create copy-deployments script for managing contract backups
- Configure hardhat to handle contract compilation and deployment

* feat(hardhat): implement contract deployment flow and deployment backup system

- Add consensus contract deployment flow with proper initialization sequence
- Implement contract backup system for deployment persistence
- Add deployment verification to prevent unnecessary redeployments
- Create copy-deployments script for managing contract backups
- Configure hardhat to handle contract compilation and deployment

* refactor: remove ghost_contract_address redundant references

- Remove ghost_contract_address from transactions_processor to avoid duplication
- Remove unused ghost_contract_address from types.py

* refactor: replace hardhat-deploy with hardhat-ignition

- Migrate deployment process from hardhat-deploy to hardhat-ignition
- Create DeployFixture module following TransactionFlow deployment order
- Update deployment tests to ensure initialization and contract connecton as TransactionFlow

* refactor: remove deployFixture from consensus_service.py

- Remove unused deployFixture function from consensus_service.py
- Deployment now handled entirely through hardhat-ignition

* refactor(consensus): clean up unused imports and methods in ConsensusService

- Remove unused eth_account Account import
- Remove unused _send_transaction method

* refactor(consensus): clean up unused imports and methods in ConsensusService

- Remove unused eth_account Account import
- Remove unused _send_transaction method

---------

Co-authored-by: kstroobants <stroobants.kristof@hotmail.com>

Author: mpaya5

.gitignore

@@ -153,11 +153,16 @@ package-lock.json
 nginx/ssl/*.key
 nginx/ssl/*.crt
 
+
 # Hardhat files
-hardhat/cache
-hardhat/artifacts
+hardhat/cache/*
+hardhat/artifacts/*
 hardhat/.openzeppelin
 hardhat/coverage
 hardhat/coverage.json
 hardhat/typechain
 hardhat/typechain-types
+hardhat/deployments/*
+
+# Geth keystore
+hardhat/geth_keystore/*

backend/protocol_rpc/server.py

@@ -23,6 +23,7 @@
 from backend.database_handler.accounts_manager import AccountsManager
 from backend.consensus.base import ConsensusAlgorithm
 from backend.database_handler.models import Base
+from backend.rollup.consensus_service import ConsensusService
 
 
 def get_db_name(database: str) -> str:
@@ -59,6 +60,7 @@ def create_app():
     accounts_manager = AccountsManager(sqlalchemy_db.session)
     validators_registry = ValidatorsRegistry(sqlalchemy_db.session)
     llm_provider_registry = LLMProviderRegistry(sqlalchemy_db.session)
+    consensus_service = ConsensusService()
 
     # Initialize validators from environment configuration in a thread
     initialize_validators_db_session = Session(engine, expire_on_commit=False)
@@ -84,6 +86,7 @@ def create_app():
         consensus,
         llm_provider_registry,
         sqlalchemy_db,
+        consensus_service,
     )
 
 
@@ -100,6 +103,7 @@ def create_app():
     consensus,
     llm_provider_registry,
     sqlalchemy_db,
+    consensus_service,
 ) = create_app()
 register_all_rpc_endpoints(
     jsonrpc,

backend/rollup/__init__.py

@@ -0,0 +1,87 @@
+import json
+import os
+from web3 import Web3
+from eth_account import Account
+from typing import Optional
+from pathlib import Path
+
+
+class ConsensusService:
+    def __init__(self):
+        """
+        Initialize the ConsensusService class
+        """
+        # Connect to Hardhat Network
+        port = os.environ.get("HARDHAT_PORT")
+        url = os.environ.get("HARDHAT_URL")
+        hardhat_url = f"{url}:{port}"
+        self.web3 = Web3(Web3.HTTPProvider(hardhat_url))
+
+        if not self.web3.is_connected():
+            raise ConnectionError(f"Failed to connect to Hardhat node at {hardhat_url}")
+
+        # Set up the default account (similar to ethers.getSigners()[0])
+        self.owner = self.web3.eth.accounts[0]
+        self.web3.eth.default_account = self.owner
+        self.private_key = os.environ.get("HARDHAT_PRIVATE_KEY")
+
+        # Load all required contracts
+        self.ghost_contract = self._load_contract("GhostContract")
+        self.ghost_factory_contract = self._load_contract("GhostFactory")
+        self.ghost_blueprint_contract = self._load_contract("GhostBlueprint")
+        self.consensus_manager_contract = self._load_contract("ConsensusManager")
+        self.mock_gen_staking_contract = self._load_contract("MockGenStaking")
+        self.queues_contract = self._load_contract("Queues")
+        self.transactions_contract = self._load_contract("Transactions")
+        self.consensus_main_contract = self._load_contract("ConsensusMain")
+
+    def _load_contract(self, contract_name: str) -> Optional[dict]:
+        """
+        Load contract deployment data from Hardhat deployments
+
+        Args:
+            contract_name (str): The name of the contract to load
+
+        Returns:
+            Optional[dict]: The contract deployment data or None if loading fails
+        """
+        try:
+            # Path to deployment file (using Docker volume path)
+            deployment_path = (
+                Path("/app/hardhat/deployments/localhost") / f"{contract_name}.json"
+            )
+
+            if not deployment_path.exists():
+                print(
+                    f"CONSENSUS_SERVICE: Deployment file not found at {deployment_path}"
+                )
+                return None
+
+            with open(deployment_path, "r") as f:
+                deployment_data = json.load(f)
+
+            # Create contract instance
+            contract = self.web3.eth.contract(
+                address=deployment_data["address"], abi=deployment_data["abi"]
+            )
+            print(
+                f"CONSENSUS_SERVICE: Loaded {contract_name} contract with address {contract.address}"
+            )
+
+            return contract
+
+        except FileNotFoundError:
+            print(
+                f"CONSENSUS_SERVICE: Warning: {contract_name} deployment file not found"
+            )
+            return None
+        except json.JSONDecodeError as e:
+            print(
+                f"CONSENSUS_SERVICE: Error decoding {contract_name} deployment file: {str(e)}"
+            )
+            return None
+        except Exception as e:
+            print(
+                f"CONSENSUS_SERVICE: Error loading {contract_name} contract: {str(e)}"
+            )
+            return None

backend/rollup/consensus_service.py

@@ -78,7 +78,8 @@ services:
     volumes:
       - ./.env:/app/.env
       - ./backend:/app/backend
-      - ./hardhat/artifacts:/app/hardhat/artifacts
+      - hardhat_artifacts:/app/hardhat/artifacts
+      - hardhat_deployments:/app/hardhat/deployments
     depends_on:
       hardhat:
         condition: service_healthy
@@ -197,13 +198,19 @@ services:
     ports:
       - "${HARDHAT_PORT:-8545}:8545"
     volumes:
-      - ./hardhat/contracts:/app/contracts
-      - ./hardhat/scripts:/app/scripts
-      - ./hardhat/test:/app/test
-      - ./hardhat/hardhat.config.js:/app/hardhat.config.js
+      - ./hardhat/contracts:/app/contracts:ro
+      - ./hardhat/scripts:/app/scripts:ro
+      - ./hardhat/test:/app/test:ro
+      - ./hardhat/hardhat.config.js:/app/hardhat.config.js:ro
+      - ./hardhat/deploy:/app/deploy:ro
+      - ./hardhat/scripts:/app/scripts:ro
       - hardhat_artifacts:/app/artifacts
+      - hardhat_cache:/app/cache
+      - hardhat_deployments:/app/deployments
+      - ignition_deployments:/app/ignition/deployments
     environment:
       - HARDHAT_NETWORK=hardhat
+    restart: always
     healthcheck:
       test: ["CMD", "curl", "-X", "POST", "-H", "Content-Type: application/json", "--fail", "http://localhost:8545", "-d", '{"jsonrpc":"2.0","method":"net_version","params":[],"id":1}']
       interval: 10s
@@ -213,4 +220,6 @@ services:
 
 volumes:
   hardhat_artifacts:
-
+  hardhat_cache:
+  hardhat_deployments:
+  ignition_deployments:

docker-compose.yml

@@ -6,7 +6,6 @@ WORKDIR /app
 
 # Install necessary packages and set up the environment
 RUN apk add --no-cache curl g++ make netcat-openbsd python3 && \
-    # Copy and install npm packages
     mkdir -p /app && \
     chown -R hardhat-user:hardhat-group /app
 
@@ -15,20 +14,36 @@ RUN npm install --ignore-scripts
 
 COPY ./hardhat .
 
-# Set up directories and permissions
-RUN mkdir -p /app/artifacts/build-info && \
+# Create directories and set permissions
+RUN mkdir -p /app/deployments/localhost && \
+    mkdir -p /app/deployments/hardhat && \
+    mkdir -p /app/artifacts/build-info && \
     mkdir -p /app/artifacts/contracts && \
+    mkdir -p /app/cache && \
+    mkdir -p /app/ignition/deployments && \
     chown -R hardhat-user:hardhat-group /app && \
-    chmod -R 755 /app/artifacts && \
-    # Create start script
-    echo -e '#!/bin/sh\necho "Compiling contracts..."\nnpx hardhat compile --force\necho "Starting Hardhat node..."\nexec ./node_modules/.bin/hardhat node --network hardhat' > /app/start.sh && \
-    chmod +x /app/start.sh
+    chmod -R 755 /app && \
+    chmod -R 775 /app/deployments && \
+    chmod -R 775 /app/artifacts && \
+    chmod -R 775 /app/cache && \
+    chmod -R 775 /app/ignition/deployments
 
 ENV PATH="/app/node_modules/.bin:${PATH}"
 
+# Add the start script
+RUN echo '#!/bin/sh' > /app/start.sh && \
+    echo 'echo "Checking and compiling contracts if needed..."' >> /app/start.sh && \
+    echo 'npx hardhat compile' >> /app/start.sh && \
+    echo 'echo "Starting Hardhat node..."' >> /app/start.sh && \
+    echo 'npx hardhat node --hostname 0.0.0.0 & sleep 5' >> /app/start.sh && \
+    echo 'echo "Deploying contracts with Ignition..."' >> /app/start.sh && \
+    echo 'npx hardhat run scripts/deploy.js --network localhost' >> /app/start.sh && \
+    echo 'npx hardhat test' >> /app/start.sh && \
+    echo 'wait' >> /app/start.sh && \
+    chmod +x /app/start.sh
+
 EXPOSE 8545
 
-# Switch to non-root user
 USER hardhat-user
 
-CMD ["/app/start.sh"]
+CMD ["/app/start.sh"]

docker/Dockerfile.hardhat

@@ -1,6 +1,6 @@
 require("@nomicfoundation/hardhat-toolbox");
+require("@nomicfoundation/hardhat-ignition-ethers");
 
-/** @type import('hardhat/config').HardhatUserConfig */
 module.exports = {
   solidity: {
     version: "0.8.24",
@@ -21,15 +21,16 @@ module.exports = {
       chainId: 31337,
       gasPrice: 0,
       initialBaseFeePerGas: 0,
-      accounts: {
-        count: 1
-      }
-    }
-  },
-  paths: {
-    sources: "./contracts",
-    tests: "./test",
-    cache: "./cache",
-    artifacts: "./artifacts"
+      blockGasLimit: 30000000,
+      ignition: {
+        blockPollingInterval: 1_000,
+        timeBeforeBumpingFees: 3 * 60 * 1_000,
+        maxFeeBumps: 4,
+        requiredConfirmations: 5,
+        disableFeeBumping: false,
+        deploymentDir: "deployments/localhost",
+      },
+    },
+
   }
 };

hardhat/hardhat.config.js

@@ -0,0 +1,65 @@
+const { buildModule } = require("@nomicfoundation/hardhat-ignition/modules");
+
+module.exports = buildModule("DeployFixture", (m) => {
+    // 1. Deploy base contracts (in order of TransactionFlow)
+    const ConsensusManager = m.contract("ConsensusManager");
+    const GhostFactory = m.contract("GhostFactory");
+    const GhostBlueprint = m.contract("GhostBlueprint");
+    const GhostContract = m.contract("GhostContract");
+    const MockGenStaking = m.contract("MockGenStaking", [m.getAccount(1)]);
+    const Queues = m.contract("Queues");
+    const Transactions = m.contract("Transactions");
+    const ConsensusMain = m.contract("ConsensusMain");
+
+    // 2. Initialize GhostFactory and configure GhostBlueprint
+    const initGhostFactory = m.call(GhostFactory, "initialize");
+    const setBlueprint = m.call(GhostFactory, "setGhostBlueprint", [GhostBlueprint], {
+        after: [initGhostFactory]
+    });
+
+    // 3. Initialize ConsensusMain and dependents
+    const initConsensusMain = m.call(ConsensusMain, "initialize", [ConsensusManager]);
+    const initTransactions = m.call(Transactions, "initialize", [ConsensusMain], {
+        after: [initConsensusMain]
+    });
+    const initQueues = m.call(Queues, "initialize", [ConsensusMain], {
+        after: [initConsensusMain]
+    });
+
+    // 4. Setup contract connections
+    m.call(ConsensusMain, "setGhostFactory", [GhostFactory], { after: [initConsensusMain] });
+    m.call(ConsensusMain, "setGenStaking", [MockGenStaking], { after: [initConsensusMain] });
+    m.call(ConsensusMain, "setGenQueue", [Queues], { after: [initConsensusMain, initQueues] });
+    m.call(ConsensusMain, "setGenTransactions", [Transactions], { after: [initConsensusMain, initTransactions] });
+
+    m.call(GhostFactory, "setGenConsensus", [ConsensusMain], { after: [initGhostFactory, initConsensusMain] });
+    m.call(GhostFactory, "setGhostManager", [ConsensusMain], { after: [initGhostFactory, initConsensusMain] });
+    m.call(Transactions, "setGenConsensus", [ConsensusMain], { after: [initTransactions] });
+    m.call(ConsensusMain, "setAcceptanceTimeout", [0], { after: [initConsensusMain] });
+
+    // 5. Setup validators
+    const validator1 = m.getAccount(1);
+    const validator2 = m.getAccount(2);
+    const validator3 = m.getAccount(3);
+    m.call(MockGenStaking, "addValidators", [[validator1, validator2, validator3]]);
+
+    // 6. Deploy beacon proxy
+    m.call(GhostFactory, "deployNewBeaconProxy", [], {
+        after: [
+            setBlueprint,
+            initGhostFactory,
+            initConsensusMain
+        ]
+    });
+
+    return {
+        GhostBlueprint,
+        GhostContract,
+        ConsensusManager,
+        GhostFactory,
+        MockGenStaking,
+        Queues,
+        Transactions,
+        ConsensusMain
+    };
+});

hardhat/ignition/modules/DeployFixture.js

@@ -20,5 +20,9 @@
   },
   "engines": {
     "node": ">=18.0.0"
+  },
+  "devDependencies": {
+    "@nomicfoundation/hardhat-ignition": "^0.15.8",
+    "@nomicfoundation/hardhat-ignition-ethers": "^0.15.8"
   }
 }

hardhat/package.json

@@ -0,0 +1,49 @@
+const hre = require("hardhat");
+const fs = require("fs-extra");
+const path = require("path");
+
+/**
+ * Save the deployment of a contract to a file.
+ * @param {string} name - The name of the contract.
+ * @param {Contract} contract - The contract instance.
+ * @param {string} folder - The folder to save the deployment file.
+ */
+async function saveDeployment(name, contract, folder = "deployments/localhost") {
+    const deploymentData = {
+        address: await contract.getAddress(),
+        abi: JSON.parse(contract.interface.formatJson())
+    };
+
+    await fs.ensureDir(folder);
+    const savePath = path.join(folder, `${name}.json`);
+    await fs.writeJson(savePath, deploymentData, { spaces: 2 });
+
+    const backupFolder = path.join("copy_deployments/localhost");
+    await fs.ensureDir(backupFolder);
+    await fs.writeJson(path.join(backupFolder, `${name}.json`), deploymentData, { spaces: 2 });
+}
+
+/**
+ * Main function to deploy contracts and save their deployments.
+ */
+async function main() {
+    console.log("\nDeploying contracts with Ignition...");
+    const DeployFixture = require("../ignition/modules/DeployFixture");
+    const result = await hre.ignition.deploy(DeployFixture);
+
+    try {
+        for (const [name, contract] of Object.entries(result)) {
+            console.log(`Saving ${name}...`);
+            await saveDeployment(name, contract);
+        }
+    } catch (error) {
+        console.error("Error saving deployment:", error);
+    }
+}
+
+main()
+    .then(() => process.exit(0))
+    .catch((error) => {
+        console.error(error);
+        process.exit(1);
+    });