feat: consensus deduct processing time appeal (#867)

* feat: add rollup transaction db table

* fix: drop audit table, change id and nonce of rollup, only make one finalized rollup transaction

* create rollup transaction for every validator

* add function to mock in consensus test

* feat: add appeal window, accepted queue, basic test

* fix: old tests were stuck on accepted state, add usage of thread

* refactor consensus into states

* feat: added appeal flow when transaction is accepted including the loop when appeal failed and succeeded

* fix: adding tests for appeals and fixing minor bugs

* refactor: merge main and PR #573 into this branch

* feat: add appeal_failed in db, select new validators when appealed based on formula

* docs: cleanup consensus mechanism base file

* test: checking the number of validators for different appeals

* feat: adding appeal window to undetermined state

* feat: change timestamp_accepted and add appeal_undetermined in database

* feat: undetermined to pending, activate frontend button in undetermined state and add button also to modal

* feat: leader only has no appeal button and no appeal window

* feat: implement the state transitions to process the appeal
- Add N+2 validators, remove leader
- Use latest data of transaction when in pending state and not the old one from the crawler
- Write consensus data before setting status to have it updated in frontend when going to transaction info modal
- Do not deploy contract when transaction was in the undetermined state

* test: add test for leader appeals

* refactor: merge 593-appeals-add-validators-when-appealed into 604-appeals-implement-sequential-appeals-fail

* refactor: merging changed file permissions

* fix: appealing a write method gave a KeyError because of wrong conversation of transaction

* docs: update transaction_processor argument description

* fix: all appeals disagreed because of pending_transactions type

* refactor: undo change because of merge

* fix: do not reset finality window when leader appeal failed, leader_only check in appeal_window

* fix: set value in database migration file

* fix: add leader_only check for modal appeal button, comment out modal appeal button

* fix: do not show appeal button when finality window is finished, checking on finalized state gave a small delay in showing when appeal failed and finality window was exceeded

* fix: we do not emit messages when the transaction goes from undetermined to finalized

* fix: complete merge

* fix: redirect leader appeal is processed by appeal queue, not pending queue

* fix: test increase wait time to get to finalized

* fix: use appeal property of the transaction in frontend

* fix: comment out appeal buttons

* refactor: clean up prints

* fix: check status typo

* fix: update consensus test

* fix: update test_get_highest_timestamp

* feat: deduct processing time appeal

* fix: solve consensus test timings

* fix: solve consensus test timings

* fix: changes check timings consensus test

* fix: add activated to status match check in consensus test

* fix: split set_transaction_appeal_processing_time

* fix: add log_terminal argument in messagehandler mock; remove messagehandler argument in set_transaction_appeal

* fix: database migration file revision duplicates

Author: kstroobants

backend/consensus/base.py

@@ -870,7 +870,11 @@ def can_finalize_transaction(
             bool: True if the transaction can be finalized, False otherwise.
         """
         if (transaction.leader_only) or (
-            (int(time.time()) - transaction.timestamp_awaiting_finalization)
+            (
+                int(time.time())
+                - transaction.timestamp_awaiting_finalization
+                - transaction.appeal_processing_time
+            )
             > self.finality_window_time
         ):
             if index == 0:
@@ -1092,7 +1096,7 @@ async def process_validator_appeal(
                 )
             )
             context.transactions_processor.set_transaction_appeal(
-                context.transaction.hash, False, self.msg_handler
+                context.transaction.hash, False
             )
             context.transaction.appealed = False
             self.msg_handler.send_message(
@@ -1107,6 +1111,9 @@ async def process_validator_appeal(
                 ),
                 log_to_terminal=False,
             )
+            context.transactions_processor.set_transaction_appeal_processing_time(
+                context.transaction.hash
+            )
         else:
             # Set up the context for the committing state
             context.num_validators = len(context.remaining_validators)
@@ -1697,6 +1704,15 @@ async def handle(self, context):
                     None,
                     context.validation_results,
                 )
+
+                # Reset the appeal processing time
+                context.transactions_processor.reset_transaction_appeal_processing_time(
+                    context.transaction.hash
+                )
+                context.transactions_processor.set_transaction_timestamp_appeal(
+                    context.transaction.hash, None
+                )
+
                 return "validator_appeal_success"
 
         else:
@@ -1851,8 +1867,20 @@ async def handle(self, context):
                 context.transaction.hash, False
             )
             context.transaction.appeal_undetermined = False
+            context.transactions_processor.reset_transaction_appeal_processing_time(
+                context.transaction.hash
+            )
+            context.transactions_processor.set_transaction_timestamp_appeal(
+                context.transaction.hash, None
+            )
+            context.transaction.timestamp_appeal = None
             return "leader_appeal_success"
         else:
+            # Increment the appeal processing time when the transaction was appealed
+            if context.transaction.timestamp_appeal is not None:
+                context.transactions_processor.set_transaction_appeal_processing_time(
+                    context.transaction.hash
+                )
             return None
 
 
@@ -1927,6 +1955,12 @@ async def handle(self, context):
             context.transaction.hash
         )
 
+        # Increment the appeal processing time when the transaction was appealed
+        if context.transaction.timestamp_appeal is not None:
+            context.transactions_processor.set_transaction_appeal_processing_time(
+                context.transaction.hash
+            )
+
         return None
 
 

backend/database_handler/migration/versions/d932a5fef8b1_appeal_processing_time.py

@@ -0,0 +1,37 @@
+"""appeal_processing_time
+
+Revision ID: d932a5fef8b1
+Revises: 2de9c3151194
+Create Date: 2025-01-31 15:39:44.618075
+
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = "d932a5fef8b1"
+down_revision: Union[str, None] = "2de9c3151194"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column(
+        "transactions", sa.Column("timestamp_appeal", sa.BigInteger(), nullable=True)
+    )
+    op.add_column(
+        "transactions", sa.Column("appeal_processing_time", sa.Integer(), nullable=True)
+    )
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column("transactions", "appeal_processing_time")
+    op.drop_column("transactions", "timestamp_appeal")
+    # ### end Alembic commands ###

backend/database_handler/models.py

@@ -98,6 +98,8 @@ class Transactions(Base):
     ghost_contract_address: Mapped[Optional[str]] = mapped_column(String(255))
     appeal_failed: Mapped[Optional[int]] = mapped_column(Integer)
     consensus_history: Mapped[Optional[dict]] = mapped_column(JSONB)
+    timestamp_appeal: Mapped[Optional[int]] = mapped_column(BigInteger)
+    appeal_processing_time: Mapped[Optional[int]] = mapped_column(Integer)
 
     # Relationship for triggered transactions
     triggered_by_hash: Mapped[Optional[str]] = mapped_column(

backend/database_handler/transactions_processor.py

@@ -68,6 +68,8 @@ def _parse_transaction_data(transaction_data: Transactions) -> dict:
             "appeal_failed": transaction_data.appeal_failed,
             "appeal_undetermined": transaction_data.appeal_undetermined,
             "consensus_history": transaction_data.consensus_history,
+            "timestamp_appeal": transaction_data.timestamp_appeal,
+            "appeal_processing_time": transaction_data.appeal_processing_time,
         }
 
     @staticmethod
@@ -178,6 +180,8 @@ def insert_transaction(
             appeal_failed=0,
             appeal_undetermined=False,
             consensus_history={},
+            timestamp_appeal=None,
+            appeal_processing_time=0,
         )
 
         self.session.add(new_transaction)
@@ -312,12 +316,15 @@ def set_transaction_appeal(self, transaction_hash: str, appeal: bool):
         )
         # You can only appeal the transaction if it is in accepted or undetermined state
         # Setting it to false is always allowed
-        if (
-            (not appeal)
-            or (transaction.status == TransactionStatus.ACCEPTED)
-            or (transaction.status == TransactionStatus.UNDETERMINED)
+        if not appeal:
+            transaction.appealed = appeal
+            self.session.commit()
+        elif transaction.status in (
+            TransactionStatus.ACCEPTED,
+            TransactionStatus.UNDETERMINED,
         ):
             transaction.appealed = appeal
+            self.set_transaction_timestamp_appeal(transaction, int(time.time()))
             self.session.commit()
 
     def set_transaction_timestamp_awaiting_finalization(
@@ -450,3 +457,29 @@ def update_consensus_history(
 
         flag_modified(transaction, "consensus_history")
         self.session.commit()
+
+    def set_transaction_timestamp_appeal(
+        self, transaction: Transactions | str, timestamp_appeal: int
+    ):
+        if isinstance(transaction, str):  # hash
+            transaction = (
+                self.session.query(Transactions).filter_by(hash=transaction).one()
+            )
+        transaction.timestamp_appeal = timestamp_appeal
+
+    def set_transaction_appeal_processing_time(self, transaction_hash: str):
+        transaction = (
+            self.session.query(Transactions).filter_by(hash=transaction_hash).one()
+        )
+        transaction.appeal_processing_time += (
+            round(time.time()) - transaction.timestamp_appeal
+        )
+        flag_modified(transaction, "appeal_processing_time")
+        self.session.commit()
+
+    def reset_transaction_appeal_processing_time(self, transaction_hash: str):
+        transaction = (
+            self.session.query(Transactions).filter_by(hash=transaction_hash).one()
+        )
+        transaction.appeal_processing_time = 0
+        self.session.commit()

backend/domain/types.py

@@ -87,6 +87,8 @@ class Transaction:
     appeal_failed: int = 0
     appeal_undetermined: bool = False
     consensus_history: dict = field(default_factory=dict)
+    timestamp_appeal: int | None = None
+    appeal_processing_time: int = 0
 
     def to_dict(self):
         return {
@@ -114,6 +116,8 @@ def to_dict(self):
             "appeal_failed": self.appeal_failed,
             "appeal_undetermined": self.appeal_undetermined,
             "consensus_history": self.consensus_history,
+            "timestamp_appeal": self.timestamp_appeal,
+            "appeal_processing_time": self.appeal_processing_time,
         }
 
     @classmethod
@@ -143,4 +147,6 @@ def from_dict(cls, input: dict) -> "Transaction":
             appeal_failed=input.get("appeal_failed", 0),
             appeal_undetermined=input.get("appeal_undetermined", False),
             consensus_history=input.get("consensus_history"),
+            timestamp_appeal=input.get("timestamp_appeal"),
+            appeal_processing_time=input.get("appeal_processing_time", 0),
         )

tests/unit/consensus/test_base.py

@@ -244,6 +244,13 @@ def get_vote():
             ]
             == timestamp_awaiting_finalization_1
         )
+
+        assert (
+            transactions_processor.get_transaction_by_hash(transaction.hash)[
+                "appeal_processing_time"
+            ]
+            > 0
+        )
     finally:
         cleanup_threads(event, threads)
 
@@ -309,6 +316,12 @@ def get_vote():
             ]
             == timestamp_awaiting_finalization_1
         )
+        assert (
+            transactions_processor.get_transaction_by_hash(transaction.hash)[
+                "appeal_processing_time"
+            ]
+            > 0
+        )
     finally:
         cleanup_threads(event, threads)
 
@@ -434,6 +447,13 @@ def get_vote():
 
         assert new_leader_address != old_leader_address
         assert new_leader_address in validator_set_addresses
+
+        assert (
+            transactions_processor.get_transaction_by_hash(transaction.hash)[
+                "appeal_processing_time"
+            ]
+            == 0
+        )
     finally:
         cleanup_threads(event, threads)
 
@@ -807,6 +827,12 @@ def get_vote():
         )
         leader_address = get_leader_address(transaction, transactions_processor)
 
+        appeal_processing_time_temp = transactions_processor.get_transaction_by_hash(
+            transaction.hash
+        )["appeal_processing_time"]
+        assert appeal_processing_time_temp == 0
+        timestamp_appeal_temp = 0
+
         for appeal_failed in range(3):
             assert (
                 transactions_processor.get_transaction_by_hash(transaction.hash)[
@@ -821,6 +847,18 @@ def get_vote():
                 transactions_processor, transaction, [TransactionStatus.ACCEPTED.value]
             )
 
+            appeal_processing_time_new = transactions_processor.get_transaction_by_hash(
+                transaction.hash
+            )["appeal_processing_time"]
+            assert appeal_processing_time_new > appeal_processing_time_temp
+            appeal_processing_time_temp = appeal_processing_time_new
+
+            timestamp_appeal_new = transactions_processor.get_transaction_by_hash(
+                transaction.hash
+            )["timestamp_appeal"]
+            assert timestamp_appeal_new > timestamp_appeal_temp
+            timestamp_appeal_temp = timestamp_appeal_new
+
             assert (
                 transactions_processor.get_transaction_by_hash(transaction.hash)[
                     "timestamp_awaiting_finalization"
@@ -1229,6 +1267,19 @@ def get_vote():
         check_validator_count(transaction, transactions_processor, nb_validators)
         assert len(created_nodes) == nb_created_nodes
 
+        assert (
+            transactions_processor.get_transaction_by_hash(transaction.hash)[
+                "appeal_processing_time"
+            ]
+            > 0
+        )
+        assert (
+            transactions_processor.get_transaction_by_hash(transaction.hash)[
+                "timestamp_appeal"
+            ]
+            is not None
+        )
+
         appeal(transaction, transactions_processor)
         assert_transaction_status_match(
             transactions_processor, transaction, [TransactionStatus.ACCEPTED.value]
@@ -1252,6 +1303,19 @@ def get_vote():
         check_validator_count(transaction, transactions_processor, nb_validators)
         assert len(created_nodes) == nb_created_nodes
 
+        assert (
+            transactions_processor.get_transaction_by_hash(transaction.hash)[
+                "appeal_processing_time"
+            ]
+            == 0
+        )
+        assert (
+            transactions_processor.get_transaction_by_hash(transaction.hash)[
+                "timestamp_appeal"
+            ]
+            is None
+        )
+
         appeal(transaction, transactions_processor)
         current_status = assert_transaction_status_match(
             transactions_processor,
@@ -1323,5 +1387,18 @@ def get_vote():
         nb_created_nodes += nb_validators**2
         check_validator_count(transaction, transactions_processor, nb_validators)
         assert len(created_nodes) == nb_created_nodes
+
+        assert (
+            transactions_processor.get_transaction_by_hash(transaction.hash)[
+                "appeal_processing_time"
+            ]
+            > 0
+        )
+        assert (
+            transactions_processor.get_transaction_by_hash(transaction.hash)[
+                "timestamp_appeal"
+            ]
+            is not None
+        )
     finally:
         cleanup_threads(event, threads)