Overview

This project is a scalable, production-grade data collection system that captures and processes SMS and call log events from a mobile device. It features:

• A Flutter App for UI and permission handling
• A custom Dart SDK for smart event processing and buffering
• A Flask Backend API to log received data

Designed for resilience, efficiency, and real-world usability, the system goes beyond the basic assignment with advanced batching, deduplication, offline support, and transactional detection.

---

Demo Video

Watch Demo Video

---

Architecture Diagram

Figure: End-to-end architecture with SMS/call log data flowing through the Flutter UI into the SDK’s processing pipeline, and finally to the backend.

---

Tech Stack

Layer	Technology
Mobile	Flutter, Dart
SDK	Pure Dart Module
Backend API	Python + Flask
Storage	Hive (Local DB)

---

✅ Key Features

✅ Flutter App

• Minimal, modern UI
• Real-time permission state
• Dashboard with live SDK event feed
• Manual data collection trigger

✅ Dart SDK

• Double buffer architecture for efficient batching
• Sliding window polling based on timestamps
• Transactional SMS detection via keyword classification
• Persistent deduplication using content hashing
• Offline support via Hive-backed storage
• Auto & manual flush on thresholds

✅ Flask Backend

• Single endpoint: POST /v1/events
• Logs incoming event batches
• Lightweight and easy to test

---

⚙️ Internal SDK Design

Sliding Window Polling

• Polls device SMS & call logs every 30 seconds
• Uses the last recorded timestamp to only fetch new data
• Improves performance and reduces duplicate reads

Deduplication

• Unique hash created per event
• Persistent set of previously seen hashes is stored in Hive
• Ensures no duplicate events even across app restarts

Buffering & Batching

• Two in-memory buffers (double buffering)
• Flush triggered by:
  • Reaching 50 events
  • Time-based (30 seconds)
  • Manual trigger from UI
• Buffered data sent as a single batch to the backend

Transactional SMS Detection

• Keywords: OTP, transaction, debited, credited, spent
• Detected transactional SMS are immediately flushed
• Non-transactional events go through the batching pipeline

Offline Support

• If offline, events are stored locally using Hive
• Retry mechanism automatically re-sends when connectivity is restored

Persistent State

• Deduplicator history
• Buffers
• Last seen timestamp
• All persisted using Hive to ensure crash safety and resumability

---

End-to-End Data Flow

1. User Launches App
   → Permissions for SMS and call logs are requested
   → Status is shown in the UI

2. Data Collection Begins
   → SDK starts polling every 30s
   → New SMS/call logs are read and processed

3. SDK Event Processing
   → Events passed through deduplication
   → Transactional SMS sent immediately
   → Others buffered (batched on 50 items or 30s timer)

4. Persistence & Recovery
   → State saved using Hive
   → App can resume after crash or restart

5. Backend API Logging
   → Event JSONs posted to Flask server
   → Stored/logged for auditing or further analysis

6. UI Monitoring
   → Live dashboard shows SDK activity (event IDs, timestamps, flush logs)

---

How to Run

Prerequisites

• Flutter SDK (3.x)
• Dart SDK (3.x)
• Python 3.x + pip

Backend (Flask API)

cd backend
python -m venv venv
source venv/bin/activate     # On Windows: venv\Scripts\activate
pip install flask
python app.py

The server listens at http://0.0.0.0:5000/v1/events

📱 Flutter App

cd Flutter/app
flutter pub get
flutter run

Update the API endpoint in dashboard_screen.dart if running backend on a different IP.

---

SDK Public Interface

// Initialize SDK with backend endpoint
SDK.initialize(String apiUrl);

// Pass an SMS or Call event
SDK.trackSMS(SMSEvent event);
SDK.trackCall(CallEvent event);

// Optional manual flush
SDK.flush();

All classes are modular, documented, and reusable.

---

🚀 Additional Enhancements Beyond Assignment

Feature	Description
Sliding Window Polling	Avoids duplicate reads using timestamp-based SMS/call filtering
Double Buffering	Non-blocking flush ensures continuous event intake
Deduplication (Persistent)	Hashing + Hive ensures no duplicates even across app restarts
Offline Storage	Events stored in Hive when offline; retried on reconnection
Real-Time UI	Every SDK event shows up on a live dashboard in the Flutter UI
Configurable Parameters	Batch size and polling interval can be adjusted (future: remote config)
Crash Resilience	SDK resumes polling from last known state after reboot

---

Folder Structure

├── backend/             # Flask API server
│   └── app.py
│
├── Flutter/
│   └── app/             # Flutter UI and main logic
│
├── sdk/                 # Reusable Dart SDK (pub-ready)
│   ├── lib/
│   └── test/
│
└── assets/
    └── architecture diagram.png

---

📫 Contact

Ashutosh Mishra
📧 am3718440@gmail.com
🔗 LinkedIn

---

This project demonstrates advanced mobile data collection techniques with efficient batching, real-time processing, and professional UI design.