Skip to content

Latest commit

 

History

History
144 lines (115 loc) · 5.56 KB

README.md

File metadata and controls

144 lines (115 loc) · 5.56 KB

Document Processing System

Overview

This project automates the handling, extraction, and processing of documents using a combination of services such as OpenAI, Dropbox, Nextcloud, and Paperless NGX. The system extracts metadata, processes document contents, and stores the results efficiently. It is designed for flexibility and configurability through environment variables, making it easily customizable for different workflows.

Features

  • Document Upload & Storage: Upload and manage documents via Dropbox and Nextcloud.
  • OCR Processing: Extract text from scanned documents.
  • Metadata Extraction: Automatically extract key information using OpenAI's API.
  • Document Management: Store processed documents and metadata in Paperless NGX for easy retrieval.
  • IMAP Integration: Fetch documents from multiple IMAP email accounts for processing.

Environment Variables

The project is configured via the .env file, where credentials and settings for different services are defined. Below is a breakdown of key configuration variables:

General Configuration

Variable Description How to Obtain
DATABASE_URL Path to the SQLite database. Example: sqlite:///./app/database.db
REDIS_URL URL for Redis connection. Example: redis://redis:6379/0
WORKDIR Working directory for the application. Example: /workdir
NEXTCLOUD_UPLOAD_URL Nextcloud WebDAV upload URL. Example: https://nextcloud.example.com/remote.php/dav/files/<USERNAME>
NEXTCLOUD_FOLDER Folder in Nextcloud for file uploads. Example: /Documents/Uploads
PAPERLESS_NGX_URL Paperless NGX API endpoint. Example: https://paperless.example.com/api/documents/post_document/
PAPERLESS_HOST Root URL for Paperless NGX. Example: https://paperless.example.com

Tokens/API Credentials

Variable Description How to Obtain
OPENAI_API_KEY API key for OpenAI services. Get from OpenAI platform.
PAPERLESS_NGX_API_TOKEN API token for Paperless NGX. Obtain from your Paperless NGX instance.
DROPBOX_APP_KEY Dropbox API key. Generate from the Dropbox Developer Console.
DROPBOX_APP_SECRET Dropbox API secret. Available in the Dropbox Developer Console.
DROPBOX_REFRESH_TOKEN Dropbox OAuth refresh token. Obtain by following Dropbox's OAuth flow.

User Credentials

Variable Description
ADMIN_USERNAME Admin username for system access.
ADMIN_PASSWORD Admin password for system access.
NEXTCLOUD_USERNAME Username for Nextcloud authentication.
NEXTCLOUD_PASSWORD Password for Nextcloud authentication.

IMAP Configuration

Variable Description
IMAP1_USERNAME IMAP username for the first email account.
IMAP1_PASSWORD IMAP password for the first email account.
IMAP1_HOST Hostname of the first IMAP server.
IMAP1_PORT IMAP server port (typically 993).
IMAP1_SSL Enable SSL (true or false).
IMAP1_POLL_INTERVAL_MINUTES Polling interval for IMAP server.
IMAP1_DELETE_AFTER_PROCESS Delete emails after processing (true or false).

Additional Services

Variable Description
GOTENBERG_URL URL for Gotenberg PDF processing.

Running as a Docker Container

This project includes a docker-compose.yml file that allows for easy deployment using Docker. The following services are defined:

  • API Service: Runs the document processing API using uvicorn.
  • Worker Service: Runs the Celery worker for handling document processing tasks.
  • Redis: Used as a message broker for Celery.
  • Gotenberg: Provides PDF processing capabilities.

Running the Application with Docker Compose

  1. Ensure Docker and Docker Compose are installed.
  2. Clone the repository and navigate to the directory: 
    git clone <repository_url>
cd <repository_name>
  3. Create and configure the .env file.
  4. Start the services: 
    docker-compose up -d
  5. The API will be available at http://localhost:8000.

Services in docker-compose.yml

services:
  api:
    image: christianlouis/document-processor:latest
    container_name: document_api
    working_dir: /workdir
    command: ["sh", "-c", "cd /app && uvicorn app.main:app --host 0.0.0.0 --port 8000"]
    environment:
      - PYTHONPATH=/app
    env_file:
      - .env
    ports:
      - "8000:8000"
    depends_on:
      - redis
      - worker
    volumes:
      - /var/docparse/workdir:/workdir

  worker:
    image: christianlouis/document-processor:latest
    container_name: document_worker
    working_dir: /workdir
    command: ["celery", "-A", "app.celery_worker", "worker", "-B", "--loglevel=info", "-Q", "document_processor,default,celery"]
    env_file:
      - .env
    environment:
      - PYTHONPATH=/app
    depends_on:
      - redis
      - gotenberg
    volumes:
      - /var/docparse/workdir:/workdir

  gotenberg:
    image: gotenberg/gotenberg:latest
    container_name: gotenberg

  redis:
    image: redis:alpine
    container_name: document_redis
    restart: always

To-Do List

  • Refactor AWS-related code to Azure.
  • Remove unnecessary environment variables.
  • Make upload targets configurable.
  • Remove S3 upload functionality.