Proposal: Support for Alternative Time-Series Database Backends

[fbeneventi]

Time-Series Database Integration Guidelines for Examon

Overview

These guidelines provide a framework for adding support for new time-series databases to Examon. The existing KairosDB connector serves as a reference implementation.

Goal

Following this strategy we can reuse all existing plugins with zero changes in the publish format

Reference Implementation

• KairosDB Connector

File Structure

For a new database connector, follow this structure (based on the KairosDB implementation):

examon/subscribers/mqtt2<dbname>_queue/
├── mqtt2<dbname>.conf       # Configuration file
├── mqtt2<dbname>.py         # Connector implementation
└── requirements.txt         # Dependencies

Implementation Guidelines

1. Configuration File

• Create a configuration file similar to mqtt2kairosdb.conf
• Include sections for:
  • MQTT broker connection parameters
  • Database connection parameters
  • Data handling settings (queue size, batch processing)
  • Logging configuration

2. Connector Implementation

• Implement a Python module that:
  • Transforms data to the database's required format
  • Handles connection management and retries
  • Implements buffering/queuing for reliability
  • Provides appropriate error handling

3. Core Requirements

• Handle connection disruptions gracefully
• Support batch operations for performance
• Include proper logging and monitoring
• Implement memory-efficient processing
• Provide clear error messages and status reporting

4. Documentation

• Document configuration options
• Explain any database-specific limitations or features
• Provide performance considerations and tuning options
• Include example queries for the new database

5. Testing

• Develop unit tests for the connector
• Create integration tests with the full Examon stack
• Include performance benchmarks alongside KairosDB comparisons

6. Docker Integration

• Update Docker Compose files to include the new database
• Document volume configuration for data persistence
• Provide environment variable templates

Pull Request Process

1. Implement the connector following these guidelines
2. Add documentation for setup and configuration
3. Include tests validating functionality
4. Submit a PR with a clear description of the new database's advantages

Collaboration Expectations

• Respond to feedback and review comments
• Be prepared to maintain the connector or find maintainers
• Document any limitations or known issues

[fbeneventi]

Of course it is possible to use other languages besides Python.

A list of languages compatible with the MQTT protocol can be found in the Client libraries section here https://mqtt.org/software/