TP2000-1407 Sqlite dump failure fix

[paulpepper-trade]

TP2000-1407 Sqlite dump failure fix

Why

Until recently, the SQLite export to S3 capability was taking over four hours to complete. With a continued increase in database size, export performance had sharply degraded and was taking in the region of three days to complete. Performance improvements were required.

What

This PR:

• Reduces the time to create a SQLite export and upload to S3 down to approximately six minutes.
• Refactors methods on storage classes that are used to save SQLite export files to allow multiple strategies to be applied for different export situations (e.g. exporting to local file system from a command line, or exporting to S3 via a Celery Beat initiated task).
• Add new storage classes SQLiteLocalStorage and SQLiteS3Storage to apply the different storage strategies.
• Adds a decorator that is used to log the start time, end time and total elapsed time taken for a function to execute. This was useful for comparing the performance of different storage strategies and discovering the issue that this PR fixes, but has been placed under the common application for wider use.
• Added command-line support for generating exports to the local file system.
• Refactored some (but not all) of the export code in order to simplify the process - more could be done, but the prospect of a more radical replacement means a fuller refactor (and simplification) is probably not worthwhile.
• Added docstrings to export classes and functions in place of the fuller refactor that could otherwise have been carried out.
• Added config support to enable / disable scheduled (via Celery Beat) SQLite exports.
• Added unit tests for the newly added storage classes and updated existing tests.
• Added unit test for dump_sqlite Django management command.
• Applied incidental minor fixes and corrections.

Checklist

• Requires migrations? No
• Requires dependency updates? No
• Ensure sufficient disk quota is available on instance from which the export is performed.

[eadpearce]

Is it possible to test this function?

[paulpepper-trade]

It's implicitly tested as part of test_s3_export_task_uploads() - called within the export_and_upload_sqlite() function, which I think gives it full and sufficient coverage.

[eadpearce]

It's showing up for me as red in codecov. Is that right? Can you check it on your end as well and confirm?

[eadpearce]

Is this command tested? This whole file is not showing up in the coverage report. Not sure if that's a bug with codecov or not

[paulpepper-trade]

It looks as though there's never been a test for this command. I can only guess that's because it's fairly simple and the function it executes is tested by exporter/tests/test_sqlite.py.

Test now added. Btw, export_and_upload_sqlite() is mocked in that test because it's tested elsewhere and is relatively expensive to execute - each execution performs a manage.py makemigrations followed by a manage.py migrate amongst other things!