Sentinel verify

Gemfile.lock

@@ -3,6 +3,7 @@ PATH
   specs:
     litestream (0.9.0)
       logfmt (>= 0.0.10)
+      sqlite3
 
 GEM
   remote: https://rubygems.org/

README.md

@@ -168,37 +168,19 @@ You can forward arguments in whatever order you like, you simply need to ensure
 
 ### Verification
 
-You can verify the integrity of your backed-up databases using the gem's provided `litestream:verify` rake task. This rake task requires that you specify which specific database you want to verify. As with the `litestream:restore` tasks, you pass arguments to the rake task via argument forwarding. For example, to verify the production database, you would run:
+You can verify the integrity of your backed-up databases using the gem's provided `Litestream.verify!` method. The method takes the path to a database file that you have configured Litestream to backup; that is, it takes one of the `path` values under the `dbs` key in your `litestream.yml` configuration file. For example, to verify the production database, you would run:
 
-```shell
-bin/rails litestream:verify -- --database=storage/production.sqlite3
-# or
-bundle exec rake litestream:verify -- --database=storage/production.sqlite3
+```ruby
+Litestream.verify! "storage/production.sqlite3"
 ```
 
-The `litestream:verify` rake task takes the same options as the `litestream:restore` rake task. After restoring the backup, the rake task will verify the integrity of the restored database by ensuring that the restored database file
+In order to verify that the backup for that database is both restorable and fresh, the method will add a new row to that database under the `_litestream_verification` table, which it will create if needed. It will then wait 10 seconds to give the Litestream utility time to replicate that change to whatever storage providers you have configured. After that, it will download the latest backup from that storage provider and ensure that this verification row is present in the backup. If the verification row is _not_ present, the method will raise a `Litestream::VerificationFailure` exception. This check ensures that the restored database file
 
 1. exists,
 2. can be opened by SQLite, and
-3. sufficiently matches the original database file.
-
-Since point 3 is subjective, the rake task will output a message providing both the file size and number of tables of both the "original" and "restored" databases. You must manually verify that the restored database is within an acceptable range of the original database.
-
-The rake task will output a message similar to the following:
-
-```
-size
-  original          21688320
-  restored          21688320
-  delta             0
-
-tables
-  original          9
-  restored          9
-  delta             0
-```
+3. has up-to-date data.
 
-After restoring the backup, the `litestream:verify` rake task will delete the restored database file. If you need the restored database file, use the `litestream:restore` rake task instead.
+After restoring the backup, the `Litestream.verify!` method will delete the restored database file. If you need the restored database file, use the `litestream:restore` rake task or `Litestream::Commands.restore` method instead.
 
 ### Introspection
 

lib/litestream.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "sqlite3"
+
 module Litestream
   class << self
     attr_accessor :configuration
@@ -16,6 +18,29 @@ class Configuration
     def initialize
     end
   end
+
+  VerificationFailure = Class.new(StandardError)
+
+  def self.verify!(database_path)
+    database = SQLite3::Database.new(database_path)
+    database.execute("CREATE TABLE IF NOT EXISTS _litestream_verification (id INTEGER PRIMARY KEY, uuid BLOB)")
+    sentinel = SecureRandom.uuid
+    database.execute("INSERT INTO _litestream_verification (uuid) VALUES (?)", [sentinel])
+    # give the Litestream replication process time to replicate the sentinel value
+    sleep 10
+
+    backup_path = "tmp/#{Time.now.utc.strftime("%Y%m%d%H%M%S")}_#{sentinel}.sqlite3"
+    Litestream::Commands.restore(database_path, **{"-o" => backup_path})
+
+    backup = SQLite3::Database.new(backup_path)
+    result = backup.execute("SELECT 1 FROM _litestream_verification WHERE uuid = ? LIMIT 1", sentinel) # => [[1]] || []
+
+    raise VerificationFailure, "Verification failed, sentinel not found" if result.empty?
+  ensure
+    database.execute("DELETE FROM _litestream_verification WHERE uuid = ?", sentinel)
+    database.close
+    Dir.glob(backup_path + "*").each { |file| File.delete(file) }
+  end
 end
 
 require_relative "litestream/version"

lib/litestream/commands.rb

@@ -88,46 +88,6 @@ def restore(database, async: false, **argv)
         execute("restore", argv, database, async: async, tabled_output: false)
       end
 
-      def verify(database, async: false, **argv)
-        raise DatabaseRequiredException, "database argument is required for verify command, e.g. litestream:verify -- --database=path/to/database.sqlite" if database.nil? || !File.exist?(database)
-        argv.stringify_keys!
-
-        dir, file = File.split(database)
-        ext = File.extname(file)
-        base = File.basename(file, ext)
-        now = Time.now.utc.strftime("%Y%m%d%H%M%S")
-        backup = File.join(dir, "#{base}-#{now}#{ext}")
-        args = {
-          "-o" => backup
-        }.merge(argv)
-        restore(database, async: false, **args)
-
-        restored_schema = `sqlite3 #{backup} "select name, type from sqlite_schema;"`.chomp.split("\n")
-        restored_data = restored_schema.map { _1.split("|") }.group_by(&:last)
-        restored_rows_count = restored_data["table"]&.sum { |tbl, _| `sqlite3 #{backup} "select count(*) from #{tbl};"`.chomp.to_i }
-
-        original_schema = `sqlite3 #{database} "select name, type from sqlite_schema;"`.chomp.split("\n")
-        original_data = original_schema.map { _1.split("|") }.group_by(&:last)
-        original_rows_count = original_data["table"]&.sum { |tbl, _| `sqlite3 #{database} "select count(*) from #{tbl};"`.chomp.to_i }
-
-        Dir.glob(backup + "*").each { |file| File.delete(file) }
-
-        {
-          "original" => {
-            "path" => database,
-            "tables" => original_data["table"]&.size,
-            "indexes" => original_data["index"]&.size,
-            "rows" => original_rows_count
-          },
-          "restored" => {
-            "path" => backup,
-            "tables" => restored_data["table"]&.size,
-            "indexes" => restored_data["index"]&.size,
-            "rows" => restored_rows_count
-          }
-        }
-      end
-
       def databases(async: false, **argv)
         execute("databases", argv, async: async, tabled_output: true)
       end

lib/tasks/litestream_tasks.rake

@@ -80,66 +80,4 @@ namespace :litestream do
 
     Litestream::Commands.snapshots(database, async: true, **options)
   end
-
-  desc "verify backup of SQLite database from a Litestream replica, e.g. rake litestream:verify -- -database=storage/production.sqlite3"
-  task verify: :environment do
-    options = {}
-    if (separator_index = ARGV.index("--"))
-      ARGV.slice(separator_index + 1, ARGV.length)
-        .map { |pair| pair.split("=") }
-        .each { |opt| options[opt[0]] = opt[1] || nil }
-    end
-    database = options.delete("--database") || options.delete("-database")
-    options.symbolize_keys!
-
-    result = Litestream::Commands.verify(database, async: true, **options)
-    original_tables = result["original"]["tables"]
-    restored_tables = result["restored"]["tables"]
-    original_indexes = result["original"]["indexes"]
-    restored_indexes = result["restored"]["indexes"]
-    original_rows = result["original"]["rows"]
-    restored_rows = result["restored"]["rows"]
-
-    same_number_of_tables = original_tables == restored_tables
-    same_number_of_indexes = original_indexes == restored_indexes
-    same_number_of_rows = original_rows == restored_rows
-
-    if same_number_of_tables && same_number_of_indexes && same_number_of_rows
-      puts "Backup for `#{database}` verified as consistent!\n" + [
-        "  tables   #{original_tables}",
-        "  indexes  #{original_indexes}",
-        "  rows     #{original_rows}"
-      ].compact.join("\n")
-    else
-      abort "Verification failed for #{database}:\n" + [
-        (unless same_number_of_tables
-           if original_tables > restored_tables
-             diff = original_tables - restored_tables
-             "  Backup is missing #{diff} table#{"s" if diff > 1}"
-           else
-             diff = restored_tables - original_tables
-             "  Backup has extra #{diff} table#{"s" if diff > 1}"
-           end
-         end),
-        (unless same_number_of_indexes
-           if original_indexes > restored_indexes
-             diff = original_indexes - restored_indexes
-             "  Backup is missing #{diff} index#{"es" if diff > 1}"
-           else
-             diff = restored_indexes - original_indexes
-             "  Backup has extra #{diff} index#{"es" if diff > 1}"
-           end
-         end),
-        (unless same_number_of_rows
-           if original_rows > restored_rows
-             diff = original_rows - restored_rows
-             "  Backup is missing #{diff} row#{"s" if diff > 1}"
-           else
-             diff = restored_rows - original_rows
-             "  Backup has extra #{diff} row#{"s" if diff > 1}"
-           end
-         end)
-      ].compact.join("\n")
-    end
-  end
 end

litestream.gemspec

@@ -26,6 +26,7 @@ Gem::Specification.new do |spec|
 
   # Uncomment to register a new dependency of your gem
   spec.add_dependency "logfmt", ">= 0.0.10"
+  spec.add_dependency "sqlite3"
   spec.add_development_dependency "rubyzip"
   spec.add_development_dependency "rails"
   spec.add_development_dependency "sqlite3"