michal/tit
Browse tree · Show commit · Download archive
Diff
7d8762c45fe0 → 393c9decda7b
.github/workflows/ci.yml
Mode 100644 → 100644; object bbd1c8fe6185 → 76a2f673d056
@@ -30,4 +30,5 @@
persist-credentials: false
- run: cargo --version
- run: cargo test --locked --all-targets --all-features
+ - run: cargo test --locked --release --test sqlite_workload -- --ignored --nocapture
- run: cargo build --locked --release
CONTRIBUTING.md
Mode 100644 → 100644; object 293305c34a2d → d10b31892330
@@ -104,6 +104,8 @@ - advertised hostname, hostname, onion hostname, and proxy command. - database constraint, foreign key, online backup, prepared statement, and schema migration. +- durability gate, local filesystem, migration backup, percentile, and + workload gate. Use the technical verbs in this list:
PLAN.md
Mode 100644 → 100644; object bd9e4a578fed → 2aa9d362c5be
@@ -27,6 +27,7 @@ System installations keep the full instance in `/srv/tit` by default. This includes `config.toml`, the metadata database, bare repositories, SSH host keys, instance secrets, and recoverable operational state. +The metadata database name is `tit.sqlite3`. ## Product boundaries @@ -535,6 +536,10 @@ Gate: restored and migrated fixtures pass SQLite integrity checks and `doctor`. No handler needs direct access to a `rusqlite` type or SQL statement. + +The M1A evidence and limits are in +`docs/adr/0001-sqlite-storage.md`. The decision stays provisional until the +Linux and macOS CI workload gates pass. ##### M1B — SSH identity
README.md
Mode 100644 → 100644; object 9403fc1c97bc → fec2dfef7042
@@ -25,6 +25,31 @@ This command formats, lints, tests, audits, and builds the release executable. +## Milestone 1A gate + +Run the SQLite durability gate on a local filesystem: + +```text +./scripts/check-m1a +``` + +This command also creates and measures a release database with 10,000 issue +records and 1,000,000 event records. Read the SQLite +[architectural decision record](docs/adr/0001-sqlite-storage.md) for the limits +and current platform evidence. + +## Database check + +An initialized instance keeps its metadata in `tit.sqlite3`. Check an existing +database with this command: + +```text +tit --config /absolute/path/to/tit/config.toml doctor +``` + +The command does not create or migrate a database. Successful validation writes +no output and returns exit code 0. + ## Configuration validation Copy `config.example.toml` into an empty instance directory. Change
docs/adr/0001-sqlite-storage.md
Mode → 100644; object → 1df67c85bf2c
@@ -1,0 +1,66 @@ +# Architectural decision record 0001: SQLite storage + +Status: Provisional + +Date: 2026-07-22 + +## Context + +`tit` needs one inspectable metadata database in the instance directory. The +database must have transactions, constraints, indexes, online backup, and a +clear recovery procedure. The executable must not need an installed database +server or an installed SQLite shared library. + +## Decision + +Use `rusqlite` with the `bundled` and `backup` features. Keep SQL and +`rusqlite` types in the `store` module. Use numbered SQL migrations and +`PRAGMA user_version`. Apply all pending migrations in one exclusive +transaction. Create an online backup before an automatic migration of an +existing database. + +Use WAL mode, `synchronous=FULL`, a five-second busy timeout, and foreign-key +enforcement on each connection. Verify each setting. Use `tit.sqlite3` as the +metadata database name. Require a local filesystem until a platform gate proves +that a different filesystem has correct WAL behavior. + +`tit doctor` opens an existing database without the create flag. It does not +migrate the database. It verifies the schema version, SQLite integrity, and +foreign keys. + +## Evidence + +The local gate used Rust 1.96.0 on arm64 macOS 27.0 and a local APFS filesystem. +It killed child processes during writes and schema migrations. After each kill, +the database contained the initial full state or the new full state. Tests also +proved constraints, indexed scans, concurrent reads, serialized writes, busy +handling, rollback, WAL checkpoint, vacuum, online backup, restore, and +migration from each committed fixture. + +The release workload used 10,000 parent records as issues and 1,000,000 child +records as events. The measured database size was 113,926,144 bytes. A safe +migration, including its required backup, took 774 ms. A subsequent online +backup took 509 ms. For 1,000 indexed queries that each read 25 events, the 50th, +95th, and 99th percentile times were 4, 5, and 8 microseconds. + +The workload gate has these limits: + +- the database must not be larger than 1 GiB. +- migration and backup must each complete in 120 seconds. +- the 99th percentile query time must not be more than 250 ms. + +The GitHub Actions release matrix runs the functional and workload gates on +Linux and macOS. This decision stays provisional until both hosted runners pass. +BSD and non-local filesystem support stay outside the current supported +platform set. + +## Consequences + +SQLite adds a small amount of explicit SQL, but it gives standard inspection +and recovery tools and a stable file format. Bundled SQLite increases the +executable size, but it removes variation in the runtime SQLite version. + +The application has one serialized SQLite writer. WAL lets readers continue +during a write. If the measured workload becomes too large for this model, the +project must review the evidence before it adds a database server, a connection +pool, or another persistence abstraction.
scripts/check-m1a
Mode → 100755; object → 60dbb96e2c8e
@@ -1,0 +1,5 @@ +#!/bin/sh +set -eu + +./scripts/check +cargo test --locked --release --test sqlite_workload -- --ignored --nocapture
src/cli.rs
Mode 100644 → 100644; object cd699f4d1c08 → bdce7f039b64
@@ -1,7 +1,7 @@
use std::net::SocketAddr;
use std::path::PathBuf;
-use clap::Parser;
+use clap::{Parser, Subcommand};
use url::Url;
#[derive(Debug, Parser)]
@@ -12,6 +12,9 @@
arg_required_else_help = true
)]
pub(crate) struct Cli {
+ #[command(subcommand)]
+ pub(crate) command: Option<Command>,
+
/// Read configuration from FILE
#[arg(long, value_name = "FILE", conflicts_with = "user")]
pub(crate) config: Option<PathBuf>,
@@ -39,4 +42,10 @@
/// Override the public SSH port
#[arg(long, value_name = "PORT")]
pub(crate) ssh_public_port: Option<u16>,
+}
+
+#[derive(Clone, Copy, Debug, Subcommand)]
+pub(crate) enum Command {
+ /// Check the instance database
+ Doctor,
}
src/main.rs
Mode 100644 → 100644; object 2dd220a09ea7 → 1a3f95b4687a
@@ -1,11 +1,12 @@
mod cli;
mod config;
+mod store;
use std::process::ExitCode;
use clap::Parser;
-use crate::cli::Cli;
+use crate::cli::{Cli, Command};
fn main() -> ExitCode {
let cli = match Cli::try_parse() {
@@ -18,7 +19,16 @@
};
match config::load(&cli) {
- Ok(_config) => ExitCode::SUCCESS,
+ Ok(config) => match cli.command {
+ None => ExitCode::SUCCESS,
+ Some(Command::Doctor) => match store::doctor(&config.instance_dir) {
+ Ok(()) => ExitCode::SUCCESS,
+ Err(error) => {
+ eprintln!("tit: {error}");
+ ExitCode::FAILURE
+ }
+ },
+ },
Err(error) => {
eprintln!("tit: {error}");
ExitCode::FAILURE
src/store/migrations/001_initial.sql
Mode → 100644; object → 4bd0d587a594
@@ -1,0 +1,15 @@ +CREATE TABLE m1a_parent ( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL UNIQUE CHECK (length(name) BETWEEN 1 AND 64), + created_at INTEGER NOT NULL CHECK (created_at >= 0) +) STRICT; + +CREATE INDEX m1a_parent_created_at ON m1a_parent (created_at, id); + +CREATE TABLE m1a_child ( + id INTEGER PRIMARY KEY, + parent_id INTEGER NOT NULL REFERENCES m1a_parent (id) ON DELETE RESTRICT, + sequence INTEGER NOT NULL CHECK (sequence >= 0), + body TEXT NOT NULL, + UNIQUE (parent_id, sequence) +) STRICT;
src/store/migrations/002_state.sql
Mode → 100644; object → c4b099d2d1b4
@@ -1,0 +1,6 @@
+ALTER TABLE m1a_child
+ADD COLUMN state TEXT NOT NULL DEFAULT 'open'
+CHECK (state IN ('open', 'closed'));
+
+CREATE INDEX m1a_child_state_parent
+ON m1a_child (state, parent_id, sequence);
src/store/mod.rs
Mode → 100644; object → 2b8fc2ddd09a
@@ -1,0 +1,261 @@
+use std::ffi::OsString;
+use std::path::{Path, PathBuf};
+use std::time::Duration;
+
+use rusqlite::OpenFlags;
+use rusqlite::backup::Backup;
+use rusqlite::{Connection, TransactionBehavior};
+use thiserror::Error;
+
+const BUSY_TIMEOUT: Duration = Duration::from_secs(5);
+const SCHEMA_VERSION: i64 = 2;
+#[allow(
+ dead_code,
+ reason = "the integration test imports this module without the CLI operation"
+)]
+const DATABASE_FILE: &str = "tit.sqlite3";
+#[allow(
+ dead_code,
+ reason = "M1A proves migrations before the M2 server calls them"
+)]
+const MIGRATIONS: [&str; 2] = [
+ include_str!("migrations/001_initial.sql"),
+ include_str!("migrations/002_state.sql"),
+];
+
+#[derive(Debug, Error)]
+pub(crate) enum StoreError {
+ #[error("SQLite error: {0}")]
+ Sqlite(#[from] rusqlite::Error),
+ #[allow(
+ dead_code,
+ reason = "M1A proves migrations before the M2 server calls them"
+ )]
+ #[error("database schema version {0} is newer than this executable")]
+ NewerSchema(i64),
+ #[allow(
+ dead_code,
+ reason = "the integration test imports this module without the CLI operation"
+ )]
+ #[error("database schema version is {actual}, expected {expected}")]
+ SchemaVersion { expected: i64, actual: i64 },
+ #[error("database integrity check failed: {0}")]
+ Integrity(String),
+ #[error("SQLite setting {name} is {actual}, expected {expected}")]
+ Setting {
+ name: &'static str,
+ expected: &'static str,
+ actual: String,
+ },
+}
+
+pub(crate) struct Store {
+ connection: Connection,
+}
+
+impl Store {
+ #[allow(
+ dead_code,
+ reason = "M1A proves migrations before the M2 server calls them"
+ )]
+ pub(crate) fn open(path: &Path) -> Result<Self, StoreError> {
+ let mut store = Self::open_unmigrated(path)?;
+ let current = store.schema_version()?;
+ if current > 0 && current < SCHEMA_VERSION {
+ store.backup(&migration_backup_path(path, current))?;
+ }
+ store.migrate()?;
+ Ok(store)
+ }
+
+ #[allow(
+ dead_code,
+ reason = "M1A proves migrations before the M2 server calls them"
+ )]
+ pub(crate) fn open_unmigrated(path: &Path) -> Result<Self, StoreError> {
+ let connection = Connection::open(path)?;
+ configure(&connection)?;
+ Ok(Self { connection })
+ }
+
+ #[allow(
+ dead_code,
+ reason = "M1A proves migrations before the M2 server calls them"
+ )]
+ pub(crate) fn migrate(&mut self) -> Result<(), StoreError> {
+ self.migrate_with_hook(|_| {})
+ }
+
+ #[allow(
+ dead_code,
+ reason = "M1A proves migrations before the M2 server calls them"
+ )]
+ pub(crate) fn migrate_with_hook(
+ &mut self,
+ mut after_migration: impl FnMut(i64),
+ ) -> Result<(), StoreError> {
+ let current = self.schema_version()?;
+ if current > SCHEMA_VERSION {
+ return Err(StoreError::NewerSchema(current));
+ }
+ if current == SCHEMA_VERSION {
+ return Ok(());
+ }
+
+ let transaction = self
+ .connection
+ .transaction_with_behavior(TransactionBehavior::Exclusive)?;
+ for version in (current + 1)..=SCHEMA_VERSION {
+ transaction.execute_batch(MIGRATIONS[(version - 1) as usize])?;
+ transaction.pragma_update(None, "user_version", version)?;
+ after_migration(version);
+ }
+ transaction.commit()?;
+ Ok(())
+ }
+
+ pub(crate) fn schema_version(&self) -> Result<i64, StoreError> {
+ Ok(self
+ .connection
+ .pragma_query_value(None, "user_version", |row| row.get(0))?)
+ }
+
+ pub(crate) fn integrity_check(&self) -> Result<(), StoreError> {
+ let result: String =
+ self.connection
+ .pragma_query_value(None, "integrity_check", |row| row.get(0))?;
+ if result != "ok" {
+ return Err(StoreError::Integrity(result));
+ }
+
+ let mut statement = self.connection.prepare("PRAGMA foreign_key_check")?;
+ let mut rows = statement.query([])?;
+ if let Some(row) = rows.next()? {
+ let table: String = row.get(0)?;
+ return Err(StoreError::Integrity(format!(
+ "foreign key violation in table {table}"
+ )));
+ }
+ Ok(())
+ }
+
+ #[allow(dead_code, reason = "M1A proves backup before the M2 server calls it")]
+ pub(crate) fn backup(&self, path: &Path) -> Result<(), StoreError> {
+ let mut destination = Connection::open(path)?;
+ let backup = Backup::new(&self.connection, &mut destination)?;
+ backup.run_to_completion(128, Duration::from_millis(1), None)?;
+ Ok(())
+ }
+
+ #[allow(
+ dead_code,
+ reason = "M1A proves maintenance before an operator command calls it"
+ )]
+ pub(crate) fn checkpoint(&self) -> Result<(), StoreError> {
+ self.connection
+ .execute_batch("PRAGMA wal_checkpoint(TRUNCATE)")?;
+ Ok(())
+ }
+
+ #[allow(
+ dead_code,
+ reason = "M1A proves maintenance before an operator command calls it"
+ )]
+ pub(crate) fn vacuum(&self) -> Result<(), StoreError> {
+ self.connection.execute_batch("VACUUM")?;
+ Ok(())
+ }
+
+ #[allow(
+ dead_code,
+ reason = "M1A tests storage behavior through this narrow test boundary"
+ )]
+ pub(crate) fn connection(&self) -> &Connection {
+ &self.connection
+ }
+
+ #[allow(
+ dead_code,
+ reason = "M1A tests storage behavior through this narrow test boundary"
+ )]
+ pub(crate) fn connection_mut(&mut self) -> &mut Connection {
+ &mut self.connection
+ }
+}
+
+#[allow(
+ dead_code,
+ reason = "the integration test imports this module without the CLI operation"
+)]
+pub(crate) fn doctor(instance_dir: &Path) -> Result<(), StoreError> {
+ let path = instance_dir.join(DATABASE_FILE);
+ let connection = Connection::open_with_flags(
+ path,
+ OpenFlags::SQLITE_OPEN_READ_WRITE | OpenFlags::SQLITE_OPEN_NO_MUTEX,
+ )?;
+ configure(&connection)?;
+ let store = Store { connection };
+ let actual = store.schema_version()?;
+ if actual != SCHEMA_VERSION {
+ return Err(StoreError::SchemaVersion {
+ expected: SCHEMA_VERSION,
+ actual,
+ });
+ }
+ store.integrity_check()
+}
+
+#[allow(
+ dead_code,
+ reason = "M1A proves migrations before the M2 server calls them"
+)]
+fn migration_backup_path(path: &Path, version: i64) -> PathBuf {
+ let mut backup = OsString::from(path.as_os_str());
+ backup.push(format!(".v{version}.backup"));
+ PathBuf::from(backup)
+}
+
+fn configure(connection: &Connection) -> Result<(), StoreError> {
+ connection.busy_timeout(BUSY_TIMEOUT)?;
+ connection.pragma_update(None, "journal_mode", "WAL")?;
+ connection.pragma_update(None, "synchronous", "FULL")?;
+ connection.pragma_update(None, "foreign_keys", true)?;
+
+ verify_text_setting(connection, "journal_mode", "wal")?;
+ verify_integer_setting(connection, "synchronous", 2, "2")?;
+ verify_integer_setting(connection, "foreign_keys", 1, "1")?;
+ Ok(())
+}
+
+fn verify_text_setting(
+ connection: &Connection,
+ name: &'static str,
+ expected: &'static str,
+) -> Result<(), StoreError> {
+ let actual: String = connection.pragma_query_value(None, name, |row| row.get(0))?;
+ if actual.eq_ignore_ascii_case(expected) {
+ return Ok(());
+ }
+ Err(StoreError::Setting {
+ name,
+ expected,
+ actual,
+ })
+}
+
+fn verify_integer_setting(
+ connection: &Connection,
+ name: &'static str,
+ expected: i64,
+ expected_text: &'static str,
+) -> Result<(), StoreError> {
+ let actual: i64 = connection.pragma_query_value(None, name, |row| row.get(0))?;
+ if actual == expected {
+ return Ok(());
+ }
+ Err(StoreError::Setting {
+ name,
+ expected: expected_text,
+ actual: actual.to_string(),
+ })
+}
tests/cli.rs
Mode 100644 → 100644; object 5de9f1298aa6 → 390a4b3163b5
@@ -9,6 +9,8 @@
};
use tempfile::TempDir;
+const V2_DATABASE: &str = include_str!("fixtures/sqlite/v2.sql");
+
#[test]
fn help_and_version_use_standard_output() {
for argument in ["--help", "--version"] {
@@ -54,6 +56,88 @@
assert_eq!(output.status.code(), Some(1));
assert!(output.stdout.is_empty());
assert!(String::from_utf8_lossy(&output.stderr).starts_with("tit: "));
+}
+
+#[test]
+fn doctor_checks_an_existing_current_database() {
+ let instance = TestInstance::new();
+ let database = rusqlite::Connection::open(instance.path().join("tit.sqlite3"))
+ .expect("open the instance database");
+ database
+ .execute_batch(V2_DATABASE)
+ .expect("create the current database");
+ drop(database);
+
+ let output = instance.run(&[
+ "--config",
+ instance.config().to_str().expect("a UTF-8 path"),
+ "doctor",
+ ]);
+
+ assert!(output.status.success());
+ assert!(output.stdout.is_empty());
+ assert!(output.stderr.is_empty());
+}
+
+#[test]
+fn doctor_does_not_create_or_migrate_a_database() {
+ let instance = TestInstance::new();
+ let database_path = instance.path().join("tit.sqlite3");
+ let arguments = [
+ "--config",
+ instance.config().to_str().expect("a UTF-8 path"),
+ "doctor",
+ ];
+
+ let missing = instance.run(&arguments);
+ assert_eq!(missing.status.code(), Some(1));
+ assert!(!database_path.exists());
+
+ let database = rusqlite::Connection::open(&database_path).expect("open the old database");
+ database
+ .execute_batch(include_str!("fixtures/sqlite/v1.sql"))
+ .expect("create the old database");
+ drop(database);
+
+ let old = instance.run(&arguments);
+ assert_eq!(old.status.code(), Some(1));
+ let database = rusqlite::Connection::open(&database_path).expect("reopen the old database");
+ assert_eq!(
+ database
+ .pragma_query_value::<i64, _>(None, "user_version", |row| row.get(0))
+ .expect("read the schema version"),
+ 1
+ );
+}
+
+#[test]
+fn doctor_reports_a_foreign_key_violation() {
+ let instance = TestInstance::new();
+ let database = rusqlite::Connection::open(instance.path().join("tit.sqlite3"))
+ .expect("open the instance database");
+ database
+ .execute_batch(V2_DATABASE)
+ .expect("create the current database");
+ database
+ .pragma_update(None, "foreign_keys", false)
+ .expect("disable foreign keys for the damaged fixture");
+ database
+ .execute(
+ "INSERT INTO m1a_child (id, parent_id, sequence, body) VALUES (2, 999, 1, 'orphan')",
+ [],
+ )
+ .expect("create a foreign-key violation");
+ drop(database);
+
+ let output = instance.run(&[
+ "--config",
+ instance.config().to_str().expect("a UTF-8 path"),
+ "doctor",
+ ]);
+
+ assert_eq!(output.status.code(), Some(1));
+ assert!(output.stdout.is_empty());
+ assert!(String::from_utf8_lossy(&output.stderr).contains("foreign key violation"));
}
#[test]
tests/fixtures/sqlite/v1.sql
Mode → 100644; object → b14a9baf53c7
@@ -1,0 +1,23 @@ +CREATE TABLE m1a_parent ( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL UNIQUE CHECK (length(name) BETWEEN 1 AND 64), + created_at INTEGER NOT NULL CHECK (created_at >= 0) +) STRICT; + +CREATE INDEX m1a_parent_created_at ON m1a_parent (created_at, id); + +CREATE TABLE m1a_child ( + id INTEGER PRIMARY KEY, + parent_id INTEGER NOT NULL REFERENCES m1a_parent (id) ON DELETE RESTRICT, + sequence INTEGER NOT NULL CHECK (sequence >= 0), + body TEXT NOT NULL, + UNIQUE (parent_id, sequence) +) STRICT; + +INSERT INTO m1a_parent (id, name, created_at) +VALUES (1, 'fixture', 1); + +INSERT INTO m1a_child (id, parent_id, sequence, body) +VALUES (1, 1, 1, 'version one'); + +PRAGMA user_version = 1;
tests/fixtures/sqlite/v2.sql
Mode → 100644; object → 62d6184de10f
@@ -1,0 +1,27 @@
+CREATE TABLE m1a_parent (
+ id INTEGER PRIMARY KEY,
+ name TEXT NOT NULL UNIQUE CHECK (length(name) BETWEEN 1 AND 64),
+ created_at INTEGER NOT NULL CHECK (created_at >= 0)
+) STRICT;
+
+CREATE INDEX m1a_parent_created_at ON m1a_parent (created_at, id);
+
+CREATE TABLE m1a_child (
+ id INTEGER PRIMARY KEY,
+ parent_id INTEGER NOT NULL REFERENCES m1a_parent (id) ON DELETE RESTRICT,
+ sequence INTEGER NOT NULL CHECK (sequence >= 0),
+ body TEXT NOT NULL,
+ state TEXT NOT NULL DEFAULT 'open' CHECK (state IN ('open', 'closed')),
+ UNIQUE (parent_id, sequence)
+) STRICT;
+
+CREATE INDEX m1a_child_state_parent
+ON m1a_child (state, parent_id, sequence);
+
+INSERT INTO m1a_parent (id, name, created_at)
+VALUES (1, 'fixture', 1);
+
+INSERT INTO m1a_child (id, parent_id, sequence, body, state)
+VALUES (1, 1, 1, 'version two', 'closed');
+
+PRAGMA user_version = 2;
tests/sqlite.rs
Mode → 100644; object → 6fdc072fc689
@@ -1,0 +1,564 @@
+#[path = "../src/store/mod.rs"]
+mod store;
+
+use std::process::{Child, Command};
+use std::sync::mpsc;
+use std::sync::{Arc, atomic::AtomicBool, atomic::Ordering};
+use std::thread;
+use std::time::{Duration, Instant};
+use std::{env, ffi::OsString, fs};
+
+use rusqlite::{Connection, ErrorCode, TransactionBehavior, params};
+use store::{Store, StoreError};
+use tempfile::TempDir;
+
+const V1_FIXTURE: &str = include_str!("fixtures/sqlite/v1.sql");
+const V2_FIXTURE: &str = include_str!("fixtures/sqlite/v2.sql");
+
+fn database(directory: &TempDir, name: &str) -> std::path::PathBuf {
+ directory.path().join(name)
+}
+
+fn migration_backup(path: &std::path::Path, version: i64) -> std::path::PathBuf {
+ let mut backup = OsString::from(path.as_os_str());
+ backup.push(format!(".v{version}.backup"));
+ backup.into()
+}
+
+fn create_fixture(path: &std::path::Path, fixture: &str) {
+ let connection = Connection::open(path).expect("open a fixture database");
+ connection
+ .execute_batch(fixture)
+ .expect("create the fixture");
+}
+
+fn spawn_crash_child(
+ mode: &str,
+ database_path: &std::path::Path,
+ ready_path: &std::path::Path,
+) -> Child {
+ Command::new(env::current_exe().expect("find the integration test executable"))
+ .args(["--exact", "crash_child", "--nocapture", "--test-threads=1"])
+ .env("TIT_M1A_CHILD_MODE", mode)
+ .env("TIT_M1A_DATABASE", database_path)
+ .env("TIT_M1A_READY", ready_path)
+ .spawn()
+ .expect("start the crash-test child")
+}
+
+fn wait_for_child(child: &mut Child, ready_path: &std::path::Path) {
+ let deadline = Instant::now() + Duration::from_secs(10);
+ loop {
+ if ready_path.exists() {
+ return;
+ }
+ if let Some(status) = child.try_wait().expect("inspect the crash-test child") {
+ panic!("crash-test child stopped before it was ready: {status}");
+ }
+ assert!(Instant::now() < deadline, "crash-test child was not ready");
+ thread::sleep(Duration::from_millis(10));
+ }
+}
+
+fn kill_child(mut child: Child, ready_path: &std::path::Path) {
+ wait_for_child(&mut child, ready_path);
+ child.kill().expect("kill the crash-test child");
+ child.wait().expect("wait for the crash-test child");
+}
+
+fn signal_ready_and_wait() {
+ let ready_path = env::var_os("TIT_M1A_READY").expect("read the ready-file path");
+ fs::write(ready_path, b"ready").expect("write the ready file");
+ loop {
+ thread::park();
+ }
+}
+
+#[test]
+fn crash_child() {
+ let Some(mode) = env::var_os("TIT_M1A_CHILD_MODE") else {
+ return;
+ };
+ let database_path = env::var_os("TIT_M1A_DATABASE").expect("read the database path");
+ let database_path = std::path::Path::new(&database_path);
+
+ match mode.to_str().expect("read the crash-test mode") {
+ "write-uncommitted" => {
+ let mut store = Store::open(database_path).expect("open the child store");
+ let transaction = store
+ .connection_mut()
+ .transaction_with_behavior(TransactionBehavior::Immediate)
+ .expect("start the child transaction");
+ transaction
+ .execute(
+ "INSERT INTO m1a_parent (id, name, created_at) VALUES (1, 'uncommitted', 1)",
+ [],
+ )
+ .expect("insert an uncommitted row");
+ signal_ready_and_wait();
+ }
+ "write-committed" => {
+ let store = Store::open(database_path).expect("open the child store");
+ store
+ .connection()
+ .execute(
+ "INSERT INTO m1a_parent (id, name, created_at) VALUES (1, 'committed', 1)",
+ [],
+ )
+ .expect("insert a committed row");
+ signal_ready_and_wait();
+ }
+ "migration-uncommitted" => {
+ let mut store = Store::open_unmigrated(database_path).expect("open the child fixture");
+ store
+ .migrate_with_hook(|version| {
+ if version == 2 {
+ signal_ready_and_wait();
+ }
+ })
+ .expect("migrate the child fixture");
+ }
+ "migration-committed" => {
+ Store::open(database_path).expect("migrate the child fixture");
+ signal_ready_and_wait();
+ }
+ unexpected => panic!("unknown crash-test mode: {unexpected}"),
+ }
+}
+
+#[test]
+fn configures_connections_and_creates_the_current_schema() {
+ let directory = TempDir::new().expect("create a temporary directory");
+ let store = Store::open(&database(&directory, "store.sqlite")).expect("open the store");
+
+ assert_eq!(store.schema_version().expect("read the schema version"), 2);
+ assert_eq!(
+ store
+ .connection()
+ .pragma_query_value::<String, _>(None, "journal_mode", |row| row.get(0))
+ .expect("read the journal mode"),
+ "wal"
+ );
+ assert_eq!(
+ store
+ .connection()
+ .pragma_query_value::<i64, _>(None, "synchronous", |row| row.get(0))
+ .expect("read the synchronization mode"),
+ 2
+ );
+ assert_eq!(
+ store
+ .connection()
+ .pragma_query_value::<i64, _>(None, "foreign_keys", |row| row.get(0))
+ .expect("read the foreign-key mode"),
+ 1
+ );
+ store.integrity_check().expect("check database integrity");
+}
+
+#[test]
+fn enforces_constraints_and_supports_crud_and_indexed_scans() {
+ let directory = TempDir::new().expect("create a temporary directory");
+ let mut store = Store::open(&database(&directory, "store.sqlite")).expect("open the store");
+ let connection = store.connection_mut();
+
+ connection
+ .execute(
+ "INSERT INTO m1a_parent (id, name, created_at) VALUES (?1, ?2, ?3)",
+ params![1, "parent", 10],
+ )
+ .expect("insert a parent");
+ connection
+ .execute(
+ "INSERT INTO m1a_child (id, parent_id, sequence, body) VALUES (?1, ?2, ?3, ?4)",
+ params![1, 1, 1, "body"],
+ )
+ .expect("insert a child");
+
+ let duplicate = connection
+ .execute(
+ "INSERT INTO m1a_parent (id, name, created_at) VALUES (?1, ?2, ?3)",
+ params![2, "parent", 11],
+ )
+ .expect_err("reject a duplicate name");
+ assert_eq!(
+ duplicate.sqlite_error_code(),
+ Some(ErrorCode::ConstraintViolation)
+ );
+
+ let orphan = connection
+ .execute(
+ "INSERT INTO m1a_child (id, parent_id, sequence, body) VALUES (?1, ?2, ?3, ?4)",
+ params![2, 999, 1, "orphan"],
+ )
+ .expect_err("reject an orphan");
+ assert_eq!(
+ orphan.sqlite_error_code(),
+ Some(ErrorCode::ConstraintViolation)
+ );
+
+ let deletion = connection
+ .execute("DELETE FROM m1a_parent WHERE id = ?1", [1])
+ .expect_err("restrict deletion of a referenced parent");
+ assert_eq!(
+ deletion.sqlite_error_code(),
+ Some(ErrorCode::ConstraintViolation)
+ );
+
+ connection
+ .execute(
+ "UPDATE m1a_child SET body = ?1, state = ?2 WHERE id = ?3",
+ params!["changed", "closed", 1],
+ )
+ .expect("update a child");
+ let child: (String, String) = connection
+ .query_row(
+ "SELECT body, state FROM m1a_child WHERE id = ?1",
+ [1],
+ |row| Ok((row.get(0)?, row.get(1)?)),
+ )
+ .expect("read the child");
+ assert_eq!(child, ("changed".to_owned(), "closed".to_owned()));
+
+ let plan: String = connection
+ .query_row(
+ "EXPLAIN QUERY PLAN SELECT id FROM m1a_parent WHERE created_at >= ?1 ORDER BY created_at, id",
+ [0],
+ |row| row.get(3),
+ )
+ .expect("read the query plan");
+ assert!(plan.contains("m1a_parent_created_at"), "query plan: {plan}");
+
+ connection
+ .execute("DELETE FROM m1a_child WHERE id = ?1", [1])
+ .expect("delete the child");
+ connection
+ .execute("DELETE FROM m1a_parent WHERE id = ?1", [1])
+ .expect("delete the parent");
+ assert_eq!(
+ connection
+ .query_row("SELECT count(*) FROM m1a_parent", [], |row| row
+ .get::<_, i64>(0))
+ .expect("count parents"),
+ 0
+ );
+}
+
+#[test]
+fn rolls_back_a_failed_unit_of_work() {
+ let directory = TempDir::new().expect("create a temporary directory");
+ let mut store = Store::open(&database(&directory, "store.sqlite")).expect("open the store");
+
+ let transaction = store
+ .connection_mut()
+ .transaction()
+ .expect("start a transaction");
+ transaction
+ .execute(
+ "INSERT INTO m1a_parent (id, name, created_at) VALUES (?1, ?2, ?3)",
+ params![1, "rolled-back", 1],
+ )
+ .expect("insert a parent");
+ transaction.rollback().expect("roll back the transaction");
+
+ assert_eq!(
+ store
+ .connection()
+ .query_row("SELECT count(*) FROM m1a_parent", [], |row| row
+ .get::<_, i64>(0))
+ .expect("count parents"),
+ 0
+ );
+}
+
+#[test]
+fn recovers_committed_state_and_discards_uncommitted_state_after_a_process_kill() {
+ for (mode, expected_rows) in [("write-uncommitted", 0), ("write-committed", 1)] {
+ let directory = TempDir::new().expect("create a temporary directory");
+ let path = database(&directory, "store.sqlite");
+ Store::open(&path).expect("create the store");
+ let ready_path = database(&directory, "ready");
+
+ let child = spawn_crash_child(mode, &path, &ready_path);
+ kill_child(child, &ready_path);
+
+ let store = Store::open(&path).expect("recover the store");
+ store.integrity_check().expect("check recovered integrity");
+ assert_eq!(
+ store
+ .connection()
+ .query_row("SELECT count(*) FROM m1a_parent", [], |row| row
+ .get::<_, i64>(0))
+ .expect("count recovered rows"),
+ expected_rows,
+ "recovered row count for {mode}"
+ );
+ }
+}
+
+#[test]
+fn permits_reads_during_a_write_and_serializes_writers() {
+ let directory = TempDir::new().expect("create a temporary directory");
+ let path = database(&directory, "store.sqlite");
+ let store = Store::open(&path).expect("open the store");
+ store
+ .connection()
+ .execute(
+ "INSERT INTO m1a_parent (id, name, created_at) VALUES (1, 'initial', 1)",
+ [],
+ )
+ .expect("insert the initial parent");
+ drop(store);
+
+ let (locked_sender, locked_receiver) = mpsc::channel();
+ let writer_path = path.clone();
+ let writer = thread::spawn(move || {
+ let mut store = Store::open(&writer_path).expect("open the first writer");
+ let transaction = store
+ .connection_mut()
+ .transaction_with_behavior(TransactionBehavior::Immediate)
+ .expect("start the first write");
+ transaction
+ .execute(
+ "INSERT INTO m1a_parent (id, name, created_at) VALUES (2, 'first-writer', 2)",
+ [],
+ )
+ .expect("write with the first writer");
+ locked_sender.send(()).expect("signal the write lock");
+ thread::sleep(Duration::from_millis(200));
+ transaction.commit().expect("commit the first writer");
+ });
+
+ locked_receiver.recv().expect("wait for the write lock");
+ let reader = Store::open(&path).expect("open a reader");
+ assert_eq!(
+ reader
+ .connection()
+ .query_row("SELECT count(*) FROM m1a_parent", [], |row| row
+ .get::<_, i64>(0))
+ .expect("read during a write"),
+ 1
+ );
+ drop(reader);
+
+ let started = Instant::now();
+ let second_writer = Store::open(&path).expect("open the second writer");
+ second_writer
+ .connection()
+ .execute(
+ "INSERT INTO m1a_parent (id, name, created_at) VALUES (3, 'second-writer', 3)",
+ [],
+ )
+ .expect("write with the second writer");
+ assert!(started.elapsed() >= Duration::from_millis(100));
+ writer.join().expect("join the first writer");
+
+ assert_eq!(
+ second_writer
+ .connection()
+ .query_row("SELECT count(*) FROM m1a_parent", [], |row| row
+ .get::<_, i64>(0))
+ .expect("count committed writes"),
+ 3
+ );
+}
+
+#[test]
+fn returns_busy_after_the_configured_timeout() {
+ let directory = TempDir::new().expect("create a temporary directory");
+ let path = database(&directory, "store.sqlite");
+ let mut first = Store::open(&path).expect("open the first store");
+ let second = Store::open(&path).expect("open the second store");
+ second
+ .connection()
+ .busy_timeout(Duration::from_millis(50))
+ .expect("set a short busy timeout");
+
+ let transaction = first
+ .connection_mut()
+ .transaction_with_behavior(TransactionBehavior::Immediate)
+ .expect("start a write transaction");
+ transaction
+ .execute(
+ "INSERT INTO m1a_parent (id, name, created_at) VALUES (1, 'locked', 1)",
+ [],
+ )
+ .expect("hold the write lock");
+
+ let started = Instant::now();
+ let error = second
+ .connection()
+ .execute(
+ "INSERT INTO m1a_parent (id, name, created_at) VALUES (2, 'busy', 2)",
+ [],
+ )
+ .expect_err("return a busy error");
+ assert_eq!(error.sqlite_error_code(), Some(ErrorCode::DatabaseBusy));
+ assert!(started.elapsed() >= Duration::from_millis(40));
+ transaction.rollback().expect("release the write lock");
+}
+
+#[test]
+fn backs_up_a_consistent_database_while_writes_continue() {
+ let directory = TempDir::new().expect("create a temporary directory");
+ let source_path = database(&directory, "source.sqlite");
+ let backup_path = database(&directory, "backup.sqlite");
+ let source = Store::open(&source_path).expect("open the source store");
+ source
+ .connection()
+ .execute(
+ "INSERT INTO m1a_parent (id, name, created_at) VALUES (1, 'parent', 1)",
+ [],
+ )
+ .expect("insert the parent");
+ for sequence in 0..1_000 {
+ source
+ .connection()
+ .execute(
+ "INSERT INTO m1a_child (id, parent_id, sequence, body) VALUES (?1, 1, ?2, 'initial')",
+ params![sequence + 1, sequence],
+ )
+ .expect("insert an initial child");
+ }
+
+ let running = Arc::new(AtomicBool::new(true));
+ let writer_running = Arc::clone(&running);
+ let writer_path = source_path.clone();
+ let writer = thread::spawn(move || {
+ let store = Store::open(&writer_path).expect("open the backup writer");
+ let mut sequence = 1_000;
+ while writer_running.load(Ordering::Relaxed) {
+ store
+ .connection()
+ .execute(
+ "INSERT INTO m1a_child (id, parent_id, sequence, body) VALUES (?1, 1, ?2, 'concurrent')",
+ params![sequence + 1, sequence],
+ )
+ .expect("insert a concurrent child");
+ sequence += 1;
+ }
+ sequence
+ });
+
+ source
+ .backup(&backup_path)
+ .expect("create an online backup");
+ running.store(false, Ordering::Relaxed);
+ let final_sequence = writer.join().expect("join the backup writer");
+ let backup = Store::open(&backup_path).expect("open the backup");
+ backup.integrity_check().expect("check backup integrity");
+ let backup_count: i64 = backup
+ .connection()
+ .query_row("SELECT count(*) FROM m1a_child", [], |row| row.get(0))
+ .expect("count backup children");
+ assert!(backup_count >= 1_000);
+ assert!(backup_count <= i64::from(final_sequence));
+}
+
+#[test]
+fn migrates_each_committed_historical_fixture() {
+ for (name, fixture, initial_version) in
+ [("v1.sqlite", V1_FIXTURE, 1), ("v2.sqlite", V2_FIXTURE, 2)]
+ {
+ let directory = TempDir::new().expect("create a temporary directory");
+ let path = database(&directory, name);
+ create_fixture(&path, fixture);
+
+ let store = Store::open(&path).expect("migrate the fixture");
+ assert_eq!(store.schema_version().expect("read the schema version"), 2);
+ store.integrity_check().expect("check migrated integrity");
+ let state: String = store
+ .connection()
+ .query_row("SELECT state FROM m1a_child WHERE id = 1", [], |row| {
+ row.get(0)
+ })
+ .expect("read the migrated state");
+ let expected = if initial_version == 1 {
+ "open"
+ } else {
+ "closed"
+ };
+ assert_eq!(state, expected);
+
+ let backup_path = migration_backup(&path, 1);
+ assert_eq!(backup_path.exists(), initial_version == 1);
+ if initial_version == 1 {
+ let backup = Store::open_unmigrated(&backup_path).expect("open the migration backup");
+ assert_eq!(backup.schema_version().expect("read the backup version"), 1);
+ backup.integrity_check().expect("check backup integrity");
+ }
+ }
+}
+
+#[test]
+fn recovers_complete_schema_versions_after_a_process_kill_during_migration() {
+ for (mode, expected_version) in [("migration-uncommitted", 1), ("migration-committed", 2)] {
+ let directory = TempDir::new().expect("create a temporary directory");
+ let path = database(&directory, "fixture.sqlite");
+ create_fixture(&path, V1_FIXTURE);
+ let ready_path = database(&directory, "ready");
+
+ let child = spawn_crash_child(mode, &path, &ready_path);
+ kill_child(child, &ready_path);
+
+ let store = Store::open_unmigrated(&path).expect("recover the fixture");
+ assert_eq!(
+ store.schema_version().expect("read the recovered version"),
+ expected_version,
+ "recovered schema version for {mode}"
+ );
+ store.integrity_check().expect("check recovered integrity");
+
+ let has_state: i64 = store
+ .connection()
+ .query_row(
+ "SELECT count(*) FROM pragma_table_info('m1a_child') WHERE name = 'state'",
+ [],
+ |row| row.get(0),
+ )
+ .expect("inspect the recovered schema");
+ assert_eq!(has_state, i64::from(expected_version == 2));
+ }
+}
+
+#[test]
+fn rejects_a_newer_schema() {
+ let directory = TempDir::new().expect("create a temporary directory");
+ let path = database(&directory, "newer.sqlite");
+ let connection = Connection::open(&path).expect("open a fixture database");
+ connection
+ .pragma_update(None, "user_version", 999)
+ .expect("set a newer version");
+ drop(connection);
+
+ assert!(matches!(
+ Store::open(&path),
+ Err(StoreError::NewerSchema(999))
+ ));
+}
+
+#[test]
+fn checkpoints_and_vacuums_without_losing_data() {
+ let directory = TempDir::new().expect("create a temporary directory");
+ let path = database(&directory, "store.sqlite");
+ let store = Store::open(&path).expect("open the store");
+ store
+ .connection()
+ .execute(
+ "INSERT INTO m1a_parent (id, name, created_at) VALUES (1, 'parent', 1)",
+ [],
+ )
+ .expect("insert a parent");
+
+ store.checkpoint().expect("checkpoint the WAL");
+ store.vacuum().expect("vacuum the database");
+ store.integrity_check().expect("check database integrity");
+ assert_eq!(
+ store
+ .connection()
+ .query_row("SELECT count(*) FROM m1a_parent", [], |row| row
+ .get::<_, i64>(0))
+ .expect("count parents"),
+ 1
+ );
+}
tests/sqlite_workload.rs
Mode → 100644; object → ab1b79ab3664
@@ -1,0 +1,146 @@
+#[path = "../src/store/mod.rs"]
+mod store;
+
+use std::fs;
+use std::time::{Duration, Instant};
+
+use rusqlite::{Connection, params};
+use store::Store;
+use tempfile::TempDir;
+
+const V1_FIXTURE: &str = include_str!("fixtures/sqlite/v1.sql");
+const ISSUE_COUNT: i64 = 10_000;
+const EVENTS_PER_ISSUE: i64 = 100;
+const EVENT_COUNT: i64 = ISSUE_COUNT * EVENTS_PER_ISSUE;
+const MAX_DATABASE_BYTES: u64 = 1_073_741_824;
+const MAX_OPERATION_TIME: Duration = Duration::from_secs(120);
+const MAX_P99_QUERY_TIME: Duration = Duration::from_millis(250);
+
+#[test]
+#[ignore = "run through scripts/check-m1a"]
+fn measures_the_m1a_workload() {
+ let directory = TempDir::new().expect("create a temporary directory");
+ let database_path = directory.path().join("workload.sqlite");
+ let backup_path = directory.path().join("workload.backup.sqlite");
+ create_version_one_workload(&database_path);
+
+ let migration_started = Instant::now();
+ let store = Store::open(&database_path).expect("back up and migrate the workload");
+ let migration_time = migration_started.elapsed();
+ store.checkpoint().expect("checkpoint the workload");
+ let database_bytes = fs::metadata(&database_path)
+ .expect("inspect the workload database")
+ .len();
+
+ let query_times = measure_queries(&store);
+ let p50_query_time = percentile(&query_times, 50);
+ let p95_query_time = percentile(&query_times, 95);
+ let p99_query_time = percentile(&query_times, 99);
+
+ let backup_started = Instant::now();
+ store.backup(&backup_path).expect("back up the workload");
+ let backup_time = backup_started.elapsed();
+ let restored = Store::open(&backup_path).expect("restore the workload backup");
+ restored
+ .integrity_check()
+ .expect("check restored workload integrity");
+ let restored_counts: (i64, i64) = restored
+ .connection()
+ .query_row(
+ "SELECT (SELECT count(*) FROM m1a_parent), \
+ (SELECT count(*) FROM m1a_child)",
+ [],
+ |row| Ok((row.get(0)?, row.get(1)?)),
+ )
+ .expect("count restored workload records");
+ assert_eq!(restored_counts, (ISSUE_COUNT, EVENT_COUNT));
+
+ eprintln!(
+ "M1A workload: database_bytes={database_bytes} migration_ms={} backup_ms={} query_p50_us={} query_p95_us={} query_p99_us={}",
+ migration_time.as_millis(),
+ backup_time.as_millis(),
+ p50_query_time.as_micros(),
+ p95_query_time.as_micros(),
+ p99_query_time.as_micros()
+ );
+
+ assert!(database_bytes <= MAX_DATABASE_BYTES);
+ assert!(migration_time <= MAX_OPERATION_TIME);
+ assert!(backup_time <= MAX_OPERATION_TIME);
+ assert!(p99_query_time <= MAX_P99_QUERY_TIME);
+}
+
+fn create_version_one_workload(path: &std::path::Path) {
+ let mut connection = Connection::open(path).expect("open the workload database");
+ connection
+ .execute_batch(V1_FIXTURE)
+ .expect("create the version-one schema");
+ connection
+ .execute_batch("DELETE FROM m1a_child; DELETE FROM m1a_parent;")
+ .expect("remove the small fixture records");
+ connection
+ .pragma_update(None, "foreign_keys", true)
+ .expect("enable foreign keys");
+
+ let transaction = connection.transaction().expect("start the workload");
+ {
+ let mut insert_issue = transaction
+ .prepare("INSERT INTO m1a_parent (id, name, created_at) VALUES (?1, ?2, ?3)")
+ .expect("prepare the issue insert");
+ let mut insert_event = transaction
+ .prepare(
+ "INSERT INTO m1a_child (id, parent_id, sequence, body) VALUES (?1, ?2, ?3, ?4)",
+ )
+ .expect("prepare the event insert");
+ let body = "A representative issue event records a bounded plain-text change.";
+
+ for issue_id in 1..=ISSUE_COUNT {
+ insert_issue
+ .execute(params![issue_id, format!("issue-{issue_id}"), issue_id])
+ .expect("insert an issue");
+ for sequence in 0..EVENTS_PER_ISSUE {
+ let event_id = (issue_id - 1) * EVENTS_PER_ISSUE + sequence + 1;
+ insert_event
+ .execute(params![event_id, issue_id, sequence, body])
+ .expect("insert an event");
+ }
+ }
+ }
+ transaction.commit().expect("commit the workload");
+}
+
+fn measure_queries(store: &Store) -> Vec<Duration> {
+ let mut times = Vec::with_capacity(1_000);
+ let mut statement = store
+ .connection()
+ .prepare(
+ "SELECT id, sequence, body FROM m1a_child \
+ WHERE state = ?1 AND parent_id = ?2 ORDER BY sequence LIMIT 25",
+ )
+ .expect("prepare the representative query");
+
+ for sample in 0_i64..1_000 {
+ let issue_id = sample % ISSUE_COUNT + 1;
+ let started = Instant::now();
+ let mut rows = statement
+ .query(params!["open", issue_id])
+ .expect("query issue events");
+ let mut row_count = 0;
+ while let Some(row) = rows.next().expect("read an issue event") {
+ let _: (i64, i64, String) = (
+ row.get(0).expect("read the event ID"),
+ row.get(1).expect("read the sequence"),
+ row.get(2).expect("read the body"),
+ );
+ row_count += 1;
+ }
+ assert_eq!(row_count, 25);
+ times.push(started.elapsed());
+ }
+ times.sort_unstable();
+ times
+}
+
+fn percentile(times: &[Duration], percentile: usize) -> Duration {
+ times[(times.len() - 1) * percentile / 100]
+}