diff --git a/ROADMAP.md b/ROADMAP.md index c495880..283913e 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -48,7 +48,7 @@ standards register names the rows that ask for them. | 031 | Session search (SQLite FTS5) | 3 Memory | S | 010 | approved | | 032 | Embeddings + semantic memory + hybrid retrieval | 3 Memory | L | 031 | planned | | 033 | Project context: AGENTS.md | 3 Memory | S | 030, 022 | done | -| 034 | Export, import, restore | 3 Memory | S | 030, 031 | planned | +| 034 | Export, import, restore | 3 Memory | S | 030, 031 | done | | 040 | Skills registry + agentskills.io format + progressive disclosure | 4 Skills | M | 020 | planned | | 041 | Skill self-management with staged approval + scanner | 4 Skills | M | 040, 021 | planned | | 050 | Scheduler: Oban cron agent tasks with delivery targets | 5 Automation | M | 012 | planned | diff --git a/coverage.tsv b/coverage.tsv index d085021..dc0ca76 100644 --- a/coverage.tsv +++ b/coverage.tsv @@ -14,3 +14,4 @@ slice_id percent sha date 031 76.96 866dc3f 2026-09-20 030 78.40 c657da8 2026-09-21 033 78.50 3b5e89e 2026-09-21 +034 78.83 3a5785f 2026-09-21 diff --git a/docs/backup.md b/docs/backup.md new file mode 100644 index 0000000..25351fa --- /dev/null +++ b/docs/backup.md @@ -0,0 +1,62 @@ +# Backup, restore, and moving to another machine + +Slice 034. One archive holds a Trinity; a fresh install takes it back. Losing the machine is a recoverable +event, not a total one. + +## What the archive holds + +`mix trinity.export --out `, or the Export button on `/settings`, writes a gzip tarball with: + +| Entry | What it is | Always | +|---|---|---| +| `manifest.json` | the format (`trinity-archive/1`), this build's version, when it was made, the migrated schema versions of each database, whether private keys are inside, and one line per file with its size and SHA-256 | yes | +| `trinity.db` | the primary database: personas, sessions, messages, the search index, memories and their change log, approvals and rules | yes | +| `receipts.db` | the receipt chains and their checkpoints | yes | +| `keys/registry.json` | the key registry: public keys, algorithms, statuses (a stranger verifies receipts with this) | yes | +| `keys/receipts-.key` | the private signing key | only with `--keys` (or `?keys=1` on the page) | +| `skills/…`, `personas/…` | files under those directories, when present (skills arrive at slice 040) | when present | + +Each database is snapshotted with `VACUUM INTO`: a consistent copy including what the write-ahead log held, +taken while Trinity runs. The manifest says `keys_included` either way, so an archive's holder knows what it +carries. + +## What it does not hold + +- Model caches, the desktop shell's state, the `LOCK` file, `RESTORED` markers from earlier restores. +- The private key, unless asked for. An archive without it verifies every receipt ever written (the registry + carries the public keys) and cannot write a new one as you: a fresh install without the key generates its + own on first boot and appends it to the registry as a new key id; the old chain stays verifiable under the + old key's row. Hand an archive to someone else without `--keys`. +- Anything from other machines: this is one data directory, not a sync. + +## Restoring + +On a fresh install, with Trinity stopped: + +``` +mix trinity.import trinity-2026-09-21.tar.gz +``` + +The task reads the manifest, checks the schema versions against this binary's migrations (an archive from a +newer Trinity refuses by name; an older one migrates on the next boot), verifies every file's digest against +the tarball's bytes, and only then writes the files. A data directory that is not empty is refused with the +paths it found; `--force` replaces them and prints each one. A directory held by a live Trinity is refused +until that process stops. A `RESTORED` file in the data directory names the archive and the time. + +Then start Trinity. The receipt chains verify under the restored registry (`mix trinity.receipts.verify --scope +`, or the Verify button on a session's receipts page), which is the sharpest check that the copy is +faithful. The search index is inside the database on SQLite; on Postgres it is a generated column. + +`--data-dir ` points either task at a directory other than the configured one. + +## Postgres + +The archive is the SQLite data directory's. A deployment on Postgres (`TRINITY_DB=postgres`) keeps its rows in +that database and backs it up with Postgres's own tools (`pg_dump`); the export there carries the key registry +and the directories, not the databases, and says so in the manifest's file list. + +## Size + +On the machine this was written on (2026-09-21), 18 sessions and 8,029 messages made a 713 KB archive (3.1 MB +of database before compression). Vectors (slice 032) and any model cache are not measured yet; the slice that +adds them reads this line. diff --git a/lib/mix/tasks/trinity.export.ex b/lib/mix/tasks/trinity.export.ex new file mode 100644 index 0000000..dff9b13 --- /dev/null +++ b/lib/mix/tasks/trinity.export.ex @@ -0,0 +1,51 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Mix.Tasks.Trinity.Export do + @shortdoc "Writes the data directory to one archive (databases, registry, skills; private keys only with --keys)" + + @moduledoc """ + Slice 034 (`docs/backup.md`): + + mix trinity.export --out trinity-2026-09-21.tar.gz + mix trinity.export --out with-keys.tar.gz --keys + + The databases are snapshotted with `VACUUM INTO`, so a running Trinity is fine. Without + `--keys` the archive carries the key registry (public) and not the private key files, so it + can be handed to someone without the ability to sign as you; the manifest records which. + """ + use Boundary, classify_to: Trinity + use Mix.Task + + @impl Mix.Task + def run(argv) do + {opts, _, _} = + OptionParser.parse(argv, strict: [out: :string, keys: :boolean, data_dir: :string]) + + opts = Map.new(opts) + + unless Map.has_key?(opts, :out) do + Mix.shell().error( + "usage: mix trinity.export --out [--keys] [--data-dir ]" + ) + + exit({:shutdown, 2}) + end + + Mix.Task.run("app.config") + + layout = + if d = opts[:data_dir], + do: Trinity.Archive.Layout.of(d), + else: Trinity.Archive.Layout.current() + + case Trinity.Archive.export(layout, opts.out, keys: opts[:keys] == true) do + {:ok, %{manifest: m, bytes: bytes}} -> + Mix.shell().info( + "exported #{length(m.files)} files, #{bytes} bytes, keys #{if m.keys_included, do: "included", else: "not included"}: #{opts.out}" + ) + + {:error, reason} -> + Mix.raise("export failed: #{inspect(reason)}") + end + end +end diff --git a/lib/mix/tasks/trinity.import.ex b/lib/mix/tasks/trinity.import.ex new file mode 100644 index 0000000..a7a5a04 --- /dev/null +++ b/lib/mix/tasks/trinity.import.ex @@ -0,0 +1,75 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Mix.Tasks.Trinity.Import do + @shortdoc "Restores an archive into an empty data directory (or replaces one with --force)" + + @moduledoc """ + Slice 034 (`docs/backup.md`): + + mix trinity.import trinity-2026-09-21.tar.gz + mix trinity.import trinity-2026-09-21.tar.gz --force + mix trinity.import trinity-2026-09-21.tar.gz --data-dir /path/to/dir + + Stop Trinity first: the task refuses while a live process holds the data directory's lock. + Every digest in the manifest is verified and the schema versions are checked against this + binary before anything is written; a non-empty directory is refused unless `--force`, and + then the task prints what it replaced. Exit 2 on usage, 1 on refusal. + """ + use Boundary, classify_to: Trinity + use Mix.Task + + @impl Mix.Task + def run(argv) do + {opts, args, _} = OptionParser.parse(argv, strict: [force: :boolean, data_dir: :string]) + + case args do + [archive] -> + Mix.Task.run("app.config") + + layout = + if d = opts[:data_dir], + do: Trinity.Archive.Layout.of(d), + else: Trinity.Archive.Layout.current() + + case Trinity.DataDir.Lock.live_holder(layout.data_dir) do + {:ok, %{pid: pid, mode: mode}} -> + Mix.shell().error( + "refused: a #{mode} Trinity (OS pid #{pid}) holds #{layout.data_dir}; stop it first" + ) + + exit({:shutdown, 1}) + + :none -> + import_into(archive, layout, opts[:force] == true) + end + + _ -> + Mix.shell().error("usage: mix trinity.import [--force] [--data-dir ]") + exit({:shutdown, 2}) + end + end + + defp import_into(archive, layout, force?) do + File.mkdir_p!(layout.data_dir) + + case Trinity.Archive.import(archive, layout, force: force?) do + {:ok, %{manifest: m, written: written, replaced: replaced}} -> + Enum.each(replaced, &Mix.shell().info("replaced #{&1}")) + + Mix.shell().info( + "restored #{length(written)} files from #{archive} (created #{m.created_at}, keys #{if m.keys_included, do: "included", else: "not included"}) into #{layout.data_dir}" + ) + + {:error, {:not_empty, present}} -> + Mix.shell().error( + "refused: #{layout.data_dir} is not empty (#{Enum.join(present, ", ")}); --force replaces them" + ) + + exit({:shutdown, 1}) + + {:error, reason} -> + Mix.shell().error("refused: #{inspect(reason)}") + exit({:shutdown, 1}) + end + end +end diff --git a/lib/trinity.ex b/lib/trinity.ex index 643c9e9..d1e0354 100644 --- a/lib/trinity.ex +++ b/lib/trinity.ex @@ -28,6 +28,9 @@ defmodule Trinity do Personas, Context, Context.AgentsMd, + Archive, + Archive.Layout, + Archive.Manifest, LLM, Memory, Memory.Tokens, diff --git a/lib/trinity/archive.ex b/lib/trinity/archive.ex new file mode 100644 index 0000000..ad46e2f --- /dev/null +++ b/lib/trinity/archive.ex @@ -0,0 +1,239 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Archive do + @moduledoc """ + Export, import, restore (slice 034): the data directory as one gzip tarball a person can + take to another machine, and back. `export/3` snapshots each database with `VACUUM INTO` + (consistent, including what the WAL holds, without stopping the app), copies the key + registry and, only on the explicit flag, the private key files, and writes a manifest with + a digest per file and the schema versions. `import/3` verifies every digest against the + tarball's bytes and the schema versions against this binary's migrations before it writes + anything, refuses a non-empty target unless forced (and then says what it replaced), and + leaves a `RESTORED` marker naming the archive. + + The layout (`Trinity.Archive.Layout`) is the set of paths by role, derived from a data + directory or given explicitly, so the same functions serve the mix tasks, the settings + page and the tests, which work on their own directories. + """ + use Boundary, deps: [Trinity], exports: [Layout, Manifest] + + alias Trinity.Archive.{Layout, Manifest} + + Module.register_attribute(__MODULE__, :sobelow_skip, persist: true) + + @marker "RESTORED" + + @type export_result :: %{path: Path.t(), manifest: Manifest.t(), bytes: non_neg_integer()} + @type import_result :: %{manifest: Manifest.t(), written: [Path.t()], replaced: [Path.t()]} + + @doc """ + Writes the archive at `out`. `opts`: `keys: true` includes the private key files (the + manifest records the choice either way). + """ + # sobelow_skip reason: Traversal.FileModule: `out` is the path the owner named on the task or + # a temporary file the controller made; the temporary directory is this function's own. + @sobelow_skip ["Traversal.FileModule"] + @spec export(Layout.t(), Path.t(), keyword()) :: {:ok, export_result()} | {:error, term()} + def export(%Layout{} = layout, out, opts \\ []) do + keys? = Keyword.get(opts, :keys, false) + tmp = Path.join(System.tmp_dir!(), "trinity-export-#{System.unique_integer([:positive])}") + File.mkdir_p!(tmp) + + try do + with {:ok, staged} <- stage(layout, tmp, keys?), + manifest = Manifest.build(staged, schema_versions(layout), keys?), + :ok <- File.write(Path.join(tmp, Manifest.file_name()), Manifest.encode(manifest)), + entries = [ + {~c"manifest.json", String.to_charlist(Path.join(tmp, Manifest.file_name()))} + | staged_entries(staged) + ], + :ok <- :erl_tar.create(String.to_charlist(out), entries, [:compressed]) do + {:ok, %{path: out, manifest: manifest, bytes: File.stat!(out).size}} + end + after + File.rm_rf!(tmp) + end + end + + @doc """ + Restores an archive into a layout. `opts`: `force: true` replaces what a non-empty target + holds (listed in the result). Nothing is written until every digest and the schema versions + have been checked. + """ + # sobelow_skip reason: Traversal.FileModule: the marker is written at the layout's data + # directory, the owner's; the archive path is the owner's argument. + @sobelow_skip ["Traversal.FileModule"] + @spec import(Path.t(), Layout.t(), keyword()) :: {:ok, import_result()} | {:error, term()} + def import(archive, %Layout{} = layout, opts \\ []) do + force? = Keyword.get(opts, :force, false) + + with {:ok, entries} <- read_tarball(archive), + {:ok, manifest} <- Manifest.from_entries(entries), + :ok <- check_schema(manifest), + :ok <- Manifest.verify(manifest, entries), + {:ok, replaced} <- clear_target(layout, force?) do + written = write_files(manifest, entries, layout) + File.write!(Path.join(layout.data_dir, @marker), marker(archive, manifest)) + {:ok, %{manifest: manifest, written: written, replaced: replaced}} + end + end + + @doc "The digest the manifest uses: SHA-256, lowercase hex." + @spec digest(binary()) :: String.t() + def digest(bytes) when is_binary(bytes), + do: :crypto.hash(:sha256, bytes) |> Base.encode16(case: :lower) + + @doc "A digest over a restored layout's files, by manifest path, to compare with the manifest." + # sobelow_skip reason: Traversal.FileModule: reads the layout's own files by manifest paths + # `Layout.safe_path?/1` accepted at import. + @sobelow_skip ["Traversal.FileModule"] + @spec digest_tree(Layout.t(), Manifest.t()) :: %{String.t() => String.t()} + def digest_tree(%Layout{} = layout, %Manifest{files: files}) do + Map.new(files, fn %{"path" => rel} -> + abs = Layout.absolute(layout, rel) + {rel, if(File.regular?(abs), do: digest(File.read!(abs)), else: nil)} + end) + end + + @doc "The migration versions this binary carries for a repo, from its migration files." + @spec available_versions(module()) :: [integer()] + def available_versions(repo) do + repo + |> Ecto.Migrator.migrations_path() + |> Path.join("*.exs") + |> Path.wildcard() + |> Enum.map(fn f -> + f |> Path.basename() |> String.split("_", parts: 2) |> hd() |> String.to_integer() + end) + |> Enum.sort() + end + + @doc "The marker file's name." + @spec marker_file() :: String.t() + def marker_file, do: @marker + + ## Export + + # Each database is snapshotted with VACUUM INTO through exqlite directly: no Ecto, no + # transaction, so it runs beside a live app and inside a test that must see disk. + # sobelow_skip reason: Traversal.FileModule: the sources are the layout's (the owner's data + # directory by role) and the destinations are under this export's own temporary directory. + @sobelow_skip ["Traversal.FileModule"] + defp stage(layout, tmp, keys?) do + Enum.reduce_while(Layout.sources(layout, keys?), {:ok, []}, fn {rel, abs, kind}, {:ok, acc} -> + dest = Path.join(tmp, rel) + File.mkdir_p!(Path.dirname(dest)) + + result = + case kind do + :sqlite -> snapshot(abs, dest) + :file -> File.cp(abs, dest) + end + + case result do + :ok -> + {:cont, + {:ok, + acc ++ + [ + %{ + path: rel, + staged: dest, + bytes: File.stat!(dest).size, + sha256: digest(File.read!(dest)) + } + ]}} + + {:error, reason} -> + {:halt, {:error, {:stage, rel, reason}}} + end + end) + end + + defp snapshot(src, dest) do + with {:ok, db} <- Exqlite.Sqlite3.open(src), + :ok <- Exqlite.Sqlite3.execute(db, "VACUUM INTO '#{String.replace(dest, "'", "''")}'") do + Exqlite.Sqlite3.close(db) + end + end + + defp staged_entries(staged), + do: Enum.map(staged, &{String.to_charlist(&1.path), String.to_charlist(&1.staged)}) + + # The migrated versions of each database, read from the source file itself. + defp schema_versions(layout) do + for {name, path, table} <- [ + {"trinity.db", layout.db, "schema_migrations"}, + {"receipts.db", layout.receipts_db, "receipts_schema_migrations"} + ], + File.regular?(path), + into: %{} do + {name, migrated(path, table)} + end + end + + defp migrated(path, table) do + with {:ok, db} <- Exqlite.Sqlite3.open(path), + {:ok, st} <- Exqlite.Sqlite3.prepare(db, "SELECT version FROM #{table} ORDER BY version"), + {:ok, rows} <- Exqlite.Sqlite3.fetch_all(db, st), + :ok <- Exqlite.Sqlite3.release(db, st), + :ok <- Exqlite.Sqlite3.close(db) do + List.flatten(rows) + else + _ -> [] + end + end + + ## Import + + defp read_tarball(archive) do + case :erl_tar.extract(String.to_charlist(archive), [:compressed, :memory]) do + {:ok, entries} -> {:ok, Map.new(entries, fn {name, bin} -> {List.to_string(name), bin} end)} + {:error, reason} -> {:error, {:archive, reason}} + end + end + + # A newer archive into an older binary refuses by name; the other way migrates on boot. + defp check_schema(%Manifest{schema_versions: versions}) do + binary = %{ + "trinity.db" => available_versions(Trinity.Repo), + "receipts.db" => available_versions(Trinity.Repo.Receipts) + } + + newer = + for {db, archived} <- versions, + have = Map.get(binary, db, []), + extra = archived -- have, + extra != [], + do: {db, extra} + + if newer == [], do: :ok, else: {:error, {:schema_newer_than_binary, newer}} + end + + defp clear_target(layout, force?) do + present = Layout.present(layout) + + cond do + present == [] -> {:ok, []} + force? -> Enum.each(present, &File.rm_rf!/1) && {:ok, present} + true -> {:error, {:not_empty, present}} + end + end + + # sobelow_skip reason: Traversal.FileModule: every path was accepted by `Layout.safe_path?/1` + # in `Manifest.verify/2` before this runs, and lands under the layout's directories. + @sobelow_skip ["Traversal.FileModule"] + defp write_files(%Manifest{files: files}, entries, layout) do + for %{"path" => rel} <- files do + abs = Layout.absolute(layout, rel) + File.mkdir_p!(Path.dirname(abs)) + File.write!(abs, Map.fetch!(entries, rel)) + if String.starts_with?(rel, "keys/"), do: File.chmod!(abs, 0o600) + abs + end + end + + defp marker(archive, %Manifest{} = m) do + "restored from #{Path.basename(archive)} (created #{m.created_at}, format #{m.format}, keys #{if m.keys_included, do: "included", else: "not included"}) at #{DateTime.utc_now() |> DateTime.to_iso8601()}\n" + end +end diff --git a/lib/trinity/archive/layout.ex b/lib/trinity/archive/layout.ex new file mode 100644 index 0000000..5a675fd --- /dev/null +++ b/lib/trinity/archive/layout.ex @@ -0,0 +1,147 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Archive.Layout do + @moduledoc """ + The files of a data directory by role (slice 034): the two databases, the key registry + and the private key files under `keys/`, and the `skills/` and `personas/` directories + (empty at this slice; 040 fills skills). `of/1` derives it from a directory; the tests and + the tasks may give the paths themselves. + """ + + @type t :: %__MODULE__{ + data_dir: Path.t(), + db: Path.t(), + receipts_db: Path.t(), + keys_dir: Path.t(), + skills_dir: Path.t(), + personas_dir: Path.t() + } + + @enforce_keys [:data_dir, :db, :receipts_db, :keys_dir] + defstruct [:data_dir, :db, :receipts_db, :keys_dir, :skills_dir, :personas_dir] + + @doc "The layout of a data directory." + @spec of(Path.t()) :: t() + def of(data_dir) do + %__MODULE__{ + data_dir: data_dir, + db: Path.join(data_dir, "trinity.db"), + receipts_db: Path.join(data_dir, "receipts.db"), + keys_dir: Path.join(data_dir, "keys"), + skills_dir: Path.join(data_dir, "skills"), + personas_dir: Path.join(data_dir, "personas") + } + end + + @doc "The running application's layout: its configured databases and keys directory." + @spec current() :: t() + def current do + data_dir = Trinity.Paths.data_dir() + + %__MODULE__{ + data_dir: data_dir, + db: + Application.get_env(:trinity, Trinity.Repo)[:database] || + Path.join(data_dir, "trinity.db"), + receipts_db: + Application.get_env(:trinity, Trinity.Repo.Receipts)[:database] || + Path.join(data_dir, "receipts.db"), + keys_dir: + Application.get_env(:trinity, :receipts, [])[:keys_dir] || Path.join(data_dir, "keys"), + skills_dir: Path.join(data_dir, "skills"), + personas_dir: Path.join(data_dir, "personas") + } + end + + @doc """ + What an export takes, as `{archive path, absolute path, kind}`: the databases (`:sqlite`, + snapshotted), the registry and, with `keys?`, the private key files, and every file under + `skills/` and `personas/` (`:file`, copied). Only what exists. + """ + @spec sources(t(), boolean()) :: [{String.t(), Path.t(), :sqlite | :file}] + def sources(%__MODULE__{} = l, keys?) do + dbs = + for {rel, abs} <- [{"trinity.db", l.db}, {"receipts.db", l.receipts_db}], + File.regular?(abs), + do: {rel, abs, :sqlite} + + registry = Path.join(l.keys_dir, "registry.json") + reg = if File.regular?(registry), do: [{"keys/registry.json", registry, :file}], else: [] + + keys = + if keys?, + do: + for( + f <- Path.wildcard(Path.join(l.keys_dir, "receipts-*.key")), + do: {"keys/" <> Path.basename(f), f, :file} + ), + else: [] + + dirs = + for {name, dir} <- [{"skills", l.skills_dir}, {"personas", l.personas_dir}], + is_binary(dir) and File.dir?(dir), + f <- Path.wildcard(Path.join(dir, "**/*")), + File.regular?(f), + do: {Path.join(name, Path.relative_to(f, dir)), f, :file} + + dbs ++ reg ++ keys ++ dirs + end + + @doc """ + True for a manifest path this layout accepts: one of the known files, or a relative path + under `keys/`, `skills/` or `personas/` with no `..` segment and no absolute part. An + archive's manifest is data; a path it names lands nowhere the layout did not say. + """ + @spec safe_path?(String.t()) :: boolean() + def safe_path?(path) when is_binary(path) do + segments = Path.split(path) + + cond do + path in ["trinity.db", "receipts.db"] -> true + Path.type(path) != :relative -> false + Enum.any?(segments, &(&1 in ["..", "."])) -> false + Enum.any?(segments, &String.contains?(&1, "\\")) -> false + hd(segments) in ["keys", "skills", "personas"] and length(segments) > 1 -> true + true -> false + end + end + + def safe_path?(_), do: false + + @doc "The absolute path a manifest path lands at in this layout (for a path `safe_path?/1` accepts)." + @spec absolute(t(), String.t()) :: Path.t() + def absolute(%__MODULE__{} = l, "trinity.db"), do: l.db + def absolute(%__MODULE__{} = l, "receipts.db"), do: l.receipts_db + def absolute(%__MODULE__{} = l, "keys/" <> rest), do: Path.join(l.keys_dir, rest) + + def absolute(%__MODULE__{} = l, "skills/" <> rest), + do: Path.join(l.skills_dir || Path.join(l.data_dir, "skills"), rest) + + def absolute(%__MODULE__{} = l, "personas/" <> rest), + do: Path.join(l.personas_dir || Path.join(l.data_dir, "personas"), rest) + + def absolute(%__MODULE__{} = l, other), do: Path.join(l.data_dir, other) + + @doc "What is present of this layout (the files and directories an import would replace)." + @spec present(t()) :: [Path.t()] + def present(%__MODULE__{} = l) do + files = + for p <- [ + l.db, + l.db <> "-wal", + l.db <> "-shm", + l.receipts_db, + l.receipts_db <> "-wal", + l.receipts_db <> "-shm" + ], + File.regular?(p), + do: p + + dirs = + for d <- [l.keys_dir, l.skills_dir, l.personas_dir], + is_binary(d) and File.dir?(d) and File.ls!(d) != [], + do: d + + files ++ dirs + end +end diff --git a/lib/trinity/archive/manifest.ex b/lib/trinity/archive/manifest.ex new file mode 100644 index 0000000..2706d2a --- /dev/null +++ b/lib/trinity/archive/manifest.ex @@ -0,0 +1,123 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule Trinity.Archive.Manifest do + @moduledoc """ + The archive's manifest (slice 034): the format, this build's version, when it was made, + the migrated schema versions of each database, whether the private keys are inside, and + one entry per file with its size and SHA-256. It is the first thing an import reads and + the thing every write is checked against. + """ + + @format "trinity-archive/1" + @file_name "manifest.json" + + @type t :: %__MODULE__{ + format: String.t(), + app_version: String.t(), + created_at: String.t(), + schema_versions: %{String.t() => [integer()]}, + keys_included: boolean(), + files: [%{String.t() => term()}] + } + + defstruct format: @format, + app_version: nil, + created_at: nil, + schema_versions: %{}, + keys_included: false, + files: [] + + @doc "The manifest's file name inside the archive." + @spec file_name() :: String.t() + def file_name, do: @file_name + + @doc "The format string." + @spec format() :: String.t() + def format, do: @format + + @doc "Builds a manifest from staged files (`%{path, bytes, sha256}`)." + @spec build([map()], map(), boolean()) :: t() + def build(staged, schema_versions, keys?) do + %__MODULE__{ + app_version: to_string(Application.spec(:trinity, :vsn)), + created_at: DateTime.utc_now() |> DateTime.to_iso8601(), + schema_versions: schema_versions, + keys_included: keys?, + files: Enum.map(staged, &%{"path" => &1.path, "bytes" => &1.bytes, "sha256" => &1.sha256}) + } + end + + @doc "JSON." + @spec encode(t()) :: String.t() + def encode(%__MODULE__{} = m) do + JSON.encode!(%{ + "format" => m.format, + "app_version" => m.app_version, + "created_at" => m.created_at, + "schema_versions" => m.schema_versions, + "keys_included" => m.keys_included, + "files" => m.files + }) + end + + @doc "From JSON; refuses another format." + @spec decode(binary()) :: {:ok, t()} | {:error, term()} + def decode(bin) do + case JSON.decode(bin) do + {:ok, %{"format" => @format} = m} -> + {:ok, + %__MODULE__{ + format: @format, + app_version: m["app_version"], + created_at: m["created_at"], + schema_versions: m["schema_versions"] || %{}, + keys_included: m["keys_included"] == true, + files: m["files"] || [] + }} + + {:ok, %{"format" => other}} -> + {:error, {:unknown_format, other}} + + {:ok, _} -> + {:error, :not_a_manifest} + + {:error, reason} -> + {:error, {:manifest_json, reason}} + end + end + + @doc "The manifest out of an extracted tarball's entries." + @spec from_entries(%{String.t() => binary()}) :: {:ok, t()} | {:error, term()} + def from_entries(entries) do + case Map.fetch(entries, @file_name) do + {:ok, bin} -> decode(bin) + :error -> {:error, :no_manifest} + end + end + + @doc "Every file named in the manifest is in the entries with the manifest's digest and size; nothing else is required." + @spec verify(t(), %{String.t() => binary()}) :: :ok | {:error, term()} + def verify(%__MODULE__{files: files}, entries) do + # Not a comprehension filter: a `nil` bound in a `for` filter skips the element, and a + # missing file is exactly what must not be skipped (found by AC4's second half). + bad = Enum.flat_map(files, &problem(&1, Map.get(entries, &1["path"]))) + if bad == [], do: :ok, else: {:error, {:verification_failed, bad}} + end + + # A path the layout would not accept is refused first: the manifest is data from the + # archive, and `keys/../x` must never be resolved, let alone written. + defp problem(%{"path" => path}, _bin) when not is_binary(path), + do: [{inspect(path), :unsafe_path}] + + defp problem(%{"path" => path} = file, bin) do + if Trinity.Archive.Layout.safe_path?(path), do: check(file, bin), else: [{path, :unsafe_path}] + end + + defp check(%{"path" => path}, nil), do: [{path, :missing}] + + defp check(%{"path" => path, "sha256" => sha, "bytes" => bytes}, bin) do + if byte_size(bin) == bytes and Trinity.Archive.digest(bin) == sha, + do: [], + else: [{path, :digest_mismatch}] + end +end diff --git a/lib/trinity/data_dir/lock.ex b/lib/trinity/data_dir/lock.ex index f977a16..022d7da 100644 --- a/lib/trinity/data_dir/lock.ex +++ b/lib/trinity/data_dir/lock.ex @@ -196,4 +196,13 @@ defmodule Trinity.DataDir.Lock do rescue _ -> true end + + @doc "The live holder of `dir`, if a live process holds it (slice 034: the import task asks before it writes)." + @spec live_holder(Path.t()) :: {:ok, holder()} | :none + def live_holder(dir) do + case holder(dir) do + {:ok, %{pid: pid} = h} -> if alive?(pid), do: {:ok, h}, else: :none + _ -> :none + end + end end diff --git a/lib/trinity_web/controllers/export_controller.ex b/lib/trinity_web/controllers/export_controller.ex new file mode 100644 index 0000000..4350978 --- /dev/null +++ b/lib/trinity_web/controllers/export_controller.ex @@ -0,0 +1,37 @@ +# SPDX-FileCopyrightText: Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +defmodule TrinityWeb.ExportController do + @moduledoc """ + `GET /settings/export.tar.gz[?keys=1]` (slice 034): the running Trinity's archive as a + download, made the same way the mix task makes it. Private keys only with `keys=1`. + """ + use TrinityWeb, :controller + + Module.register_attribute(__MODULE__, :sobelow_skip, persist: true) + + # sobelow_skip reason: Traversal.FileModule and Traversal.SendDownload: the file sent and + # removed is the temporary archive this action just made, at a path it chose; nothing in + # the request names a path. + @sobelow_skip ["Traversal.FileModule", "Traversal.SendDownload"] + def download(conn, params) do + keys? = params["keys"] in ["1", "true"] + stamp = DateTime.utc_now() |> DateTime.to_iso8601(:basic) |> String.slice(0, 15) + name = "trinity-#{stamp}#{if keys?, do: "-with-keys", else: ""}.tar.gz" + + out = + Path.join( + System.tmp_dir!(), + "trinity-download-#{System.unique_integer([:positive])}.tar.gz" + ) + + case Trinity.Archive.export(Trinity.Archive.Layout.current(), out, keys: keys?) do + {:ok, _} -> + conn = send_download(conn, {:file, out}, filename: name, content_type: "application/gzip") + File.rm(out) + conn + + {:error, reason} -> + conn |> put_status(500) |> text("export failed: #{inspect(reason)}") + end + end +end diff --git a/lib/trinity_web/live/session_live/index.ex b/lib/trinity_web/live/session_live/index.ex index 450e30c..7eb4234 100644 --- a/lib/trinity_web/live/session_live/index.ex +++ b/lib/trinity_web/live/session_live/index.ex @@ -66,6 +66,9 @@ defmodule TrinityWeb.SessionLive.Index do <.link navigate={~p"/personas"} class="underline opacity-70">{gettext("edit")} <.link navigate={~p"/memory"} class="underline opacity-70">{gettext("memory")} + <.link id="settings-link" navigate={~p"/settings"} class="underline opacity-70">{gettext( + "settings" + )}