System architecture · v1 kernel complete · updated 2026-07-04

B2 — how the system is built

B2 is a local-first Markdown vault with an AI layer that surfaces semantically similar, not-yet-linked notes for you to connect. This repo is the index engine + CLI: a SQLite store treated as a disposable projection of your Markdown — nothing durable outside your notes — a real local embedder behind the one AI seam, and the typed Vault API the b2 binary drives. This page traces the whole architecture through one real vault — the two-note fixture the test suite runs on — so every claim is concrete and checkable.

model-free · deterministic suite 4 crates: b2-core · b2-embed · b2-cli · b2-desktop Rust 1.96 · edition 2021 v1 kernel: reindex · search · neighbors · explain · add · mv · similar · link

The suite is integration-first: most tests open a real SQLite database, ingest a real vault on disk, and assert on the resulting projection — so it describes the built system, not mocks of it. One additional #[ignore] test exercises the real embedder and runs out of CI. The LLM relator was cut 2026-07-04 (per-pair cost didn't scale) — its crate, seam, and suggestion queue are gone; connection discovery is now local, free, and instant.

The governing equation

One equation shapes every decision. Keep only this:

index = projection of ( Markdown )
Markdown vault → ingest → b2.sqlite (disposable) search · neighbors · explain · similar

Delete b2.sqlite, rebuild it from the Markdown — it must come back identical. Exactly one thing is durable and un-derivable: your Markdown (the source of truth for both knowledge and every committed connection). There is no state anywhere outside your notes. Everything downstream is this equation defended. (Through 2026-06-30 a third tier — a durable .b2/log/ event log holding the suggestion queue + rejection memory — closed an ∪ log term; the 2026-07-04 relator cut removed the only thing it was for, so the log tier is gone and the equation simplified.)

Tier 1 · Markdown — source of truth

Notes + every committed edge, plain .md, fully usable in Obsidian with no B2. Stays pristine — B2's only writes are surgical and in frontmatter; the body is 100% the human's.

Tier 2 · b2.sqlite — disposable cache

FTS5 + sqlite-vec + the typed graph. Disposable: holds nothing that can't be reconstructed from the Markdown. Drop it, reindex, get an identical index.

A whole vault is one portable folder: the index lives under <root>/.b2/, a dotfolder Obsidian and the vault scanner both ignore. Point B2 at a folder of Markdown — that is the entire setup.

Four crates, cleanly layered

The engine owes nothing to the two adapters above it or the one model beside it. The AI-heavy crate sits behind the Embedder trait defined in b2-core, so the engine and its whole fast suite never import a tensor — and adding the desktop UI added one adapter, not new architecture.

b2-core · the engine

Turns a folder of Markdown into a queryable index and keeps it a pure function of disk. Model-free (no candle, no network) so its suite is fast and deterministic. rusqlite (bundled SQLite + FTS5), sqlite-vec, blake3, ulid, yaml.

  • note · ingest · chunk · link · relation
  • db · graph · search · discover
  • embed — the one seam + fake
  • add · mv · vault (the façade)
b2-cli · the adapter

The b2 binary. Holds no engine logic: parse args, pick + inject the embedder, call the Vault façade, print (human, or --json for agents). Funnels every error through user_message so nothing internal leaks.

b2-desktop · the GUI adapter

The Tauri host — the GUI sibling of b2-cli, equally logic-free: each #[tauri::command] is deserialize → one Vault call → serialize, reusing the CLI's --json view types as the IPC contract. The ui/ frontend (Vite + vanilla TS) renders the read → discover → link loop; two adapters over one façade can't drift.

b2-embed · the real embedder

LocalEmbedder: candle-backed BERT, BAAI/bge-base-en-v1.5 (768-dim, ungated), pure-Rust inference. b2 init downloads + verifies it into a shared XDG cache. Implements embed::Embedder — the only AI seam left.

Ground truth: the layering is enforced by where deps live. b2-core's Cargo.toml has no candle and no HTTP client; the whole engine suite runs on the deterministic fake embedder. The real model is reached only by b2 init + the out-of-CI retrieval eval. Cut 2026-07-04: a former crate, b2-relate (the Claude-backed relator), was removed with the LLM discovery pipeline — see Flow ③.

One vault, traced end to end

Everything below follows this exact vault — the hand-authored golden fixture the tests run on. Two notes, engineered to exercise both edge kinds at once. Watch what each layer does to them.

concepts/memory.md
---
b2id: 01JMEM0000000000000000000A
type: concept
title: "Human memory"
created: 2026-06-20
---
The brain encodes, stores, and retrieves information…
notes/spaced-repetition.md
---
b2id: 01JSRS0000000000000000000B
type: concept
title: "Spaced repetition"
created: 2026-06-20
---
Spaced repetition exploits the [[concepts/memory|Human memory]]
retrieval curve.

## Relations
- elaborates [[concepts/memory|Human memory]] — applies the forgetting curve

The prose [[…]] is a bare, untyped references edge. The - elaborates [[…]] — … list item is a typed edge with an explanation. Both are human-authored (origin=inline) — B2 reads them, never writes them. One ingest yields a known-good graph the tests match whole.

FLOW ①  Ingest — vault becomes index

crates/b2-core/src/ingest.rs

b2 reindex is the write path that realizes the governing equation. Its one subtlety: a note can link to one the walker hasn't reached yet, so link resolution must not depend on file order. Ingest runs in two phases.

walk *.md (dotfolders skipped, sorted) Phase 1 · notes + chunks + vectors → fills the resolver → Phase 2 · edges
  • Phase 1 projects each note's frontmatter, chunks its body (a minimal paragraph splitter, one chunk per maximal run of non-blank lines), and embeds each chunk through the seam. This fills notes(b2id, path) — the path ⇄ b2id resolver — for the whole vault. Our two notes yield one chunk each; memory.md's chunk lands in chunks_vec ready to be found.
  • Phase 2 derives edges by parsing each body + frontmatter relations: and resolving every [[path]] against the now-complete resolver. From spaced-repetition.md that's two directed edges to memory.md: a bare references and a typed elaborates. A typed line consumes its wikilink, so it isn't double-counted as a bare reference.
  • If a note lacks a b2id, phase 1 stamps one — mint a ULID, write a single surgical line into the note's frontmatter. That's the whole record: the id travels in the file, so there is no separate log to keep. B2's one always-allowed write during ingest. (Both fixture notes are pre-stamped, so nothing is written here.)

Incremental by default, ≡ a full rebuild. A note whose body_hash is unchanged and whose chunks all still have vectors reuses them verbatim — only edited notes re-embed. Both paths replace rather than accumulate: re-projecting a note deletes its old chunks (FTS triggers fire, stale chunks_vec rows are cleared) and all its edges, then re-derives every edge from the current Markdown (body inline ∪ frontmatter). Because every edge is Markdown-derived, a one-note re-parse is exactly a one-note slice of a full rebuild — there is no log term to reconcile. b2 reindex --dry-run previews the same decision, writing nothing.

Ground truth: graph.rs · one_note_reindex_equals_full — incremental re-index of one note yields a byte-identical edge set to a full rebuild, and a second reindex is identical again (idempotent). chunks.rs proves replace-not-accumulate; ingest_resolver.rs confirms the two-phase resolver works both directions; dry_run.rs proves the preview writes nothing. ↳ the indexing deep dive covers incremental skip, --force, batching.

The schema — a derived projection

Every table is a projection of (Markdown); nothing here is a source of truth. The schema is derived from the data model — it must satisfy the model, never the reverse.

TableHoldsDerived from
notesb2id (PK), path (UNIQUE), type, title, created/updated, body_hash — the resolver is this tableMarkdown
chunksper-note paragraph chunks with char offsets + token_countMarkdown
chunks_ftsFTS5 BM25 index, kept in lockstep by triggersderived (FTS5)
chunks_vecvec0 KNN table, created at the embedder's dimnot in the base migrationderived (sqlite-vec)
edgestyped graph: src→dst, type, origin∈{inline,frontmatter}, dst_id (nullable → dangling), explanation — every edge authored + activeMarkdown
metaschema_version, embed_model_id, embed_dimbookkeeping

Two schema details do load-bearing work. There is no status column and no lifecycle — an edge exists iff it is written in the Markdown, so origin only records which home it came from (inline body link, or a frontmatter relations: entry). And chunks_vec is created outside the base migration because its FLOAT[N] dimension is a DDL literal pinned to meta.embed_dim: the vector space can only be shaped once the embedder's dim is known. (Removed 2026-07-04: the edge_provenance table, the status column, the origin='suggested' value, and the origin/status CHECK — all gone with the LLM relator.)

The substrate bet. One embedded store answers every query at once — FTS5, vectors, and the typed graph — so b2 similar can join all three in one transaction. That only works if FTS5 and sqlite-vec share one connection: sqlite-vec is registered as a static auto-extension (no runtime load_extension), so the whole thing is one statically-linked binary. substrate.rs proves a BM25 query and a vector KNN run against the same connection.

FLOW ③  Similar — surface the connections you haven't made

crates/b2-core/src/discover.rs

b2 similar <note> surfaces the notes most semantically similar to an anchor that you haven't linked yet — instantly, locally, free: a vector KNN over the embeddings already in chunks_vec, with no model call and no network. The machine finds the candidates; you supply the judgment and the type (Flow ③ cont.). This is the reason B2 exists — and after the 2026-07-04 pivot it costs nothing per run.

The one idea — a candidate is near ∖ connected

A candidate is a note semantically near the anchor but not already connected — the complement of the graph, not the intersection. Mechanism, per anchor, in discover::candidates:

anchor's stored chunk vectors KNN over chunks_vec max-sim per candidate note reachable_within(anchor, 1) rank → top-N (default ~10)
  • No re-embedding, no model call. Each anchor chunk's already-stored vector is the KNN query — discovery is passage↔passage, so the anchor is its indexed vectors, never an embed_query whose asymmetric prefix is the wrong side. The run reads only the vectors reindex already wrote.
  • Max-sim to the note. Every other note scores by its single best chunk-pair similarity, carrying the winning chunk as the evidence passage B2 shows you.
  • Subtract the graph — the only graph read. reachable_within(anchor, 1) (anchor + direct neighbors) is removed. Distance is exclusion-only: a 2-hop note (triadic closure — "you linked A–B and B–E but never A–E") survives, ranked purely by semantic score.

In our fixture the two notes are directly linked, so each is excluded as the other's candidate — a correctly empty result, which is exactly the assertion discover.rs pins.

what b2 similar shows you per candidate
  • path + title — the note to consider connecting to.
  • similarity score — best chunk-pair max-sim (negated sqlite-vec distance; higher is nearer).
  • evidence passage — the candidate's chunk that achieved the score, so you can judge the link at a glance.

You are the precision gate. There is no LLM adjudicating pairs and nothing is proposed on your behalf — B2 ranks, you decide, and you commit the ones worth a real connection with b2 link (or a body link of your own).

Ground truth: discover.rs pins the complement — direct neighbor excluded, disconnected near note kept, 2-hop note survives, deterministic ranking (ties broken by b2id), and an empty result when the anchor has no vector space yet. (Removed 2026-07-04: the Relator seam, the ClaudeRelator, the generate/suggest pipeline, and their tests — the per-pair LLM cost didn't scale to a 1000+ note vault. Candidate generation is all that remains, and it was always the free, deterministic stage.)the discovery run deep dive walks a full b2 similar.

The write discipline — losslessness & surgical edits

crates/b2-core/src/note.rs · add.rs · mv.rs

The vault stays the human's. A parsed note keeps its raw text verbatim and records only the byte spans of the frontmatter block; serialization returns those raw bytes. So parse → serialize → parse is byte-identical — unknown keys, comments, odd whitespace, a missing final newline all survive. Against that backdrop, B2 makes only four narrow writes:

Frontmatter, surgical
  • stamp_b2id — one b2id: line; strict no-op if present, never re-stamped.
  • add_relation — one YAML-quoted typed-link appended to relations: (the b2 link path).
Whole-file ops (CRUD)
  • b2 add — create a note (minimal frontmatter, no b2id; ingest stamps it on first sight).
  • b2 mv — move/rename + rewrite inbound [[oldpath]] link text.
b2 mv never breaks the graph. Edges key on b2id, and a move leaves the target's id untouched — so neighbors/backlinks show the same set before and after. Only the human convenience-copy [[oldpath|alias]] text is repaired, and only in the exact inbound files the materialized graph names (db::inbound_edge_targets — bounded, never an O(vault) scan). A byte-preserving rewrite swaps only the target token; a prefix-sharing [[foo-bar]] is never touched when moving foo.
Ground truth: roundtrip.rs pins losslessness; stamp.rs · frontmatter.rs pin the surgical writes; mv.rs (15 tests) pins graph-unchanged, byte-exact inbound diffs, prefix-safety, and clean errors on clobber/invalid dest; add.rs pins stamp-on-create and clobber refusal.

The one AI seam — a fake for CI, a real model for users

The one AI part left sits behind a swappable trait defined in b2-core. The engine is built and tested against a deterministic fake; a real model drops in through the identical seam with no schema or flow change. This is the "build for tomorrow's model" tenet made mechanical — and it is why the whole suite runs with no live model, no wall clock, and no random ids.

Embedder — embed::Embedder

Fake: FakeEmbedder blake3-hashes text into a vector — identical text → identical vector, deterministically. KNN is reproducible; not semantic. Real: b2-embed::LocalEmbedder, candle BERT, CLS-pool + L2-normalize (so sqlite-vec's L2 ranks by cosine), asymmetric query prefix, dim read from the model's own config.json.

The swap is safe by construction. meta records (embed_model_id, embed_dim); on a model/dim change chunks_vec is dropped + recreated empty so a full re-embed follows. Opening a vault never mutates the vector space; a stale search fails fast with ModelMismatch rather than returning silently-wrong results.

The relator seam was cut 2026-07-04. B2 briefly had a second seam — a Relator (real: a Claude-backed ClaudeRelator in b2-relate) that typed and explained every candidate pair. On a real 1000+ note vault the per-pair latency and dollar cost didn't scale, and a per-pair LLM adjudicator is exactly the "model-compensating machinery" the Bitter-Lesson tenet says to defer. So the seam, the crate, and the suggestion queue are gone; the human is the precision gate. If a seam lands next it's a reranker (post-fusion, changes ordering not the store) — not a return of the relator.
the relation vocabulary — closed core + tolerated tail · relation.rs
  • references → referenced-by
  • relates ↔ (symmetric)
  • elaborates → elaborated-by
  • supports → supported-by
  • refutes → refuted-by
  • contradicts ↔ (symmetric)
  • example-of → has-example
  • part-of → has-part
  • supersedes → superseded-by
  • derived-from → source-of

A closed 10-verb core is now your typing palette on b2 link --type and all queries rely on it; each ships a display-only inverse label and a symmetry flag (inbound neighbors render the inverse; only one directed row is ever stored). A tail verb a human writes (e.g. inspired-by) is kept verbatim, never dropped, treated as opaque. Expressiveness lives in the tail; reliability lives in the core.

The command surface — a dumb adapter over one typed façade

crates/b2-core/src/vault.rs · crates/b2-cli/src/main.rs

Every engine module is called directly only by the integration tests. The Vault façade is the single typed entry point the CLI (and any future GUI) is the sole client of — it owns the connection, the injected embedder, and the id generator, and returns display-ready view structs (NeighborView, SearchResult, ExplainView, the discovery candidate view, …). The b2 binary parses args, picks the embedder, calls the façade, and prints — human, or --json for agents.

CommandDoesModel needed
initdownload + verify the embedder into the shared cache
reindexre-project the vault (incremental; --force, --dry-run)embedder
addcreate a note, project it (stamps the b2id)embedder
mvmove/rename + repair inbound link textembedder
searchhybrid keyword ⊕ semantic ⊕ graphembedder
neighborsa note's typed neighbors— (pure graph)
explainevery typed edge + its "why", with an orphan flag— (pure graph)
similarsurface the nearest not-yet-linked notes (path/title/score/evidence)— (stored vectors)
linkcommit a chosen connection to frontmatter relations:, re-project itembedder
Honest by construction. A note ref resolves by path or b2id. Commands that embed fail fast with "run b2 init" if the model is absent; pure-graph commands (and similar, which reads the vectors already stored) need no live call. B2_EMBEDDER=fake forces the fake everywhere (offline/dev, and what the suite runs under) — under the fake the CLI prints a loud caveat on stderr, keeping --json stdout pure data. B2_DEBUG opts into internal error detail; otherwise errors are generic and never leak sqlite/io/serde.
Ground truth: vault.rs pins the façade — open creates .b2/, reindex is idempotent and reports counts, all ref forms resolve to the same set, search is note-level with a snippet. cli.rs drives the built binary for every command, both human and --json, clean non-zero exits on unknown notes/ids, and that the not-yet-semantic caveat rides stderr — including the similarlink discovery flow and its idempotent, invalid-verb, and both-directions cases.

The invariants that hold it together

Almost every choice above serves one of these. Each is pinned by the suite.

InvariantWhat it means · where it's defended
Lossless round-tripparse → serialize → parse is byte-identical; B2 only ever makes surgical edits. → roundtrip.rs
B2 never authors the bodyWrites are surgical and in frontmatter: the b2id stamp, and a committed edge in relations: via b2 link. → stamp · frontmatter
Idempotent re-projectionRe-ingest changes nothing; an incremental one-note re-index equals a full rebuild; no stale FTS/vector rows accumulate. → chunks · graph · embed
Disposable indexDrop the DB, rebuild from Markdown → an identical index. Every committed edge returns because it lives in the Markdown; there is no log to replay. → graph · ingest
The graph never breaks on a moveEdges key on b2id, so a rename repairs only link text — backlinks resolve unchanged. → mv
index = projection of (Markdown)No durable state outside your notes; an edge exists iff it is authored in the Markdown (body or frontmatter). → graph · ingest

Grounded in the tests

Nothing here is aspirational: the suite is the executable specification, integration-first, and every property above is asserted in it. Most tests share the golden vault, copied into a temp dir before every run so ingest never mutates the committed fixture, with two fixed b2ids so exact ids are assertable. Determinism is a hard requirement of the core — no wall clock, no randomness; timestamps and ids are passed in.

Coverage at a glance — model-free, all green (129 tests under cargo test)
PropertyPinned bytests
Substrate — FTS5 + sqlite-vec coexist; pragmas persistsubstrate.rs2
Lossless parse · b2id stamp · resolverroundtrip · stamp · ingest_resolver10
Chunks (+FTS) · link classification · typed graphchunks · links · graph15
Vectors & the embedder seamembed.rs8
Hybrid retrieval — RRF + graph⨝vector joinsearch.rs8
Candidate generation (near ∖ connected)discover.rs7
Committing a connection — b2 link → frontmatterfrontmatter · graph10
Note-authoring CRUD — add · mv · explain · dry-runadd · mv · explain · dry_run · pathspec39
The Vault façade · the b2 CLI (similar/link included)vault.rs · cli.rs31
Embedder config & provisioningb2-embed4

Run under cargo test (rustc 1.96, the fake embedder). One #[ignore] test — a batched-embed equality check (b2-embed) — exercises the real model out of CI. Open any file in the right column; it is the ground truth this page paraphrases.

Not yet built

An honest architecture names its boundaries. Connection discovery is built — it is now b2 similar (surface) + b2 link (commit), local and free; the LLM relator was cut, not deferred. What remains is tuning, scale, and packaging — tracked in planning/tasks.md, not gaps that slipped through.

  • Semantic quality in CI. The engine suite runs on the deterministic fake embedder, so it proves plumbing, not ranking quality. The real model is measured by one separate out-of-CI eval — retrieval precision/MRR (cargo run -p b2-embed --example eval). Model quality never flakes CI.
  • Cold-index scale. Incremental reindex is done; the first embed of a large vault still blocks. Deferred approaches — a background reindex + b2 status, a keyword-first progressive index, a smaller/quantized embedder — all compose behind the existing seam.
  • Deferred, behind clean seams. A cross-encoder reranker (the likely next seam — changes ordering, not the store), query expansion, the qmd-style ~900-token overlap-aware chunker (a pure re-projection — the chunks schema already carries token_count + heading_path), and the actual packaging/distribution build.

Describes the repository as of the 2026-07-04 pivot — the v1 kernel across three crates, discovery as similar + link. Source of truth for every claim: crates/{b2-core,b2-embed,b2-cli}/src/ and the test suite under each crate's tests/. Design rationale lives in planning/ (data-model.md · index-engine.md · specs/). Next phase: the first UI — a Tauri + CodeMirror desktop app (a fourth crate, b2-desktop, as the second dumb adapter over the façade), speced in planning/specs/completed/desktop-ui-mvp.md. · Home · Quick start · Indexing pipeline · Connection discovery