B2 — how the system is built
B2 is a local-first Markdown vault with an AI layer that surfaces semantically similar,
not-yet-linked notes for you to connect. This repo is the index engine + CLI: a SQLite
store treated as a disposable projection of your Markdown — nothing durable outside your notes — a
real local embedder behind the one AI seam, and the typed Vault API the
b2 binary drives. This page traces the whole architecture through one real vault
— the two-note fixture the test suite runs on — so every claim is concrete and checkable.
The suite is integration-first: most tests open a real SQLite
database, ingest a real vault on disk, and assert on the resulting projection — so it describes the built
system, not mocks of it. One additional #[ignore] test exercises the real embedder and runs out
of CI. The LLM relator was cut 2026-07-04 (per-pair cost didn't scale) — its crate, seam, and
suggestion queue are gone; connection discovery is now local, free, and instant.
The governing equation
One equation shapes every decision. Keep only this:
Delete b2.sqlite, rebuild it
from the Markdown — it must come back identical. Exactly one thing is durable and
un-derivable: your Markdown (the source of truth for both knowledge and every committed
connection). There is no state anywhere outside your notes. Everything downstream is this equation
defended. (Through 2026-06-30 a third tier — a durable .b2/log/
event log holding the suggestion queue + rejection memory — closed an ∪ log term; the
2026-07-04 relator cut removed the only thing it was for, so the log tier is gone and the equation
simplified.)
Notes + every committed edge, plain .md, fully usable
in Obsidian with no B2. Stays pristine — B2's only writes are surgical and in frontmatter; the body
is 100% the human's.
FTS5 + sqlite-vec + the typed graph. Disposable:
holds nothing that can't be reconstructed from the Markdown. Drop it, reindex, get an identical
index.
<root>/.b2/,
a dotfolder Obsidian and the vault scanner both ignore. Point B2 at a folder of Markdown — that is the entire setup.Four crates, cleanly layered
The engine owes nothing to the two adapters above it or the one model beside it. The AI-heavy
crate sits behind the Embedder trait defined in b2-core, so the engine and
its whole fast suite never import a tensor — and adding the desktop UI added one adapter, not new
architecture.
Turns a folder of Markdown into a queryable
index and keeps it a pure function of disk. Model-free (no candle, no network) so its suite is fast
and deterministic. rusqlite (bundled SQLite + FTS5), sqlite-vec, blake3, ulid, yaml.
note·ingest·chunk·link·relationdb·graph·search·discoverembed— the one seam + fakeadd·mv·vault(the façade)
The b2 binary. Holds no
engine logic: parse args, pick + inject the embedder, call the Vault façade, print
(human, or --json for agents). Funnels every error through user_message so nothing
internal leaks.
The Tauri host — the GUI sibling of
b2-cli, equally logic-free: each #[tauri::command] is deserialize → one
Vault call → serialize, reusing the CLI's --json view types as the IPC contract.
The ui/ frontend (Vite + vanilla TS) renders the read → discover → link loop; two
adapters over one façade can't drift.
LocalEmbedder: candle-backed
BERT, BAAI/bge-base-en-v1.5 (768-dim, ungated), pure-Rust inference. b2 init downloads +
verifies it into a shared XDG cache. Implements embed::Embedder — the only AI seam left.
b2-core's
Cargo.toml has no candle and no HTTP client; the whole engine suite runs on the
deterministic fake embedder. The real model is reached only by b2 init + the out-of-CI retrieval
eval. Cut 2026-07-04: a former crate, b2-relate (the Claude-backed relator), was removed
with the LLM discovery pipeline — see Flow ③.One vault, traced end to end
Everything below follows this exact vault — the hand-authored golden fixture the tests run on. Two notes, engineered to exercise both edge kinds at once. Watch what each layer does to them.
--- b2id: 01JMEM0000000000000000000A type: concept title: "Human memory" created: 2026-06-20 --- The brain encodes, stores, and retrieves information…
--- b2id: 01JSRS0000000000000000000B type: concept title: "Spaced repetition" created: 2026-06-20 --- Spaced repetition exploits the [[concepts/memory|Human memory]] retrieval curve. ## Relations - elaborates [[concepts/memory|Human memory]] — applies the forgetting curve
The prose [[…]] is a bare, untyped
references edge. The - elaborates [[…]] — … list item is a typed edge with
an explanation. Both are human-authored (origin=inline) — B2 reads them, never writes them. One
ingest yields a known-good graph the tests match whole.
FLOW ① Ingest — vault becomes index
b2 reindex is the write path that realizes the governing equation. Its one subtlety:
a note can link to one the walker hasn't reached yet, so link resolution must not depend on file order. Ingest
runs in two phases.
- Phase 1 projects each note's frontmatter, chunks its body (a minimal paragraph splitter, one chunk
per maximal run of non-blank lines), and embeds each chunk through the seam. This fills
notes(b2id, path)— thepath ⇄ b2idresolver — for the whole vault. Our two notes yield one chunk each;memory.md's chunk lands inchunks_vecready to be found. - Phase 2 derives edges by parsing each body + frontmatter
relations:and resolving every[[path]]against the now-complete resolver. Fromspaced-repetition.mdthat's two directed edges tomemory.md: a barereferencesand a typedelaborates. A typed line consumes its wikilink, so it isn't double-counted as a bare reference. - If a note lacks a
b2id, phase 1 stamps one — mint a ULID, write a single surgical line into the note's frontmatter. That's the whole record: the id travels in the file, so there is no separate log to keep. B2's one always-allowed write during ingest. (Both fixture notes are pre-stamped, so nothing is written here.)
Incremental by default, ≡ a full rebuild.
A note whose body_hash is unchanged and whose chunks all still have vectors reuses them verbatim —
only edited notes re-embed. Both paths replace rather than accumulate: re-projecting a note deletes its
old chunks (FTS triggers fire, stale chunks_vec rows are cleared) and all its edges, then
re-derives every edge from the current Markdown (body inline ∪ frontmatter). Because every edge is
Markdown-derived, a one-note re-parse is exactly a one-note slice of a full rebuild — there is no log term to
reconcile. b2 reindex --dry-run previews the same decision, writing nothing.
--force, batching.The schema — a derived projection
Every table is a projection of (Markdown); nothing here is a source of truth.
The schema is derived from the data model — it must satisfy the model, never the reverse.
| Table | Holds | Derived from |
|---|---|---|
| notes | b2id (PK), path (UNIQUE), type, title, created/updated, body_hash — the resolver is this table | Markdown |
| chunks | per-note paragraph chunks with char offsets + token_count | Markdown |
| chunks_fts | FTS5 BM25 index, kept in lockstep by triggers | derived (FTS5) |
| chunks_vec | vec0 KNN table, created at the embedder's dim — not in the base migration | derived (sqlite-vec) |
| edges | typed graph: src→dst, type, origin∈{inline,frontmatter}, dst_id (nullable → dangling), explanation — every edge authored + active | Markdown |
| meta | schema_version, embed_model_id, embed_dim | bookkeeping |
Two schema details do load-bearing work. There is no
status column and no lifecycle — an edge exists iff it is written in the Markdown, so
origin only records which home it came from (inline body link, or a
frontmatter relations: entry). And chunks_vec is created outside
the base migration because its FLOAT[N] dimension is a DDL literal pinned to
meta.embed_dim: the vector space can only be shaped once the embedder's dim is known.
(Removed 2026-07-04: the edge_provenance table, the
status column, the origin='suggested' value, and the origin/status
CHECK — all gone with the LLM relator.)
b2 similar can join all three in one transaction. That only works if FTS5 and
sqlite-vec share one connection: sqlite-vec is registered as a static
auto-extension (no runtime load_extension), so the whole thing is one statically-linked binary.
substrate.rs proves a BM25 query and a vector KNN run against the same connection.FLOW ② Search — hybrid retrieval
BM25 keyword search and brute-force vector KNN run in parallel and fuse with Reciprocal Rank
Fusion (Σ 1/(k + rank + 1), k=60). Results resolve up from chunks to notes.
# keyword + semantic + graph, one command $ b2 search "how does forgetting affect recall" notes/spaced-repetition.md Spaced repetition …exploits the [[concepts/memory|Human memory]] retrieval curve. concepts/memory.md Human memory The brain encodes, stores, and retrieves information…
- The keyword half sanitizes raw natural language into a safe FTS5 expression — each alphanumeric term double-quoted so nothing is read as an operator, OR-joined for recall. (Punctuation is FTS5 syntax and would otherwise crash the parse — a real bug the eval surfaced.)
- A graph-filtered variant — "nearest chunks whose note is within k typed hops of note X" —
is the
chunks_vec ⨝ chunks ⨝ edgesjoin. It searches within an already-related neighborhood (near ∩ connected). Discovery is its complement (near ∖ connected) — the two are opposites, and confusing them is the trap the engine turns on avoiding. - The façade resolves hits to note level: pulls a wide chunk pool, dedups to the best chunk per note, attaches a one-line snippet.
cargo run -p b2-embed --example eval).FLOW ③ Similar — surface the connections you haven't made
b2 similar <note> surfaces the notes most semantically similar to an anchor
that you haven't linked yet — instantly, locally, free: a vector KNN over the embeddings already
in chunks_vec, with no model call and no network. The machine finds the candidates; you
supply the judgment and the type (Flow ③ cont.). This is the reason B2 exists — and after
the 2026-07-04 pivot it costs nothing per run.
A candidate is a note semantically near the anchor but not
already connected — the complement of the graph, not the intersection. Mechanism, per anchor,
in discover::candidates:
- No re-embedding, no model call. Each anchor chunk's already-stored vector is the KNN query
— discovery is passage↔passage, so the anchor is its indexed vectors, never an
embed_querywhose asymmetric prefix is the wrong side. The run reads only the vectorsreindexalready wrote. - Max-sim to the note. Every other note scores by its single best chunk-pair similarity, carrying the winning chunk as the evidence passage B2 shows you.
- Subtract the graph — the only graph read.
reachable_within(anchor, 1)(anchor + direct neighbors) is removed. Distance is exclusion-only: a 2-hop note (triadic closure — "you linked A–B and B–E but never A–E") survives, ranked purely by semantic score.
In our fixture the two notes are directly linked, so each is excluded as the other's candidate — a correctly empty result, which is exactly the assertion discover.rs pins.
b2 similar shows you per candidate- path + title — the note to consider connecting to.
- similarity score — best chunk-pair max-sim (negated
sqlite-vecdistance; higher is nearer). - evidence passage — the candidate's chunk that achieved the score, so you can judge the link at a glance.
You are the precision gate. There is no LLM
adjudicating pairs and nothing is proposed on your behalf — B2 ranks, you decide, and you commit the ones worth
a real connection with b2 link (or a body link of your own).
b2id), and an empty result when the anchor has no vector space yet.
(Removed 2026-07-04: the Relator seam, the ClaudeRelator,
the generate/suggest pipeline, and their tests — the per-pair LLM cost didn't scale to a
1000+ note vault. Candidate generation is all that remains, and it was always the free, deterministic stage.)
↳ the discovery run deep dive walks a full b2 similar.FLOW ③ cont. Link — commit a chosen connection
You saw a candidate worth connecting; now you commit it. There are exactly two ways a connection becomes real, both authored in Markdown — B2 never proposes an edge on its own, and there is no inert queue to accept from.
b2 similar→ your judgment →
b2 link (frontmatter)
or
a [[link]] you write in the body
→ reindex →
active edge
b2 link <src> <dst> [--type <verb>] [--explanation …]. B2 resolves both refs (path or b2id), builds the typed-link string from the target's current path + title, and appends it to the source note's frontmatterrelations:(Markdown-first — never the body).--typedefaults toreferences; the palette is the closed 10-verb core.- Then it re-projects the note (Flow ①), so the edge materializes as
origin=frontmatterstraight from the line just written. Committing is a projection of an authored line, not a bespoke index write — which is what keeps every edge traceable to a real line in the Markdown. - Or write a body link yourself — a bare
[[path|title]](areferencesedge) or a typed- <verb> [[…]] — …line. B2 reads it on the next reindex; it never writes the body. Nostatus, no tombstone, no provenance — an edge exists iff it is written in the Markdown.
b2 link writes into the source note--- b2id: 01JSRS…B type: concept title: "Spaced repetition" relations: - "elaborates [[concepts/memory|Human memory]]" # ← appended ---
Frontmatter only. The body is byte-unchanged. On
the next reindex this line re-derives as an active origin=frontmatter edge — no separate record
anywhere.
relations:, the body byte-unchanged (frontmatter.rs,
roundtrip.rs). The committed edge then re-derives as origin=frontmatter on
reindex, identical to a hand-authored relations: line (graph.rs).The write discipline — losslessness & surgical edits
The vault stays the human's. A parsed note keeps its raw text verbatim and records only the
byte spans of the frontmatter block; serialization returns those raw bytes. So
parse → serialize → parse is byte-identical — unknown keys, comments, odd whitespace, a missing
final newline all survive. Against that backdrop, B2 makes only four narrow writes:
stamp_b2id— oneb2id:line; strict no-op if present, never re-stamped.add_relation— one YAML-quoted typed-link appended torelations:(theb2 linkpath).
b2 add— create a note (minimal frontmatter, no b2id; ingest stamps it on first sight).b2 mv— move/rename + rewrite inbound[[oldpath]]link text.
b2 mv never breaks the graph. Edges key on b2id, and a move
leaves the target's id untouched — so neighbors/backlinks show the same set before and after. Only
the human convenience-copy [[oldpath|alias]] text is repaired, and only in the exact inbound files
the materialized graph names (db::inbound_edge_targets — bounded, never an O(vault) scan). A
byte-preserving rewrite swaps only the target token; a prefix-sharing [[foo-bar]] is never touched
when moving foo.The one AI seam — a fake for CI, a real model for users
The one AI part left sits behind a swappable trait defined in b2-core. The engine is
built and tested against a deterministic fake; a real model drops in through the identical seam with no schema or
flow change. This is the "build for tomorrow's model" tenet made mechanical — and it is why the whole
suite runs with no live model, no wall clock, and no random ids.
embed::EmbedderFake: FakeEmbedder blake3-hashes text into a vector
— identical text → identical vector, deterministically. KNN is reproducible; not semantic.
Real: b2-embed::LocalEmbedder, candle BERT, CLS-pool + L2-normalize (so sqlite-vec's L2
ranks by cosine), asymmetric query prefix, dim read from the model's own config.json.
The swap is safe by construction.
meta records (embed_model_id, embed_dim); on a model/dim change
chunks_vec is dropped + recreated empty so a full re-embed follows. Opening a vault never
mutates the vector space; a stale search fails fast with ModelMismatch rather than
returning silently-wrong results.
Relator
(real: a Claude-backed ClaudeRelator in b2-relate) that typed and explained every
candidate pair. On a real 1000+ note vault the per-pair latency and dollar cost didn't scale, and a per-pair LLM
adjudicator is exactly the "model-compensating machinery" the Bitter-Lesson tenet says to defer. So the seam, the
crate, and the suggestion queue are gone; the human is the precision gate. If a seam lands next it's a
reranker (post-fusion, changes ordering not the store) — not a return of the relator.references→ referenced-byrelates↔ (symmetric)elaborates→ elaborated-bysupports→ supported-byrefutes→ refuted-bycontradicts↔ (symmetric)example-of→ has-examplepart-of→ has-partsupersedes→ superseded-byderived-from→ source-of
A closed 10-verb core is now your typing
palette on b2 link --type and all queries rely on it; each ships a display-only inverse label and a
symmetry flag (inbound neighbors render the inverse; only one directed row is ever stored). A
tail verb a human writes (e.g. inspired-by) is kept verbatim, never dropped, treated as
opaque. Expressiveness lives in the tail; reliability lives in the core.
The command surface — a dumb adapter over one typed façade
Every engine module is called directly only by the integration tests. The Vault
façade is the single typed entry point the CLI (and any future GUI) is the sole client of — it owns the
connection, the injected embedder, and the id generator, and returns display-ready view structs
(NeighborView, SearchResult, ExplainView, the discovery candidate view, …).
The b2 binary parses args, picks the embedder, calls the façade, and prints — human, or
--json for agents.
| Command | Does | Model needed |
|---|---|---|
| init | download + verify the embedder into the shared cache | — |
| reindex | re-project the vault (incremental; --force, --dry-run) | embedder |
| add | create a note, project it (stamps the b2id) | embedder |
| mv | move/rename + repair inbound link text | embedder |
| search | hybrid keyword ⊕ semantic ⊕ graph | embedder |
| neighbors | a note's typed neighbors | — (pure graph) |
| explain | every typed edge + its "why", with an orphan flag | — (pure graph) |
| similar | surface the nearest not-yet-linked notes (path/title/score/evidence) | — (stored vectors) |
| link | commit a chosen connection to frontmatter relations:, re-project it | embedder |
b2 init" if the model is absent; pure-graph commands (and similar,
which reads the vectors already stored) need no live call. B2_EMBEDDER=fake forces the fake
everywhere (offline/dev, and what the suite runs under) — under the fake the CLI prints a loud caveat on
stderr, keeping --json stdout pure data. B2_DEBUG opts into internal error detail;
otherwise errors are generic and never leak sqlite/io/serde..b2/, reindex is idempotent and reports counts, all ref forms resolve to the same set, search is
note-level with a snippet. cli.rs drives the built binary for every
command, both human and --json, clean non-zero exits on unknown notes/ids, and that the
not-yet-semantic caveat rides stderr — including the similar → link discovery
flow and its idempotent, invalid-verb, and both-directions cases.The invariants that hold it together
Almost every choice above serves one of these. Each is pinned by the suite.
| Invariant | What it means · where it's defended |
|---|---|
| Lossless round-trip | parse → serialize → parse is byte-identical; B2 only ever makes surgical edits. → roundtrip.rs |
| B2 never authors the body | Writes are surgical and in frontmatter: the b2id stamp, and a committed edge in relations: via b2 link. → stamp · frontmatter |
| Idempotent re-projection | Re-ingest changes nothing; an incremental one-note re-index equals a full rebuild; no stale FTS/vector rows accumulate. → chunks · graph · embed |
| Disposable index | Drop the DB, rebuild from Markdown → an identical index. Every committed edge returns because it lives in the Markdown; there is no log to replay. → graph · ingest |
| The graph never breaks on a move | Edges key on b2id, so a rename repairs only link text — backlinks resolve unchanged. → mv |
| index = projection of (Markdown) | No durable state outside your notes; an edge exists iff it is authored in the Markdown (body or frontmatter). → graph · ingest |
Grounded in the tests
Nothing here is aspirational: the suite is the executable specification, integration-first, and every property above is asserted in it. Most tests share the golden vault, copied into a temp dir before every run so ingest never mutates the committed fixture, with two fixed b2ids so exact ids are assertable. Determinism is a hard requirement of the core — no wall clock, no randomness; timestamps and ids are passed in.
cargo test)| Property | Pinned by | tests |
|---|---|---|
| Substrate — FTS5 + sqlite-vec coexist; pragmas persist | substrate.rs | 2 |
| Lossless parse · b2id stamp · resolver | roundtrip · stamp · ingest_resolver | 10 |
| Chunks (+FTS) · link classification · typed graph | chunks · links · graph | 15 |
| Vectors & the embedder seam | embed.rs | 8 |
| Hybrid retrieval — RRF + graph⨝vector join | search.rs | 8 |
| Candidate generation (near ∖ connected) | discover.rs | 7 |
Committing a connection — b2 link → frontmatter | frontmatter · graph | 10 |
| Note-authoring CRUD — add · mv · explain · dry-run | add · mv · explain · dry_run · pathspec | 39 |
The Vault façade · the b2 CLI (similar/link included) | vault.rs · cli.rs | 31 |
| Embedder config & provisioning | b2-embed | 4 |
Run under cargo test (rustc 1.96,
the fake embedder). One #[ignore] test — a batched-embed equality check (b2-embed) —
exercises the real model out of CI. Open any file in the right column; it is the ground truth this page
paraphrases.
Not yet built
An honest architecture names its boundaries. Connection discovery is built — it is now
b2 similar (surface) + b2 link (commit), local and free; the LLM relator was cut, not
deferred. What remains is tuning, scale, and packaging — tracked in planning/tasks.md, not gaps that
slipped through.
- Semantic quality in CI. The engine suite runs on the deterministic fake embedder, so it proves
plumbing, not ranking quality. The real model is measured by one separate out-of-CI eval — retrieval
precision/MRR (
cargo run -p b2-embed --example eval). Model quality never flakes CI. - Cold-index scale. Incremental reindex is done; the first embed of a large vault still blocks.
Deferred approaches — a background reindex +
b2 status, a keyword-first progressive index, a smaller/quantized embedder — all compose behind the existing seam. - Deferred, behind clean seams. A cross-encoder reranker (the likely next seam — changes
ordering, not the store), query expansion, the qmd-style ~900-token overlap-aware chunker (a pure
re-projection — the
chunksschema already carriestoken_count+heading_path), and the actual packaging/distribution build.
Describes the repository as of the 2026-07-04 pivot — the v1 kernel across three crates, discovery as
similar + link. Source of truth for every claim: crates/{b2-core,b2-embed,b2-cli}/src/
and the test suite under each crate's tests/. Design rationale lives in planning/
(data-model.md · index-engine.md · specs/). Next phase: the
first UI — a Tauri + CodeMirror desktop app (a fourth crate, b2-desktop, as the second dumb
adapter over the façade), speced in planning/specs/completed/desktop-ui-mvp.md. ·
Home · Quick start ·
Indexing pipeline · Connection discovery