Indexing & the embedding pipeline

1The shape of a reindex

Ingest is the write path that realizes the governing equation index = projection of (Markdown). It runs in two phases so link resolution never depends on file order: phase 1 projects every note + its chunks (filling the path → b2id resolver for the whole vault), phase 2 derives edges against that now-complete resolver. Everything expensive — embedding — happens in phase 1, and that is where the heuristics live.

2The incremental heuristic — embed only what changed

A full re-embed of a large vault is the one genuinely slow step (a real transformer, on CPU). So a routine reindex must not redo finished work. Per note, the decision is: if the body is unchanged and its chunks already all have vectors, reuse them and embed nothing. Otherwise re-chunk and re-embed. --force bypasses the skip; a model swap forces it implicitly (see §4).

Note the asymmetry: edges are always re-derived (phase 2, cheap) because a link's target may have moved even when this note's body didn't — only embedding is skipped. A frontmatter-only edit (e.g. a relations: entry written by b2 link) therefore re-projects the note and its edges but embeds nothing, which is what makes b2 link cheap too.

3Chunking — anchored paragraphs

Today's chunker is deliberately minimal: a paragraph splitter — maximal runs of non-blank lines, one chunk each. Every chunk records char_start..char_end addressing the exact body slice that produced it, so a chunk stays anchored for later highlight/explain. The chunks schema already carries token_count and heading_path for a future qmd-style ~900-token, overlap-aware chunker — swapping it in is a pure re-projection (drop & rebuild), no schema or invariant change. Chunk text is also the FTS5 source: the chunks_ai/ad/au triggers keep chunks_fts in lockstep, so keyword search sees a chunk the moment it lands — before it is embedded.

4The embedder seam — one interface, batched

Producing embeddings is orthogonal to the store, so it sits behind a trait. The engine is built and tested against a deterministic fake; the real local model drops in through the same seam with no schema or flow change. The trait is five methods — model_id, dim, embed (one passage), embed_query (asymmetric-ready, defaults to symmetric), and embed_batch (many passages at once).

Embedding is the reindex hot path, so the write side hands whole batches of chunks to embed_batch (up to EMBED_BATCH = 32). The real model turns a batch into one padded forward pass instead of N single ones — a large CPU win — and on macOS candle's matmuls run on Apple's Accelerate BLAS.

5The embedding space & the model swap

The vector column's dimension is a DDL literal, so chunks_vec is created at embed time at the model's dim, not in the base migration. meta records (embed_model_id, embed_dim) — the only place a model swap is detectable.

• On a swap, ensure_embedding_space drops and recreates chunks_vec empty. Every note is then "not fully embedded", so the incremental heuristic (§2) re-embeds the whole vault on that reindex — automatically, no --force needed.
• Opening a vault never mutates the vector space. Shaping and any re-embed happen only on reindex, so changing the configured model can't silently wipe vectors on the next read command.
• A stale read fails fast. search compares the index's recorded model to the active one and returns ModelMismatch ("run b2 reindex") rather than fusing incomparable vectors into silently-wrong results.

6Progress & force — the operator view

Because a cold index of a large vault is real work, the embed phase reports ReindexProgress after every batch, and the CLI renders a live line — on an interactive stderr only, so --json and piped/captured output stay pure data. The count tracks the notes that actually embed (3/3 below, matching the report's embedded), not position in the full list — an incremental run reuses most notes untouched, so the line reflects real work rather than jumping to something like 812/1200.

# incremental by default — only changed notes re-embed; reindex needs an explicit vault
b2 reindex ~/vault
Indexing /Users/you/vault
  embedding 3/3 · notes/spaced-repetition.md (11 chunks)     # live, in place (TTY only)
Indexed 1200 notes (3 embedded, 0 stamped)

# re-embed everything in place, without deleting the index
b2 reindex --force

Timestamps like indexed_at come from SQLite, keeping b2-core free of any wall-clock — the façade is the one boundary where real ids and time enter, exactly as it is for the id generator.

7What's deferred (and why it's clean)

Incremental reindex makes the steady state cheap. Making the first cold index of a large vault non-blocking is a separate, composable step — documented in the task backlog:

• Background reindex + b2 status — detach and return immediately; search/similar read the index live (SQLite WAL already permits one writer + concurrent readers across processes).
• Progressive (keyword-first) index — chunk text + FTS land instantly (keyword search works at once); vectors fill in behind so the semantic half improves over time.
• Faster / smaller embedder — bge-small (384-dim) or a quantized/ONNX path behind the same Embedder seam; a raw-speed lever, measured against the eval before it becomes the default.

Each drops in behind an existing seam or invariant — none needs a schema or flow change, which is the whole point of the deterministic-seam discipline.