System One Experiments
Lab notebook · AlteredCraft · opened 2026-09-26

What can a model do when it can't write?

System One models don't generate text. They answer a typed question about some state with a probability for every answer, in milliseconds. These experiments test where that's enough to stand in for a parser, a grader, a release manager, or the choice of which model to call.

Read the latest results → Routing results All four experiments
Score · route a sub-agent task
Which description fits the work this task asks for?
Replaying real answers from TypeSafe Jev (jev-1.13.0), model-router run of 2026-10-08.
01

Why I ran these

I wanted a quick, reasonably informed gut check on whether this class of model is viable for small decisions that fixed rules can't make: which move a player meant, which model a sub-agent task needs, which misconception an answer shows.

Two of the four experiments have live runs so far. In both, a hosted model and two running on a laptop handled a small set of mostly clear cases without a costly mistake, and differed in how sure they were. That tells me the mechanism works. It isn't an accuracy figure, and the note on rigor says why.

02

The premise: what Jev is

Jev is TypeSafe's System One model. It never writes a reply. You send it some state and typed questions about that state, it returns a probability for every possible answer, and ordinary code decides what to do with those numbers. Open Jev serves the same API from open weights on your own machine.

It takes three kinds of question, and the experiments below use all three. TypeSafe's introduction covers the rest.

Noul yes / no

The probability that a statement about the state is true.

attempt: "a genuine attempt?" → 0.97 example from misconception-gates' design

Choice one of ≤ 255

A probability for each option. It can only pick something that's on the list.

"grab the torch" → take__torch godot-jev: options are the moves possible right now

Score 2–10 levels

Where the input sits on an ordered scale, with a probability per level.

lookup 0.00 · standard 1.00 · open_ended 0.00 model-router, task 11, TypeSafe Jev
03

The experiments

Each is a standalone directory with its own stack, a hypothesis, the checks that would confirm it, and tests that run without an API key. Skeleton means the design is in code and tested against scripted answers. Live means runs against real models are recorded, with dates. If one is close to a problem you have, the code is there to run against your own data.

EXP-04 · model-router · Node Live · 3 models

Which model should this sub-agent run on?

An agent harness asks one Score question about a sub-agent's task, then takes the cheapest model whose chance of being too weak is acceptable. The orchestrator spends no tokens deciding.

The results page's route explorer: Open Jev Flash 9B's answer for task 15, moved up from lookup to standard The results page's route explorer: Open Jev Flash 9B's answer for task 15, moved up from lookup to standard
One decision, from the results page
One Jev call over the text of a sub-agent task chooses a model tier well enough to move a large share of work off the most capable model, while rarely sending a task to a model too weak for it.
First live run · 2026-10-08 · 24 tasks
0
tasks routed too low, by any of the three decision models
67%
kept off the most capable model, by all three
93 ms
median decision, TypeSafe Jev hosted (on one laptop: Flash 9B 181, 27B 567)

Timings describe this setup, not how fast the models are.

Walk through the results → Code
EXP-02 · godot-jev · Godot 4.7 Live · 3 models

The Sunken Crypt

A dungeon crawl whose parser understands whatever you type but can only choose moves the author wrote. Each turn, the possible moves become one Choice, plus "none of these".

The Sunken Crypt: a pixel-art hall with a skeleton, the transcript, and a panel showing what Jev heard
The game, asking "did you mean?"
Players type naturally with few dead ends, the game stays fully authored, and it's fast enough to feel like a parser, not a chat.
Three-model run · 2026-10-08 · 16 lines × 3 rounds
0
wrong moves in 144 turns, by any of the three decision models
6
turns where the game asked "did you mean?", all Flash 9B, on 2 lines
165 ms
median turn, TypeSafe Jev hosted (on one laptop: Flash 9B 185, 27B 579)

Timings describe this setup, not how fast the models are.

Walk through the results → Code
EXP-01 · misconception-gates · Node Skeleton

Grade the misunderstanding

Read a learner's free-text answer for which misconception it shows, and send them to the part of the lesson written for it.

With four to six authored misconceptions per gate, one Jev call diagnoses free-text answers accurately enough to act on most of them automatically.
Next
Accuracy at a useful coverage on ≥ 50 labelled answers per gate.
README & code
EXP-03 · semver-judge · Python Skeleton

The next version number

Decide the bump from the commits since the last tag. Code reads what messages declare, Jev judges what they don't, and a human sees only commits that are uncertain and could change the answer.

A single Jev call per commit agrees with the bump a maintainer would choose on commits without Conventional prefixes.
Next
Replay the tagged history of public repos and compare with the bumps maintainers made.
README & code
04

More about my process

Each experiment's README states a hypothesis, the checks that would confirm it, and what is still open. An AI coding agent does most of the building, tests first, working to a short set of rules kept in the repository's AGENTS.md. One of them: loosening a check, or changing what an experiment is trying to show, is my decision and not the agent's.

A live run is a script that pins the model version and writes every answer to a file. Each results page is generated from those files, and the generator refuses to write if its totals differ from the ones the run saved. When a check gets evidence, the README records it with the date.

The code follows the same five rules in every experiment, so the model can be swapped and no test needs the network.

  1. A decider port

    Code asks decide(state, questions) and gets typed answers back. One adapter calls Jev and a scripted one drives the tests, so no unit test calls the network or needs a key.

  2. Policy in code

    Thresholds, arithmetic, dates and counting stay in ordinary code. The model only makes the judgments, so a policy change is a code change, not a prompt rewrite.

  3. Ask only what's open

    Deterministic rules go first: a Conventional Commit prefix, the moves possible in a room. The model answers only what they leave undecided.

  4. Confidence decides the path

    Act, ask or escalate by how sure the model is, with the bar set by the cost of being wrong. The crypt asks "did you mean?"; the router moves a task up a tier.

  5. Fail safe

    If the model is unreachable, each experiment falls back to what it did before: an authored "not possible" line, Conventional Commits only, the harness's default model.

05

Notebook

Dated entries, newest first. Each experiment's README records its evidence the same way.

  1. The Sunken Crypt on three decision models TypeSafe Jev, Open Jev 27B and Open Jev Flash 9B parse the same 16 lines, three rounds each. No wrong move in 144 turns; Flash 9B is less sure, so the game asks on two lines.
  2. model-router's first live run TypeSafe Jev, Open Jev 27B and Open Jev Flash 9B route 24 sub-agent tasks. None goes too low; the three differ in how sure they are.
  3. The Sunken Crypt on TypeSafe Jev and Open Jev Same 16 lines, three rounds each. Both landed 48 of 48.
  4. A scripted 16-turn session through the crypt Every turn 197–300 ms on jev-latest, once the HTTP request moved off the main thread.
  5. godot-jev becomes a dungeon crawl Six rooms, five items, pixel-art rooms painted by code, and a status panel that shows what Jev heard.
  6. Three experiment skeletons Picked from areas awesome-jev had no entries for.
  7. Proof of concept: pattern matching on meaning The fuzzymatch library, now on the archive/fuzzymatch branch.
06

A note on the current level of rigor

The goal was a gut check, so the rigor is pragmatic: enough to tell whether an idea deserves more work, not enough to quote a number from. I plan to formalize it as the experiments continue, and each README's Open list says what comes next.

  • The samples are small. The live runs cover 24 tasks and 16 lines, written for these experiments and mostly clear cases.
  • AI did most of the work. It wrote the code and these pages, and in model-router it also wrote and labelled the tasks. Nobody else has checked the labels.
  • The timings describe one laptop, not how fast the models are. Each results page gives the setup.
  • Jev has limits of its own. It can't count or compare dates, and text in its input can argue for an answer, so keep the stakes of any automatic action low.