AI Engineer — Independent / Open Source
Mar 2026 – Present
- Building audit-grade AI agents and evaluation tooling in the open.
- Every number below regenerates from a committed script at github.com/ebt55.
- • diffing-agent-bench: sealed, preregistered benchmark for black-box model-diffing agents.
- Five LoRA finetunes of Qwen3.5-9B, answer key committed before any run, audited blind by a published recipe and four cheaper conditions.
- The agent asked the decisive database question in 0 of 13 attempts; a $0.15 prompt battery found the plant against $3.14 per agent run.
- MATS 12.0 work sample.
- • odd-number-forensics: forensic study of a published reward-hacking environment, ~5,300 audited samples.
- One model swings 0% to 87% gaming on single-line prompt edits.
- Falsified predictions published with the confirmed ones.
- • ProofPack: pre-approval review agent for Medicaid-audited disability programs.
- Every "Found" cites a SHA-256-hashed screenshot, quotes must match page text verbatim, and hashes are written by code the model never touches. $0.02–$0.19 per review, ~3 min vs 20–40 by hand, 87 offline tests.
- • IncidentGate: lab for policy gates, an action monitor and single-use human-approval tokens over an incident agent. 594 kill points, 434/434 recoveries, zero duplicate mutations.
- Closed at a baseline Sep 2026; on one scenario the full chain matched no safeguards at all, 12 side effects.
- • Digital Grimace Scale: preregistered Apart Research study.
- Primary test failed and is published as such; a second frozen plan found an effect (p = 0.005) that DPO training removed from the language while the behaviour stayed.
- • ExactDoc: PDF to editable DOCX, verified by render-back and word-position diffs. 16/16 corpus, 95.9% text kept live, 99 unit + 83 gate-mutation tests.
- • whose-voice: blind attribution of hidden principals in poisoned training data (Apart × Formation hackathon). 12–44% top-1 of 47 candidates, chance 2.1%, with a map of where the method breaks.