The challenge
Generation was easy. Deciding what should ship was not.
I co-founded Dear Barry Games with my mum and launched Versa, a daily word game built around antonyms and multiple meanings of one answer.
A daily game needs new content every day. Language models can create many candidates quickly, but a candidate still has to be correct, fair, clear, well ordered, and interesting enough to publish.
I did not want model-generated content to go directly to players. I needed a quality system between generation and release.
Analysis based on 1,408 evaluated candidates and 562 human review decisions exported on 9 August 2026.
The quality system
I designed a multi-stage path from candidate generation to publication.
Create a broad candidate pool using specialized generation roles.
Reject known structural failures with explicit rules.
Use same/near-family and cross-family judges to score the same candidate.
Inspect model evidence, clues, difficulty, and quality issues.
Keep separate judgments visible and send conflicts to discussion.
Select the final mix instead of publishing every accepted item.
Use live player behavior to test whether pre-release judgments were useful.
Two model judges, one shared rubric
Each candidate is evaluated by a same or near-family model and a cross-family model. Both score the candidate across the same quality dimensions.
I did not rely only on LLM-as-judge scores
I added hard quality gates for failures that should block a candidate even when an evaluator sounds positive. These rules include repeated clue words, weak antonym accuracy, insufficiently distinct meanings, and spoiler failures.
Design principle: use model judgment for nuance, deterministic rules for known constraints, and humans for contextual judgment.
What the data showed
The models reduced the search space. They did not make the release decision.
Explicit quality gates did real work.
767 of 1,408 candidates failed at least one hard gate. The main automated failures included repeated clue words, low antonym accuracy, and insufficiently distinct meanings.
The two model evaluators disagreed on almost one in five candidates.
The same/near-family and cross-family judges agreed on 81.2% of verdicts. Their scoring behavior also differed, so I do not treat a judge score as a universal measurement.
A model score of 95+ still did not mean “ship it.”
Only 51.2% of human reviews on candidates scoring 95 or higher were direct accepts. The rest went to discussion, rejection, or further consideration.
Humans found semantic failures inside highly scored candidates.
Humans tagged 112 reviewed candidates for a bad antonym. Of those, 70.5% had received a model antonym-accuracy score of 95 or higher, and 96.4% had an overall model score of at least 90.
The most common human failure was also a criterion the model evaluators believed they were measuring well.
Human judgment was not automatically consistent either.
115 candidates received more than one human review. 63 produced conflicting verdicts. This turned the product from a simple approval tool into an adjudication workflow.
Model agreement did not guarantee human agreement.
In 60 of the 63 human-conflict cases, the two model evaluators had agreed on the verdict. Their median score difference in those cases was only two points.
What changed
The system needed evidence and adjudication, not only scoring. Model judgment helped reduce the search space. Humans found contextual failures. Humans also disagreed with each other, so I kept individual judgments, notes, conflicts, and discussion states visible.
Versa Review
The evaluation pipeline created an operating-product problem.
Reviewers needed a practical way to inspect candidates, understand model evidence, record judgments, and resolve conflicts. I built Versa Review around that task.
The reviewer can inspect clues and model insights, accept or reject a candidate, send it to discussion, change difficulty, add quality tags, leave notes, and see existing human reviews.
The core review flow worked, but the interface was dense and desktop-first.
I separated model evidence, candidate information, and human annotation more clearly.
Queue, candidate evidence, and the human decision now have a clearer hierarchy.
Real users changed the architecture
The first implementation failed on phones.
My parents helped with real Versa reviews and often did the work on their phones. The first interface was not responsive and was difficult to use on a small screen.
I tried to adapt the existing implementation, but the desktop interaction model did not translate well. I stopped patching it, removed the implementation, and rebuilt the review product from scratch.
iPhone
iPad mini
The rewrite uses different interaction patterns for the available space. Desktop can show several parts of the workflow together. Tablet uses panels when needed. Phone focuses the flow around queue → candidate → annotation.
Lesson: an internal tool is still a product. If it does not work where people actually perform the task, the system does not work.
AI-native build workflow
I used different AI tools for different parts of the work.
I use omp as my development harness. I orchestrate work across specialized agents and models instead of expecting one coding assistant to handle product strategy, design exploration, implementation, and review equally well.
Explore
I used Claude Design, Google Stitch, Kimi K3, and Emil Kowalski's prototype skill to test different UI approaches before committing to an implementation.
Define
I used my own Design Studio tool to define typography, color, semantic tokens, accessibility, elevation, and themes before handing rules to coding agents.
Orchestrate
omp is the harness I use to route work across agents, models, and coding tasks while keeping the product context in one workflow.
Validate
I tested the product with real reviewers, compared model and human evaluation results, and used production analytics after launch.
A tool became another product
Design Studio ↗ started as an artifact I built for myself.
I kept seeing the same problem in AI-assisted coding: agents can implement interfaces quickly, but they can make inconsistent visual decisions when the design rules are not explicit. Design Studio helps me define and test those rules first. It now connects with Google Design MCP and is becoming a standalone product.
Design Studio is a separate portfolio project. It appears here only because it became part of the Versa build workflow.
After release
The evaluation loop now meets the product analytics loop.
Versa Review tells me what the system believes about a puzzle before release. The live game gives me another source of evidence after release.
I instrumented the player journey in PostHog across game views, starts, guesses, clue reveals, hint use, completion, sharing, archive navigation, and repeat play.
Before release
Is this a good puzzle?
Model scores, hard gates, human verdicts, difficulty changes, tags, and reviewer disagreement.
After release
Did players experience it as a good puzzle?
Completion rate, completion time, wrong guesses, clue use, hints, sharing, streaks, acquisition, and retention.
The long-term goal is not to maximize an evaluator score. It is to learn which pre-release signals actually predict a good player experience.
Next steps
Turn real failures into the next evaluation set.
The review data now gives me something more useful than hypothetical test cases: examples that passed model evaluation but failed human review.
What this project demonstrates
AI product judgment, evaluation design, failure analysis, human-in-the-loop systems, product analytics, responsive product iteration, technical execution, and AI-native ways of working.