Quality is a UI problem

Every AI project starts in one of two familiar places: a spreadsheet or a JSON file. These are every developer's favorite dumping grounds for data. You query for the rows you want, serialize them, and hand the file to your annotation team to start labeling.

At first it works fine. When your prompt accuracy is sitting at 60%, the errors are obvious. You iterate fast, narrow down the failure modes, close the glaring gaps. 

Then you hit 80 or 90% and the whole game changes. Now you're hunting for marginal improvements. You notice a row that isn't labeled right. You look a little closer, and one by one, more errors start showing up.

This is where most teams quietly stall, and the reason usually has nothing to do with the model.

The real bottleneck in AI development isn't modeling power. It's the speed and fidelity of the human feedback loop.

When that loop breaks, it tends to break in two ways. The first is a loss of trust, the old "garbage in, garbage out" trap. 

Your model is only ever as good as its ground truth, so if your benchmark is carrying a 5% labeling error rate, a 2% gain in model performance is basically indistinguishable from noise. At that point you can't tell whether your prompt engineering is failing or whether your "gold" dataset is actually lead. 

The second way is the sloppy ceiling, the point where you can't hire your way out of the problem. If the environment itself encourages errors, throwing more people at it just gets you more inconsistent data.

Row-drift is a UI failure, not a discipline failure

When our team was still living in spreadsheets, we kept running into a specific cognitive pattern we started calling row-drift.

Humans are pattern-matching machines. When a labeler is staring at a thousand near-identical rows in a grid, the brain eventually flips to autopilot. Row 452 starts to look like Row 451. Context from the previous entry bleeds into the current one and the labeler starts anticipating the answer before they've actually read the data.

It's easy to write this off as carelessness. It isn't. Row-drift is a UI failure. Spreadsheets throw too much peripheral information at you all at once, which drives up cognitive load and creates this "aiming fatigue" where the mental effort of finding the right cell in Column AF ends up costing more than the actual work of evaluating the AI's output.

Compared to a focused view, which keeps one row spotlighted at a time, that blur becomes more pronounced. 

And once you see it that way, the fix stops being "try harder" and starts being "build a better interface."

The era of disposable engineering

For most of software history, building a custom interface for a one-off labeling task would have been a ridiculous use of time.

Before 2023, the answer to row-drift was stricter checks and more auditing, because, in truth, anything else had negative ROI. Building a bespoke tool meant pulling an engineer off work that "directly contributed" to the bottom line and burning a sprint on something you'd throw away in a month, and then there was the cost of maintaining the thing as the needs kept changing.

That math has completely collapsed. We're now in what I'd call the era of disposable engineering. You don't build software that's meant to last forever. You build hyper-specific software for a single project and you're fine letting it die afterward. 

If an LLM can scaffold a React UI on top of a JSON file in under an hour, the distance between noticing a gap and shipping a fix is basically zero. Andrej Karpathy made a version of this point: the future is highly custom, ephemeral apps glued together by LLMs.

The principle underneath all of this is centralized logic, decentralized interface. The source of truth stays put as your master instruction set, and the disposable tool just enforces it. 

A spreadsheet hopes a labeler remembers the labeling guide. A bespoke UI forces it. Mark an intent as "Frustrated" and a required "Reasoning" field pops up before you can move on.

And the nice part is that you don't need a senior engineer for any of this. You don't need much seniority at all to notice a spreadsheet is failing. 

If an annotation lead sees the team fighting a bad layout, they can just prompt a new version of the tool themselves. In this workflow the reusable thing isn't the code, it's the prompt. 

If the task changes on Tuesday you don't refactor anything, you just ask for Version 2.0. Start by working a representative batch yourself. Notice where attention gets wasted or errors creep in, then prompt a small tool that removes that friction and encodes the rules. Put it in the hands of the team, watch for the next bottleneck, and prompt the next version. Repeat until the sprint is done—or the tool earns a longer life

What this looks like in practice

The first time this really clicked for us was an email labeling project. We regularly use data from past conversations to train the classifiers that improve our collection accuracy, and this one needed thousands of user utterances labeled to pull out valid email addresses wherever they appeared. 

Simple task. But the old playbook was either to hire more annotators or just accept that one person's cognitive capacity was going to stretch it out over days.

So instead, the UI idea came from thinking about the whole thing like Tinder. We built a one-card-at-a-time interface. One row in focus, keyboard shortcuts for fast navigation, color coding so errors jump out at you. 

A labeling exercise that used to eat a full week got done in under a day with the same team, and accuracy went up by around 30%. It didn't stay a one-off either. As similar requests came in, the annotation team kept refactoring it with AI to fit whatever the new task was, and it's a regular tool in their arsenal now, including for other modalities like audio.

The second case was uglier. We get asked a lot to dig through past conversations to surface bad agent behaviors for improving our LLM guards and future Instruction following projects, and that's genuinely a needle in a haystack. 

To evaluate a single bad turn an annotator has to cross-reference three different things: the exact bot instructions the agent saw at that moment, which are a messy mix of freeform guidance and deterministic action-chaining and can shift mid-conversation, the full state of the conversation up to that point, and then the call itself, which lives at its own separate URL in an internal tool. 

Done by hand, one call takes 5 to 10 minutes, and roughly 80% of the candidates turn out to be false positives anyway.

The tool we built lays the conversation out turn by turn, and clicking any turn highlights the exact system instructions that were sent to the agent at that point. Turns can be flagged for review, and an engineer can sweep the flagged set afterward to confirm them. The whole cross-referencing tax just goes away.

The takeaway

In both cases the win didn't come from a better model or from more people. It came from getting rid of the cognitive tax of the work.

We spent years treating annotation quality as a function of effort and headcount, and it just isn't. Quality is a UI problem. The moment building the right interface costs you an afternoon instead of a sprint, quality stops being some aspiration you audit your way toward and becomes the default that the tool enforces for you.

Efficiency was never really about hiring more people. It's about lowering the cognitive tax of the work, and that's finally something you can build on demand.

This post is part three of a blog series on How We Test. You can read part one on “The golden rule of evaluation here” and part two on “A multi-layered testing strategy for Replicant AI agents” here. 

Schedule time with an expert to learn more about how expertly evaluated AI agents can transform your contact center.

‍