Traditional software engineering uses the concept of the Testing Pyramid: many small unit tests at the base, fewer integration tests above them, and a small number of comprehensive end-to-end simulations at the top. 

At Replicant we incorporate the fundamental philosophy underlying the Testing Pyramid, and adapt it to the realities of testing a scalable platform of AI agents.

AI testing: Our systems combine deterministic software with stochastic AI components. Testing an AI-based component is intrinsically a lot messier than testing traditional code because you cannot in principle expect to see the same behavior every time.

Platform-level vs. agent-specific logic: Every Replicant agent is a mix of bespoke adaptations to solve a specific customer’s problem, layered on top of a robust platform that abstracts away many of the universal considerations of AI agents. Things like general steps to follow, appropriate language to use, and which back-end systems to interface with are agent-specific. Things like guardrails, how to query a knowledge base, how to handle pushbacks, dynamic behavior based on business hours, and so on, are solved at the platform level so that we only solve it once.

Platform-level evaluation: are the fundamentals sound?

Before evaluating whether an agent solves a customer problem, we need to ensure the foundational capabilities that every agent depends on are behaving correctly across all agents.

What we are looking for:

Information extraction reliability

Can we consistently collect key pieces of information from the caller (addresses, authentication details, names, dates, etc.) despite:

• ASR noise
• Accents and language variation
• Unstructured responses

The importance of this cannot be overstated; failures here will cascade into everything downstream, so the bar is set incredibly high. 

Basic information extraction is a fundamental challenge that has already been “solved” in the industry, but remains a continuous focus at Replicant, because we know that even a very difficult piece of information like an address or a password needs to be collected extremely reliably. 

You can’t afford to fail 20% of the time on a step that your entire workflow depends on.

Escalation and handoff behavior

Do we:

• Escalate when we should?
• Avoid escalation when we shouldn’t?
• Provide a warm, context-rich handoff when transferring to a human?

Escalations and handoffs are a critical part of a great customer experience, ensuring a live agent can support customers when needed. As such, it’s imperative that AI agents can handle them gracefully to prevent a disjointed journey that requires customers to repeat themselves. 

Knowledge base usage

Are we:

• Retrieving the correct information?
• Using it appropriately in context?
• Avoiding hallucination when knowledge is missing?

A company’s knowledge base is the bedrock of accurate, standardized information. It is also a living resource that changes constantly. If an AI agent cannot confidently navigate, understand, and communicate knowledge base information, brands are at risk of providing customers with inaccurate responses that can quickly spiral into larger security mishaps. 

Cross-language robustness

Do all of the above behaviors hold across the languages we support? If an AI agent only works consistently in English, it can leave a large swath of a brand’s customer base with fewer options to get prompt, accurate support. At a platform level, it is imperative that cross-language proficiency is consistently monitored in order to serve global, enterprise brands. 

How we test for it:

First and foremost, we iterate in real time using production examples of issues. With a combination of LLM-based and procedural analysis we find the calls where mistakes are made. This feeds into a continually evolving dataset for each of the above capabilities and others, giving us the directional pointer to experiment with the many different strategies needed to solve them better and better over time.

Agent-level evaluation: is the job getting done?

On top of a robustly-evaluated platform, we can now meaningfully evaluate the agent itself: the part that is actually solving a customer’s problem.

What we are looking for:

• Are the primary goals of the agent being achieved? Can the user complete the entire workflow? Do we actually capture their information correctly? 
• Are all steps executed in the correct order?
• How does the agent behave when things go off-script? What if the user is asking about unsupported capabilities? 
• Can we answer non-sequiturs and get back on track to a workflow smoothly? Can we repair failures?

How we test for it:

The two primary strategies we employ for testing agent-level behavior are end-to-end workflow tests and single-turn interaction tests.

End-to-end testing is the more “obvious” approach to testing AI agents. You can use a second AI to roleplay as a user and go through an entire scenario with the agent, then check at the end to see if basic criteria were met (the workflow completed, the agent was professional, no hallucinations, etc.) 

This is an excellent approach for setting a baseline performance expectation for an agent because it touches on every part of the agent’s behavior. 

It also tends to be more stable in the face of non-deterministic AI behavior; most mistakes are small and recoverable, and thus it is possible to write an end-to-end test that passes very consistently even when it incorporates the messy realities of AI agents.

Single-turn testing is achieved by fully recreating the snapshot state that the agent is in at a certain point in a workflow, and analyzing the behavior at that specific moment. 

Single-turn testing is an incredibly powerful debugging tool because agent failures do not tend to be evenly distributed across all possible steps of all workflows; rather, there will often be a couple of very specific steps in very specific scenarios that behave unpredictably. 

Instead of running hundreds of end-to-end tests hoping to find more needles in the haystack, a laser-focused single-turn evaluation focuses exactly on why the agent doesn’t work in that one specific context and allows faster iteration towards a fix.

That said, it is more fragile than end-to-end testing for two reasons:

• As the behavior of the bot evolves, the snapshot state for a certain step in a workflow may need to evolve with it. Maybe the bot doesn’t ask for the user’s phone number at the start of the call anymore, so a history that shows that is now wrong.
• There is no opportunity for repair. Minor mistakes that are likely recoverable are still failures from the perspective of a single turn evaluation.

These two testing strategies are clearly complementary. Contrary to traditional software testing, end-to-end testing tends to form the baseline for evaluating agents with targeted single-turn testing providing scaffolding for tricky scenarios identified in the wild. 

Together they provide a complete picture of agent performance.

Bringing It All Together

Testing Replicant agents is not just about adding more tests, it is about structuring evaluation along the right dimensions:

• Platform vs. agent responsibilities
• Broad vs. narrow test coverage
• Deterministic vs. stochastic behavior

This focus on many different layers allows us to always use the right tool to fix the right problem.

This post is part two of a blog series on How We Test. You can read part one on "The golden rule of evaluation" here. 

Schedule time with an expert to learn more about how expertly evaluated AI agents can transform your contact center.