Most of what breaks in an agentic process has nothing to do with the model.
A deterministic step in the process has one expected result. Use the input data to call system X, get the data back, and route accordingly. It all can be tested and asserted.
An AI agent step doesn't work that way. The model decides which tools to call, in what order, and what to write, so two runs over the same case can take different routes and both be right.
That changes what "tested" has to mean. You still want the ordinary assurances: that the agent does what its task requires, that it respects the guardrails you gave it, and that what it produces looks good. But you also want something a deterministic process never made you ask for: whether the agent is working well across a hundred runs and what it costs on a single run.
None of that is easy to pin down. A green run tells you the agent "worked well" once, which isn't the same as testing over multiple runs. Assertions written tightly enough to be meaningful will fail on runs where the model makes a different, equally correct choice, so they are loosened until they no longer fail. And every run that touches a real model costs tokens and takes minutes, so the tests that would catch the most are the ones you run the least.
Agentic test suites tend to end up either disabled because they flake or meaningless because nobody trusts what they assert.
Three layers, and only one of them is the model
An agentic process isn't uniformly non-deterministic. Most of it is ordinary process logic, and separating that from the model is what makes the whole suite affordable.
Agent testing has multiple layers, where each one answers a different question, and each has a very different price:
- The process model around the agent. Can the tools the agent uses actually run? Does the data mapping and business logic work? Does its output land on the variables that downstream routing, escalation, and error handling read? Fully deterministic. Mock the services, mock the model, and assert on the wiring.
- The model's behavior. Given a real case, does the agent call the tools its task requires, avoid the actions its instructions forbid, and produce good output? This needs a real model, and it's graded with evals rather than exact matches, because there's no single correct string to compare against.
- Consistency and cost across runs. Is a behavior stable, or does it drift? What does an average case cost in tokens, and is that number drifting? Deterministic process testing never needed this. Agentic testing can't skip it.
Most of what actually breaks day-to-day sits in the first layer: a renamed tool, an edited input mapping, a guardrail wired to the wrong path. None of it needs a model to detect. Answering all of it with an expensive live-model test is how a suite gets slow enough that someone turns it off.
Camunda Process Test, the JUnit-based library for testing Camunda 8 processes, helps you navigate through different test layers and use the model call only when you need it. It's the library I used to test the agentic process and check the business logic and the model's reasoning.
The process under test
The examples come from a KYC customer onboarding process. A customer applies, a screening agent gathers what it needs and forms a recommendation, a decision table routes the case by risk and approval authority, and anything the agent can't clear goes to a human reviewer under an SLA timer.
The KYC screening agent in this process is the part worth testing carefully. It has tools it can call in any order it likes, instructions that forbid escalating to compliance on a risk signal alone, an error boundary for its own failures, and a reporting tool that hands its findings to the rest of the process. Every one of those is something you'd want to verify, and they don't all need the same kind of test.
Testing the process around the agent, with no model at all
An AI agent in Camunda runs as a job that can be completed by a job worker. In real cases, this job is completed by the connector runtime. While that runtime is up, its worker always claims the agent's job, so a test can't get in front of it. Switch it off, and nobody is doing the agent's job, leaving the test framework free to do it instead: pretend the agent chose the identity check with this argument.
That one switch is what makes the deterministic layer possible. The test decides which tools the agent "called" and with what, then asserts on what the process did with them.
// Stand in for the agent: "call IdentityVerification with this customer ID."
processTestContext.completeJobOfAdHocSubProcess(
byElementId("KYC_ScreeningAgent"),
result -> result
.activateElement("IdentityVerification")
.variable("toolCall", Map.of("customerId", "CUST-1001")));
// The tool ran, and the agent's argument arrived in the tool's own scope.
assertThat(processInstance).hasCompletedElement("IdentityVerification", 1);
assertThat(processInstance)
.hasLocalVariable(byId("IdentityVerification"), "customerId", "CUST-1001");The second assertion is the one worth understanding. It checks the mapping that carries the model's argument into the tool, which is the contract that quietly breaks when someone renames a property. A live-model test can't isolate that, because from the outside, a broken mapping and a confused model look identical.
The same layer covers the rest of the agent's contract with the process. You can throw the agent's own error to prove that a token-limit failure routes to human review instead of stalling. With the same testing library, you can also test the rest of the process - advance a virtual clock to check that an unanswered review reroutes when the SLA expires or evaluate the routing decision table directly.
None of it calls a model, so it costs nothing and runs in seconds.
Testing what the model actually does
Turn the connector runtime back on, and the same library drives a real conversation with a real model. These tests are slow and they spend tokens on every run, so they earn their place only on the questions a mocked model can't answer: does the agent pick the tools the task requires, does it stay inside the instructions you gave it, and is what it writes any good.
// Did it do the job? Element assertions prove the agent called every tool its
// task requires, not just that the process reached an end event.
assertThat(processInstance).hasCompletedElement("SanctionsPepScreening", 1);
// Did it respect its instructions? Its prompt forbids escalating on a risk
// signal alone. The process can still end correctly even if the agent didn't.
assertThat(processInstance).hasNotActivatedElements("EscalateToCompliance");
// Is the output sound? An LLM judge grades the generated report against a
// plain-language expectation, instead of a string you'd have to guess.
assertThat(processInstance).hasVariableSatisfiesJudge(
"screeningReport",
"Confirms identity verification passed, no sanctions or PEP match, and a"
+ " low risk tier, consistent with a clean approval.");The first two assertions are ordinary process assertions. The third is the one built for generated output, and Camunda gives you two ways to grade it.
Judge assertions hand a variable and a plain-language expectation to an LLM, which scores how well one satisfies the other. You get to describe the outcome the way the business rule describes it, instead of guessing at a string the model might produce. What you configure: which model does the judging, the pass threshold, and optionally your own evaluation prompt. The judging model doesn't have to be the one driving your agent, and a smaller one is usually enough, because scoring an answer is an easier job than choosing tools. When the assertion fails it returns the judge's score and its reasoning, so the failure message tells you which part of the expectation wasn't met rather than handing you a diff.
Semantic similarity assertions are the other option, and they swap the judging model for embeddings:
// No generative call: the report and the expected text are both embedded and
// compared.
assertThat(processInstance).hasVariableSimilarTo(
"screeningReport", "Identity verified, no sanctions or PEP match, low risk.");They're cheaper and faster than a judge call, and deterministic once you fix the embedding model, which makes them the better fit when the phrasing can vary but the meaning shouldn't. You configure the embedding model and a threshold, the same shape as the judge.
The threshold is the knob to set deliberately. It defaults to 0.5, which means an expectation that's half satisfied passes. The same expectation, word for word, can pass at 0.5 and fail at 0.9 against one unchanged run, so that number decides how strict your suite really is, and someone has to own it.
Testing the path the agent took
Judge and similarity assertions both grade the agent's output. Neither one says anything about how the agent got there, and two runs can reach the same acceptable answer by completely different routes. In one conversation my agent needed the same person's details twice: first it listed every user and scanned for the name, then it loaded that user directly by ID, having already resolved it. Both are correct, and an expectation loose enough to accept either path passes every time while that choice changes underneath it.
Modeling the agent as part of a process is what makes its path assertable at all. Camunda governs an agent through two patterns: outer orchestration (outside the agent) coordinates it as one participant alongside the people and systems in the end-to-end process, and inner orchestration (inside the agent) puts enforceable steps between the agent's reasoning and the tools it reaches for. Both leave the same artifact behind. The agent's tool calls, its escalation path, and its error boundary are all elements in one process model, so the same assertions you'd use on any process tell you which route the agent took.
LLM eval frameworks grade a model's output well, and the better ones trace its tool calls too. What they can't show is the route through a governed business process, because there's no process to look at. Use both: element assertions for the path, an eval for the quality.
Measuring the agent across many runs
One run tells you about one run.
Every time agents in Camunda are executed, the platform records an agent instance: a structured record of that conversation. It carries how many times the model was called, how many tool calls the agent made, and how many input and output tokens it consumed, measured against the limits configured on the agent in the process model. Because the record is written when the agent actually calls a model, these numbers come from the live layer rather than the mocked one.
A test can read that record, which means a Camunda Process Test isn't limited to asserting on process state. It can also assert on what the agent cost and how hard it worked to get there: model calls under a ceiling, tool calls within the configured limit, tokens inside a budget.
One record is one sample, and the questions worth asking about an agent are questions about many runs. Was that tool called on every run or only most of them? How often does the case take each branch? Is the average token cost holding steady, or creeping up as the prompt and the model change underneath? So run the scenario repeatedly and treat the collected records as a sample rather than a verdict. Collecting and averaging them is something you assemble around the test today.
Who gets to write the tests
Camunda Process Test is a Java library, so by default, the tests sit with the developers. It also runs test cases written as JSON against a published schema, and that changes who can produce them.
The JSON file can be written by hand, generated, or assembled in a UI. Test Studio, arriving in Camunda 8.10's Web Modeler, is the UI option, and it emits the same schema, so a scenario built by the person who actually knows the policy runs unchanged in a build pipeline.
Where to start
An agentic process is only partially non-deterministic, and treating it entirely as the model's decisions makes agent testing slow, expensive, and unreliable.
So separate the layers and let the price set the cadence. The deterministic layer costs nothing and runs in seconds: tool wiring, data mappings, guardrails, SLA behavior, routing decisions. Run it on every commit and it will catch most of what actually breaks.
Bring in a real model when the question genuinely needs one, which is whether the agent picks the right tools, respects its instructions, and writes something good. And when the claim you need to make is about consistency or cost rather than correctness, run the scenario enough times to have something to average.
That's all three layers, at three very different prices, and only one of them belongs in every build.
The full worked example is public if you'd rather start from something that already runs: camunda-ai-agent-testing.
Next in this series: once the agent is live, how do you see what it's doing while it's doing it?
