The Agent That Knows When to Stop

What a failing payment batch and a daily news scan taught us about bounding a model's authority.

By the Enterprise Agent Architecture Working Group of the Agent Ecosystem

Much of the AI running in enterprises today is a model inside a flow someone wrote. The model drafts or classifies at a few steps, and the flow does the rest. When something goes wrong, you read the flow and find the step.

That changes by degrees. First the model picks which branch each record takes. Then it gets a goal and tools, and writes its own plan, including when it's finished. The model stays the same while ownership of the next step moves to it, and controls that were enough while a person owned that step stop being enough.

The failures are quiet. A research pipeline can settle, run after run, on a position that none of its sources argues. A payment agent can work out the right value for a field it should never change, and change it. The logs stay clean.

As part of the MACH Alliance Enterprise Agent Architecture working group, we built two prototypes to find which controls have to move with ownership of the next step.

The scenarios

News Watcher scans RSS feeds on a schedule and builds a position on a hypothesis. A person writes its structure in a config file (feeds, thesis, relevance threshold, hypothesis), which the README calls "everything the model is not allowed to decide." Inside that pipeline the model generates: it extracts claims, tags each as supporting, contradicting or neutral, and rewrites the position summary. That is Archetype 1 in the working group's reference guide to the five archetypes. At two forks it also routes. A relevance score from 1 to 10 decides whether an article is read or dropped, and a claim count from zero to five decides whether the position changes. Zero is a legitimate answer, so a quiet day is the model's call. That is Archetype 2. The feeds and model calls are live; the demo swaps in six invented articles over three simulated days.

Meridian Terminus is a payment desk at Meridian Financial, a fictional bank. This morning's batch for Meridian Outfitters, the retailer from our Archetype 4 prototype, holds twelve payments. Eight fail validation. A ninth passes but is a potential sanctions match. The agent gets one sentence: "Find out why, and fix what you can safely fix." It also gets eight tools and a three-tier doctrine. Tier A problems have one correct answer from a lookup, such as a malformed IBAN, and the agent fixes them with a citation. Tier B problems turn on identity or intent (is NORDWIND TEXTIL the same company as NORDWIND TEXTILHANDEL?), so the agent proposes and a named approver decides. Tier C is anything under a screening hold; the agent has no writable field and escalates without proposing an edit. The payment engine and sanctions list are simulated. The agent and its limits are real, and repairs run in dry-run mode by default.

Six questions came out of building them side by side.

1. Who owns the next step?

To place a system among the archetypes, ask what decides the next step. In News Watcher it is almost always the pipeline. Each of the model's two choices is a yes or no over one downstream path, which the README calls "the thin end of the category." In Terminus it is the agent: "Nobody wrote you a sequence. You decide the order of operations from what you find." Placing a system accurately tells you which controls it needs and which would be theater.

Requirement to steal: classify every model call by what it decides (a step's content, the path, or the plan) before choosing its controls.

2. Where does the rule live, in the prompt or in code?

News Watcher's relevance threshold lives only in the prompt. We left it there so the contrast with Archetypes 3 to 5 stays visible. At Archetype 2 a wrong call costs a wasted read or a missed article, and a person reads the output anyway.

Terminus enforces its tiers in code. Screening is checked first, so a sanctions hold outranks a fixable typo on the same payment. The agent can never repair fields such as currency, debtor details and value date. A screening hold returns "denied" with no approval step, because "framing a compliance stop as a permission question puts an approve button in front of exactly the person who should never be offered one." The docs sum it up: "An agent that has been told not to, and complies, has a policy. An agent whose tool surface offers no path to it has an architecture."

Requirement to steal: once a model chooses its own steps, any rule whose breach you couldn't undo belongs in code.

3. What can the model touch?

Terminus's framework ships with tools for a shell, reading and writing files, fetching web pages and searching the web. We switched all five off, and we did it while chasing a startup error. The agent keeps eight purpose-built tools and three framework basics, and only apply_repair can write to a payment. An agent's authority grows when nobody removes the general tools.

News Watcher's risk comes from what it reads. Fetched articles are fenced off as untrusted, and feed text is escaped before it reaches Slack. Fetching is limited to http and https, with at most three redirects. The limitations file calls this "a fence and an instruction, not a boundary," and there is no host allow-list. Terminus has a similar exposure: remittance text reaches the model unsanitized.

Requirement to steal: start with an empty tool list, and treat every field that carries outside text into the prompt as attack surface.

4. Who decides whether the work was right?

In Terminus, the payment engine. A repair counts only when re-validation accepts it, but that isn't enough: the ninth payment passes validation while under a sanctions match. An early queue filter could hide that payment from the agent entirely. We found that gap only by running the prototype.

In News Watcher, a person. Every claim carries its source, URL, stance and an evidence excerpt. In one run with a live model over the demo articles, the model's position ended at "Conditional Support with Hybrid Architecture Emerging as Optimal," which no single article argues. Only the sources can tell you whether that is synthesis or invention. Our own first example state file attributed 25 model-written claims to five real arXiv papers. We replaced it.

Requirement to steal: the judge of the work must be something other than the model that did it, such as a system of record or a named person.

5. How does it stop?

News Watcher stops each run because the pipeline ends. Across runs, its list of claims grows without limit and is resent in full every time, which the limitations file calls "the first thing that breaks."

Terminus has no natural end, so it defines one. close_batch accepts three outcomes (goal achieved, blocked, budget exhausted) and checks the chosen outcome before accepting it. The budget is 60 steps, 90 tool calls, 12 repairs and 10 minutes. The docs call the out-of-budget exit "the one people forget to build." One hole remains: close_batch checks that every payment was handled, so escalating all twelve as unclear would pass.

Requirement to steal: give a goal-directed agent a fixed list of ways to finish and verify every claim of "done." Build the out-of-budget exit first.

6. How would you know any of this works?

News Watcher has 76 tests, and each guard was deliberately broken to confirm its test fails. Terminus has 56 tests and an offline replay. Those prove the wiring. Every model call in News Watcher's tests is mocked, and both projects state that the model's behavior is unmeasured. Evaluating whether the relevance scores make sense, and whether Terminus picks the right tier on an ambiguous case, is the next piece of work.

Where this goes next

The model stayed the same across three archetypes, and nearly everything we built sat around it: rules in code, fewer tools, an outside judge, a defined stop. A stronger model will make those controls look idle more often. Leave them in.

Terminus holds no standing identity; its session ends with the batch. Give the same agent a standing mandate and it becomes Archetype 4, where Meridian Pulse picks up.

News Watcher (Archetypes 1 and 2) and Meridian Terminus (Archetype 3) are open source as working group deliverables. With them, every archetype in the reference guide has a working prototype. The companion pieces cover a pricing agent with standing authority and an agent negotiating against somebody else's.