📰 Key Summary

This is a news summary translation task related to an advertising service — just process the content directly.

Even when an AI agent succeeds once in testing, that doesn’t mean it’ll succeed again on the same request next time. Standard benchmarks usually only report the “mean success rate” (Mean@k) — the average after running a task k times — but this number hides the agent’s underlying instability. Using the AppWorld test set as an example, the research team had a GPT-4.1-powered ReAct agent repeat the same batch of tasks 5 times each. The mean success rate (Mean@5) came in at a solid-looking 77.4%. But when they required the same task to succeed in all 5 repeated runs (Pass^5, a much stricter consistency metric), the success rate dropped to just 53.0% — a 24.4 percentage point gap between the two. On harder tasks, that gap widened to 30 percentage points. In other words, nearly a quarter of tasks that look good on average can’t actually be counted on to succeed every single time they’re rerun — and for mission-critical work like financial reconciliation or contract obligation checks, that kind of instability can be an unacceptable risk. To tackle this, the team built a diagnostic tool called the “Consistency Analyzer.” It only needs the trajectory log from a single agent run: at each decision point, it resamples k candidate outputs (k=5 by default) to pinpoint the decision nodes prone to “flipping” — the critical steps where the model could easily have landed on a different choice with just one token’s difference in sampling — all without rerunning the entire task or needing a ground-truth answer. Turning these diagnostics into guidelines shrank the consistency gap from 24.4 percentage points down to 12.0 (Pass^5 on the same task improved by 16.0 points, and by 13.0 points on similar tasks), without sacrificing mean accuracy. For the full methodology and evaluation results, see the arXiv technical report linked in the original article.

💬 JudyAI Lab Take

This research highlights a blind spot that’s easy to overlook: even when an AI agent looks great in testing, repeating the exact same task doesn’t guarantee success every time — and mean success rate quietly papers over that instability.

The team’s AppWorld results show a GPT-4.1-powered ReAct agent hitting a Mean@5 of 77.4%, but when you demand the same task succeed 5 times in a row, Pass^5 drops to just 53.0% — a 24.4 point gap that widens to 30 points on harder tasks. This points to a broader industry issue: average scores alone aren’t enough to judge whether an AI agent is reliable. For use cases that can’t tolerate mistakes — financial reconciliation, contract obligation checks — rerun stability is the metric that actually matters. What’s especially worth noting is the “Consistency Analyzer” the team built: it only needs the trajectory from a single run, resampling candidate outputs at each decision point to surface the nodes prone to flipping — no full task rerun, no ground truth needed. That’s a lightweight diagnostic design worth paying attention to in its own right. Applying it cut the consistency gap from 24.4 points to 12.0, without losing any mean accuracy.

If you’re evaluating or deploying AI agents, it’s worth checking whether your current benchmarks only report Mean@k — adding a consistency metric like Pass^k will give you a much more realistic picture of real-world usage.

📅 Original Source

🔗 Further Reading