What is EVA-Bench Data 2.0? How does it differ from the original version?

EVA-Bench Data 2.0 is ServiceNow AI's enterprise-grade benchmark for voice agents. It expands from a single domain to three major scenarios — airline customer service, enterprise IT service management (ITSM), and healthcare human resources services (HRSD) — totaling 213 scenarios and 121 tools, roughly four times the coverage of the original version. Each data point was filtered from real phone-based customer service workflows, and tool schemas were modeled after production API specs.

How do you get EVA-Bench Data 2.0? How do you get started?

All three datasets are fully open-sourced on Hugging Face Datasets and can be loaded directly with the datasets library — just search for the corresponding repos under the ServiceNow-AI account. Each domain can be pulled separately, with CSM at 50 scenarios, ITSM at 80, and HRSD at 83. The scenario generation process is fully documented in the official blog, so anyone building their own eval set can adapt the workflow.

How is EVA-Bench Data 2.0 different from typical LLM conversation benchmarks?

The difference lies in the dual constraint of 'voice agent + enterprise workflow.' Typical benchmarks are mostly text-based Q&A or single-turn tool calls, while EVA-Bench is built from actual phone customer service scripts, including multi-turn dialogue, voice segmentation, and chained tool calls — all tied to real compliance rules like healthcare NPI, FMLA, and insurance coverage. This exposes gaps in production-grade voice agents rather than just testing raw model language ability.

Why cross-validate with GPT-5.4, Gemini 3.1 Pro, and Claude Opus 4.6? What does that mean?

Having three frontier models simultaneously verify that each scenario is solvable prevents the benchmark from being biased toward any one model family, and filters out score noise caused by 'unsolvable' questions. This means all 213 scenarios have at least one reasonable solution path — a low score can't be blamed on a trick question, but reflects a real gap in agent design or tool usage, making results more comparable and fair.

What are the current limitations of EVA-Bench Data 2.0?

Right now it only supports English, so enterprises deploying multilingual customer service can't apply it directly — the team has announced a multilingual expansion but it hasn't shipped yet. The healthcare domain is built around US regulations (NPI, FMLA), so other regions would need to rewrite the compliance fields themselves. The scenarios focus on phone-based customer service workflows — pure text chatbots or visual multimodal scenarios aren't covered.

Who is EVA-Bench Data 2.0 best suited for? Is it useful for individual developers too?

It's best suited for teams building voice agents inside enterprises — especially those responsible for customer service, ITSM, or healthcare HR — who can use it directly to stress-test their agent's tool-calling accuracy. For individual developers designing eval datasets, it's a solid reference for 'reverse-engineering test scenarios from business workflows,' which exposes production gaps better than forward-designing from model capability dimensions.

What are common mistakes when running EVA-Bench Data 2.0?

The most common mistake is only measuring single-turn tool-call success rate while ignoring multi-turn dialogue state management, which inflates scores. Another is applying your own tool schema without aligning it to the benchmark's definitions, which produces meaningless low scores. A third pitfall is extrapolating English-language scores directly to Chinese deployments — differences in voice segmentation and ASR mean real-world performance often falls short of the benchmark.

EVA-Bench Data 2.0 Benchmark Release: Covering 3 Domains, 121 Tools, and 213 Test Scenarios

This article is a deep-dive from JudyAI Lab — an AI engineering playbook series with 100+ published guides, 5,000+ weekly readers across 60+ countries, focused on the practical side of running AI agents, trading systems, and content pipelines in production.

📰 Key Summary

ServiceNow AI’s research team has released EVA-Bench Data 2.0, an enterprise-grade benchmark designed specifically for voice agents. This release dramatically expands the scope, moving from a single domain to three major enterprise scenarios: airline customer service management (CSM), enterprise IT service management (ITSM), and healthcare human resources service delivery (HRSD). Together, the three domains cover 213 evaluation scenarios and 121 tools, roughly four times the coverage of the original version. Broken down by domain: airline has 50 scenarios, ITSM has 80, and HRSD has 83.

This benchmark places particular emphasis on real-world voice scenarios — every data point was filtered starting from actual phone-based customer service workflows, with tool schemas modeled after production API specs. The healthcare HRSD domain goes even deeper, tying into real US healthcare policy details like NPI provider identifiers, FMLA family leave regulations, and insurance coverage rules, ensuring the evaluation scenarios match what practitioners actually deal with day to day. All 213 scenarios were cross-validated for solvability by three frontier models — OpenAI’s GPT-5.4, Google’s Gemini 3.1 Pro, and Anthropic’s Claude Opus 4.6 — to keep the benchmark challenging while ensuring results stay fair and trustworthy.

All three datasets are fully open-sourced and can be loaded directly through Hugging Face Datasets. The team has also announced an upcoming multilingual expansion, which will push the benchmark’s scope beyond its current English-only enterprise deployment limitations. The full design principles and generation process are documented in the original post — a solid implementation reference for anyone looking to build their own eval dataset.

💬 JudyAI Lab Take

ServiceNow expanding its voice agent benchmark from a single domain to three major enterprise scenarios — airline, ITSM, and healthcare — and open-sourcing all of it signals, in our view, that standardizing enterprise AI voice evaluation has moved from a conceptual discussion to something you can actually deploy as a tool.

The most notable design principle here is filtering scenarios “starting from real phone-based customer service workflows” rather than dreaming up test questions from scratch. This approach reflects a consensus that’s taking shape: if enterprise voice AI evaluation is disconnected from actual business workflows, the resulting scores often fail to predict production performance. Having all 213 scenarios cross-validated for solvability across GPT-5.4, Gemini 3.1 Pro, and Claude Opus 4.6 — this multi-model consensus design ensures the benchmark stays both challenging and fair, rather than being friendly to just one model family. The healthcare HRSD scenarios incorporating details like NPI identifiers, FMLA leave rules, and insurance coverage also show that eval data for highly regulated domains needs to hit a certain density of business-level detail before it can actually measure meaningful differences.

If you’re designing an enterprise AI eval dataset, the scenario generation files in this open-source release are a solid starting point to reference directly — reverse-engineering test scenarios from business workflows exposes production gaps more effectively than forward-designing from model capability dimensions.

📅 Original Article Info

Published: 2026-06-04T12:24
Source: https://huggingface.co/blog/ServiceNow-AI/eva-bench-data

EVA-Bench Data 2.0 Benchmark Release: Covering 3 Domains, 121 Tools, and 213 Test Scenarios

📰 Key Summary

💬 JudyAI Lab Take

📅 Original Article Info

🔗 Further Reading

References

📰 Key Summary#

💬 JudyAI Lab Take#

📅 Original Article Info#

🔗 Further Reading#

References#

Get our weekly AI digest:

📰 Key Summary

💬 JudyAI Lab Take

📅 Original Article Info

🔗 Further Reading

References