📰 Key Summary

Design Arena co-founder Grace Li says the company started a few weeks before graduation in 2025, when a group of college friends were trying to get their own AI game engine working. The models could generate playable games, but none of them were actually fun, which got the team thinking about how to judge whether a game is enjoyable. They concluded human judgment can’t be replaced, so they set out to figure out how to collect real human feedback at scale, and that’s how Design Arena — the AI evaluation tool they eventually built — came to be. It now has 5.3 million users worldwide. Plenty of AI companies are looking for a scalable way to get user feedback, and they’re willing to pay for it. Li describes this as a key bottleneck many models struggle to break through on the design side, and the team closed its first big deal with a frontier lab about a week after launch.

For everyday users, Design Arena works kind of like an advanced model router: there’s a ChatGPT-style prompt box, and you can pick from more than a dozen visual formats — websites, images, and so on. You type in what you want, the format, and the style, and the system shows you a series of “A vs. B” comparisons until you’ve ranked multiple outputs from best to worst. The real business value comes from the enterprise side — models on the platform can use it as a real-time feedback source to keep improving what they generate. Since users generally don’t care which model is behind the output and just want the best result, that ranking data ends up reflecting genuine user preferences pretty accurately. Li says the platform’s annual recurring revenue (ARR) has hit $60 million now, cementing its position as a key source of human evaluation data for the AI industry.

Since users have to log in to get outputs, Intelligence (the company’s name) can also track how aesthetic preferences shift across regions and over time — Li mentioned, for example, that web dashboard design in Asia tends to lean maximalist. This kind of human evaluation is seen as an important complement to automated benchmarks, which can run at scale but are easy to game, as last week’s Hugging Face breach showed. That said, crowdsourced human feedback isn’t a guaranteed win of a market — a similar startup, Yupp, backed by a16z crypto’s Chris Dixon with $33 million in funding, shut down earlier this year less than a year after launch, even though it had landed some frontier model customers too and claimed over 1.3 million users. It just couldn’t build a sustainable long-term business model.

This round for Design Arena was a $7.9 million seed led by Index Ventures, with participation from Conviction (Sarah Guo and Mike Vernal), A*, and Valkyrie. See the original article for more details.

💬 JudyAI Lab Take

Design Arena’s story shows the AI industry is looking for a scalable answer to “is this model output actually good?” This company built up 5.3 million users through A/B comparisons across more than a dozen formats, and its ARR has hit $60 million — a sign that frontier labs are willing to pay for real human preference data instead of relying solely on automated benchmarks.

That’s worth noting if you’re building AI products: automated benchmarks can run at scale, but as the Hugging Face breach showed, they’re easy to game — and human evaluation fills exactly that gap. At the same time, the regional taste differences Li mentioned (like Asian web design leaning maximalist) show that user preference isn’t one-size-fits-all — product design needs to account for local context. But Yupp shutting down in under a year, despite $33 million in funding and 1.3 million users, is a reminder that crowdsourced human feedback as a category doesn’t guarantee a sustainable business on its own.

If you’re building an AI product, it’s worth asking: beyond automated testing, is there a low-cost way to collect real “A vs. B” preference feedback from actual users?

📅 Original Article Info

🔗 Further Reading