Nvidia published new research on Friday showing that the “harness” — the scaffolding of tools, memory, and runtime around an AI model — is often more critical to agentic performance than the model itself. In tests on the interactive reasoning benchmark ARC-AGI-3, researchers got Anthropic’s Claude Opus 5 to a perfect 100% score by pairing it with a custom harness that included a “supervisor” component. Without that harness, the same model scored just 30%, which was still the best among all models tested.
The findings add to a growing body of evidence that enterprises and developers focused solely on picking the “smartest” model may be missing the bigger lever for building reliable AI agents. For long-horizon tasks — work that requires stringing together many decisions over hours or days — the harness is what keeps the agent on track, manages its context window, and feeds it the right tools at the right time.
Also read: Harvard’s $699 Startup Bootcamp Deploys AI Avatars of Its Instructors
The harness is what makes a model an agent
“Generally speaking the world interprets an agent almost as an API of the model,” Adel El Hallack, vice president of product in Nvidia’s AI unit, told TechCrunch. “But an agent is actually more than that. It is the model. It is the scaffolding around the model, which we call the harness, i.e. the set of tools that it utilizes. It is the runtime and the associated skills and libraries that we give it access to.”
This distinction has practical consequences. Microsoft published research in April that tested 19 large language models on long-horizon document editing tasks and found that all of them, including frontier models, filled the documents with errors. In other documented cases, models acting autonomously have deleted users’ files, dropped entire databases, or resorted to collusion and hacking to achieve their objectives. A well-designed harness, Nvidia argues, can mitigate these failure modes by adding guardrails and supervision.
ARC-AGI-3: A benchmark that stings OpenAI
The choice of ARC-AGI-3 for Nvidia’s tests is notable. The benchmark consists of a set of 2D games with no instructions — the model has to figure out how to play and win on its own. A 100% score means the model performs at human level. OpenAI has been particularly sensitive about this benchmark; its models scored below 10%, prompting the company to conduct its own research last month. OpenAI found that tweaking just two harness settings tripled its models’ scores — but none reached 100%.
Nvidia’s researchers achieved the perfect score by adding a “supervisor” agent that monitors the main agent’s progress. “The more interesting part was introducing a supervising agent in addition to your main agent that’s doing the work,” El Hallack said. It “almost acts like a CEO to nudge the agent when it goes off direction or starts exploring a path that it might lead to a dead end, or re-explore a path that it had previously trod.”
The concept of a supervising agent isn’t new, but most current agentic tools — such as Claude Code, Codex, or Hermes — rely on a single layer. Nvidia’s research harness, called Agentic Variation Operators (AVO), is not a commercial product. Instead, the company provides open building blocks under its Nemo brand, some commercial and much of it openly available.
Why this matters for AI costs and control
Nvidia’s results reinforce the idea that model choice is far from the only factor in agentic performance — or cost. In July, Databricks published research showing that the harness can dramatically impact AI spend. “You can pick the same model but different harnesses, and you get significantly more cost if you use the wrong harness,” Databricks CEO Ali Ghodsi told TechCrunch. “So you think, oh, this is an expensive model. This is a cheap model. But wait, which harness are you using? That itself can 2x your cost.”
For enterprises, the takeaway is that building or choosing a sturdy harness may be as important as selecting the underlying model. Nvidia’s broader point is that open harnesses give users more control. “We believe, and we’re demonstrating with the ecosystem, how open harnesses allow you to turn a lot more knobs to drive up that accuracy,” El Hallack said. He also linked the approach to security: “We believe in having an open agent stack — where you have control across the harness, across the infrastructure, across the runtime — is what’s required for us to usher the ecosystem forward and securely.”
As AI agents move from demos to production, the harness is becoming the battleground for reliability, cost, and safety. Nvidia’s research suggests that the industry’s focus on model benchmarks may be obscuring a more practical truth: the way you wire the model up matters just as much as the model itself.
Disclaimer: This article discusses AI research and industry trends. It does not constitute financial advice or a recommendation for any specific technology investment. The AI market is volatile and uncertain; readers should conduct their own research before making decisions.
Be the first to comment