The landscape of software development is rapidly evolving, with coding agents increasingly taking the reins. Imagine describing a task, and an agent intelligently selects the right library, writes the necessary code, executes it, and even debugs its own errors. This shift introduces a fascinating new dimension to library design: beyond correctness and speed, our tools must now be inherently “agentic” – easy for AI to discover and operate effectively.
A clunky API or outdated documentation doesn’t just frustrate human developers anymore; it can send an AI agent down a costly and inefficient path. This realization spurred our investigation into how we can optimize software for these new AI collaborators. Most traditional benchmarks only measure the final outcome, but we wanted to understand the entire process: not just if an agent succeeded, but how much effort it took, and how that varied across models, library versions, and tasks.
Benchmarking Agentic Efficiency
To tackle this challenge, we developed a specialized tool-specific benchmark designed to scrutinize the “how” behind an agent’s success. Using the popular Hugging Face Transformers library as our case study, we implemented a simple harness that runs entirely on open models, driven by the pi coding agent. All tests are fanned out across Hugging Face Jobs, ensuring every run operates on identical hardware for consistent results.
Our core belief is that software should be easy to use and easy to debug. For agentic-optimized tooling, these principles are more intertwined than ever. An agent needs to discover your tool effortlessly, your API must be crystal clear, and your documentation should be comprehensive and structured for rapid access to useful information and examples. Essentially, if you want your tool to work for an agent, you must test it for agentic-use.
Optimizing Transformers for AI Agents
Our intuition suggested that using the Transformers library could be significantly simplified for agents with a few key enhancements: a dedicated Command Line Interface (CLI), a “Skill” (a packaged set of curated docs and examples), and self-contained, task-specific examples. We’d seen similar wins with the hf CLI redesign, where agents used 1.3–1.8 times fewer tokens (and up to 6 times fewer in some cases) for specific tasks. We aimed to discover if such improvements could generalize to Transformers.
Intuition is valuable, but before proposing thousands of lines of code to a widely used codebase like Transformers, we needed concrete evidence. It’s not enough for two agents to achieve the same correct classification; the path they take matters immensely. One agent might pipe a complex Python script, while another achieves the same result with a single, elegant CLI command. The latter is far more efficient in terms of cost, latency, token usage, and potential for failures.
Our benchmark aims to quantify the “work” an agent expends on a task and assess whether changes to the library genuinely improve performance. We categorize agent interactions into three variants, or “tiers”:
- Bare: A minimal setup with just
pip install transformers. - Clone: The agent has access to the full Transformers source code in its working directory.
- Skill: The agent is provided with a curated “Skill” – a package containing the CLI’s documentation and task examples.
These tiers aren’t nested; each offers a distinct type of assistance to the agent, allowing us to pinpoint the most effective support mechanisms. Interestingly, a model might sometimes perform better on the ‘clone’ variant than ‘skill’, highlighting the nuances of agent interaction.
Benchmarking Different Models and Revisions
The choice of agent model significantly impacts benchmarking strategy. For large, highly capable open models, task completion often nears 100%, making “effort” metrics more relevant. How many turns did it take? How many tokens? Was the path clean, or did it use deprecated APIs? For smaller, local models, “match %” is crucial, revealing how model size and capabilities affect performance on your specific tool.
Our harness scores every run on multiple axes, including match percentage, median new tokens, and median time per turn. This comprehensive data is then compiled into an interactive report, allowing users to delve into specific runs and even examine the agent’s exact command-by-command trace via the Hugging Face Hub’s agent-traces viewer.
When benchmarking large open models, we primarily focused on varying the Transformers revision. Since these models usually find the correct answer, we measured the effort required. Our findings showed that introducing a dedicated CLI and Skill significantly reduced the time agents spent on tasks. The “Skill” commit, in particular, proved to be the fastest path to task completion.
However, an interesting trade-off emerged: while the Skill commit reduced time, cloning the repository with the CLI and examples led to a substantial increase in token consumption. This is because approximately a third of the runs in the ‘clone’ variant involved agents reading the newly added CLI implementation and example scripts to learn the interface. This raised the median input from ~4k to ~6.4k tokens. This trade-off between time saved and tokens consumed is a crucial insight for library maintainers.
It’s important to note a caveat for the CLI: our benchmark evaluates single, fresh runs. In real-world usage, an agent learns the interface once and then applies that knowledge across multiple tasks within the same session, amortizing the discovery cost. The token bump we observed is therefore closer to a worst-case scenario.
Assessing Smaller Models
For smaller open models, the focus shifts. Here, we held the revision constant and varied the model itself. This helps identify which models can reliably handle tool calls and tasks, not just based on token count or time, but on their ability to get the correct answer. Our intuition that smaller models would struggle more with both tool use and complex tasks was largely confirmed by the results.
Introducing the concept of “markers” allows us to look beyond success and quantify “what happened under the hood.” A marker is a named pattern that the harness’s profile (a tool-specific plugin) matches against a run, providing a one-line label for specific agent behaviors, such as the shell commands executed or the code written. These markers offer invaluable insights into the agent’s decision-making process, helping us understand not just if an agent succeeded, but how.
Source: Hugging Face Blog
