The buzz around artificial intelligence agents is undeniable, promising a new era of autonomous operations and efficiency. Yet, beneath the surface of this excitement, many enterprises are grappling with a fundamental question: are these powerful tools truly delivering a tangible return on investment? This critical query comes into sharper focus following a stark prediction from tech analyst Gartner.
Gartner anticipates that a significant 40% of enterprises will scale back or entirely decommission autonomous AI agents by 2027. The primary culprit? Governance gaps that often remain hidden until these agents are fully operational and incidents inevitably occur. So, how can organizations navigate this challenge and ensure their AI agent deployments are not only successful but also sustainable?
At the recent Snowflake Summit in San Francisco, three prominent digital leaders shared their real-world experiences in deploying AI agents into production. Their collective wisdom offered crucial insights, boiling down to three actionable lessons for any professional seeking to harness the true potential of AI. These pillars of success involve implementing robust frameworks, leveraging human expertise, and strategically monetizing data.
Moving Beyond Hype: Realizing AI Agent ROI
Matt Luizzi, VP of Analytics at wearable technology innovator Whoop, provided a compelling case study for successful AI agent integration. Whoop collects biometric data around the clock, using it to fuel insightful health and wellness analytics, with Snowflake serving as the backbone for their internal analytics services. Luizzi highlighted how AI agents are becoming increasingly central to this process.
Specifically, Whoop has embraced Snowflake CoCo, an advanced coding agent designed for developers and data engineers. Luizzi explained their initial cautious rollout: “We’ve been using CoCo for several months now, and started with just the analytics team, which is people who could quickly look at a query response and say this is correct or not, and trying to figure out how to scale that process out.” This measured approach allowed them to build confidence and refine their methods.
Now, CoCo is being scaled across the organization, supported by more formalized evaluation frameworks. Whoop’s software engineers, for instance, utilize CoCo to analyze A/B test results, propose subsequent features, and rapidly iterate on product development. This method is dramatically accelerating the delivery of both business and customer value by automating key aspects of the experimentation framework.
Luizzi emphasized that Whoop was fortunate to have a robust data infrastructure already in place, with all their data centralized on the Snowflake platform. This solid foundation enabled them to quickly begin testing agents with Snowflake’s Cortex AI service and learn invaluable lessons. A key takeaway was that “context was everything,” necessitating a deep reliance on the semantic layer to ensure context was meticulously structured.
For Luizzi, the ultimate lesson is the indispensable role of comprehensive frameworks in successful agentic AI exploration. He stressed the importance of building repeatable processes, just as they’ve done with their data architecture for a decade. This strategic commitment to repeatable frameworks is enabling Whoop to scale its AI workloads efficiently and effectively.
The Foundation of Success: Data & Expertise
Madeleine Want, VP of Data at sports giant Fanatics, manages the intricate data engineering, data science, and machine learning operations for the company’s betting and gaming division. Like Whoop, Fanatics relies on the Snowflake platform to support these extensive activities. Want’s team embarked on their AI agent journey with an experimental mindset, ready to discover what truly worked.
They quickly learned a crucial lesson: the success of AI agents is directly tied to the condition and governance of the underlying data. Want noted, “the better the condition of the underlying data and the better the governance of it, the more easily the LLM was able to derive meaning and answer questions effectively.” While this might seem obvious now, she pointed out it was less so just 18 months prior.
Fanatics, accustomed to building bespoke machine learning models, found it initially challenging to trust importing third-party models directly onto their data for analysis. However, this approach has now become deeply embedded in their operational DNA. Their early successes emerged in domains with well-defined contexts, where expert analysts could effectively “coach” the agents, guiding them to accurate insights.
Over time, Fanatics has seen continued progress. The investment required in the context layer is decreasing, and agents need less supervision before autonomously answering questions. Moreover, their ability to measure the accuracy of agent responses is significantly improving, thanks to the introduction of scaled evaluation frameworks. This provides critical confidence in agents’ performance, even when operating unsupervised.
These mounting successes mean the scope of agents at Fanatics is rapidly expanding beyond mere analytics. Other departments are now recognizing the immense potential and are eager to explore agent-driven solutions. While still leveraging Snowflake’s interfaces and agents, Fanatics is increasingly embedding APIs and responses into other third-party tools, allowing users to interact with data-powered insights wherever they work.
Balancing Automation and Autonomy for Impact
Sriram Sitaraman, CIO at software specialist Synopsys, detailed how his long-standing Snowflake customer organization utilizes the platform and its agentic services, including CoCo, to drive decision-making. About 18 months ago, Synopsys recognized the significant potential of AI agents to streamline tasks typically performed by junior employees, such as running quick queries, generating graphs, and extracting insights.
This led them to deploy various specialized “knowledge agents.” For instance, a revenue agent supports the finance department by automating report generation, while a debug agent assists with their data center’s ticketing system. Sitaraman’s team evaluated the impact of AI across three crucial dimensions: the quality of results, the time taken to achieve results, and the overall cost. Remarkably, AI demonstrated a positive impact in all three areas.
“In the past, you had to sacrifice one or the other,” Sitaraman reflected, highlighting this as a significant breakthrough. Now, instead of constantly reprogramming systems for context changes, the focus can shift squarely to deriving insights without the underlying operational concerns. His advice for those embarking on an agentic journey is clear: “Start with data — monetize your data using AI.”
Sitaraman emphasized the linear scalability of AI: “It doesn’t matter how much volume you throw at the initiative, because AI is just truly a linear scale. The more data AI has, the better decisions it makes.” However, he also issued a vital caution about the subtle yet critical distinction between automation and autonomy. “One thing we realized is there’s not a lot of difference today between automation and autonomy, and so you have to be careful.”
It’s crucial to identify whether a process truly requires automation or full agentic autonomy, as each comes with different cost structures, usage patterns, and governance requirements. Sitaraman urged professionals to meticulously identify the right use cases, build robust frameworks, and never underestimate an agent’s potential scope. An agent designed for sales operations, for instance, might inadvertently evolve into a sales analyst agent, demanding careful oversight.
Therefore, thorough thought processes, strong frameworks, and appropriate skill sets are paramount for strategic and successful AI agent deployment. By following these lessons – prioritizing robust frameworks, leveraging expert human guidance, and strategically monetizing data – organizations can navigate the complexities of AI agents and build solutions that truly deliver lasting value.
Source: ZDNet – AI
