The primary factor hindering AI agents is legacy infrastructure, not the models themselves. This consensus emerged among three infrastructure leaders at VB Transform 2026, representing LinkedIn, Walmart, and Zendesk.
The panel featured Animesh Singh, Senior Director of AI Platforms and Infrastructure at LinkedIn; Desiree Gosby, SVP of Corporate Technology Services and Strategy at Walmart; and Sami Ghoche, VP of Applied AI at Zendesk. Each expert shared insights on the challenges encountered while transitioning agents from pilot projects to full-scale production, concluding that none of these challenges stemmed from model limitations.
A common theme connected their insights: enterprise infrastructure is typically designed for human workflows, not for agent operations. This disparity in efficiency is where significant engineering advancements are required.
When asked about empowering Walmart’s internal staff, Gosby emphasized the necessity of ensuring that “engineering never becomes a bottleneck for our initiatives.”
Identifying the Bottleneck
Each company faced unique variations of a familiar challenge. Infrastructure designed for human tasks becomes inefficient when agents are expected to execute those tasks autonomously.
At LinkedIn, the initial stumbling block was Kubernetes, which depends on demand-based container start-ups, causing delays. Singh explained that this latency was problematic for the agents. The solution involved shifting from on-demand provisioning to a pool of pre-provisioned containers that could manage agent workloads in real-time.
Once LinkedIn granted agents control over orchestration, a second challenge arose. Despite a seemingly straightforward 5-point rating system, persistent “hallucinations” occurred. Singh noted that structural deficiencies were at play, indicating that an LLM evaluating another LLM would encounter similar issues.
“We developed our own harness and control flow, pushing the LLM to the output instead of modifying the loop,” Singh explained. Currently, about 80% of workflows utilize deterministic coded scripts, engaging LLMs solely for inference, with all evidence carefully logged before proceeding.
Walmart’s bottleneck emerged from its success. By empowering employees with the agent harness, “citizen developers” began independently creating agents to resolve issues previously requiring formal engineering, leading to true innovation but also overlap and redundancy. The solution was not to limit the harness but to recognize duplication, standardize the most effective agents, and establish governance for streamlined production.
At Zendesk, the bottleneck was data-oriented. Ghosh, part of Zendesk’s Acquisition of Forethought team, disclosed that they manage a rich repository of 20 billion customer conversations. The initial idea was to harness this data through a large language model for agent generation, but Ghosh emphasized, “This approach won’t work. Instead, substantial investment in the underlying data pipeline and infrastructure is crucial.”
The Importance of Open Source
The three leaders shared similar perspectives on the role of open-source solutions: prioritize ownership while resorting to Frontier Labs only when distinct advantages exist.
Ghosh indicated that most companies favor owning their models and infrastructure, which informs Zendesk’s exclusive approach. Exceptions remain for frontier inference, previously managed by labs, although these use cases have diminished compared to other AI applications.
LinkedIn’s strategy involved developing two independent subsystems: the AI Gateway, which provides a singular interface for all outbound model calls, and a memory subsystem designed to retain context independently from model providers.
“Every outbound request to LLM adheres to uniform semantics and API calls, whether hosted on public cloud or on-premises, allowing seamless transitions between providers,” Singh stated.
Walmart has comprehensive internal gateways accommodating three workload categories: fully deterministic workflows, those incorporating planning and inference, and hybrid options. Compliance-focused tasks are inherently deterministic. Governance, security, and evaluations occur through this gateway, regardless of the model employed. For Gosby, choosing between frontier models or open-weight models hinges on the best fit for each workload, rather than adhering to a fixed policy.
Recommendations for Modernization
Three crucial pieces of advice surfaced, each linked to obstacles faced by the leaders:
Prioritize evaluation from the start. Ghosh suggested that this is vital for both internal and customer-facing applications. “What unites them is evaluation, forcing a decomposition of the problem. A robust evaluation system accelerates progress,” he remarked.
Take ownership of your agent harness from day one. Gosby advocated for empowering employees with the AI harness early on, supplemented by infrastructure to oversee AI outputs. “This approach unlocks vast innovation potential,” she asserted.
Develop model and context agnostically. Flexibility is crucial for success. “Whether utilizing today’s frontier model or tomorrow’s open-source alternative, we maintain an independence-focused approach,” Singh concluded. “Preserve that context within your enterprise, allowing reuse when deploying future models or harnesses.”
Source: venturebeat.com