This week, top business leaders and technology experts gathered at the luxurious Hotel Nia in Menlo Park for VB Transform 2026, the leading conference focused on leveraging generative AI agents to enhance business outcomes.
Rachad Alao, the Vice President of Product Engineering at the emerging Canadian enterprise AI startup Cohere, engaged with VentureBeat CEO and Editor-in-Chief Matt Marshall to delve into the complexities of building agent systems while safeguarding sensitive data, maintaining infrastructure control, and ensuring vendor flexibility.
Alao, who previously spearheaded responsible AI initiatives and reliability engineering teams at Google and Meta, emphasized that AI sovereignty transcends merely downloading open models or operating applications behind corporate firewalls.
When discussing Cohere’s definition of sovereignty, Alao highlighted organizations managing mission-critical systems, such as banks, hospitals, and government entities.
“It’s crucial to maintain strict control over data storage and AI operations,” he stated, noting that AI functionalities should only occur in jurisdictions that organizations fully understand or can directly govern.
This encompasses everything from GPUs and private cloud infrastructure to governance systems orchestrating requests between models, as well as connectors and tools that interface with enterprise data.
“We aim to control the entire stack,” Alao asserted.
Agent Workload May Surpass Declining Token Prices
Marshall challenged the prevailing economic notion regarding small-scale, locally deployed models, pointing out that inference costs continue to decline swiftly, which could diminish the rationale for optimizing all tokens.
Alao rebutted this, indicating that overall consumption is escalating even more rapidly as companies transition from basic chatbots to sophisticated agents capable of multi-step reasoning, invoking tools, and efficiently searching internal systems.
“Token usage is growing exponentially as we tackle more complex agent use cases,” he noted, highlighting that these processes require extensive computation and tool interactions.
Alao also contrasted Cohere’s approach with other providers who bill customers based on token consumption.
“If your pricing model revolves around token usage, your goal is to maximize that usage,” he explained. “We do not plan to sell our model or platform in this manner.”
Instead, Cohere is dedicated to aiding companies in resolving their most challenging problems privately and securely while minimizing unnecessary model utilization. His recommendation is straightforward: deploy the appropriate model for each specific task.
Rather than directing every request to the largest frontier model, businesses should categorize tasks based on the intelligence required and the associated sensitivity or regulatory requirements.
Alao pointed to an anonymous Canadian bank utilizing Cohere’s on-premises model for sensitive, regulated workloads, while routing less critical tasks that demand higher intelligence to a larger frontier model via Cohere’s North platform.
“Model routing proves extremely beneficial,” he concluded.
Small Models Ideal for Enterprise Operations
In response to a question regarding how Cohere’s open-source North Mini Code, launched last month, could compete with proprietary coding models, Alao acknowledged that while the larger Frontier model might excel on the toughest challenges, its advantages may not warrant indiscriminate usage.
“For approximately 80% of our use cases, it was significantly more efficient and cost-effective,” Alao remarked about the developers adopting this model.
Cohere’s North Mini Code operates on a single Nvidia H100 GPU and is designed for agent software engineering, covering areas such as terminal work, code review, and tool usage.
The company also unveiled Command A+, a 218 billion parameter expert mixture model, engaging only 25 billion parameters per generation step.
The compressed 4-bit version lessens the resource demands for private deployments, while the Apache 2.0 license provides enterprises with greater operational freedom and modification capabilities.
Integrating Search into Agent Workflows
When asked about Cohere’s extensive work in embedding and enterprise search, Alao shared that the focus has shifted beyond merely capturing text to a comprehensive understanding of multimodal search.
“Today’s state of the art embraces multimodal search,” he said. “It encompasses more than just text.”
Searching through documents, images, and varied information sources is evolving into a vital aspect of agent workflows, he added. Models will determine the optimal use of search capabilities, much like any other tool.
When queried about what would motivate companies to transition away from bundled AI services provided by existing cloud vendors, Alao reiterated the importance of data control and portability.
“If you value sovereignty, it’s essential to have greater control over your data,” he explained. He highlighted that Cohere’s governance layer enables clients to channel traffic to the most suitable models, thereby alleviating vendor lock-in issues prevalent among many customers.
Source: venturebeat.com