From mainframes to microservices, enterprises have evolved their software architectures to meet growing demands for scalability, flexibility, and adaptability. Today’s reality demands more than just distribution or decomposition. In a world of real-time market shifts and ever-rising customer expectations, businesses need systems that can reason, adapt, and act on their own. This is the promise of agentic systems. This promise comes with a profound shift in what we expect from the infrastructure, especially the network.
Legacy networking was built around deterministic workflows – static APIs, predictable service calls and pre-defined paths. On the other hand, agentic systems are dynamic and emergent. They interpret user intent in natural language, orchestrate chains of agents and tools in real time, and make decisions on the fly. Security, observability, and routing is no longer just about headers or URLs, it’s about semantics and context. This shift demands a new kind of networking, one that supports intelligent adaptive policies at runtime, and provides observability across multi-hop workflows. We should future proof investments like this by supporting standard protocols. This new paradigm needs an “agent mesh”—a platform that enables security, observability, discovery, and governance across all agent interactions no matter where those agents, LLMs or tools are deployed. This post lays out that vision, and how we at Solo.io are ready to deliver this infrastructure.
An Agent Mesh
We believe enterprises building and deploying agentic systems will need to solve common security, observability, tenancy, and guardrail concerns. An agent mesh solves these challenges consistently for enterprise agent deployments (self-built, SaaS, developer tools like Cursor, etc) leveraging an agent gateway, a data plane tailored for these use cases.
An agent mesh consistently solves AI-specific networking challenges across three main interactions:
- Agent to LLM communication
- Agent to Tools / MCP
- Agent to Agent / A2A
An agent mesh has the following properties:
- Secure by default: Agent identity, mTLS, and pluggable auth (OIDC, API keys, etc.)
- Layer 7 native: Supports agent-to-agent (A2A) and model control plane (MCP) communication
- Fine-grained access control: Authorization control for all agent and tool interactions
- End-to-end observability: Unified tracing across LLMs, agents, and tools
- Registration and discovery: Runtime agent/tool registration and lookup
- Resilience and safety: Guardrails, tool poisoning protection, and tenancy isolation
- Modern ops model: Declarative config and GitOps workflows
An agent mesh uses a specialized, highly performant data plane, an agent gateway, that’s been tuned and optimized for AI communication patterns.
Let’s dig into the details of an agent mesh.
Controlling Agent to LLM Communication
Agents often exchange sensitive prompts and payloads with an LLM—customer data, access tokens, internal documents—and this traffic shouldn’t go directly over the wire or bypass enterprise policies. That’s where an agent gateway must act as an “LLM gateway”. It can enforce policies such as guardrails, caching, and failover based on semantic understanding of the prompt. It can also do token-based rate limiting and usage tracking per user, tenant, or agent. Tracing what calls are sent to the LLM (and what responses were returned) is paramount. Tracking metrics like number of tokens consumed, guardrails tripped, and failovers detected is necessary to support enterprise observability.
An agent gateway can also make smart decisions for self-hosted inference backends (ie, AI models) backed by GPUs. It can use real-time telemetry to route prompts to the best available model taking into account metrics such as GPU KV cache, work queue depth, adapter deployment and criticality. The Inference Extension to Gateway API specifies an API for doing this.
Enabling Agents with Enterprise Tools
Agents individually cannot do interesting things without access to tools. Connecting LLMs to tools is a very open-ended problem which is recently solved with the Model Context Protocol (MCP). Describing tools, exposing them to agents, and allowing them to be invoked in a consistent manner (protocol) regardless of what the backing tool (database, API, file system, SaaS, etc) is addressed by MCP. The challenge with MCP is that it is a nice foundation for RPC communication but it lacks enterprise readiness. Security, discovery, registration, tenancy and observability are left out of the protocol. Since agents select tools based on their names and descriptions, semantic understanding and guardrailing of these tools is crucial. Overlooking this fact can lead to serious security issues.
Multi-Agent Task Based Workflows
The challenge with building agents isn’t just making them smart—it’s right-sizing them for the specific workflows they’re meant to tackle. Giving a single, monolithic agent access to too many tools or responsibilities often leads to agent confusion. The agent will hallucinate, wander off-task, or fail entirely. Instead, we’re finding that organizations benefit more from smaller, focused agents aligned to well-bounded goals or tasks. But increasing the number of agents introduces complexity. Agents may need to communicate. How do we ensure observability, traceability, and secure interactions—things like authentication, authorization, and behavioral guardrails?
Google recently announced a new specification for agent-to-agent communication, called the A2A protocol. This spec defines how agents declare their capabilities and skills to coordinate tasks, get task updates, and execute on tasks regardless of the underlying framework used to build them. The protocol publishes skills and capabilities of agents through Agent Cards which are discoverable at runtime and contain both technical and semantic details for calling a specific agent. While the specification mentions transport security and observability, it intentionally keeps these outside of the core protocol. Registration, discovery, and routing are also left out.
Digging into the Details: Securing the Foundation
To work through an assigned task, agents will be passing sensitive information, retrieving data from enterprise APIs, and potentially communicating with customers. While users may have identities represented with tokens and keys, agents don’t have any specific identity. This is where non-human, workload identity becomes crucial. Identity frameworks like SPIFFE become foundational.
Agent identity should be combined with user identity to enforce fine-grained authorizations. All of this traffic should be cryptographically secured with mTLS and rooted in an enterprise’s PKI. All traffic should be “deny-all” except for those communication paths explicitly allowed within the guardrails of an agentic system. In other words, agentic systems should be built on a zero-trust architecture. To accomplish this, we can deploy Istio ambient mode (ztunnel). This will give us agent identity based on mTLS for all calls between agents and tools transparently (without any special libraries or sidecars). From there we can layer on the composite identity to include users to build fine-grained authorizations.
An enterprise will likely have many agents and each of those agents will need access to tools. They will need a way to register, discover, and access these agents or tools in a secure, observable way with guardrails. Agents will need to publish approved AgentCards to specify their capabilities.
Enterprises should expose curated lists of MCP tools via MCP proxies, but the MCP protocol does not lend itself to proxying with a traditional reverse proxy. Traditional reverse proxies are built with a stateless “one request to one backend” model. MCP is both stateful and allows sessions to handle traffic initiated by the client or the server. Since MCP is stateful and requires multiplexing and demultiplexing to proxy, we should not try to shoehorn this into the reverse proxy model. The A2A protocol falls into a similar situation.
In an effort to apply consistent authentication, authorization, and observability policies to these protocols, we will need a dedicated agent data plane that understands these types of protocols. That’s where agent gateway comes into the picture. Agent gateway is built to natively understand MCP and A2A and provide the following:
- Virtual MCP tool endpoints
- Authentication
- Fine-grained authorizations (user, machine, agent, combinations, etc)
- Tracing, observability (based on semantic OTeL)
- Agent routing / task status observability
- Multiplexing across multiple backend MCP servers
- Version management and decoupling
- Guardrails, prevent tool poisoning attacks, etc
- Manage health and failover of backend MCP servers or agents
Registration
In dynamic agent ecosystems, managing the registration of agents and tools is crucial to prevent issues like name collisions and tool poisoning. The agent mesh supports a curated registration processes, where agents and tools are registered with unique identifiers and metadata, such as AgentCards. These AgentCards provide semantic descriptions of capabilities, facilitating discovery and interoperability. We need a way to detect tool or agent semantic capability or description drift to prevent prompt injection or semantic poisoning. The mesh must also implement validation mechanisms to detect and prevent malicious registrations, ensuring the integrity and security of the system.
Any Agents across any cloud
A significant advantage of the agent mesh is its ability to support a diverse range of agents—whether self-built, off-the-shelf, or hosted—across any cloud environment. Unlike solutions that are tightly coupled with specific cloud providers, the agent mesh offers flexibility and interoperability, allowing enterprises to integrate various agent types without vendor lock-in. This openness accelerates innovation, enabling organizations to adopt the best tools and models for their specific needs, regardless of their origin or hosting environment.
Pieces are Composable
The agent mesh is designed with composability in mind, allowing enterprises to assemble a tailored infrastructure that meets their unique requirements. Its architecture supports unified and consistent access to models, tools, and agents, ensuring seamless integration and communication. Policy enforcement, observability, and guardrails are uniformly applied across all components, enhancing security and compliance. The specialized data plane is optimized for performance, focusing on AI-specific communication patterns. Semantic understanding is embedded at every layer, from ingress to east-west traffic and egress, providing intelligent routing and decision-making capabilities. This composable approach empowers enterprises to build scalable, adaptable, and secure agentic systems.
Conclusion
The evolution of enterprise software architectures demands systems that are not only scalable and flexible but also intelligent and autonomous. Agentic systems fulfill this need, but they introduce new challenges in networking, security, and observability. The agent mesh addresses these challenges by providing a robust infrastructure that supports secure, observable, and governed interactions among agents, tools, and models. With components like Kagent and support for protocols like MCP and A2A, the agent mesh enables enterprises to harness the full potential of agentic AI across any cloud environment. By adopting an agent mesh, organizations can build intelligent systems that are ready to meet the demands of today's dynamic business landscape.
Check out the Agent Gateway website to learn more and join the Agent Gateway conversation on Discord.
.webp)
