We recently built a data plane called agent gateway that sits between agents and LLMs, tools, and other agents, adding the critical capabilities Model Context Protocol (MCP) and Agent to Agent (A2A) leave out. We covered an agentic communication platform called Agent Mesh in another post which leverages agent gateway. In this post, we’ll explore why traditional API gateways don’t fit well with MCP and A2A, and how agent gateway enables secure, scalable agentic systems in the enterprise.
MCP has become the de facto standard for connecting LLMs to tools, giving agents a structured way to plan, act, and interact with real systems. A2A is a newly introduced complementary protocol to MCP that solves for long-running tasks and state management across multiple agents. But while these protocols define an RPC communication protocol, it stops short of addressing the realities of enterprise environments—things like authentication, authorization, tracing, tenancy, and guardrails.
Agent RPC: A Foundation, Not a Full Stack
MCP and A2A are both RPC protocols that define the structure of interactions—how an agent describes what it wants to do, how it calls tools, and how it hands off tasks to other agents. However, while these protocols offer a solid starting point, they intentionally leave many real-world concerns to the implementer. In a real enterprise environment, these interactions will likely be “remote” vs “local”. In a “remote” scenario, requests flow over the network, and as Kevin Swiber notes in this recent blog on MCP architecture, you will have to pay the “distributed systems tax”.
Solving for latency, retries, timeouts, failures/partial failures must be implemented, but where? Microservice environments have these same issues, so can we just re-use the same proxies, gateways, and service mesh from that environment? To understand that, we need to dig a little deeper into the respective protocols.
MCP and A2A both introduce scenarios that don’t play well with common API gateway or reverse proxy infrastructure.
For example:
- Older versions of MCP required multiple TCP connections per tool session, leading to heavy connection overhead at scale.
- Session fan-out use cases—where a single client request (like "list available tools") needs to fan out across multiple backend MCP servers, collect responses, and assemble a unified answer
- Server-initiated events, such as backend tools pushing real-time updates via Server-Sent Events (SSE), need to be properly routed and multiplexed back through the client session without breaking the JSON-RPC context.
- Protocol upgrades and fallback mechanisms must be negotiated gracefully to avoid client or server failures as MCP/A2A evolves.
To make MCP and A2A viable for real enterprise adoption, we need a purpose-built data plane that understands these sessions, manages stateful connections efficiently, and enforces enterprise-grade security, resilience, observability, and multi-tenancy.
That’s exactly the gap agent gateway fills.
- A single endpoint to federate multiple backend MCP servers
- Virtualization of tool servers on a per-client basis
- Multiplexing tool calls to backend servers with strict authorization enforcement
- Support for multiple security protocols between clients and MCP servers
- Support for diverse security protocols between MCP servers and backend tools
- Registration and approval workflows for MCP servers
- Tracing and observability of incoming MCP client and for all backend tool interactions
- Protection against tool poisoning attacks (e.g. direct tampering, shadowing, rug pulls)
- Seamless interoperability with existing enterprise APIs (e.g. automatic OpenAPI ←> MCP translation)
- Support for MCP version negotiation to decouple client and backend upgrades
- Multi-tenancy support to enable scalable, enterprise-wide adoption
Why not use an existing reverse proxy?
Traditional API gateways and reverse proxies aren’t built for MCP, and adapting them isn’t straightforward. These systems are optimized for stateless REST-style interactions, common in microservices architectures. They operate with a simple model: one request in, pick a backend endpoint, one request out, with no session context or ongoing connection state.
MCP, by contrast, is a stateful protocol based on JSON-RPC. MCP clients and servers maintain long-lived sessions, and every request is tied to that session context. Servers must track each client session and can even initiate messages back to the client asynchronously. Properly routing these interactions requires a deep understanding of the JSON-RPC message body—something traditional proxies aren't equipped to handle.
Trying to virtualize MCP servers behind a generic proxy quickly runs into multiplexing and demultiplexing challenges. A single client request (e.g., "list all available tools") may need to fan out across multiple backend MCP servers, aggregate the responses, and return a single coherent result. And since different clients may have access to different tools, the proxy must dynamically adjust what gets exposed on a per-session basis.
Moreover, MCP servers can initiate events or requests independently of any inbound call. This means the proxy not only needs to track stateful sessions, but also maintain a two-way mapping between client sessions and backend servers. Without purpose-built session and message awareness, traditional proxies fall short.
Safe, efficient implementation
We built agent gateway in Rust because performance and memory safety are non-negotiable for this kind of system. We went through the same exercise when building the Istio Ambient Ztunnel. Using a dedicated data plane for the specific needs of the networking layer has worked out best.
Stateful protocols are a different layer altogether. Managing stateful sessions, long-lived connections, and fan-out patterns—where a single client can drive multiple concurrent sessions across backend servers—can quickly become resource-intensive. Every millisecond and megabyte counts. Rust gives us fine-grained control over system resources without compromising safety, allowing us to build a proxy that’s both fast and reliable under heavy load. In an environment where throughput, latency, and stability matter, Rust is the right foundation.
Getting Started with agent gateway
We are excited to work on this project with the open-source community. Please check out the agentgateway.dev site to get started. Take a look at the GitHub project and open issues, submit documentation or bug fixes/features. Lastly, you can ask questions in the Discord server and get live feedback from the team.
.webp)
