AI Gateway

An AI Gateway is a tool that streamlines the interactions between applications and LLM (Large Language Model) API providers like Gemini, OpenAI, Mistral, and others. Unlike traditional API Gateways, an AI Gateway offers advanced capabilities tailored for AI applications. It improves security, governance, and control over AI applications consuming LLMs by enforcing policies and monitoring AI traffic for threats between LLM APIs. An AI Gateway achieves this by tracking LLM usage, performance, and traffic through API requests on AI applications and workloads utilizing LLMs.

Organizations are increasingly developing applications that utilize third-party AI models, necessitating a solution to expose these AI services for broader consumption. The rise of the AI Gateway category underscores the need for features specifically designed for AI developers—features often missing in traditional API Gateway solutions.

API Gateways serve as the frontline for APIs, providing a unified point of contact for API consumers, including microservices and internal and external users. The primary role of API Gateways is to manage API traffic, securing, operationalizing, and efficiently managing application networks through functionalities such as authorization, access control, rate limiting, and observability.

In contrast, AI Gateways are API Gateways with extended, purpose-built functions designed for AI applications and LLM scenarios. Beyond the standard features of API Gateways, AI Gateways provide specialized functionalities, including token-based observability, LLM usage tracking, and token-based rate limiting. They also support advanced features such as prompt enrichment, insights into retrieval-augmented generation (RAG), semantic caching, and model failover mechanisms. Both API and AI Gateways aim to enhance performance, functionality, security, and observability across applications and networks.

For enterprises, AI APIs, such as LLM APIs, must adhere to stringent requirements to prevent unauthorized access and ensure data integrity and confidentiality. Here are some essential features of AI Gateways:

1. Unified Access Point: Simplify access for consumers to backend LLM APIs or other generative AI models approved by the organization. This centralized access point streamlines the management and utilization of various AI services.
   
2. Authentication and Authorization: Integrate with existing access control mechanisms such as API keys, OAuth, and JWTs. Use advanced authentication and authorization strategies to control access to AI models, ensuring secure and compliant AI usage.
   
3. Credential Management: Increase developer productivity by shifting the control of key management (including tracking, revocation, and refresh) to the gateway. This approach reduces individual and team API key sprawl and centralizes credential management within the AI infrastructure rather than with developers.
   
4. Consumption Control: Implement rate limiting for requests to public LLMs to avoid excessive charges. Set provider-specific and client-specific consumption limits to manage AI usage effectively.
5. Observability: Provide developers insights into token usage, quotas, error rates, and other usage metrics. Track usage by clients across multiple LLM providers with access logging for cost control and chargeback, enhancing visibility into AI traffic.
   
6. Enrichment: Enrich requests with additional headers for usage reporting and tracking purposes or append/transform the request body to add context, screen for unwanted text, or reject inappropriate or sensitive response content.
   
7. Canonical LLM API Definition: Customize a client-facing LLM API definition that can map to multiple providers. The gateway transforms provider-specific request and response data to a canonical model, facilitating consistent and efficient AI application development.

While many of the security and governance controls in API Gateways are necessary for AI Gateways, generative AI Gateways present unique challenges that must be addressed:

1. Fine-Grained Access Policy for AI Gateways: Since AI Gateway calls differ from most API Gateway calls, tighter control over LLM API access policies and usage is crucial. A robust policy engine allows fine-tuning of endpoints to ensure compliance and prevent accidental or malicious usage. These workloads are often linked to critical company data, making a well-defined LLM data security policy enforced at the AI Gateway essential.
2. AI-Specific Observability: Beyond uptime, latency, and successful calls, an AI Gateway must track metrics specific to AI LLMs. Generative AI workloads can extend metrics to track aspects like token usage and user consumption rates. These metrics help defend against bad actors by setting baselines and detecting out-of-band usage.
3. Threat Detection and Input Sanitization for AI Gateways: Detecting malicious usage of an endpoint involves AI Gateway-specific requirements. Input sanitization and token counts (with metrics) help prevent attacks. Detecting and removing prompt abuse is crucial. Coupling input sanitization with fine-grained rate limiting (e.g., calls per user) helps safeguard workloads from misuse.
4. Obfuscation of Keys: Ensuring that account keys for existing LLM API providers (like OpenAI, Hugging Face, etc.) are never exposed in the codebase or shared among developers is critical. A well-designed AI Gateway built with infrastructure-as-code helps achieve this by ensuring keys are securely managed through existing key storage mechanisms (e.g., Vault, AWS KMS, GCP Cloud Key Management, Azure Key Vault).
5. PII Removal Policies: Similar to API Gateways, it is crucial to scrub PII to prevent accidental storage or inappropriate exposure. Given that many AI API calls involve potentially sensitive data, having an AI Gateway capable of parsing fields is even more critical.
6. Securing Workloads Beyond the AI Gateway: Once an AI Gateway is in place, all downstream calls should be secured with mutual TLS encryption. A hardened front door is a significant first step, but ensuring the same level of defense is maintained throughout the entire chain is essential.

To successfully integrate LLMs into your AI applications, follow these best practices:

1. Secure Access and Credential Management of LLM APIs:
   • Store credentials securely for LLM API Providers at the infrastructure or gateway level, not with individual developers.
   • Generate API keys per client in the AI Gateway that map to one or more LLM API provider secrets.
   • Restrict LLM API provider and capability access by client using external authentication mechanisms.
   • Implement per-client authentication and authorization of LLM API access.
   • Implement fine-grained authorization of LLM API access by model.
   • Use fine-grained access controls with OPA and API key metadata.
2. Consumption Control and Visibility of LLMs:
   • Rate limit requests to LLM API providers to control costs.
   • Rate limit based on API key metadata for consumers of the AI Gateway.
   • Implement token-based rate limiting of LLM APIs by client or team.
   • Pre-configure token limits on requests.
   • Capture client context in access logging for downstream analytics and usage reporting.
3. Prompt Management:
   • Use the AI gateway’s transformation capabilities to enrich request/response bodies by injecting organizational security prompts and adding context.
   • If using multiple backend generative AI models, create a canonical input schema for your organization to simplify access to various local or third-party models.
   • Establish prompt guards to protect sensitive data from being leaked due to sophisticated attacks. Reject requests matching specific patterns, such as credit card information or other PII.