Skip to content

A Blazing Fast AI Gateway. Route to 100+ LLMs with 1 fast & friendly API.

License

Notifications You must be signed in to change notification settings

alexsherstinsky/gateway

Β 
Β 

Repository files navigation

English | δΈ­ζ–‡

AI Gateway

Reliably route to 200+ LLMs with 1 fast & friendly API

Gateway Demo

License Discord Twitter npm version Better Stack Badge

The AI Gateway streamlines requests to 250+ language, vision, audio and image models with a unified API. It is production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.

βœ…Β  Blazing fast (9.9x faster) with a tiny footprint (~100kb build)
βœ…Β  Load balance across multiple models, providers, and keys
βœ…Β  Fallbacks make sure your app stays resilient
βœ…Β  Automatic Retries with exponential fallbacks come by default
βœ…Β  Configurable Request Timeouts to easily handle unresponsive LLM requests
βœ…Β  Multimodal to support routing between Vision, TTS, STT, Image Gen, and more models
βœ…Β  Plug-in middleware as needed
βœ…Β  Battle tested over 480B tokens
βœ…Β  Enterprise-ready for enhanced security, scale, and custom deployments

Setup & Installation

Use the AI gateway through the hosted API or self-host the open-source or enterprise versions on your environment.

πŸ‘‰ Hosted Gateway on portkey.ai (Fastest)

The hosted API is the fastest way to setup an AI Gateway for your Gen AI application. We process billions of tokens daily and is in production with companies like Postman, Haptik, Turing, MultiOn, SiteGPT, and more.

Get API Key

πŸ‘‰ Self-hosting the OSS version (MIT License)

To run the AI gateway locally, execute the following command in your terminal. (Needs npx installed) Or, explore deployment guides for Cloudflare, Docker, Node.js and more here.

npx @portkey-ai/gateway

Your AI Gateway is now running on http://localhost:8787 πŸš€

πŸ‘‰ Self-hosting the Enterprise Version

The AI Gateway's enterprise version offers enterprise-ready capabilities for org management, governance, security and more out of the box. Compare the open source, hosted and enterprise versions here.

The enterprise deployment architecture, supported platforms is available here - Enterprise Private Cloud Deployments

Book an enterprise AI gateway demo


Making requests through the AI gateway

Compatible with OpenAI API & SDKs

The AI Gateway is compatible with the OpenAI API & SDKs, and extends them to call 200+ LLMs reliably. To use the Gateway through OpenAI, update the client to include the gateway's URL and headers and make requests as usual. The AI gateway can translate requests written in the OpenAI format to the signature expected by the specified provider. View examples

Using the Python SDK Β Β 

Portkey Python SDK is a wrapper over the OpenAI Python SDK with added support for additional parameters across all other providers. If you're building with Python, this is the recommended library to connect to the Gateway.

pip install -qU portkey-ai

Using the Node.JS SDK

Portkey JS/TS SDK is a wrapper over the OpenAI JS SDK with added support for additional parameters across all other providers. If you're building with JS or TS, this is the recommended library to connect to the Gateway.

npm install --save portkey-ai

Using the REST APIs

The AI gateway supports OpenAI compatible endpoints with added parameter support for all other providers and models. View API Reference.

Other Integrations

Language Supported SDKs
JS / TS LangchainJS
LlamaIndex.TS
Python
Langchain
LlamaIndex
Go go-openai
Java openai-java
Rust async-openai
Ruby ruby-openai

Gateway Cookbooks

πŸ“ˆ Trending Cookbooks

✨ Latest Cookbooks

Supported Providers

Explore Gateway integrations with 25+ providers and 6+ frameworks.

Provider Support Stream
OpenAI βœ… βœ…
Azure OpenAI βœ… βœ…
Anyscale βœ… βœ…
Google Gemini & Palm βœ… βœ…
Anthropic βœ… βœ…
Cohere βœ… βœ…
Together AI βœ… βœ…
Perplexity βœ… βœ…
Mistral βœ… βœ…
Nomic βœ… βœ…
AI21 βœ… βœ…
Stability AI βœ… βœ…
DeepInfra βœ… βœ…
Ollama βœ… βœ…
Novita AI βœ… βœ…

View the complete list of 200+ supported models here



Agents

Gateway seamlessly integrates with popular agent frameworks. Read the documentation here.

Framework Call 200+ LLMs Advanced Routing Caching Logging & Tracing* Observability* Prompt Management*
Autogen βœ… βœ… βœ… βœ… βœ… βœ…
CrewAI βœ… βœ… βœ… βœ… βœ… βœ…
Phidata βœ… βœ… βœ… βœ… βœ… βœ…
Llama Index βœ… βœ… βœ… βœ… βœ… βœ…
Control Flow βœ… βœ… βœ… βœ… βœ… βœ…
Build Your Own Agents βœ… βœ… βœ… βœ… βœ… βœ…

*Only available on the hosted app. For detailed documentation click here.

Features

Fallbacks
Fallback to another provider or model on failed requests. You can specify the errors on which to trigger the fallback. Improves reliability of your application

Automatic Retries
Automatically retry failed requests up to 5 times. An exponential backoff strategy spaces out retry attempts to prevent network overload.

Load Balancing
Distribute LLM requests across multiple API keys or AI providers with weights to ensure high availability and optimal performance.

Request Timeouts

Manage unruly LLMs & latencies by setting up granular request timeouts, allowing automatic termination of requests that exceed a specified duration.

Multi-modal LLM Gateway
Call vision, audio (text-to-speech & speech-to-text), and image generation models from multiple providers β€” all using the familiar OpenAI signature

Guardrails

Verify your LLM inputs AND outputs to adhere to your specified checks. Build your own checks or choose from the 20+ pre-built guardrails.

These features are configured through the Gateway Config added to the x-portkey-config header or the config parameter in the SDKs.

Here's a sample config JSON showcasing the above features. All the features are optional

{
	"retry": { "attempts": 5 },
	"request_timeout": 10000,
	"strategy": { "mode": "fallback" }, // or 'loadbalance', etc
	"targets": [{
		"provider": "openai",
		"api_key": "sk-***"
	},{
		"strategy": {"mode": "loadbalance"}, // Optional nesting
		"targets": {...}
	}]
}

Then use the config in your API requests to the gateway.

Using Gateway Configs

Here's a guide to use the config object in your request.


Gateway Enterprise Version

Make your AI app more reliable and forward compatible, while ensuring complete data security and privacy.

βœ…Β  Secure Key Management - for role-based access control and tracking
βœ…Β  Simple & Semantic Caching - to serve repeat queries faster & save costs
βœ…Β  Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments
βœ…Β  PII Redaction - to automatically remove sensitive data from your requests to prevent indavertent exposure
βœ…Β  SOC2, ISO, HIPAA, GDPR Compliances - for best security practices
βœ…Β  Professional Support - along with feature prioritization

Schedule a call to discuss enterprise deployments


Contributing

The easiest way to contribute is to pick an issue with the good first issue tag πŸ’ͺ. Read the contribution guidelines here.

Bug Report? File here | Feature Request? File here


Community

Join our growing community around the world, for help, ideas, and discussions on AI.

Rubeus Social Share (4)

About

A Blazing Fast AI Gateway. Route to 100+ LLMs with 1 fast & friendly API.

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • TypeScript 99.7%
  • Other 0.3%