A Go library that provides a unified interface for routing requests across multiple LLM providers. Write against one API, deploy across OpenAI, Anthropic, Google Gemini, and any OpenAI-compatible service.
- Go 1.25+
- API key for at least one supported provider
go get github.com/bluefunda/llmrouterpackage main
import (
"context"
"fmt"
"time"
llmrouter "github.com/bluefunda/llmrouter"
"github.com/bluefunda/llmrouter/middleware"
"github.com/bluefunda/llmrouter/providers/anthropic"
"github.com/bluefunda/llmrouter/providers/openai"
)
func main() {
router := llmrouter.New(
llmrouter.WithProvider("openai", openai.NewFromEnv("openai", "OPENAI_API_KEY")),
llmrouter.WithProvider("anthropic", anthropic.NewFromEnv()),
llmrouter.WithMiddleware(
middleware.Retry(3, time.Second),
middleware.Timeout(60*time.Second),
),
)
resp, err := router.Complete(context.Background(), &llmrouter.Request{
Model: "gpt-4o-mini",
Messages: []llmrouter.Message{
{Role: llmrouter.RoleUser, Content: "Hello!"},
},
})
if err != nil {
panic(err)
}
fmt.Println(resp.Choices[0].Message.Content)
}Each provider is configured via environment variables or explicit options.
| Provider | Package | Env Variable | Models |
|---|---|---|---|
| OpenAI | providers/openai |
OPENAI_API_KEY |
gpt-4o, gpt-4o-mini, gpt-4.1, o4-mini |
| Anthropic | providers/anthropic |
ANTHROPIC_API_KEY |
claude-opus-4, claude-sonnet-4, claude-haiku-3.5 |
| Gemini | providers/gemini |
GEMINI_API_KEY |
gemini-1.5-pro, gemini-1.5-flash, gemini-2.0-flash-exp |
| DeepSeek | providers/openai (preset: deepseek) |
DEEPSEEK_API_KEY |
deepseek-chat, deepseek-coder |
| Groq | providers/openai (preset: groq) |
GROQ_API_KEY |
llama-3.3-70b-versatile, mixtral-8x7b |
| Together | providers/openai (preset: together) |
TOGETHER_API_KEY |
llama-3.3-70b, mixtral-8x7b |
| Ollama | providers/openai (preset: ollama) |
— | Any locally hosted model |
| Sarvam | providers/openai (preset: sarvam) |
SARVAM_API_KEY |
sarvam-m, sarvam-30b, sarvam-105b |
DeepSeek, Groq, Together AI, Ollama, and Sarvam all use the OpenAI provider with a preset name:
openai.NewFromEnv("deepseek", "DEEPSEEK_API_KEY")
openai.NewFromEnv("groq", "GROQ_API_KEY")
openai.NewFromEnv("ollama", "") // no key needed for local
openai.NewFromEnv("sarvam", "SARVAM_API_KEY")Gemini requires explicit error handling at construction time and holds a gRPC connection — call router.Close() (or provider.Close() directly) on shutdown:
geminiProvider, err := gemini.NewFromEnv()
if err != nil {
log.Fatal(err)
}
defer router.Close()router := llmrouter.New(
llmrouter.WithProvider("openai", openaiProvider),
llmrouter.WithProvider("anthropic", anthropicProvider),
llmrouter.WithModelMapping("gpt-4o", "openai"),
llmrouter.WithModelMapping("claude-sonnet-4-20250514", "anthropic"),
llmrouter.WithFallback("openai", "anthropic"),
llmrouter.WithMiddleware(retryMw, cb.Wrap, timeoutMw),
)| Option | Description |
|---|---|
WithProvider |
Register a named provider |
WithModelMapping |
Route a model name to a specific provider |
WithFallback |
Set fallback provider order on primary failure |
WithMiddleware |
Attach middleware to the processing chain |
The router resolves a model to a provider in this order:
- Explicit mapping —
WithModelMapping("gpt-4o", "openai") - Provider name match — model name equals a registered provider name
- Provider model list — iterates providers in registration order and checks
Models()
Middleware is a MiddlewareFunc — a plain func(Provider) Provider. It is applied in declaration order (first declared = outermost wrapper).
Exponential backoff with configurable max attempts. Non-retryable errors (auth failures, invalid requests, context cancellation) short-circuit immediately.
middleware.Retry(3, time.Second)
middleware.Retry(3, time.Second, middleware.WithMaxDelay(10*time.Second))
middleware.Retry(3, time.Second, middleware.WithRetryFunc(myRetryPolicy))Stdlib-only three-state circuit breaker (Closed → Open → HalfOpen). Opens after consecutive failures exceed the threshold; recovers after the timeout period. No external dependencies.
Because the circuit breaker has observable state, it is constructed separately and passed via cb.Wrap:
cb := middleware.NewCircuitBreaker(5, 30*time.Second)
router := llmrouter.New(
llmrouter.WithMiddleware(cb.Wrap),
)
fmt.Println(cb.State()) // CBStateClosed / CBStateOpen / CBStateHalfOpenEnforces a deadline on both Complete and Stream calls. On timeout, Stream surfaces the error through StreamResult.Err().
middleware.Timeout(60 * time.Second)Any func(llmrouter.Provider) llmrouter.Provider satisfies MiddlewareFunc directly:
func Logging(next llmrouter.Provider) llmrouter.Provider {
return &loggingProvider{Provider: next}
}
router := llmrouter.New(llmrouter.WithMiddleware(Logging))Stream returns a *StreamResult iterator. Advance it with Next(), read the current event with Event(), and check errors after the loop with Err(). Always defer stream.Close() to release resources.
stream, err := router.Stream(ctx, &llmrouter.Request{
Model: "claude-sonnet-4-20250514",
Messages: []llmrouter.Message{
{Role: llmrouter.RoleUser, Content: "Write a haiku about Go."},
},
})
if err != nil {
log.Fatal(err)
}
defer stream.Close()
for stream.Next() {
event := stream.Event()
switch event.Type {
case llmrouter.EventContentDelta:
fmt.Print(event.Content)
case llmrouter.EventToolCallDelta:
// handle tool call delta
case llmrouter.EventDone:
// event.Response holds the final response with usage stats
}
}
if err := stream.Err(); err != nil {
log.Fatal(err)
}Define tools once and use them across any provider that supports function calling:
tool := llmrouter.Tool{
Type: "function",
Function: llmrouter.Function{
Name: "get_weather",
Description: "Get current weather for a location",
Parameters: json.RawMessage(`{
"type": "object",
"properties": {
"location": {"type": "string"}
},
"required": ["location"]
}`),
},
}
resp, _ := router.Complete(ctx, &llmrouter.Request{
Model: "gpt-4o-mini",
Messages: messages,
Tools: []llmrouter.Tool{tool},
})Messages support text, images, and documents via ContentParts:
msg := llmrouter.Message{
Role: llmrouter.RoleUser,
ContentParts: []llmrouter.ContentPart{
{Type: "text", Text: "What's in this image?"},
{Type: "image_url", ImageURL: &llmrouter.ImageURL{URL: "https://..."}},
},
}Mark static content for provider-level caching. Anthropic uses explicit CacheControl annotations; OpenAI and Gemini cache automatically. Observe savings via Usage.CachedPromptTokens:
req := &llmrouter.Request{
Model: "claude-sonnet-4-20250514",
Messages: []llmrouter.Message{
{
Role: llmrouter.RoleSystem,
Content: longSystemPrompt,
CacheControl: &llmrouter.CacheControl{Type: "ephemeral"},
},
{Role: llmrouter.RoleUser, Content: userQuery},
},
}The library classifies errors for intelligent retry and routing decisions:
| Error | Retryable | Description |
|---|---|---|
ErrRateLimited |
Yes | Provider rate limit (429) |
ErrAuthFailed |
No | Invalid API key (401/403) |
ErrInvalidRequest |
No | Malformed request (400) |
ErrCircuitOpen |
No | Circuit breaker is open |
ErrMaxRetriesExceeded |
No | All retry attempts exhausted |
ErrUnknownModel |
No | Model not found in any provider |
ErrNoProviders |
No | No providers registered |
Use llmrouter.IsRetryable(err) and llmrouter.IsRateLimited(err) for programmatic checks.
router.go # Core router — provider registry, model resolution, middleware chain
provider.go # Provider interface and MiddlewareFunc type
types.go # Unified request/response types, streaming events, tool definitions
options.go # Functional options for router configuration
errors.go # Error types and retryability classification
middleware/
retry.go # Retry with exponential backoff
timeout.go # Request timeout enforcement
breaker.go # Circuit breaker state machine (stdlib only)
circuitbreaker.go # Circuit breaker middleware wrapper
providers/
openai/ # OpenAI + compatible providers (DeepSeek, Groq, Together, Ollama, Sarvam)
anthropic/ # Anthropic Claude
gemini/ # Google Gemini
examples/
simple/ # Basic completion
streaming/ # Streaming responses
tools/ # Function calling
fallback/ # Multi-provider with middleware
Apache 2.0 — see LICENSE.
Built by BlueFunda — open-sourced under Apache 2.0.