AI Gateway
The AICR AI Gateway is the single entry point for all AI capabilities across the platform ecosystem. All applications in ~/dev should route AI requests through this gateway.
Why a Central Gateway?
┌─────────────────────────────────────────────────────────────────┐
│ ~/dev Ecosystem │
├──────────────┬──────────────┬──────────────┬───────────────────┤
│ IntelligentSPM│ SDA Demo │ SPARCC Apps │ Other Apps │
└──────┬───────┴──────┬───────┴──────┬───────┴────────┬──────────┘
│ │ │ │
└──────────────┴──────┬───────┴────────────────┘
▼
┌────────────────┐
│ AI Gateway │ ← Single Point
│ (aicr) │
└────────┬───────┘
│
┌─────────────────────┼─────────────────────┐
▼ ▼ ▼
┌────────────┐ ┌────────────┐ ┌────────────┐
│ OpenAI │ │ Anthropic │ │ Ollama │
│ gpt-4o │ │ Claude │ │ (local) │
└────────────┘ └────────────┘ └────────────┘Benefits
| Feature | Without Gateway | With Gateway |
|---|---|---|
| API Keys | Scattered across .env files | Single source of truth |
| Usage Tracking | None | Per-tenant metering |
| Budget Control | None | Automatic limits |
| Caching | None | Shared response cache |
| Fallbacks | Manual per-app | Automatic failover |
| Audit Trail | None | Full provenance |
Core Components
1. ai-router Package
Location: packages/ai-router/
The routing layer that handles:
- Provider selection (OpenAI, Anthropic, Ollama)
- Model routing and fallbacks
- Response caching
- Rate limiting
- Budget enforcement
2. agent-conductor Service
Location: services/agent-conductor/
Orchestrates AI agent workflows:
- Multi-step reasoning
- Tool execution
- Context management
- Ollama local inference
3. Orbs Package
Location: packages/orbs/
Reusable AI UI components:
- AskOrb: Chat interface for any context
- KBOrb: Knowledge base query interface
Quick Start
For Application Developers
// In your app, import from ai-router
import { createAIClient } from '@aicr/ai-router';
const ai = createAIClient({
tenantId: 'your-tenant',
// Credentials flow through gateway
});
// Make requests - routing handled automatically
const response = await ai.chat({
model: 'gpt-4o-mini', // or 'claude-3-sonnet', 'llama3'
messages: [{ role: 'user', content: 'Hello' }]
});For API Consumers
# Direct API call to gateway
curl -X POST https://api.aicoderally.com/v1/chat \
-H "Authorization: Bearer $AICR_API_KEY" \
-H "X-Tenant-ID: your-tenant" \
-d '{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Hello"}]
}'Environment Setup
Required environment variables (set once in aicr, propagated to all apps):
# Primary AI Providers
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
# Local Inference (optional)
OLLAMA_BASE_URL=http://localhost:11434
# Gateway Configuration
AICR_AI_GATEWAY_URL=https://api.aicoderally.com
AICR_GATEWAY_SECRET=... # For inter-service authSupported Models
| Provider | Models | Use Case |
|---|---|---|
| OpenAI | gpt-4o, gpt-4o-mini | General purpose, function calling |
| Anthropic | claude-3-opus, claude-3-sonnet | Long context, analysis |
| Ollama | llama3, mistral, nomic-embed-text | Local/free inference, embeddings |
Integration Patterns
Pattern 1: Direct Import (Monorepo Apps)
For apps within the aicr monorepo:
import { AIRouter } from '@aicr/ai-router';Pattern 2: HTTP Gateway (External Apps)
For apps outside the monorepo (IntelligentSPM, etc.):
const response = await fetch(`${AICR_GATEWAY_URL}/v1/chat`, {
method: 'POST',
headers: {
'Authorization': `Bearer ${AICR_API_KEY}`,
'Content-Type': 'application/json'
},
body: JSON.stringify({ model, messages })
});Pattern 3: Vercel AI SDK Integration
import { createAICR } from '@aicr/vercel-ai-provider';
const aicr = createAICR({
apiKey: process.env.AICR_API_KEY,
});
// Use with Vercel AI SDK
const { text } = await generateText({
model: aicr('gpt-4o-mini'),
prompt: 'Hello'
});