diegosouzapw/OmniRoute

Routing is orchestrated by handleComboChat() in open-sse/services/combo.ts, which implements 19+ strategies (priority, weighted, round-robin, random, least-used, cost-optimized, reset-aware, reset-window, strict-random, auto, fill-first, p2c, lkgp, context-optimized, context-relay, fusion, pipeline, quota-share, chaos). The dispatch passes through a prelude sequence: pinned model → fusion (parallel judge-synthesis) → chaos (multi-model parallel) → pipeline → runtime-unit → round-robin → target iteration with cooldown-aware retry. Target resolution in targetResolution.ts performs provider-wildcard expansion, weighted step-group resolution, prompt-cache affinity, session stickiness, eval scores, and request-compatibility filtering. The auto strategy generates scored candidates via buildAutoCandidates() using 16 factors: cost per token, historical p95 latency, error rate, circuit-breaker state, quota remaining, session affinity, reset-window affinity, OAuth session availability, connection pool size, quality scores, and speed telemetry. Fallbacks operate at three resilience layers (detailed in AGENTS.md): provider circuit breakers (src/shared/utils/circuitBreaker.ts — 4 states: CLOSED/DEGRADED/OPEN/HALF_OPEN, failure-kind-aware thresholds, DB-persisted), connection cooldown (accountFallback.ts — per-credential exponential backoff), and model lockout (per provider+connection+model). Retries use full-jitter backoff with status-code-aware classifiers (408/500/502/503/504 trip the provider breaker; 401/403/429 route to connection cooldown). Model aliases resolve through open-sse/config/providerRegistry.ts where user-supplied names map to provider-specific model IDs. Health checks run via circuit-breaker lazy recovery: expired OPEN states auto-transition to HALF_OPEN on the next getStatus(), and canExecute() gates target selection.

Translation follows a hub-and-spoke architecture centered on OpenAI format. translateRequest() in open-sse/translator/index.ts first normalizes thinking budget and reasoning-routing directives, then checks for direct translators (e.g., Claude→Gemini). When none exist, it converts source→OpenAI via a registered translator, then OpenAI→target. translateResponse() reverses the pipeline. The translator registry (translator/registry.ts) maps (from, to) format pairs to request/response functions. 12+ request translators exist: claude-to-openai.ts, gemini-to-openai.ts, openai-to-claude.ts, openai-to-gemini.ts, openai-to-kiro.ts, openai-to-cursor.ts, openai-to-clova.ts, antigravity-to-openai.ts, and the openai-responses.ts converter. Streaming is handled by translateStreamChunk() which converts provider-native SSE into unified OpenAI-style deltas. Tool calls are translated via translator/helpers/toolCallHelper.ts (ensureToolCallIds, fixMissingToolResponses, stripOrphanedToolResults) and schema coercion helpers (coerceToolSchemas, sanitizeToolDescriptions). Multimodal content (images, audio, video) passes through the same format pipeline. The responsesTransformer.ts bridges OpenAI Chat Completions and the Responses API via a TransformStream. A critical subsystem is reasoning replay: for thinking models (DeepSeek V4, Kimi K2, Qwen), cached reasoning_content is re-injected on subsequent turns to avoid 400 errors. Reasoning cache uses services/reasoningCache.ts with lookupReasoning()/recordReplay(). Provider support: 325+ providers registered across auth types (OAuth: 25 entries, No-Auth: 10, Local: 14, WebCookie: ~50+), served by 161 executor files in open-sse/executors/. Role normalization handles developer→system mapping, and OpenAI-incompatible echo fields (reasoning_content, refusal, annotations) are stripped for strict upstreams.

API keys are managed through src/lib/db/apiKeys.ts which stores keys in a SQLite api_keys table with fields for id, key prefix/hash, provider/model permissions, usage limits, spend tracking, rate limits, access schedules, IP allowlists, and tenant/user associations. Key lookup uses constant-time comparison (timingSafeCompare from src/shared/utils/timingSafeCompare). Virtual keys allow multiple users/teams to share upstream credentials while maintaining distinct rate/spend limits. Upstream credentials (OAuth tokens, API keys for downstream providers) live in the provider_connections table managed by src/lib/db/providers.ts, encrypted at rest with AES-256-GCM via the encryptConnectionFields()/decryptConnectionFields() pipeline. Credentials are fetched by getProviderCredentialsByUserIdAndConnectionId(). Auth methods are diverse: API-key auth (bearer token validated by extractApiKey()/isValidApiKey() in route handlers, with requireApiKey: true flag), OAuth flows via src/lib/oauth/, JWT session tokens (src/lib/auth/jwt.ts with signJwt/verifyJwt), and refresh tokens (src/lib/db/tokens.ts). The user/team/tenant model is multi-tenant with src/lib/db/tenants.ts, src/lib/db/teams.ts, and src/lib/db/users.ts. Each API key carries fine-grained permissions: allowed models (with wildcard support via modelPermissions.ts), allowed combos, allowed connections, allowed quotas, access schedules, per-minute/per-day rate limits, spend limits (daily/weekly USD caps), and key groups (apiKeyGroups.ts). The dashboard at src/app/(dashboard)/ provides a UI for key management, while the /v1/* API routes serve the programmatic interface. Auth method discovery is also driven by src/shared/constants/providers/ which classifies providers as OAuth (25), No-Auth (10), Local (14), API-key (bulk), WebCookie (~50+), and Cloud Agent types.

Rate limiting is implemented via src/shared/utils/rateLimiter.ts using a Redis-backed (or in-memory fallback) leaky bucket strategy. API keys carry maxRequestsPerDay, maxRequestsPerMinute, and rich rateLimits rules (parsed by parseRateLimits()). Cost tracking centers on open-sse/services/providerCostData.ts which provides a KNOWN_MODEL_PRICING map covering ~25 key models (e.g., gpt-4o at $2.50/$10 per 1M tokens, claude-sonnet-4.6 at $3/$15) and a getModelPricing() resolver that falls back through: provider-specific KNOWN match → default pricing DB table → generic model name match → free-model check → estimated fallback ($5/$15). The tier resolver (tierResolver.ts) layers operator-configured pricing over the defaults using a DB snapshot from pricing/pricing_synced/models_dev_pricing key-value namespaces and caches the merged result. Usage accounting is stored in two SQLite tables: usage_history (per-request rows with tokens_input/tokens_output and ISO timestamps) and daily_usage_summary (rolled-up aggregates), queried by src/lib/db/usageSummary.ts and src/lib/db/usageLogs.ts. The GET /v1/usage endpoint exposes aggregated usage. Spend limits are enforced by apiKeyPolicyService.ts (enforceApiKeyLimits()) which checks daily/weekly USD caps before allowing requests. Circuit breakers, connection cooldown (exponential backoff on per-credential failures), and model lockout all act as runtime cost-control mechanisms by stopping traffic to failing-expensive or exhausted endpoints. The connectionBillingCatalog.ts classifies connections as subscription (plan-included) vs metered (per-token) to budget routing.

Caching uses a dual-layer architecture in open-sse/services/cache/semanticCacheManager.ts. Layer 1 is exact-match: messages are normalized and hashed with SHA-256 for O(1) hits. Layer 2 is semantic: conversation text is embedded via embeddingClient.ts and matched by cosine similarity against a vector store (in-memory MemoryVectorStore or Redis RedisVectorStore). Cache entries store full response data and support SSE streaming replay. Cache configuration (semanticCacheConfig.ts), TTLs, and namespace isolation are provider-specific. The old deterministic caching also lives in the proxyDispatcherCache.ts for HTTP connection reuse. Guardrails are modular in src/lib/guardrails/. The promptInjection.ts guardrail detects system-override attempts with regex patterns (e.g., system: override, markdown code-block system blocks), classifies severity (low/medium/high), and supports block/warn/log modes with configurable block thresholds. piiMasker.ts provides PII redaction but is strictly opt-in — disabled by default (Hard Rule #20: both PII_REDACTION_ENABLED and PII_RESPONSE_SANITIZATION feature flags default to "false"). Moderation is handled by moderationProvider.ts/moderationProviderOpenAI.ts. The guardrail pipeline is assembled by buildGuardrailDependencyGraph() from guardrails/onRequest.ts (restructured into the current codebase) which sequences execution order. Additional guardrails cover audio/video bridges (audioBridge.ts, videoBridgeHelpers.ts), vision (visionBridgeRouter.ts), and compliance/no-log markers. Custom guardrail/eval/webhook extensions follow documented patterns: guardrails at src/lib/guardrails/ → docs/security/GUARDRAILS.md, evals at src/lib/evals/ → docs/frameworks/EVALS.md. The anti-ReDoS learnings from AGENTS.md mandate strictly bounded regex sequences to prevent catastrophic backtracking on untrusted inputs.

Observability is three-tiered. Structured JSON logging uses pino through src/lib/logger/, with a log-export system (src/lib/logExport/) that can ship call logs to BigQuery or other destinations. OpenTelemetry support is optional and lightweight: open-sse/services/routing/otel.ts exports GenAI semantic-convention spans (gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.*) to an OTLP/HTTP collector. It uses global fetch (no SDK dependency), a bounded buffer with background flush, and is disabled by default unless OMNIROUTE_OTEL_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT is set. Internal metrics (latency, request counts, error rates) are collected in src/lib/logger/metrics.ts. Deployment supports multiple modes: container via Dockerfile (multi-stage Node.js), Fly.io via fly.toml, Docker Compose, and tunnel-based deployment (src/mitm/tunnel/) for local model endpoints. The dashboard at src/app/(dashboard)/ provides visual management of providers, API keys, usage, logs, and settings. Performance architecture is standard Node.js async I/O on Next.js 16, single-threaded event loop with 161 executor modules and connection pooling via Undici (proxyDispatcherCache.ts with round-robin dispatchers to prevent SSE stream monopolization). HA mechanisms include the three resilience layers: provider circuit breakers (persisted to SQLite for restart survival), connection cooldown, and model lockout — all lazily self-recovering. Rate limiting uses Redis when available (falling back to in-memory). The _artifacts/ directory stores temporary working files per CLAUDE.md conventions. TypeScript strict mode is off (target ES2022), with 193 DB migrations for schema evolution.