Skip to main content

ADR-012 — LiteLLM bundled as Python sidecar inside noxys-proxy (no standalone service)

1. Context​

Noxys needs to track AI consumption — tokens, cost per tenant, per user, per department, per service, per model — to power both billing (Stripe usage-based metering) and the console dashboard (cost analytics, attribution, routing rules). LiteLLM (BSD-2-Clause, Python) exposes a unified API across 100+ LLM providers, with built-in cost calculation, caching, and provider fallback.

The initial proposal in earlier brainstorming was to ship a standalone noxys-llm-gateway service (a fourth deployment artifact alongside extension, endpoint-agent, and proxy). Jerome rejected this: the customer-facing deployment surface MUST stay limited to three methods per the deployment-methods scoping rule (see memory feedback_three_deployment_methods.md). Adding a fourth public service inflates GTM messaging, sales onboarding, documentation, and operations cost.

2. Decision​

Bundle LiteLLM as a Python sidecar inside the noxys-proxy deployment unit:

  • Docker Compose: a litellm service co-located with noxys-proxy
  • Kubernetes / Helm: a sidecar pod inside the noxys-proxy chart (single Helm subchart)
  • Healthcheck wired into the proxy readiness probe
  • Configuration (BYOK provider keys, routing rules) injected via env + KMS

The customer-facing surface remains noxys-proxy only. LiteLLM is an internal implementation detail — invisible to the operator and to the buyer.

Two deployment modes supported (see §3.bis): Mode A embedded sidecar (default, all tiers) and Mode B BYO chain (Enterprise opt-in — noxys-proxy forwards to the customer's existing LiteLLM/Helicone/Portkey gateway).

3. Architecture​

Customer AI app
    │
    ▼
noxys-proxy (Go MITM)
  ├── auth (JWT, mTLS)
  ├── plan_gate (Stripe tier check)
  ├── policy verdict (PII, secret, jurisdiction) — pre-call
    │
    ▼ localhost:LITELLM_PORT
noxys-proxy LiteLLM sidecar (Python)
  ├── provider routing (EU-aware)
  ├── token counter
  ├── cost calc (provider price tables)
    │
    ▼
Real provider (OpenAI / Anthropic / Mistral / Google / Bedrock / ...)

Per-call flow:

  1. noxys-proxy receives the LLM request from the customer app
  2. Auth + plan_gate + policy verdict (PII redact, secret scan) run pre-call
  3. Request forwarded to localhost:LITELLM_PORT
  4. LiteLLM dispatches to the real provider, returns response with token/cost
  5. noxys-proxy emits NATS event llm.usage.{tenant_id} with token/cost/model/latency
  6. noxys-api consumes the event → persists llm_usage_logs (Postgres)
  7. noxys-console reads llm_usage_logs for the dashboard

For customers without proxy mode (extension only / API only):

  • noxys-api exposes /v1/llm/* (OpenAI-compatible) — same auth/plan_gate/PII redact path
  • Internally forwards to noxys-proxy → LiteLLM
  • Not advertised as a fourth deployment method — it's the API surface of the existing noxys-api service

3.bis Deployment modes — Embedded vs BYO chain​

Two modes supported, configured per-tenant via noxys-proxy config:

Mode A — Embedded (default)​

Customer app → noxys-proxy → localhost LiteLLM sidecar → real provider

  • Default for all tenants (Free / Pro / Enterprise)
  • Noxys controls BYOK keys (KMS-backed envelope encryption)
  • Noxys controls provider routing (sovereign EU defaults)
  • Cost calculation accuracy guaranteed (LiteLLM upstream price tables refreshed via cron)
  • Required for usage-based Stripe billing (LITELLM-8) — accurate tokens needed for overage metering

Mode B — BYO chain (Enterprise tier)​

Customer app → noxys-proxy → customer's existing LiteLLM/Helicone/Portkey → real provider

  • Enterprise tier only (config flag llm_gateway.mode: byo)
  • Customer keeps their gateway investment (no rip-and-replace)
  • Customer keeps provider keys (Noxys never sees them)
  • Noxys = pure policy/observability layer (MITM in front of customer gateway)
  • Multi-vendor enterprise scenarios supported (customer mix gateways)
  • Cost/token data = best-effort (depends on customer gateway response shape adherence to OpenAI usage spec)
  • Stripe usage-based metering (LITELLM-8) NOT guaranteed in BYO mode — disclaimer in contract

Configuration shape​

noxys-proxy/config.yaml:

llm_gateway:
  mode: embedded   # or "byo"
  byo:
    upstream_url: https://customer-litellm.example.com
    auth_header_env: CUSTOMER_LITELLM_TOKEN  # env var holding bearer token
    timeout_ms: 30000
    schema_version: openai-compat-v1  # tolerance hint

Token/cost extraction (both modes)​

After upstream returns response, noxys-proxy extracts response.usage.{prompt_tokens, completion_tokens, total_tokens} from OpenAI-compatible body. Cost computed:

  • Embedded: LiteLLM _response_cost field (authoritative)
  • BYO: Noxys-side lookup against embarked price table (best-effort)

Same NATS event shape llm.usage.{tenant_id} emitted in both modes.

Tier matrix update​

ModeFreeProEnterprise
Embedded— (no LLM gateway access)✅ (default)✅ (default)
BYO chain——✅ (config opt-in)

Tradeoffs accepted for BYO​

  • Schema variations across customer gateway versions → tolerance tests in PROXY-LITELLM-2
  • Cost data quality lower than embedded → Stripe billing disclaimer
  • Test matrix expands (per-gateway-version compatibility)

4. Why bundle vs separate service​

  • One less moving piece in the deploy graph (no new repo, no new CI, no new Helm chart, no new SLA)
  • Reuses noxys-proxy auth, MITM TLS termination, and policy engine — single source of truth for AI traffic
  • No new repo bootstrap (CI matrix, security scans, RTK config, ADR backlog, docs)
  • Customer-facing GTM stays at exactly 3 methods (extension / endpoint-agent / proxy)
  • LiteLLM is replaceable: if Helicone or Portkey become a better fit, swap the sidecar with no customer-facing change

5. Why LiteLLM specifically​

  • 100+ providers unified (OpenAI, Anthropic, Mistral, Google Gemini, AWS Bedrock, Cohere, Together, Perplexity, etc.)
  • Mature provider price tables, refreshed per model release
  • Native token counter per model
  • Python proxy server mode is well-suited for a sidecar (single binary container)
  • BSD-2-Clause license — sovereignty-friendly, no copyleft contamination
  • Active community, frequent releases

6. Tradeoffs accepted​

(Mode-specific tradeoffs for BYO chain are documented in §3.bis.)

  • Python in the stack: noxys-proxy is Go. Adding Python adds a runtime — but the precedent already exists (Presidio is a Python sidecar to noxys-api), so build/deploy patterns are reusable.
  • Cost-table accuracy: Cost calc depends on LiteLLM's price tables, which lag new model releases by hours/days. Mitigation: cron job pulls upstream weekly; manual override path for new models.
  • Sidecar memory footprint: ~200 MB peak per proxy replica. Acceptable in Helm deploys (proxy nodes already sized for MITM TLS buffers).

7. Sovereignty​

  • LiteLLM container hosted on Scaleway EU (Paris/AMS) following ADR-009 three-deployment-modes constraint
  • Default provider routing is EU-aware: prefer Mistral, Albert (gov), Anthropic EU regions, OpenAI EU residency where customer paid for it
  • BYOK provider keys stored encrypted at rest in noxys-api with KMS-backed envelope encryption, per ADR-009 Three Deployment Modes when shipped
  • Every LLM call emits an immutable audit entry conforming to LOG-TIER-1 pattern (LOG-TIER-1 is the cross-cutting logging policy, not LLM-specific)
  • No outbound calls leave EU borders unless the customer explicitly enabled a US/UK provider in BYOK config

8. Tier matrix​

TierLLM gateway access
FreeNot available — observability only via extension passive capture (no API gateway)
ProManaged gateway access, 10k req/month quota, BYOK provider keys, 7d cost analytics retention
EnterpriseUnlimited + per-user/dept attribution + routing rules (cheapest fallback, region-pin) + self-hosted gateway option + 90d analytics + CSV export

BYOK is mandatory for Pro+ — Noxys does not resell provider tokens. Overage billing on Pro: 0.10 EUR per 1M tokens beyond the 10k req/month soft cap (see LITELLM-8 sprint).

9. Consequences​

8 sprints (LITELLM-1 through LITELLM-8), 20j total (PROXY-LITELLM-1 4j + PROXY-LITELLM-2 5j include BYO chain mode support), spread W21-22 → W27-28:

  • W21-22: LITELLM-1 (spike, this ADR ratification), PROXY-LITELLM-1 (sidecar bundle)
  • W23-24: LITELLM-3 (DB schema), PROXY-LITELLM-2 (proxy interception + NATS emit)
  • W25-26: LITELLM-5 (API gateway endpoint), LITELLM-6 (console dashboard)
  • W27-28: LITELLM-7 (plan_gate + tier limits), LITELLM-8 (Stripe metering)

LITELLM-1 + PROXY-LITELLM-1 land alongside MM-GUARD-1 and LOG-TIER-1 in the W21-22 anti-Varonis push (see MASTER-ROADMAP-2026 W21-22 section).

10. References​