ADR-012 — LiteLLM bundled as Python sidecar inside noxys-proxy (no standalone service)

1. Context

Noxys needs to track AI consumption — tokens, cost per tenant, per user, per department, per service, per model — to power both billing (Stripe usage-based metering) and the console dashboard (cost analytics, attribution, routing rules). LiteLLM (BSD-2-Clause, Python) exposes a unified API across 100+ LLM providers, with built-in cost calculation, caching, and provider fallback.

The initial proposal in earlier brainstorming was to ship a standalone noxys-llm-gateway service (a fourth deployment artifact alongside extension, endpoint-agent, and proxy). Jerome rejected this: the customer-facing deployment surface MUST stay limited to three methods per the deployment-methods scoping rule (see memory feedback_three_deployment_methods.md). Adding a fourth public service inflates GTM messaging, sales onboarding, documentation, and operations cost.

2. Decision

Bundle LiteLLM as a Python sidecar inside the noxys-proxy deployment unit:

Docker Compose: a litellm service co-located with noxys-proxy
Kubernetes / Helm: a sidecar pod inside the noxys-proxy chart (single Helm subchart)
Healthcheck wired into the proxy readiness probe
Configuration (BYOK provider keys, routing rules) injected via env + KMS

The customer-facing surface remains noxys-proxy only. LiteLLM is an internal implementation detail — invisible to the operator and to the buyer.

Two deployment modes supported (see §3.bis): Mode A embedded sidecar (default, all tiers) and Mode B BYO chain (Enterprise opt-in — noxys-proxy forwards to the customer's existing LiteLLM/Helicone/Portkey gateway).

3. Architecture

Customer AI app
    │
    ▼
noxys-proxy (Go MITM)
  ├── auth (JWT, mTLS)
  ├── plan_gate (Stripe tier check)
  ├── policy verdict (PII, secret, jurisdiction) — pre-call
    │
    ▼ localhost:LITELLM_PORT
noxys-proxy LiteLLM sidecar (Python)
  ├── provider routing (EU-aware)
  ├── token counter
  ├── cost calc (provider price tables)
    │
    ▼
Real provider (OpenAI / Anthropic / Mistral / Google / Bedrock / ...)

Per-call flow:

noxys-proxy receives the LLM request from the customer app
Auth + plan_gate + policy verdict (PII redact, secret scan) run pre-call
Request forwarded to localhost:LITELLM_PORT
LiteLLM dispatches to the real provider, returns response with token/cost
noxys-proxy emits NATS event llm.usage.{tenant_id} with token/cost/model/latency
noxys-api consumes the event → persists llm_usage_logs (Postgres)
noxys-console reads llm_usage_logs for the dashboard

For customers without proxy mode (extension only / API only):

noxys-api exposes /v1/llm/* (OpenAI-compatible) — same auth/plan_gate/PII redact path
Internally forwards to noxys-proxy → LiteLLM
Not advertised as a fourth deployment method — it's the API surface of the existing noxys-api service

3.bis Deployment modes — Embedded vs BYO chain

Two modes supported, configured per-tenant via noxys-proxy config:

Mode A — Embedded (default)

Customer app → noxys-proxy → localhost LiteLLM sidecar → real provider

Default for all tenants (Free / Pro / Enterprise)
Noxys controls BYOK keys (KMS-backed envelope encryption)
Noxys controls provider routing (sovereign EU defaults)
Cost calculation accuracy guaranteed (LiteLLM upstream price tables refreshed via cron)
Required for usage-based Stripe billing (LITELLM-8) — accurate tokens needed for overage metering

Mode B — BYO chain (Enterprise tier)

Customer app → noxys-proxy → customer's existing LiteLLM/Helicone/Portkey → real provider

Enterprise tier only (config flag llm_gateway.mode: byo)
Customer keeps their gateway investment (no rip-and-replace)
Customer keeps provider keys (Noxys never sees them)
Noxys = pure policy/observability layer (MITM in front of customer gateway)
Multi-vendor enterprise scenarios supported (customer mix gateways)
Cost/token data = best-effort (depends on customer gateway response shape adherence to OpenAI usage spec)
Stripe usage-based metering (LITELLM-8) NOT guaranteed in BYO mode — disclaimer in contract

Configuration shape

noxys-proxy/config.yaml:

llm_gateway:
  mode: embedded   # or "byo"
  byo:
    upstream_url: https://customer-litellm.example.com
    auth_header_env: CUSTOMER_LITELLM_TOKEN  # env var holding bearer token
    timeout_ms: 30000
    schema_version: openai-compat-v1  # tolerance hint

Token/cost extraction (both modes)

After upstream returns response, noxys-proxy extracts response.usage.{prompt_tokens, completion_tokens, total_tokens} from OpenAI-compatible body. Cost computed:

Embedded: LiteLLM _response_cost field (authoritative)
BYO: Noxys-side lookup against embarked price table (best-effort)

Same NATS event shape llm.usage.{tenant_id} emitted in both modes.

Tier matrix update

Mode	Free	Pro	Enterprise
Embedded	— (no LLM gateway access)	✅ (default)	✅ (default)
BYO chain	—	—	✅ (config opt-in)

Tradeoffs accepted for BYO

Schema variations across customer gateway versions → tolerance tests in PROXY-LITELLM-2
Cost data quality lower than embedded → Stripe billing disclaimer
Test matrix expands (per-gateway-version compatibility)

4. Why bundle vs separate service

One less moving piece in the deploy graph (no new repo, no new CI, no new Helm chart, no new SLA)
Reuses noxys-proxy auth, MITM TLS termination, and policy engine — single source of truth for AI traffic
No new repo bootstrap (CI matrix, security scans, RTK config, ADR backlog, docs)
Customer-facing GTM stays at exactly 3 methods (extension / endpoint-agent / proxy)
LiteLLM is replaceable: if Helicone or Portkey become a better fit, swap the sidecar with no customer-facing change

5. Why LiteLLM specifically

100+ providers unified (OpenAI, Anthropic, Mistral, Google Gemini, AWS Bedrock, Cohere, Together, Perplexity, etc.)
Mature provider price tables, refreshed per model release
Native token counter per model
Python proxy server mode is well-suited for a sidecar (single binary container)
BSD-2-Clause license — sovereignty-friendly, no copyleft contamination
Active community, frequent releases

6. Tradeoffs accepted

(Mode-specific tradeoffs for BYO chain are documented in §3.bis.)

Python in the stack: noxys-proxy is Go. Adding Python adds a runtime — but the precedent already exists (Presidio is a Python sidecar to noxys-api), so build/deploy patterns are reusable.
Cost-table accuracy: Cost calc depends on LiteLLM's price tables, which lag new model releases by hours/days. Mitigation: cron job pulls upstream weekly; manual override path for new models.
Sidecar memory footprint: ~200 MB peak per proxy replica. Acceptable in Helm deploys (proxy nodes already sized for MITM TLS buffers).

7. Sovereignty

LiteLLM container hosted on Scaleway EU (Paris/AMS) following ADR-009 three-deployment-modes constraint
Default provider routing is EU-aware: prefer Mistral, Albert (gov), Anthropic EU regions, OpenAI EU residency where customer paid for it
BYOK provider keys stored encrypted at rest in noxys-api with KMS-backed envelope encryption, per ADR-009 Three Deployment Modes when shipped
Every LLM call emits an immutable audit entry conforming to LOG-TIER-1 pattern (LOG-TIER-1 is the cross-cutting logging policy, not LLM-specific)
No outbound calls leave EU borders unless the customer explicitly enabled a US/UK provider in BYOK config

8. Tier matrix

Tier	LLM gateway access
Free	Not available — observability only via extension passive capture (no API gateway)
Pro	Managed gateway access, 10k req/month quota, BYOK provider keys, 7d cost analytics retention
Enterprise	Unlimited + per-user/dept attribution + routing rules (cheapest fallback, region-pin) + self-hosted gateway option + 90d analytics + CSV export

BYOK is mandatory for Pro+ — Noxys does not resell provider tokens. Overage billing on Pro: 0.10 EUR per 1M tokens beyond the 10k req/month soft cap (see LITELLM-8 sprint).

9. Consequences

8 sprints (LITELLM-1 through LITELLM-8), 20j total (PROXY-LITELLM-1 4j + PROXY-LITELLM-2 5j include BYO chain mode support), spread W21-22 → W27-28:

W21-22: LITELLM-1 (spike, this ADR ratification), PROXY-LITELLM-1 (sidecar bundle)
W23-24: LITELLM-3 (DB schema), PROXY-LITELLM-2 (proxy interception + NATS emit)
W25-26: LITELLM-5 (API gateway endpoint), LITELLM-6 (console dashboard)
W27-28: LITELLM-7 (plan_gate + tier limits), LITELLM-8 (Stripe metering)

LITELLM-1 + PROXY-LITELLM-1 land alongside MM-GUARD-1 and LOG-TIER-1 in the W21-22 anti-Varonis push (see MASTER-ROADMAP-2026 W21-22 section).

10. References

Memory: feedback_three_deployment_methods.md (deployment-methods scoping rule)
adr-009-three-deployment-modes
adr-010-pricing-three-tiers (internal)
adr-011-multi-modal-pipeline
../00 - Inbox/MASTER-ROADMAP-2026
Parent issue: noxys-api#514 (Stripe billing) — LITELLM-8 dep
LiteLLM: https://github.com/BerriAI/litellm (BSD-2-Clause)

1. Context​

2. Decision​

3. Architecture​

3.bis Deployment modes — Embedded vs BYO chain​

Mode A — Embedded (default)​

Mode B — BYO chain (Enterprise tier)​

Configuration shape​

Token/cost extraction (both modes)​

Tier matrix update​

Tradeoffs accepted for BYO​

4. Why bundle vs separate service​

5. Why LiteLLM specifically​

6. Tradeoffs accepted​

7. Sovereignty​

8. Tier matrix​

9. Consequences​

10. References​