ADR-019 — Prompt capture, storage, retention & sovereignty

1. Context

Noxys public stance has been "SHA-256 hashes, never raw prompts" (privacy FAQ + ADR-014 §3). This is too rigid and conflicts with capabilities we are now actively building:

• AI-PLACE-3 Semantic policy engine — needs prompt embedding to evaluate intent rules
• AI-PLACE-1 Console copilot RAG — operates over what content if hashes only?
• AI-PLACE-4 Smart auto-redaction — processes raw text in flight, but stores what?
• AI-PLACE-5 Compliance narrator — credible incident narration requires detail
• COMP-1..9 Compliance reports — auditor/regulator-grade evidence requires more than hashes
• CISO incident investigation — "who leaked what?" requires content, not just a hash
• GDPR Art 15 (right of access) — user requests "show me my data" returns... a hash?

Beyond the conflict, no ADR currently defines: capture point, storage classes, retention durations, encryption-at-rest, access controls (incl. break-glass), backup retention, region pinning, right-of-erasure SLA, air-gap mode behavior. Every existing migration silently makes implicit assumptions that may not survive a serious enterprise security review.

This ADR defines the storage architecture explicitly, with tier-gated defaults aligned to ADR-010 pricing, and binding constraints on Noxys staff access to customer prompt data.

2. Decision summary

• 4 storage classes: aggregate-only, hashed, redacted, raw-encrypted. Tier-gated; tenant-configurable within their tier ceiling.
• Default per tier: Free=aggregate+hashed, Pro=+redacted, Enterprise=+raw-encrypted (opt-in).
• Encryption at rest: per-tenant KMS keys via OVHcloud KMS (default sovereign FR) or AWS KMS-EU (if non-FR EU pin) or HashiCorp Vault self-hosted (Enterprise air-gap). PG pgcrypto per-row for prompt fields; volume-level TDE everywhere else.
• Encryption in transit: TLS 1.3 minimum everywhere; mTLS optional for Enterprise extension/agent.
• Noxys staff access to raw prompts: forbidden by default. Break-glass requires 2-person rule + DPO notification to tenant within 24h + immutable audit log.
• Retention defaults: aggregate 90d, hashed 1y, redacted 1y, raw 90d. Enterprise: configurable up to 7y. Free: hard caps (no extension).
• Right of erasure (GDPR Art 17): 30-day SLA. Implementation = key-shred per-tenant (renders ciphertext unrecoverable) + 30-day data purge job.
• Right of access (GDPR Art 15): tenant-side "Export my data" via console + API (ZIP).
• Region pinning: per-tenant eu-fr-ovh (default), eu-de-ovh, eu-multi, customer-vpc (Enterprise). Cross-region replication forbidden without explicit opt-in.
• Air-gap mode (Enterprise): zero Noxys-staff access; updates via signed bundles; support via screensharing only.

3. Capture pipeline

Three capture surfaces (per the 3-deployment-methods rule):

1. Extension — DOM interception in browser tabs of LLM SaaS sites (chatgpt.com, claude.ai, etc.). Captures prompt text + multi-modal attachments (images uploaded, code blocks pasted). In-memory processing → batch send to noxys-api or noxys-proxy.
2. Endpoint agent — native process on workstation. Captures from Slack desktop, Teams desktop, Cursor, VSCode Copilot, etc. via OS hooks (macOS Accessibility / Windows UIA). Same processing pipeline as extension, transports to backend.
3. Proxy — HTTP-level interception of LLM API traffic. Captures requests + responses (token streams reassembled). Transports to noxys-api.

All three flows funnel into the same noxys-api ingest endpoint with identical schema. Multi-modal handling per ADR-011 (image OCR text, audio transcript, video keyframe OCR, code block extraction).

4. Storage classes

Class	Content stored	Reversibility	Use cases	Default tier
aggregate	counts, classifications, intent labels, scores, model + provider, costs	none — only metrics	dashboards, BOM, billing	all tiers
hashed	SHA-256(prompt) + metadata (length, language, has_pii flag, has_secret flag)	none — dedup detect only	dedup, "have we seen this hash before?", basic anomaly	Free, Pro, Enterprise
redacted	prompt text with PII/secrets/IP masked via smart redaction (AI-PLACE-4) — [PERSON_A], [PHONE_FR], [CURRENCY_5DIGIT]	partial — type structure preserved, identities lost	semantic policy (embed redacted), compliance reports evidence, RAG copilot	Pro, Enterprise
raw-encrypted	full original prompt, encrypted at rest with per-tenant KMS key	full — but only via tenant-owned key + break-glass	incident investigation, audit-grade evidence, GDPR Art 15 export	Enterprise opt-in

Same prompt may produce records in multiple classes simultaneously (e.g. aggregate + hashed + redacted always; raw only if Enterprise opt-in).

5. Tier matrix (storage × pricing tier)

Capability	Free	Pro	Enterprise
Aggregate metrics	yes (90d)	yes (1y)	yes (configurable)
Hashed prompts	yes (90d)	yes (1y)	yes (up to 7y)
Redacted prompts	—	yes (1y default)	yes (up to 7y)
Raw encrypted prompts	—	—	opt-in (default 90d, up to 7y)
Tenant break-glass to own raw	—	—	yes (admin role + audit)
Noxys-staff break-glass to raw	—	logged + denied	logged + 2-person + DPO notif (only if tenant authorizes per-incident)
Air-gap deployment	—	—	yes
Region pin choice	EU broad	EU broad	per-tenant select
Custom retention	—	—	yes
GDPR Art 15 export	basic stats CSV	aggregate + hashed + redacted ZIP	+ raw if tenant has key access
GDPR Art 17 erasure	yes 30d	yes 30d	yes 30d

6. Encryption architecture

At rest

• PG pgcrypto for prompt fields (raw_encrypted column, redacted_text column for Pro+ tenants). Per-row encryption with key reference to tenant KMS.
• Volume-level TDE on all DB volumes (OVHcloud managed PG, or self-hosted with LUKS).
• Per-tenant master key issued at tenant creation. Derived encryption keys (DEKs) wrap per-table or per-record values; KEK = tenant master in KMS.
• Key shred = erasure: GDPR Art 17 implemented by destroying tenant master key in KMS → all ciphertext renders permanently unrecoverable instantly. 30-day data purge follows for clean storage reclaim.

KMS providers

Provider	When	Sovereignty
OVHcloud KMS (Roubaix/Strasbourg)	default for eu-fr-ovh + eu-multi regions	sovereign FR, ANSSI compatible
AWS KMS-EU (Frankfurt/Paris)	when tenant pin = eu-de-ovh and OVH KMS unavailable in DE	EU-hosted but US vendor — not for SecNumCloud workloads
HashiCorp Vault (self-hosted)	Enterprise air-gap	tenant-controlled, no third-party

OVH KMS is the default. AWS KMS-EU and Vault are explicit opt-ins per tenant.

In transit

• TLS 1.3 minimum on all customer-facing endpoints.
• mTLS optional for extension/agent ↔ backend on Enterprise (extension has long-lived extension_token; mTLS adds device-cert layer).
• Inter-service (sidecar ↔ proxy ↔ api): mTLS mandatory in production deployments.

7. Access control

Default state — Noxys staff cannot read raw prompts

The noxys-api service account encrypts on write, decrypts on read only when:

1. The requesting context has a tenant-owned token (tenant admin or tenant API key), OR
2. The request is part of a documented break-glass procedure.

Database admins, DBAs, and SRE staff have query access to all rows but the prompt fields are ciphertext and they do not hold tenant master keys. Reading is technically blocked.

Tenant break-glass (tenant audits own data)

Tenant admin can request decryption of own raw via console "Audit prompt detail" button. Each request:

• Logged in prompt_access_events (immutable, append-only)
• Generates email to all tenant admins + DPO contact
• Rate-limited (default 100 reads/day per tenant, configurable)

Noxys-staff break-glass (incident response only)

Triggered ONLY when:

• An active P0/P1 incident requires content review for diagnosis
• Tenant has explicitly authorized per-incident access (signed authorization in support ticket)
• Two Noxys staff members co-sign the access request (one engineer + DPO/Security Officer)
• DPO notifies tenant DPO contact within 24h with scope + duration

Implementation: short-lived (≤4h) decrypt token issued by KMS, scoped to specific tenant + specific time window. Audit log immutable, mirrored to S3 EU compliance bucket with object lock.

For air-gap Enterprise: Noxys-staff break-glass is impossible by design. Support is on-call screensharing only.

8. Retention defaults + customization

Class	Free	Pro	Enterprise default	Enterprise max
aggregate	90d	1y	1y	7y
hashed	90d	1y	1y	7y
redacted	—	1y	1y	7y
raw	—	—	90d	7y

Retention enforced by daily purge job. Backups retained max 35d post-purge (separate cron) but use the same KMS key — key-shred propagates to backups instantly.

Free tier retention is hard-capped (no extension via config); Pro retention is hard-capped at 1y (no extension); Enterprise retention is configurable up to 7y.

9. Right of access (GDPR Art 15)

Tenant admin → Console → Settings → Privacy → "Export my data".

Generates ZIP within 24h:

• metadata.json — tenant info, subscription, users
• aggregate.csv — counts, classifications, scores
• hashed.csv — hash + per-prompt metadata
• redacted.jsonl — redacted prompt records (if Pro+)
• raw.jsonl.encrypted — raw if Enterprise + tenant key (still encrypted; tenant decrypts client-side using their KMS key export, or via tool we provide)
• audit.csv — all subscription_events + prompt_access_events for that tenant

Also exposed via API for programmatic export.

10. Right of erasure (GDPR Art 17)

Tenant admin → Console → Settings → Privacy → "Delete tenant" (with multi-step confirmation).

Sequence:

1. Tombstone tenant record (hidden from all UIs, no further writes accepted).
2. Trigger KMS key destruction within 1h — all ciphertext immediately unreadable.
3. Async purge job deletes rows + cleans backups within 30 days.
4. Audit log retained 6 years (legal requirement for GDPR compliance proof).

SLA: 30 days for full purge, but ciphertext is unrecoverable from hour 1.

11. Sovereignty & region pinning

tenant.region_pin enum:

• eu-fr-ovh — default. PG + KMS in OVHcloud Roubaix or Strasbourg.
• eu-de-ovh — Frankfurt fallback (some EU customers prefer DE legal jurisdiction).
• eu-multi — multi-region active-passive within EU (Enterprise HA).
• customer-vpc — Enterprise on-prem or customer-owned cloud account; Noxys deploys via Helm.

Cross-region replication forbidden unless tenant explicitly opts in. Default = single-region with daily off-region encrypted backups (key still in primary region — backup unreadable without primary KMS).

Sub-processors (per ADR-017 Trust Center) must declare region. List enforced at runtime — adding a sub-processor outside tenant region triggers contract review.

12. Air-gap mode (Enterprise)

When tenant.region_pin == customer-vpc and air_gap=true:

• Zero outbound calls from Noxys components except customer-allowlisted endpoints (e.g. customer's internal license server).
• Noxys-staff break-glass impossible. Customer DBA holds KMS keys.
• Updates: Noxys publishes signed bundles; customer IT downloads, validates signature, deploys.
• Support: phone + screensharing only. No log shipping to Noxys.
• License heartbeat (per ADR-018 SUB-6 enforcement): replaced by offline license file with absolute expiry + seat counts; Noxys signs, customer validates locally.
• Telemetry to Noxys: aggregate-only opt-in (CSV export tenant-controlled). Default = no telemetry.

13. Multi-modal storage

Per ADR-011 detection lanes:

• Images: OCR text → stored at storage class. Original image → never stored (ephemeral pass-through processing).
• Audio: transcript text → stored at class. Original audio → never stored.
• Video: keyframe OCR text → stored at class. Original video → never stored.
• Code blocks: extracted text → stored at class.

Rationale: file storage is huge + privacy nightmare; text-equivalents capture detection signal. If customer needs original, capture must be at customer-side (e.g. DLP system upstream).

14. Smart redaction mapping (AI-PLACE-4 dependency)

Smart redaction creates a mapping [PERSON_A] → "Jerome Soyer", [PHONE_FR] → "+33 6 12 34 56 78". Storage decision:

• Mapping NEVER stored alongside redacted text in same row.
• Mapping stored in separate table redaction_mappings, encrypted with same per-tenant key.
• Access to mapping = same access controls as raw (effectively turns redacted back into raw).
• Free + Pro tiers: redaction one-way (mapping discarded after redaction).
• Enterprise: mapping stored if raw_encrypted opt-in is on.

This means Pro tier redacted prompts are truly anonymized — even Noxys break-glass cannot recover identities.

15. Influence on currently-active sprints

Decisions to encode now:

1. EXT-UX-4 + ENROLL (auto-enrollment): JWT must include storage_tier + region_pin claims so extension/agent know what to send + where.
2. AI-PLACE-3 Semantic policy: requires Pro+ (needs redacted minimum to embed). Free tier policies are regex/recognizer only.
3. AI-PLACE-1 Copilot RAG: gracefully degrade if tenant=hashed-only (no RAG, only stats narration).
4. COMP-1..9 PDF reports: evidence section adapts depth to storage tier — hashed-only tenant gets count-based evidence, redacted tenant gets pattern excerpts, raw tenant gets full quote excerpts (with explicit tenant authorization for inclusion).
5. DT- sprints existing*: classifier inputs/outputs persistence requires Pro+ (redacted minimum).
6. SUB-1..12 (ADR-018): tenant model already adds current_subscription_id — also add storage_tier + region_pin columns now.
7. AI-PLACE-4 Smart redaction: implementation must include the mapping-storage decision (separate table + same KMS).

16. Sprint plan — 9 sprints, 25d

ID	Title	Repo	Window	Effort	Priority
STORE-1	DB schema — multi-tier prompt storage (aggregate/hashed/redacted/raw cols + retention metadata + region_pin/storage_tier on tenants)	noxys-api	W23-24	2d	P0
STORE-2	Per-tenant KMS integration (OVH KMS default + AWS KMS-EU + Vault adapter)	noxys-api + noxys-infra	W25-26	3d	P1
STORE-3	Redacted prompt storage pipeline (consumes AI-PLACE-4 output)	noxys-api	W25-26	2d	P1
STORE-4	Raw encrypted opt-in (Enterprise) + tenant break-glass UX	noxys-api + noxys-console	W27-28	3d	P2
STORE-5	Noxys-staff break-glass workflow (2-person + DPO notif + audit log)	noxys-api + noxys-console	W27-28	2d	P2
STORE-6	Tenant data export (GDPR Art 15) — ZIP generator	noxys-api + noxys-console	W29-30	2d	P1
STORE-7	Tenant erasure (GDPR Art 17) — KMS key shred + 30d purge	noxys-api	W29-30	2d	P0
STORE-8	Region pinning (eu-fr-ovh / eu-de-ovh / eu-multi / customer-vpc)	noxys-infra + noxys-api	Backlog	4d	P2
STORE-9	Air-gap mode bundle pipeline (Enterprise on-prem)	noxys-infra	Backlog	5d	P3

P0 sprints (STORE-1, STORE-7) are non-negotiable — schema + GDPR erasure must land before public Pro launch (W27-28 per ADR-010).

17. Open questions

• KMS choice OVH vs Vault: OVH KMS is default sovereign. Vault self-hosted is air-gap. AWS KMS-EU is fallback only. Confirm OVH KMS API supports per-tenant master key issuance + revoke at our scale (~10k tenants target Y2). Need POC.
• Hashed-only utility: SHA-256 of prompt is dedup-only. If we keep hashed indefinitely on Free, what value justifies it? Recommend: keep but reduce Free retention to 30d (saves storage, still useful for "have we seen this exact prompt before").
• Backup strategy: backup retention 35d, but if customer requests erasure on day 1, backups still exist 34d (encrypted). Key shred makes them unreadable. OK by GDPR 32-letter (technical impossibility of recovery), but ICO/CNIL guidance evolving — confirm legal.
• Smart redaction mapping for Free: Free tier doesn't have redacted storage but the in-flight redaction still creates a mapping briefly. Confirm no leakage path (mapping must be discarded same request lifecycle).
• Multi-modal future: if customers demand original media storage (e.g. for legal evidence in IP theft cases), revisit. Keep "no original media" as default — escape valve via Enterprise customer-vpc.

18. Non-goals

• End-to-end encryption from end-user device to console (would block detection — we need to inspect content in flight).
• Customer-managed keys for Free or Pro (only Enterprise; cost + complexity unjustified at lower tiers).
• Forensic-grade chain-of-custody (we provide audit logs, not legal forensic packaging).
• Real-time inter-DC replication for sub-second RPO (RPO 1h via async replication is sufficient for our SLA).

19. Refs

• ADR-010 Pricing tiers (storage capabilities map 1:1 to tiers)
• ADR-011 Multi-modal pipeline (capture lanes feed storage)
• ADR-014 AI placement (storage tier gates AI capabilities)
• ADR-016 Compliance reports (evidence depth depends on storage tier)
• ADR-017 Trust Center (sub-processors per region declared here)
• ADR-018 Subscription lifecycle (tenant model adds storage_tier + region_pin)
• Memory: feedback_three_deployment_methods.md
• Future ADR-027 CRM choice (renumbered through ADR-019..ADR-026 — all taken; ADR-026 = API/OpenAPI policy)
• ADR-020 Interaction lineage visualization (lineage detail depth gated by storage tier per this ADR)
• ADR-021 TanStack Query + Table adoption (console pages consuming storage-tier-gated data use TanStack Query)
• ADR-022 UsersPage V2 spec (consumes DataTable wrapper for users table; sources entitlements/cost from backend rollups)
• ADR-026 API completeness & OpenAPI policy (every endpoint defined in this ADR must have OpenAPI 3.1 spec entry; backfill obligation in §9)
• ADR-024 Authentication & MFA strategy (TOTP/WebAuthn/backup codes encrypted via KMS pattern; tenant token reveal + tenant-wide MFA enforcement)