Skip to main content

ADR-011 — Multi-modal detection pipeline (image / audio / video / code)

Context​

Noxys today detects PII only in text prompts (extension regex + backend NER via CamemBERT/Presidio). Competitors (Cyberhaven, Varonis Atlas) are moving fast on text but largely text-bound. Meanwhile, multi-modal AI usage is exploding in production:

  • ChatGPT image upload (vision models)
  • Claude file/PDF analysis with embedded images
  • Whisper-based voice prompts (mic capture in browser)
  • Copilot / Cursor code-block paste with embedded secrets
  • Google AI Studio / Gemini multi-modal uploads

The EU AI Act, Annex III (high-risk systems) effectively requires multi-modal coverage — a DLP/guardrail product that only sees text leaves a structural compliance gap for regulated buyers (banks, public sector, healthcare). Our current architecture has no path for image/audio/video/code-block PII detection. The MM-GUARD-1 sprint exists in the roadmap (row 9.47, promoted to W21-22) but lacks an architectural ADR establishing the pipeline shape.

This ADR defines that shape.

Decision​

Adopt a client-first, sidecar-fallback multi-modal pipeline organised into 4 lanes, all reusing the existing PII recognizer chain as the single source of truth:

Lane 1 — Image​

  • Extension intercepts file upload (per-site adapter in noxys-extension/src/sites/)
  • Tesseract.js WASM OCR client-side, < 2 MB bundle, < 2s p95 on 1080p
  • OCR output → existing PII recognizers (Presidio-style regex pack) + CamemBERT NER
  • Verdict: coach / block / redact (apply EXIF strip on redact)
  • Sidecar fallback: PaddleOCR for backend proxy mode (when extension cannot run)

Lane 2 — Audio​

  • Extension intercepts mic capture or audio file upload
  • Whisper.cpp WASM (Whisper-tiny, 39 MB lazy-loaded) for transcription
  • Transcript → existing PII pipeline
  • Sidecar fallback: backend Whisper sidecar in proxy mode

Lane 3 — Video​

  • Extract keyframes at 1 fps client-side via ffmpeg.wasm (25 MB lazy on first video)
  • Each keyframe → Image lane (caption/whiteboard PII)
  • Audio track → Audio lane
  • Unified PII verdict across both tracks

Lane 4 — Code blocks​

  • Detected via Markdown fence parser + AI-tool-specific selectors (chat.openai.com, claude.ai, copilot, cursor)
  • Secret scanner: gitleaks-style regex pack (AWS keys, OpenAI tokens, Stripe, JWT, GitHub PAT, GCP service-account JSON)
  • Coach with redact suggestion (replace with <REDACTED:AWS_KEY>)

Why client-first​

  1. Privacy by design — raw media never leaves the browser; only verdicts + minimal metadata
  2. Latency — no network hop, sub-2s for image
  3. Sovereignty — no US cloud OCR/STT dependency (no AWS Textract, no Google Cloud Speech)
  4. Cost — zero per-call inference cost; bundle download is one-time

Sidecar fallback only for proxy-mode tenants who cannot deploy the extension (compliance-heavy fleets, BYOD policies blocking extensions).

Why this architecture works​

  • Reuses existing PII recognizer chain — single source of truth for entity definitions; no parallel ML stack
  • Extension stays primary detection point — consistent with ADR-009 three-deployment-modes
  • ML-3/4/5 NER work directly applies — multi-modal is a feeder, not a replacement
  • AI Act Annex III compliance achievable without spinning up a separate model stack
  • Aligned with adr-010-pricing-three-tiers (internal) — Pro tier includes multi-modal, Free is text-only

Performance budgets​

LaneBudget (p95)Notes
Image< 2s1080p OCR, Tesseract.js WASM
Audio< 5s30s clip, Whisper-tiny
Video< 10s30s clip @ 1fps + audio
Code block< 50msPure regex

All client-side, all WASM. Bundles:

  • Tesseract.js: 2 MB (eager)
  • Whisper-tiny: 39 MB (lazy, behind feature flag multimodal.audio=true)
  • ffmpeg.wasm: 25 MB (lazy on first video upload)

Tradeoffs accepted​

  • Whisper-tiny accuracy lower than Whisper-medium — acceptable: we need to flag PII, not transcribe perfectly
  • Bundle size growth — mitigated by lazy loading per lane
  • Extension memory footprint — peak ~70 MB when audio + video active; documented in extension perf budget
  • Browser file API limits — not all sites use standard form submit; per-site adapters required in src/sites/

Consequences​

  • MM-GUARD-1 sprint scope clarified — first lane to ship is image (highest volume, lowest bundle cost)
  • New sprint candidates implied:
    • AUDIO-1 — Whisper.cpp WASM integration + lazy-load
    • VIDEO-1 — ffmpeg.wasm keyframe extraction
    • CODE-1 — gitleaks-style secret scanner rules + per-site adapters
  • All gated on ML-5 NER quality (F1 ≥ 0.92) — fall back to OCR + regex-only if ML-5 slips

Risks accepted​

  • Bundle growth slows extension load → lazy load per feature flag
  • Per-site adapter coverage gap → start with top-5 AI sites, expand with telemetry
  • OCR fails on low-contrast / handwriting → documented limitation; hard-block on coach UI when confidence < 0.6

Verification​

MM-GUARD-1 ships first lane (image OCR + PII scan), then iterates. Acceptance test: 50 multi-modal fixtures stored in noxys-extension/tests/fixtures/multimodal/:

  • 20 images with embedded PII (SIRET on document, email in screenshot)
  • 10 audio clips with spoken email / phone
  • 10 videos with whiteboard SIRET in keyframes
  • 10 code blocks with AWS / OpenAI / Stripe keys

Pass criteria: ≥ 90% detection across the fixture set.

References​

  • Roadmap: ../00 - Inbox/MASTER-ROADMAP-2026|MASTER-ROADMAP-2026 — W21-22 promoted section + row 9.47 (original MM-GUARD-1 slot)
  • Foundational rows: 9.29 (OCR foundation), ML-3 / ML-4 / ML-5 (NER chain)
  • Pricing: adr-010-pricing-three-tiers (internal) — Pro tier includes multi-modal
  • Deployment modes: adr-009-three-deployment-modes — extension primary, sidecar fallback