ADR-007: ML Training Forge Choice
- Status: Accepted
- Date: 2026-04-27
- Owner: Jerome Soyer (CEO / Tech Lead)
- Supersedes: —
- Related: prd-ml-training-pipeline, prd-detection-tiering, adr-006-detection-tiering
Context
Noxys positions itself as the European sovereign AI Firewall. EU sovereignty is not a marketing claim — it is the load-bearing differentiator versus US-based competitors (Lakera, Prompt Security, Protect AI). Every layer of the stack must be defensible under GDPR, the EU AI Act, and Schrems II.
We need to lock the ML training infrastructure that produces our detection models (NER for PII, prompt-injection classifier, jailbreak detector, policy-violation scorer). The forge will:
- Train small-to-medium transformer models (DistilBERT, CamemBERTa-v2, XLM-RoBERTa base, custom heads, ≤300M params).
- Run nightly fine-tuning loops on red-team / customer feedback corpora.
- Produce versioned, reproducible model artifacts consumed by the Detection Tiering stack (ADR-006).
- Store training data (potentially containing replayed prompts) under EU jurisdiction only.
GPU hardware has been acquired and self-hosted. The remaining decisions are operational stack lock-in (orchestration, tracking, dataset/model versioning, registry), capacity planning under VRAM constraint, burst fallback, and cost guardrails.
Decision
We adopt a self-hosted single-GPU forge with an MLflow-centric, EU-sovereign stack, and a Scaleway H100 burst fallback for jobs exceeding local VRAM.
Hardware (acquired) — codename synapse
| Component | Spec |
|---|---|
| GPU | 1× NVIDIA RTX 5070 (non-Ti), 12 GB GDDR7 VRAM, Blackwell sm_120 |
| CPU | AMD Ryzen 7 3700X (8c/16t) |
| RAM | 32 GB DDR4 |
| Network | 1 Gbps symmetric + Tailscale mesh VPN |
| Storage | 1 TB NVMe (LUKS2) |
| OS | Fedora 43 |
| Location | France (FR), operator residence |
| Uptime | 24/7 |
| Driver stack | NVIDIA proprietary, CUDA 13.0, nvidia-container-toolkit + CDI |
| Provisioning | noxys-ai-forge repo (codename synapse) — modules 00-system → 10-operations |
Sovereignty grade: ✅ Full — bare metal, EU-resident, single operator, no cloud subprocessor in the training path. CLOUD Act exposure: zero. This is a marketing-grade differentiator vs every cloud-based competitor.
Locked stack
| Layer | Choice | Rationale |
|---|---|---|
| Primary compute | synapse (RTX 5070, Fedora 43, provisioned by noxys-ai-forge) | Sunk hardware cost, full sovereignty, 24/7 availability |
| Burst compute | Scaleway H100-PCIe-80GB, fr-par-1, on-demand | Used only when local VRAM insufficient (>300M params or large batches) |
| Orchestration | MLflow Projects + Podman (rootless) | Solo-dev simplicity, declarative MLproject files, no Kubernetes overhead |
| Experiment tracking | MLflow Tracking on synapse (Podman compose, see noxys-ai-forge/compose/) | Air-gapped by default, accessible via Tailscale, no external telemetry |
| Tracking backend DB | PostgreSQL on synapse (Podman, same compose) | Local, no external dependency |
| Dataset versioning | DVC (artifacts) + HF Datasets (loaders) | DVC primary remote on synapse MinIO; nightly mirror to Scaleway Object Storage |
| Model registry | MLflow Model Registry on synapse | Single source of truth, signed artifacts, stage transitions |
| Container registry | Scaleway Container Registry (rg.fr-par.scw.cloud) | EU-hosted, used for serving images pushed to prod |
| Object storage (primary) | MinIO on synapse (Podman compose) | Air-gapped, S3-compatible, zero cost, full sovereignty |
| Object storage (offsite) | Scaleway Object Storage fr-par — nightly mirror only | Disaster recovery RPO 24h; primary remains air-gapped |
| Inference engines | Ollama (native), LM Studio (native), vLLM (venv) — already in forge | Reusable for eval / red-team / RAG side-experiments |
| Python environments | uv venvs training / serving / rag / dev (forge module 03-venvs) | Isolated, reproducible; noxys_train ships into training venv |
| Remote access | Tailscale mesh + Caddy reverse proxy (forge module 08-networking) | Operator + CI access without exposing public ports |
| Secrets | sops-encrypted .env + age key on hardware token | No KMS dependency |
Capacity planning under 12 GB VRAM
12 GB VRAM is the binding constraint. RTX 5070 Blackwell sm_120 — fp8 supported, ~111 TFLOPS fp16 (sparse 222 TFLOPS), ~250W TDP. Documented model-size envelope:
| Model | Params | fp16 | LoRA r16 | Verdict |
|---|---|---|---|---|
| DistilBERT base | 66M | ✅ batch 64 | ✅ batch 128 | synapse |
| CamemBERTa-v2 base | 110M | ✅ batch 32 | ✅ batch 64 | synapse |
| XLM-RoBERTa base | 270M | ✅ batch 16 (grad ckpt) | ✅ batch 32 | synapse |
| XLM-RoBERTa large | 550M | ❌ OOM | ⚠️ batch 4 (grad ckpt + offload) | burst → Scaleway |
| 7B+ foundation FT | — | ❌ | ❌ | out of scope |
Rule: if a job needs more than 12 GB or wall-clock >24h on synapse, route to Scaleway H100 burst.
Cost guardrails
- Local compute (synapse): sunk hardware. Operating cost = electricity only, ~€0.18/kWh × 0.25 kW realistic load = ~€32/month at 24/7 50% utilization (worst case 100% load: ~€50/mo).
- Burst budget: 20 GPU-hours/month max on Scaleway H100 = €50/month hard cap, billing alerts at 50% / 80% / 100%.
- Offsite mirror: Scaleway Object Storage nightly sync ~€7/month (~500 GB).
- Auto-shutdown: Burst jobs auto-terminate after 12h wall-clock (MLflow
max_durationhook). - No idle Scaleway GPUs: Instances spawned per-burst-job via Scaleway API, destroyed on completion.
- Total ML forge OPEX: ~€90/month (vs €510/month for pure-cloud alternative). Note MLflow VPS line removed — MLflow runs on synapse.
Alternatives Considered
| Option | Sovereignty | Cost | DX | EU Compliance | Verdict |
|---|---|---|---|---|---|
| Self-hosted RTX 5070 Ti + Scaleway burst | ✅ Full | €75/mo | Medium | Full + Schrems-II-immune | RECOMMENDED |
| Pure Scaleway H100 EU-PAR1 | EU (FR HQ) | €500/mo | Good | Full GDPR + AI Act | Originally picked, replaced |
| Pure self-hosted no burst | ✅ Full | €25/mo | Bad (blocked on >300M models) | Full | Rejected — caps roadmap |
| OVHcloud GPU EU | EU (FR HQ) | €560/mo | Medium | Full | Rejected — no cost benefit vs Scaleway |
| AWS SageMaker / GCP Vertex / Azure ML | US-owned | €700-900/mo | Excellent | Schrems II risk | Rejected |
| LambdaLabs / RunPod / Vast.ai | US-leaning | €250/mo | Good | Non-compliant | Rejected |
Stack-level alternatives
| Layer | Considered | Picked | Why not the other |
|---|---|---|---|
| Orchestration | Kubeflow, Metaflow, MLflow | MLflow | Kubeflow = K8s overhead; Metaflow = AWS-coupled |
| Tracking | W&B, MLflow, Neptune | MLflow self-hosted | W&B sends telemetry to US servers (sovereignty fail) |
| Dataset versioning | DVC, LakeFS, HF Datasets | DVC + HF Datasets | LakeFS heavy; pure HF Datasets lacks binary artifact versioning |
| Model registry | HF Hub private, custom, MLflow | MLflow Registry | HF Hub = US-hosted (sovereignty fail for prod artifacts) |
| Container registry | GHCR, Docker Hub, Scaleway | Scaleway Registry | GHCR is US-hosted; pulls from prod must be EU-resident |
| Container runtime | Docker, Podman | Podman | Fedora 43 default, rootless, no daemon |
Consequences
Positive
- Sovereignty defensible end-to-end with marketing edge: "trained on bare metal in France, never touched a US cloud" is a unique vs every competitor.
- Cost ~7× lower than pure-cloud (€75/mo vs €510/mo). Pre-seed budget friendly.
- 24/7 availability: no cold-start, no quota wait, no capacity-blocked training rounds.
- Schrems II immunity: zero US subprocessor for primary path. Burst path is EU-only too.
- Audit-ready: MLflow + DVC give reproducible lineage (data hash → code hash → model hash) for AI Act conformity.
- Solo-dev friendly: no Kubernetes, no platform team needed.
Negative
- 12 GB VRAM ceiling: caps single-GPU jobs at ~300M params fp16. Anything larger must burst.
- Single-point-of-failure hardware: GPU dies → training stops until RMA or burst takeover. Mitigation: burst fallback always reachable, MLflow runs resumable from checkpoint.
- Operator-managed driver/CUDA stack: Fedora 43 + NVIDIA proprietary requires manual updates and occasional kernel-driver mismatch resolution.
- Power/thermal: 24/7 operation in residential setting needs cooling and UPS planning (out of scope here, but flagged).
- No managed AutoML: worse DX than SageMaker/Vertex for hyperparameter sweeps.
Neutral
- RTX 5070 Ti 12 GB consumer card, not data-center grade — no ECC VRAM. Acceptable for our model sizes; flagged in eval harness (re-run on burst H100 before any prod release to confirm no silent corruption).
- MLflow Registry stage transitions are manual — acceptable at this scale.
- 1 TB NVMe holds ~5-10 dataset versions + 50-100 model checkpoints; DVC garbage collection scheduled monthly.
Cost Estimate
| Item | Unit cost | Volume | Monthly |
|---|---|---|---|
| synapse electricity (RTX 5070, avg load) | €0.18/kWh | ~180 kWh | €32 |
| Scaleway H100 burst | €2.50/hr | 20 hrs cap | €50 |
| Scaleway Object Storage (offsite mirror) | €0.014/GB | ~500 GB | €7 |
| Scaleway Container Registry | included up to 75 GB | — | €0 |
| MLflow / Postgres / MinIO on synapse | sunk (forge) | — | €0 |
| Total ML forge OPEX | ~€90 / month |
Hardware CAPEX (sunk): synapse workstation. Out of OPEX cap.
Per-training-round marginal cost: ~€0 on synapse, ~€15 on Scaleway burst (6h × €2.50). synapse rounds essentially unlimited within local capacity.
Sovereignty Implications
- Data residency: All datasets (red-team prompts, customer feedback, synthetic PII) stored on local NVMe (FR) and mirrored to Scaleway Object Storage
fr-paronly. No replication outside EU. - Model artifacts: Stored in MLflow Registry on noxys-vps (EU) and mirrored to Scaleway Object Storage
fr-par. Never pushed to HF Hub public/private. - Training logs & metrics: MLflow tracking DB (PostgreSQL on noxys-vps, EU). No external telemetry (W&B, Neptune, Comet disabled).
- Container images: Built on local Podman, pushed to Scaleway Container Registry
rg.fr-par.scw.cloud. Production pulls only from EU registry. - Operator access: Only EU-resident operators (currently: Jerome) hold credentials and physical access. Documented in
INVARIANTS.md. - Marketing claim: "trained on EU bare metal, never on US-owned cloud" — defensible and verifiable via MLflow tags + Scaleway invoices.
Compliance Notes
GDPR
- Training data containing personal data (replayed customer prompts) is processed under legitimate interest with explicit DPA addendum. Customers opt-in via Business/Enterprise contract clause.
- Right-to-erasure: dataset versioning via DVC means we can rebuild any historical model without the erased samples and retrain. Documented in
runbook-gdpr-erasure(TODO). - Data minimization: prompts are tokenized and hashed where possible before persisting.
- Physical access to training hardware logged (operator key card / disk full-disk encryption with LUKS2).
EU AI Act
- Our models are narrow-purpose classifiers/NER (PII detection, injection scoring). They are not General-Purpose AI (GPAI) under Article 51 — no foundation model training, no >10²⁵ FLOPs (RTX 5070 Ti at peak ~177 TFLOPS fp16, well below threshold), no broad capabilities.
- Therefore the GPAI obligations (model card, training data summary, copyright policy) do not formally apply at our compute scale. We voluntarily publish lightweight model cards via MLflow tags for transparency.
- If we ever cross into GPAI territory (e.g. fine-tuning a >7B foundation model — would require external compute beyond our forge anyway), this ADR must be revisited.
Schrems II
- No US-based subprocessor in primary or burst training path. CLOUD Act exposure: none.
Migration Path
If the local forge is lost (theft, hardware failure, fire):
- Immediate: route all training to Scaleway H100 burst, lifting €50/mo cap to €500/mo temporarily. RPO < 24h thanks to nightly DVC mirror to Scaleway Object Storage.
- Short term: procure replacement GPU (RTX 5090 32 GB or used H100 PCIe). Estimated downtime: 1-2 weeks for procurement.
- MLflow tracking server (on noxys-vps): independent of training hardware, unaffected.
If we need to swap the burst provider (e.g. Scaleway sunsets H100):
- GPU provider swap (Scaleway → OVH): rebuild Podman images, push to new registry, update MLflow
MLprojectbackend. Estimated effort: 1 day. - Tracking swap (MLflow → alternative): export MLflow runs via
mlflow.client.search_runs()to Parquet, replay into new tool. Estimated effort: 2-3 days. - Registry swap: download all artifacts from MLflow Registry, re-upload via new tool's API. Estimated effort: 1 day.
- Dataset versioning swap (DVC → LakeFS): DVC remotes are S3-compatible; point LakeFS at the same bucket. Estimated effort: 2 days.
Total worst-case full-stack migration: ~1 week. Acceptable lock-in.
Verification
- synapse provisioned via
noxys-ai-forgemodules 00..10 (./install.sh --checkclean) -
nvidia-smishows RTX 5070 12 GB on Fedora 43, CUDA 13.0 toolkit - Podman rootless GPU passthrough working (
podman run --device nvidia.com/gpu=all ... nvidia-smi) - LUKS2 full-disk encryption on 1 TB NVMe.
- UPS connected, auto-shutdown on power loss configured.
- MLflow on synapse reachable via Tailscale + Caddy reverse proxy.
- DVC primary remote on synapse MinIO + nightly mirror to Scaleway Object Storage
fr-par. - Scaleway account in
fr-par-1with H100 burst quota approved (1× on-demand). - Scaleway Container Registry namespace
noxys-mlcreated. - Burst billing alerts set at €25 / €40 / €50.
- First end-to-end smoke test: train DistilBERT on 1k-sample PII corpus on synapse, log to MLflow, register v0.1.0 — all air-gapped + Tailscale.
- First burst smoke test: route XLM-RoBERTa-large LoRA fine-tune to Scaleway H100, verify checkpoint round-trip back to synapse MinIO via DVC.
Links
- prd-ml-training-pipeline
- prd-detection-tiering
- adr-006-detection-tiering
- MASTER-ROADMAP-2026 § 9.22 (ML forge bootstrap)
- runbook-mlflow-ops (TODO)
- runbook-gdpr-erasure (TODO)
- runbook-forge-hardware (TODO — synapse ops, driver updates, thermal/power)
- External repo:
noxys-eu/noxys-ai-forge(codename synapse — provisioning scripts, Podman compose for MLflow/Postgres/MinIO/Qdrant/Redis/Open WebUI)