Skip to main content

Upgrade & Maintenance

Procedures for upgrading Noxys, rolling back if needed, and maintaining your deployment.

Before Upgrading​

Prerequisites​

  1. Backup current state

    make backup
    # Or manually:
    docker compose exec postgres pg_dump -U noxys noxys | gzip > backup-$(date +%Y-%m-%d).sql.gz

  2. Review release notes

    • Check for breaking changes
    • Review migration requirements
    • Understand new features and fixes

  3. Test in staging

    • Always test upgrades in a non-production environment
    • Verify all functionality works
    • Check integrations (webhooks, SSO, etc.)

  4. Schedule maintenance window

    • Inform users of planned downtime
    • Choose low-traffic period
    • Allocate 30-60 minutes for upgrade

  5. Verify system health

    docker compose ps
    docker compose logs api | tail -20
    docker compose exec postgres psql -U noxys -d noxys -c "SELECT COUNT(*) FROM interactions;"

Docker Compose Upgrades​

Minor Version Upgrade (0.2.x → 0.2.y)​

Typically zero-breaking-change updates.

1. Pull Latest Images​

docker compose pull

# Verify images downloaded
docker images | grep noxys

2. Stop Current Version (Graceful Shutdown)​

# Give running requests 30 seconds to complete
docker compose stop -t 30 api console

3. Start New Version​

docker compose up -d api console

# Verify health
sleep 5
docker compose exec api curl -f http://localhost:8080/healthz

4. Verify Functionality​

# Check logs for errors
docker compose logs api | grep -i error

# Test API
curl -X POST http://localhost:8080/api/v1/interactions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"test": true}'

# Verify dashboard access
# Open http://localhost:3000 in browser

Major Version Upgrade (0.2.x → 0.3.0)​

May include database migrations and breaking changes.

1. Check for Database Migrations​

# Review migration requirements in release notes
# Example: v0.3.0 adds new `risk_score` column to interactions table

# Noxys automatically runs pending migrations on startup

2. Backup Before Migrating​

make backup
docker compose exec postgres pg_dump -U noxys noxys | gzip > backup-pre-v0.3.0.sql.gz

3. Update .env File​

# Check if any new variables added in release notes
# Example: v0.3.0 adds NOXYS_RISK_SCORING_ENABLED

NOXYS_RISK_SCORING_ENABLED=true

4. Pull Images & Update Compose​

# Update docker-compose.yml to new version
nano docker-compose.yml
# Change image tags: noxys/proxy:v0.2.5 → noxys/proxy:v0.3.0

# Or use a version override
export NOXYS_VERSION=v0.3.0
docker compose pull

5. Run Migrations​

# Stop API to avoid conflicts
docker compose stop api

# Noxys will run migrations automatically on first start
# For large databases, migrations may take several minutes
docker compose up -d api

# Monitor migration progress
docker compose logs -f api | grep -i "migrat"

# Wait for "Migration complete" message

6. Verify Migration Success​

# Check for errors
docker compose logs api | grep -i error

# Verify data integrity
docker compose exec postgres psql -U noxys -d noxys << 'EOF'
SELECT COUNT(*) as interaction_count FROM interactions;
SELECT COUNT(*) as policy_count FROM policies;
SELECT COUNT(*) as user_count FROM users;
EOF

# Expected: all counts should match pre-upgrade

7. Start All Services​

docker compose up -d

# Wait 10 seconds for services to initialize
sleep 10

# Verify readiness
docker compose exec api curl -f http://localhost:8080/readyz

Kubernetes Upgrades​

Using Helm​

# Update Helm repository
helm repo update noxys

# Dry-run to preview changes
helm upgrade noxys noxys/noxys \
  --namespace noxys \
  --values values.yaml \
  --dry-run --debug

# Apply upgrade
helm upgrade noxys noxys/noxys \
  --namespace noxys \
  --values values.yaml \
  --wait \
  --timeout 10m

# Verify rollout
kubectl rollout status deployment/noxys-api -n noxys
kubectl rollout status deployment/noxys-console -n noxys

Zero-Downtime Rolling Update​

Kubernetes performs rolling updates by default (one pod at a time):

# Monitor the upgrade in real-time
kubectl get pods -n noxys -w

# After upgrade, verify all pods are running
kubectl get pods -n noxys

Manual Manifest Update​

# Update YAML files with new image versions
nano api-deployment.yaml
# Change image: noxys/proxy:v0.2.5 → noxys/proxy:v0.3.0

# Apply changes
kubectl apply -f api-deployment.yaml

# Monitor rollout
kubectl rollout status deployment/noxys-api -n noxys

Rollback Procedures​

Docker Compose Rollback​

If issues occur after upgrade:

# Option 1: Use the previous version's docker-compose file
git checkout HEAD~1 docker-compose.yml
docker compose pull
docker compose up -d api console

# Option 2: Manually specify the previous version
docker pull noxys/proxy:v0.2.5
docker compose stop api
# Edit docker-compose.yml to use v0.2.5
docker compose up -d api

# Verify
docker compose logs api

Restore Database Backup​

If database migrations caused issues:

# Stop API
docker compose stop api

# Restore from backup
docker compose exec postgres dropdb -U noxys noxys
docker compose exec postgres createdb -U noxys noxys
docker compose exec postgres psql -U noxys noxys < backup-pre-v0.3.0.sql.gz

# Start API with previous version
docker compose up -d api

# Verify data
docker compose exec postgres psql -U noxys -d noxys -c "SELECT COUNT(*) FROM interactions;"

Kubernetes Rollback​

# View rollout history
kubectl rollout history deployment/noxys-api -n noxys

# Rollback to previous version
kubectl rollout undo deployment/noxys-api -n noxys

# Or rollback to specific revision
kubectl rollout undo deployment/noxys-api -n noxys --to-revision=2

# Or with Helm:
helm rollback noxys 1 -n noxys

Maintenance Tasks​

Regular Backups​

Daily automated backups:

# Docker Compose
make backup KEEP=30  # Keep 30 days

# Manual
docker compose exec postgres pg_dump -U noxys noxys | \
  gzip > backups/noxys_$(date +%Y-%m-%d_%H-%M-%S).sql.gz

Disk Space Management​

# Check storage usage
df -h
du -sh /var/lib/docker/volumes/*/

# Remove old backups
find ./backups -name "*.sql.gz" -mtime +30 -delete

# Prune Docker
docker system prune -a

Database Maintenance​

# Analyze query plans for slow queries
docker compose exec postgres psql -U noxys -d noxys << 'EOF'
EXPLAIN ANALYZE SELECT * FROM interactions WHERE created_at > NOW() - INTERVAL '7 days';
EOF

# Reindex tables (optional, for fragmentation)
docker compose exec postgres psql -U noxys -d noxys << 'EOF'
REINDEX TABLE interactions;
REINDEX TABLE policies;
EOF

# Vacuum (reclaim space)
docker compose exec postgres psql -U noxys -d noxys << 'EOF'
VACUUM FULL interactions;
VACUUM FULL policies;
EOF

Log Rotation​

# View Docker logs
docker compose logs --tail=100 api

# For high-volume logs, configure log rotation
docker compose logs --follow api > noxys-api.log &

# Or configure in daemon.json
cat > /etc/docker/daemon.json << 'EOF'
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}
EOF

systemctl restart docker

Certificate Renewal​

For Let's Encrypt:

# Renew certificate
sudo certbot renew --force-renewal

# Verify renewal
sudo certbot certificates

# Restart Noxys to load new cert
docker compose restart api

Version Pinning Strategy​

Pin to Specific Version​

For production stability, always pin versions:

# docker-compose.yml (DO NOT use 'latest')
services:
  api:
    image: noxys/proxy:v0.3.0  # Specific version
  console:
    image: noxys/console:v0.3.0
  postgres:
    image: postgres:16.2  # Pin database too!
  redis:
    image: redis:7.2

Version Upgrade Path​

Upgrade conservatively:

v0.2.0 → v0.2.1 (patch, safe)
       → v0.3.0 (minor, test first)
       → v1.0.0 (major, extensive testing)

Always skip major versions incrementally (0.3 → 0.4 → 1.0, not 0.3 → 1.0).

Monitoring Upgrades​

Health Checks​

# Create a monitoring script
cat > monitor-upgrade.sh << 'EOF'
#!/bin/bash
while true; do
  api_status=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/healthz)
  db_status=$(docker compose exec postgres psql -U noxys -d noxys -c "SELECT 1" 2>&1)

  echo "API: $api_status, DB: $([ -z "$db_status" ] && echo "OK" || echo "ERROR")"
  sleep 5
done
EOF

chmod +x monitor-upgrade.sh
./monitor-upgrade.sh

Alert on Failure​

# Slack notification on upgrade failure
curl -X POST $SLACK_WEBHOOK_URL \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "Noxys upgrade failed! API status: '$api_status'",
    "attachments": [{"color": "danger"}]
  }'

Troubleshooting Upgrades​

Services Won't Start After Upgrade​

# Check logs
docker compose logs api

# Common issues:
# 1. Port in use
lsof -i :8080

# 2. Database connection error
docker compose logs postgres

# 3. Environment variable missing
grep "variable not found" docker compose logs api

Database Migration Hangs​

# Check migration status
docker compose exec postgres ps aux | grep postgres

# View long-running queries
docker compose exec postgres psql -U noxys -d noxys << 'EOF'
SELECT query, now() - query_start AS duration
FROM pg_stat_activity
WHERE query NOT LIKE '%pg_stat_activity%'
ORDER BY duration DESC;
EOF

# Kill long-running migration (if safe)
# docker compose exec postgres psql -U noxys -d noxys -c "SELECT pg_terminate_backend(pid);"

API Crashes After Startup​

# Check memory/CPU limits
docker stats noxys-api

# Increase limits in docker-compose.prod.yml
deploy:
  resources:
    limits:
      memory: 8G
      cpus: '4'

# Restart with new limits
docker compose up -d api

Supported Upgrade Paths​

FromToTypeMigration
v0.2.0v0.2.1PatchNone
v0.2.xv0.3.0MinorAuto (adds columns)
v0.3.xv1.0.0MajorManual (review notes)

Never skip major versions (e.g., 0.2 → 1.0 directly is unsupported).

Support During Upgrades​

If you encounter issues:

  1. Check release notes: https://github.com/noxys-io/noxys/releases
  2. Review migration guide in release
  3. Contact support: support@noxys.eu
  4. For critical issues: security@noxys.eu

Next Steps​

Questions? Email support@noxys.eu