Security Best Practices for GPU Servers

Overview

GPU servers running ML workloads have unique security considerations: large datasets, model weights, and API keys are high-value targets.

Network Security

Firewall Rules

Restrict access to the minimum required ports:

Use SSH Tunnelling for Jupyter

Instead of exposing port 8888:

$ssh -L 8888:localhost:8888 root@<SERVER_IP>

VPN for Multi-Server Clusters

Use WireGuard or Tailscale for inter-node communication:

$apt-get install -y wireguard

Credential Management

Never Hardcode API Keys

# ❌ Bad
openai.api_key = "sk-abc123..."

# ✅ Good
import os
openai.api_key = os.environ["OPENAI_API_KEY"]

Use Environment Files

$# .env (never commit to git)
$OPENAI_API_KEY=sk-abc123...
$HF_TOKEN=hf_abc123...

$# .gitignore
$.env
$*.key
$secrets/

Data Security

Encrypt Sensitive Datasets

$# Encrypt with GPG
$gpg --symmetric --cipher-algo AES256 sensitive_data.csv
$
$# Decrypt
$gpg --decrypt sensitive_data.csv.gpg > sensitive_data.csv

Secure Model Weights

If your model weights are proprietary:

1. Store in private object storage with access keys
2. Download at runtime, not baked into Docker images
3. Delete from disk after use if not needed persistently

Monitoring

Audit Login Attempts

$# Recent successful logins
$last -n 20
$
$# Failed login attempts
$grep "Failed password" /var/log/auth.log | tail -20
$
$# Currently logged-in users
$who

Monitor GPU Usage

$# Detect unexpected GPU processes
$nvidia-smi --query-compute-apps=pid,used_memory,name --format=csv

Security Checklist

• SSH key authentication only (no passwords)
• Firewall restricts all ports except SSH
• Jupyter accessed via SSH tunnel only
• API keys in environment variables, not code
• .env files in .gitignore
• fail2ban installed
• System packages updated weekly
• Snapshots taken before major changes