Key Takeaways
- HIPAA (healthcare): Patient data cannot touch cloud APIs. Local LLM on isolated network, encrypted storage, access logs.
- PCI-DSS (payment cards): Payment card data cannot be processed by LLMs at all. Use for analytics only, never full PAN (card numbers).
- Legal (attorney-client privilege): Privileged documents cannot leave attorney's control. Air-gapped machine, no network, hardcopy output only.
- Setup: Ollama or vLLM on isolated Linux server, encrypted filesystem (LUKS), audit logging (ELK), no internet. Cost: $3K-5K hardware + $2K/year updates.
- EU/GDPR: Article 32 (Security), Article 35 (DPIA required), AI Act 2024/1689 (risk assessment). Data residency = local storage only.
- vs Cloud APIs: Cloud = vendor controls data + breach liability. Local = you control + zero breach risk.
Why Local LLMs for Compliance?
Cloud APIs (ChatGPT, Claude, Gemini) cannot be used with regulated data:
- Data transmission to cloud = breach of confidentiality (HIPAA, legal privilege).
- No "private mode" option. Data trains models eventually (vendor ToS allows it).
- Vendor lock-in: if vendor breaches or shuts down, you lose data + compliance standing.
Local LLM guarantees:
- Zero data egress (air-gapped = no network, no cloud).
- Audit trail (every access logged, cryptographically signed, immutable).
- Control (you own the data, encryption keys, and the entire stack).
- Cost predictability (no per-token charges after initial $5K hardware investment).
HIPAA-Compliant Setup (Healthcare)
PHI (Protected Health Information) cannot be processed by untrusted systems. HIPAA requires 45 CFR 164.312 (encryption, access controls, audit logging).
- 1Isolate the server: Dedicated Linux machine (no shared resources), no internet, encrypted USB for data transfer in/out.
- 2Encrypt storage: LUKS full-disk encryption, AES-256, passphrase-protected. Prevents data leakage if hardware is stolen.
- 3Network isolation: Dedicated VLAN or air-gapped (no network). Access via VPN with MFA, or physical terminal only.
- 4Audit logging: Every LLM query logged: timestamp, user ID, document hash (not plaintext), response length, model used. Logs stored on separate encrypted syslog server.
- 5Access control: Role-based (doctor vs. admin vs. researcher). MFA for login. No shared passwords. Disable account on termination.
- 6Retention policy: Delete inference logs after 6 years (HIPAA requirement: 45 CFR 164.312). Automated purging scripts with cryptographic verification.
- 7Business Associate Agreement (BAA): Only applies to vendors. Open-source models (Llama, Mistral) require internal compliance documentation, not vendor signatures.
- 8Annual penetration test: Third-party security audit to verify no data exfiltration, no default credentials, no unpatched vulnerabilities.
PCI-DSS Compliant Setup (Finance)
Payment card data (PAN = Primary Account Number) cannot be processed by LLMs. PCI-DSS v4.0 requires 12 core requirements.
- 1Never input full card numbers into LLM. PCI-DSS prohibits it entirely. Use tokenized representations (last-4-digits only, no expiry date).
- 2Encrypt at rest & in transit: AES-256 encrypted files, TLS 1.3 for network. All card data encrypted before leaving the terminal.
- 3Network segmentation: LLM server on isolated VLAN with firewall rules. No access to internet, no access to corporate network.
- 4Hardware security module (HSM): Store encryption keys in tamper-evident device (Thales, Yubico HSM). Separate from LLM server.
- 5Logging & monitoring: Real-time alerts on file access, login attempts, failed authentication, data exfiltration. SIEM integration (Splunk, ELK).
- 6Quarterly compliance scan: Automated PCI-DSS scanning (Qualys, Nessus, Rapid7) for vulnerabilities. Remediate within 30 days.
- 7Vendor documentation: Model providers (Ollama, vLLM) are not payment processors and are not PCI vendors. Your local deployment is in-scope.
Air-Gapped Deployment
Most secure option: machine has zero network connection (Faraday cage-level isolation).
- 1Physical isolation: Server in locked room (physical access control), no Ethernet cable, WiFi disabled in BIOS, Bluetooth disabled.
- 2Model loading: Pre-download models on a connected machine (download from HuggingFace), transfer via encrypted USB (encrypted with GPG, AES-256).
- 3Data transfer in: Users transfer documents via encrypted USB (GPG-encrypted or 7z encrypted). Scan USB for malware on air-gapped machine.
- 4Inference: Run LLM locally (Ollama, vLLM), output saved to USB. No network calls, no external API access.
- 5Data transfer out: Encrypted USB returned to user, decrypted on separate non-sensitive machine. Original encrypted USB destroyed after decryption verification.
- 6Trade-off: Latency (manual USB transfer takes minutes) vs. absolute security (zero network risk, zero remote breach possibility).
- 7Use case: Legal discovery (attorney review), healthcare image analysis, financial model training (batch processing OK with 1-hour latency).
EU/GDPR/AI Act Compliance
EU data protection law (GDPR, EU AI Act 2024/1689) requires explicit security and risk assessments.
- 1GDPR Article 32 (Security): Ensure appropriate encryption, pseudonymization, confidentiality, integrity, resilience. Local LLM satisfies this by default (no cloud transmission = no risk).
- 2GDPR Article 35 (Data Protection Impact Assessment): Required for high-risk processing. Document: data types (medical, financial), retention (6 years), access controls, breach scenarios. Local LLM = low risk (contained).
- 3GDPR Article 17 (Right to Deletion): "Right to be forgotten." Local logs must be purged on request. Implement automated anonymization scripts (remove user ID, hash documents).
- 4EU AI Act 2024/1689 (High-Risk Systems): Classification depends on use case. Medical diagnosis = high-risk (requires explainability). Legal document review = medium-risk (requires audit trail). Implement risk assessment, human review process.
- 5DPIA (Data Protection Impact Assessment): Mandatory before deployment. Document: processing purpose (diagnosis, case review), data categories (medical records, contracts), third parties (none), retention (6 years), safeguards (encryption, access logs).
- 6German BDSG (Bundesdatenschutzgesetz): Extends GDPR with additional requirements: system provider must be German/EU (local deployment = compliant), data processing contracts mandatory (self-processing = compliant).
Recommended Models & Hardware Sizing
Choose models based on compliance requirements and infrastructure size.
| Use Case | Model | VRAM | Hardware | Why |
|---|---|---|---|---|
| Document Review (Legal) | LLaMA 4 Scout (7B) | 8-12 GB | RTX 4060 / M4 Max | Fast, accurate legal reasoning, small footprint |
| Medical Records (HIPAA) | Mistral Large (34B) | 32-40 GB | RTX 4090 / A100 (40GB) | High accuracy, medical knowledge, meets HIPAA requirements |
| Financial Analytics (PCI) | Llama 3.1 70B | 70-80 GB | A100 (80GB) / H100 | Financial reasoning, compliance audit trails |
| Small Teams (<10) | Mistral 7B Instruct | 8-16 GB | M3/M4 Pro or RTX 4070 | Cost-effective, sufficient for basic document handling |
Local LLM vs Cloud API Comparison
Direct comparison of deployment models.
| Factor | Local LLM | Cloud API |
|---|---|---|
| Data Security | Zero egress. Data stays on-premise. Encrypted at rest and in transit. | Data sent to vendor servers. Vendor may train on it (ToS allows). Breach liability on vendor. |
| Compliance | HIPAA/PCI/GDPR compliant. Audit logs under your control. DPIA required but low-risk. | Not compliant with regulations. Vendor is data processor, you are liable. |
| Cost | $3K-5K upfront hardware. $0-500/year maintenance. Predictable. | $0 upfront. $500K+/year at scale (tokens × 2026 pricing). Unpredictable. |
| Breach Liability | $0 (data never leaves your control). Insurance not required. | $50K-5M+ (vendor breach = you are liable to affected parties under HIPAA/GDPR). |
Audit Logging & Data Governance
What to log: Every LLM query (timestamp, user, prompt hash, response length, model version), file access (open/read/modify), login/logout (IP, MFA status).
Where to store: Encrypted syslog server, separate physical machine from LLM server. Prevents data breach from compromising logs.
Tamper-evidence: Cryptographic signatures on logs (SHA-256, signed with admin key). No deletion without breaking chain of trust. Implement append-only storage.
Tools: ELK Stack (Elasticsearch/Logstash/Kibana) for aggregation and search; Splunk for enterprise (60-day retention standard).
Retention policy: HIPAA = 6 years, GDPR = right to deletion (anonymize after 6 months), PCI-DSS = 1 year. Automate purging with cryptographic verification.
Monthly compliance verification: Log review (spot-check 5% of logs). Quarterly data lineage audit (trace queries back to source). Annual third-party assessment (penetration test, log verification).
Common Compliance Failures
- Using cloud ChatGPT with healthcare data. Immediate HIPAA violation. Penalty: $10K-$50K per incident (up to $1.5M per year). Example: hospital staff using ChatGPT to draft discharge summaries (exposed PHI).
- Air-gapped server with unlocked door. Physical security = zero if anyone can walk in. Example: compliance auditor found LLM server in publicly accessible server room with no badge access control.
- Logs stored on same server as data. Breach of logs = breach of audit trail. Separate systems required (45 CFR 164.312(b)). Example: ransomware encrypted both data and logs, audit trail destroyed.
- No encryption of data in transit. Unencrypted USB transfers on shared network. Sniffer attack captures medical records. Use GPG-encrypted files, verify checksums.
- BAA with open-source models. Open-source models (Llama, Mistral) have no vendor to sign BAA. Instead, document your compliance internally (audit logs, risk assessment, DPIA). BAA is only for vendors.
- No retention policy. HIPAA requires purging after 6 years. Failing to delete = violation. Implement automated scripts that delete logs on schedule with cryptographic proof.
FAQ
Can I use cloud LLMs with compliance data if I hash PII?
No. Hashed data is still regulated. HIPAA/GDPR prohibit transmission to any vendor, including cloud APIs. Hashing does not remove the data from the vendor's control. Use local LLM only.
Do I need a BAA with Llama or Mistral models?
No. Open-source models have no vendor to sign a BAA. Instead, document your compliance internally: risk assessment, data processing procedures, audit logs, retention policy. BAA is only required if using a vendor (OpenAI, Anthropic, Google).
Is air-gapped overkill for HIPAA?
Not overkill if data is highly sensitive (genetics, psychiatric records). Best practice per HIPAA Omnibus Rule (2013). For less sensitive data (basic consultations), a VPN-protected local deployment is acceptable with monthly audits.
How do I handle employee termination securely?
Immediately disable VPN access. Audit all LLM queries by that user in past 6 months (compliance requirement). Verify no confidential data was exported. Archive logs (read-only) for 6 years (HIPAA retention). Remove user from access control lists.
Can I use local LLMs for legal discovery?
Yes. Air-gapped + attorney supervision maintains attorney-client privilege (no third-party access). Document: chain of custody, data handling procedures, access logs. Meet e-discovery requirements (FRCP 34).
What if there's a breach of the local server?
Encrypted-at-rest = limited damage (attacker cannot read data). Audit logs reveal what was accessed (compromised queries only). Notify affected parties within 30 days (HIPAA/GDPR requirement). Incident response: isolate server, forensics, update passwords, penetration test.
Is local inference slower than cloud APIs?
Latency is slightly higher (200ms local vs. 50ms cloud), but throughput is comparable. Batch processing (legal review, medical image analysis) sees no practical difference. Real-time chat is acceptable for most use cases.
Can I store local LLM outputs in the cloud after inference?
Only if encrypted end-to-end (you hold encryption key, cloud provider cannot access plaintext). Recommended: store locally, backup to encrypted cloud storage (AWS S3 with server-side encryption). Comply with data residency requirements (EU = data stays in EU).
Sources
- HIPAA Privacy Rule: 45 CFR 164.300-318 (US Department of Health & Human Services, 2013)
- HIPAA Security Rule: 45 CFR 164.300-318 (Encryption, access controls, audit logging)
- PCI Data Security Standard v4.0 (PCI Security Standards Council, 2022) — Payment card data handling
- GDPR Articles 32, 35, 17 (General Data Protection Regulation, EU, 2016) — Security, DPIA, right to deletion
- EU AI Act 2024/1689 (European Union, 2024) — High-risk AI system governance
- German BDSG (Bundesdatenschutzgesetz, 2018) — Data residency and processing requirements