Leak Detection

Leak detection is an optional feature that detects PII, PHI, and other sensitive data in the logs of watched workloads. When findings exist with sufficient severity, the Incident controller routes subsequent investigations for that workload to ManualReviewRequired — preventing the LLM from being shown logs you haven't reviewed.

What's always installed vs. what's optional

Always installed:

• The SensitiveDataFinding CRD
• The SensitiveDataFinding controller
• The Incident controller's blocking logic — if a finding exists with sufficient severity, investigations route to ManualReviewRequired

Optional:

• The palena-sre-leakdetector Deployment — the component that actually tails logs and calls classifier/NER endpoints

When you enable leak detection, findings start appearing immediately and the Incident routing activates without any additional setup. When leak detection is disabled (the default), the CRD and controller sit idle — nothing creates findings, so nothing blocks investigations.

Why not regex

Regex pattern catalogs don't scale to real log formats. Logs vary across teams: JSON blobs, stringified objects, custom formats, stack traces with embedded sensitive data, debug prints from libraries you didn't write. Per-customer pattern catalogs become unmaintainable. Model-based detection handles the long tail of formats and emerging data types.

The three-tier pipeline

Log line arrives (sampled from a watched workload)
   │
   ▼
┌─────────────────────────────────────────┐
│ Tier 1: Lightweight binary classifier   │
│   Role: Classifier (OpenAI-compatible)  │
│   "contains sensitive data? y/n"        │
└─────────────────────────────────────────┘
   │ (positive — typically <5% of lines)
   ▼
┌─────────────────────────────────────────┐
│ Tier 2: NER entity extraction           │
│   Role: NER (OpenAI-compatible)         │
│   Emits entities with categories        │
└─────────────────────────────────────────┘
   │ (entities match configured sensitive categories)
   ▼
┌─────────────────────────────────────────┐
│ Tier 3 (optional): LLM contextualizer   │
│   Role: Triage (OpenAI-compatible)      │
│   "is this actually sensitive in this   │
│    context?"                            │
│   Reduces false positives on log noise  │
└─────────────────────────────────────────┘
   │
   ▼
Create or update SensitiveDataFinding

All three tiers talk to their endpoints through the same OpenAI-compatible client the investigator uses — no special protocol.

You host the models

The operator ships zero model weights. You host:

• A classifier at some URL (DistilBERT-based, MiniLM, or similar)
• An NER model at some URL (GLiNER, Piiranha, or a custom model)
• Optionally, a contextualizer — any chat-completion-capable LLM

Two integration styles, both supported:

Native OpenAI chat completions. Classifier and NER accept chat messages and return structured output. The leak detector formats prompts appropriately for the role. Works with any chat-completion endpoint.

Custom NER adapted. You wrap a non-chat NER model like GLiNER with a thin shim that accepts chat completions and returns entity JSON. The shim is your responsibility. Some LLM gateways — LiteLLM, for example — already ship adapters for common NER models.

The operator assumes OpenAI chat completions end-to-end. No special-case protocols.

Category taxonomy

Built-in categories the detector recognizes from NER output:

Category	Typical entity
Email	Email addresses
PhoneNumber	International and domestic
CreditCard	13–19 digit card numbers
SSN	US Social Security numbers
IBAN	International bank account numbers
SWIFT	Bank codes
IPAddress	IPv4 / IPv6 (context-dependent)
PersonName	Full names
DateOfBirth	Dates in person context
Address	Physical addresses
APIKey	High-entropy strings in sensitive positions
AWSAccessKey	AKIA-prefixed keys
JWT	JWT tokens
PrivateKey	PEM-formatted private keys

Vertical packs are YAML configs that add domain-specific recognizers:

Pack	Added categories
healthtech	MRN, NPI, ICD10Code, CPTCode, DEANumber, PatientID
fintech	AccountNumber, RoutingNumber, CVV
govtech	PassportNumber, DriverLicense, TaxID

Packs are YAML only — no code ships per-pack. Load via DetectionPolicy.leakDetection.recognizerPackRef.

Ingestion

Loki (preferred). The leak detector runs tail queries against Loki per watched workload and batches lines into the pipeline.

Pod logs API fallback. When Loki isn't available, the detector falls back to streaming via the Kubernetes pod logs API. Heavier on kube-apiserver; use only when Loki isn't an option.

Sampling. Configurable per DetectionPolicy:

• Default: 5% random
• While an active Incident is open on the workload: 100%
• While a high-severity finding is active: 20% until acknowledged

SensitiveDataFinding emission

One CRD per (workload, category-set, severity) tuple — not per log line.

Dedup key. hash(workload.uid + categories_sorted + severity). If a matching finding exists in Active phase, occurrenceCount increments and lastSeen moves forward. Otherwise a new finding is created.

Excerpt safety. The excerpt field stores a sample log line with detected entities replaced by [CATEGORY_REDACTED] tokens. Raw sensitive values never reach etcd. An invariant test in CI enforces this.

Entity positions. Start and end offsets are recorded so a reviewer sees what was detected where, without the raw value ever being stored.

False positive handling

A human marks a finding as FalsePositive via kubectl annotation or a UI. Two effects:

1. The finding transitions to FalsePositive — it no longer blocks investigations.
2. A record is written to a LeakDetectorFeedback ConfigMap for offline retraining by whatever training pipeline you operate. (Model training is out of scope for the operator.)

How findings affect Incidents

See Incident lifecycle. The Incident controller consults active findings on the Detected → next phase decision:

findings = list SensitiveDataFindings in the workload's namespace
activeBlocking = filter
    f.status.phase == "Active"
    f.spec.workload.uid == incident.spec.workload.uid
    f.spec.severity >= policy.leakDetection.blockSeverityThreshold

If activeBlocking is non-empty and policy.onSensitiveDataActive == ManualReviewOnly, the Incident routes to ManualReviewRequired and the LLM never sees the workload's logs until a human has reviewed the finding.

Enable / disable

# Helm values
leakDetector:
  enabled: true                 # default: false
  classifierEndpointRef:
    name: pii-classifier
    namespace: palena-sre-system
  nerEndpointRef:
    name: pii-ner
    namespace: palena-sre-system
  contextualizerEndpointRef:    # optional tier 3
    name: pii-contextualizer
    namespace: palena-sre-system

When leakDetector.enabled: false, the Deployment is not created. No classifier or NER endpoints are needed. The CRD and controller sit idle — nothing creates findings. If you later enable the workload, findings start appearing immediately and the Incident routing activates without further configuration.

Compliance framing

The detector emits evidence, not legal determinations.

• "Detected a value matching the SSN category in logs" — correct.
• "SSN was leaked" or "HIPAA breach detected" — not claims the operator makes.

Breach determination involves BAA scope, access control analysis, intent, and legal context the operator doesn't know about. The operator's job is to give your compliance team structured evidence they can act on.

What's out of scope

• Masking data in logs at source — a log-pipeline concern (Fluent Bit filters, etc.)
• Storing actual sensitive values anywhere — an invariant test prevents this
• Making legal or compliance determinations
• Blocking workloads from running
• Model training or fine-tuning — you operate this separately