Architecture

palena-sre is three Kubernetes workloads plus a set of CRDs. A fourth workload — the leak detector — is optional and only deployed when you enable it.

Component topology

┌──────────────────────────────────────────────────────────────────┐
│  palena-sre-operator  (Deployment, 1–3 replicas, leader-elected) │
│  ──────────────────────────────                                  │
│  - Reconcilers for every CRD                                     │
│  - Orchestrates dispatch to the detector and investigator        │
│  - Runs the notification dispatcher                              │
└──────────────────────────────────────────────────────────────────┘
         │ owns / reconciles
         ▼
┌──────────────────────────────────────────────────────────────────┐
│  palena-sre-detector  (Deployment, 1–N replicas)                 │
│  ──────────────────────────────                                  │
│  - Kubernetes event informers                                    │
│  - PromQL rule evaluation (when Prometheus configured)           │
│  - metrics-server polling (when available)                       │
│  - MAD/EWMA anomaly detection                                    │
│  - Drain3-based log pattern drift                                │
│  - AlertManager webhook receiver                                 │
│  - Emits and updates Incident CRDs                               │
└──────────────────────────────────────────────────────────────────┘
         │ creates / updates
         ▼
┌──────────────────────┐               ┌────────────────────────────────┐
│  Incident CRD        │ ◄──── updates │  palena-sre-investigator       │
│  (durable artifact)  │               │  (Deployment, autoscaled)      │
│                      │ ────────────► │  - ReAct loop                  │
└──────────────────────┘    reads      │  - Evidence bundle             │
                                       │  - Calls configured LLM URL    │
                                       └────────────────┬───────────────┘
                                                        │
                                                        ▼
                                       ┌────────────────────────────────┐
                                       │  External (customer-configured)│
                                       │  - LLM endpoint (OpenAI-compat)│
                                       │  - Prometheus (optional)       │
                                       │  - Loki / Elasticsearch        │
                                       │  - ArgoCD / GitHub (optional)  │
                                       └────────────────────────────────┘

┌──────────────────────────────────────────────────────────────────┐
│  palena-sre-leakdetector  (OPTIONAL Deployment)                  │
│  ──────────────────────────────                                  │
│  - Tails logs of watched workloads                               │
│  - Calls customer-configured classifier + NER endpoints          │
│  - Emits SensitiveDataFinding CRDs                               │
└──────────────────────────────────────────────────────────────────┘

Why separate workloads

Each workload has a different load profile and blast radius, so each is a separate Deployment with its own resource requests, scaling behavior, and restart semantics.

Operator. Low CPU, low memory, leader-elected. Reconcilers are predictable and boring. Restart = a short reconcile delay.

Detector. Stateful — holds rolling windows for anomaly detection. CPU-heavy on PromQL evaluation and Drain3 log mining. Shardable in the future by consistent hashing on SREWatch UID; single replica is fine for v1 workloads. Restart = a short gap in anomaly baselines, recovered from periodic ConfigMap checkpoints.

Investigator. Bursty, LLM-latency-bound, scales on queue depth. Can scale to zero with KEDA if that fits your cluster. Restart = in-flight investigations re-enqueued via heartbeat staleness; completed work is durable on the Incident CRD.

Leak detector. Only runs when enabled. Isolated because log tailing is I/O-heavy and the NER pipeline is CPU-heavy — neither should disturb the core operator.

Data flow — happy path

1. You create an SREWatch selecting app=payments pods. You also create an LLMEndpoint describing the URL to call.
2. The operator reconciles the SREWatch and publishes the workload list to a detector ConfigMap.
3. The detector sees a payments-abc-xyz pod enter CrashLoopBackOff. It computes a fingerprint (workload, signal class, 5-minute bucket). There's no open Incident with that fingerprint — it creates one in phase Detected.
4. The operator's Incident controller sees the new Incident. It consults the InvestigationPolicy: severity meets the threshold, budget is available, no blocking SensitiveDataFinding — so it transitions the phase to Investigating.
5. The investigator picks up the Incident. It builds an evidence bundle: pod spec, recent events, last N log lines, metrics, recent deploys.
6. The investigator calls the configured LLM endpoint using OpenAI chat completions. It runs a ReAct loop using the matched Runbook. Tool calls (kubectl, PromQL, Loki, etc.) happen as the model requests them.
7. The investigator writes hypothesis, confidence, and recommendations back to the Incident and transitions to Hypothesized.
8. The notification dispatcher sees the phase change and delivers to every NotificationChannel the watch lists.
9. An engineer reviews, sets status.verdict, and the Incident transitions to Resolved.
10. An IncidentTrace CRD is emitted for palena-eval consumption.
11. After the retention window, the Incident transitions to PostMortemed and becomes eligible for deletion.

Data flow — no LLM endpoint configured

Detection, notification, correlation, and leak detection all work without an LLMEndpoint. Incidents are created and notifications fire, but the Incident never leaves Detected — the controller surfaces a NoLLMEndpoint condition and waits.

Use this mode when you want detection without LLM spend, or while you're bringing up the LLM infrastructure.

External dependencies

Dependency	Required?	Interface	Failure mode
Kubernetes API	Yes	controller-runtime client	Operator retries
metrics-server	Auto-discovered	metrics.k8s.io	Evidence enrichment skipped
Prometheus	Optional	HTTP query API	PromQL detection skipped
Loki / Elasticsearch	Optional	HTTP API	Log-based detection + toolsets degraded
Tempo	Optional	HTTP API	Trace toolset degraded
LLM endpoint	Required for investigation only	OpenAI-compatible HTTP	Investigations queue; notifications still fire
ArgoCD	Optional	HTTP API	ArgoCD correlation skipped
GitHub	Optional	HTTP API	GitHub correlation skipped
Classifier + NER endpoints	Required only if leak detection enabled	OpenAI-compatible HTTP	Leak detector degraded; everything else unaffected
Slack / Teams / PagerDuty / SMTP	Optional	HTTP / SMTP	Notifications queued + retried

No Palena sibling product appears in this list because none is required.

Leader election and HA

The operator uses controller-runtime lease-based leader election. Two replicas in production; one in dev is fine.

The detector runs as a single replica in v1. The design accommodates consistent-hashing sharding on SREWatch UID when the time comes; no v1 customer hits that scale.

The investigator scales on a custom queue-depth metric derived from the count of Incidents in Investigating. It can scale to zero when KEDA is installed.

The leak detector, when present, scales roughly one replica per 20 watched workloads at typical log rates.

Multi-cluster

v1 is single-cluster. The Incident CRD carries a clusterID field from day one so multi-cluster federation — a IncidentAggregation hub CRD — can aggregate across clusters without schema changes later.

Namespaces

Default installation namespace: palena-sre-system. The operator, detector, and investigator live here along with their ServiceAccounts.

Most CRDs are namespaced. LLMEndpoint and Runbook can live in any namespace and be referenced across namespaces via explicit refs — typical practice is to keep LLMEndpoint resources in palena-sre-system.

Resource footprint

Component	CPU request	CPU limit	Memory request	Memory limit
Operator	100m	500m	128Mi	512Mi
Detector	500m	2000m	512Mi	2Gi
Investigator	200m	1000m	256Mi	1Gi
Leak detector	500m	2000m	512Mi	2Gi

The LLM model footprint is entirely your problem — the operator never hosts a model.

Relation to other Palena components

None is required at runtime. If you install palena-shield as a gateway in front of your LLM, you point the operator's LLMEndpoint at it — from the operator's perspective, palena-shield is indistinguishable from any other OpenAI-compatible URL.

The operator emits IncidentTrace CRDs on every investigated Incident. palena-eval is the intended consumer; if you haven't installed palena-eval, the traces simply accumulate until you do.