Detection

The detector watches cluster state and external observability sources, deduplicates signals, and writes Incident CRDs. It never calls an LLM. Detection is deterministic, bounded, and cheap — LLM work happens only after an Incident exists.

Signal sources

Six sources feed the same Incident pipeline:

Source	Mechanism	Requires	Typical latency
Kubernetes API events	controller-runtime informers	Kubernetes only	sub-second
PromQL rules	periodic eval against Prometheus HTTP API	Prometheus configured	scrape interval
metrics-server	polling the Metrics API	metrics-server available	~15s
Classical anomaly	rolling window + MAD/EWMA	any metric source	scrape interval
Log pattern drift	Drain3 via a Python sidecar	Loki or Elasticsearch	streaming
AlertManager webhook	HTTP receiver	AlertManager pushing to the detector	push

All six feed a single fingerprint-based dedup pipeline, so one root cause produces one Incident even when it surfaces across multiple sources.

Kubernetes events

Primary mechanism. Informers filtered at cache setup time by the label selectors on active SREWatch resources.

Signal	Trigger
CrashLoopBackOff	Pod containerStatus.waiting.reason == "CrashLoopBackOff"
OOMKilled	Pod lastState.terminated.reason == "OOMKilled"
Evicted	Pod phase=Failed AND status.reason == "Evicted"
FailedScheduling	Event.reason == "FailedScheduling" on a Pod
ImagePullBackOff	Pod waiting reason ImagePullBackOff or ErrImagePull
UnhealthyProbe	Event.reason == "Unhealthy"
BackoffLimitExceeded	Job condition BackoffLimitExceeded
ReplicaSetNotProgressing	Deployment Progressing condition false for > 10m

Debouncing. The same signal on the same workload within 30s aggregates into an existing signal — count goes up, last-seen moves forward. Different workloads or different signal types get separate evaluation.

Pod-to-workload resolution. Incidents are workload-scoped, not pod-scoped. The detector resolves pod → ReplicaSet → Deployment or StatefulSet via owner references and caches the result.

PromQL rules

When Prometheus is configured, each DetectionPolicy with PromQL rules runs a goroutine evaluating expressions on the configured evaluation interval (default 30s).

Rule expressions are validated against the Prometheus parser at admission time — malformed expressions fail the CRD apply. Hold-for logic ensures a rule has to satisfy its threshold for the for: duration before firing.

Results are post-filtered to watched workloads: the rule must produce a series with a recognizable workload label (pod, deployment, etc.), and the detector drops anything not matching an active SREWatch.

metrics-server

metrics-server ships on every major managed Kubernetes. The detector auto-discovers metrics.k8s.io/v1beta1 at startup.

When available, two features unlock:

Evidence enrichment (always on). On Incident creation, the detector captures a snapshot of CPU/memory usage plus requests and limits for every pod in the watched workload. The snapshot lives on Incident.status.resourceSnapshot and gives the investigator an instant "was the pod at 98% of its memory limit?" answer without waiting on a Prometheus range query.

metrics-server-backed anomaly (opt-in). The same MAD/EWMA logic used against Prometheus, sourced from PodMetrics and NodeMetrics at 15–30s resolution. Four supported targets: PodCPU, PodMemory, NodeCPU, NodeMemory. Useful when Prometheus isn't available or you want a lightweight tier.

Classical anomaly detection

Source-agnostic. Runs against whatever metric source is configured — Prometheus or metrics-server — using the same two algorithms:

MAD (Median Absolute Deviation). Robust to outliers. Good default for most metrics.

EWMA (Exponentially Weighted Moving Average). Lightweight, good for slow drift.

STL decomposition is a v2 feature — there's no mature Go implementation and the operator won't block v1 on porting one.

State management. Each detector holds a rolling window per (workload, metric) in memory. Windows checkpoint to ConfigMaps every 5 minutes so a detector restart loses at most five minutes of baseline.

Log pattern drift

Drain3 groups log lines into templates. When a template emerges that wasn't seen before, or a rare template spikes, the detector fires a signal.

Drain3 runs as a Python sidecar inside the detector Pod, talking to the Go detector over gRPC.

Input. Sampled log lines tailed from Loki or Elasticsearch. Sampling is adaptive: 5% random by default, 100% while an Incident on the workload is open, and 0% while a high-severity SensitiveDataFinding is active on the workload.

Signals.

Signal	Meaning
NewTemplate	Template not seen in windowSize
TemplateSpike	Existing template's rate exceeds threshold × rollingAverage
KnownErrorTemplate	Template matches a curated always-error list

AlertManager webhook

Every detector exposes /alertmanager. Point AlertManager at it and alerts become palena-sre signals.

Each alert's workload is derived from its labels — the label keys are configurable (pod and namespace by default). Alerts without a matching SREWatch are dropped silently. The same fingerprint dedup collapses AlertManager alerts with native Kubernetes event signals or PromQL rule signals for the same root cause.

Fingerprint and dedup

The fingerprint is the deduplication primitive:

fingerprint = sha256(workload.uid + ":" + signalClass + ":" + 5-minute-bucket)

Same workload UID + same signal class in the same bucket → same fingerprint → same Incident.

Signal emitted
  │
  ▼
Compute fingerprint
  │
  ▼
Look up existing Incident by label palena.ai/fingerprint
(filter: phase NOT IN {Resolved, PostMortemed})
  │
  ├─── Found? ──► Append signal, update lastSeen
  │
  ▼
Create a new Incident; capture resource snapshot if metrics-server available

Signal classes. What counts as "the same class" for dedup.

Class	Members
CrashLoop	CrashLoopBackOff, BackoffLimitExceeded, KubePodCrashLooping
Memory	OOMKilled, MemoryAnomaly, KubeMemoryOvercommit
Scheduling	FailedScheduling, UnschedulablePod
ImagePull	ImagePullBackOff, ErrImagePull
Probe	UnhealthyProbe, PodNotReady
ErrorRate	HighErrorRate, LogPatternDrift (error-class templates)
Latency	LatencyAnomaly, SLOLatencyBudgetBurn

What the detector doesn't do

• Call LLMs
• Send notifications
• Execute recommendations
• Write anything outside of Incident CRDs and its own metrics

It is deliberately small: read cluster state and external sources, write Incident CRDs.