Incident Lifecycle

The Incident CRD is the operator's central artifact. Its state machine coordinates the detector, the investigator, the notification dispatcher, and human verdict.

State machine

                 ┌──────────────┐
                 │   Detected   │ initial — created by the detector
                 └──────┬───────┘
                        │
         ┌──────────────┼──────────────┐
         │              │              │
         ▼              ▼              ▼
  ┌────────────┐ ┌───────────────┐ ┌─────────────────────┐
  │ Suppressed │ │ Investigating │ │ ManualReviewRequired│
  └──────┬─────┘ └──────┬────────┘ └──────────┬──────────┘
         │              │                     │
         │              ▼                     │
         │      ┌───────────────┐             │
         │      │ Hypothesized  │             │
         │      └──────┬────────┘             │
         │             │                      │
         └─────────────┼──────────────────────┘
                       │
                       ▼
                ┌───────────┐
                │ Resolved  │ (requires verdict)
                └─────┬─────┘
                      │
                      ▼
               ┌──────────────┐
               │ PostMortemed │ terminal
               └──────────────┘

Phase definitions

Phase	Meaning	Transitions in	Transitions out
Detected	Initial; signals accumulated	Detector (on create)	Incident controller
Suppressed	Below severity threshold or budget exhausted — notified, not investigated	Incident controller	Incident controller (re-evaluate)
ManualReviewRequired	Blocking SensitiveDataFinding active, or human override	Incident controller	Human
Investigating	Investigator working	Incident controller	Investigator
Hypothesized	Investigation complete; awaiting human verdict	Investigator	Human
Resolved	Human-confirmed resolution	Incident controller (on verdict)	Incident controller (retention)
PostMortemed	Terminal; retention elapsed	Incident controller	—

Transition rules

Enforced by a validating admission webhook:

Detected              → Suppressed | ManualReviewRequired | Investigating
Suppressed            → Detected | Investigating          (re-evaluate)
ManualReviewRequired  → Detected | Resolved               (human clears)
Investigating         → Hypothesized | Investigating | Resolved
Hypothesized          → Investigating | Resolved          (re-investigate or resolve)
Resolved              → PostMortemed
PostMortemed          → (terminal)

Severity is monotonically non-decreasing. A verdict is required before the transition from Hypothesized to Resolved.

What moves an Incident through the machine

SREWatch controller

Reconciles SREWatch resources. Ensures the detector and investigator are configured to watch the selected workloads.

• Resolves detectionPolicyRef and investigationPolicyRef; falls back to cluster defaults
• Resolves and validates every NotificationChannel
• Publishes the workload list to a detector ConfigMap
• Sets status.watchedPods and status.openIncidents

Incident controller

Reconciles every Incident transition. The Detected → next phase decision tree:

if blocking SensitiveDataFinding AND policy.onSensitiveDataActive == ManualReviewOnly:
    → ManualReviewRequired
if severity < policy.minSeverityForInvestigation:
    → Suppressed
if budget exhausted:
    → Suppressed   (re-evaluated next reconcile)
if investigation policy has no LLMEndpointRef:
    stay Detected  (notify via phase-change; no investigation)
if LLMEndpoint not Ready:
    stay Detected with condition NoLLMEndpoint
else:
    → Investigating

Pull-based dispatch. The operator doesn't call the investigator directly. It sets phase = Investigating and records dispatchedAt. The investigator watches for Investigating-phase Incidents with stale or missing heartbeat and claims them.

Verdict handling. When a human sets status.verdict on a Hypothesized-phase Incident, the controller transitions it to Resolved, triggers IncidentTrace emission, and triggers the resolution notification.

Retention. The controller compares closedIncidentRetentionDays against the age of Resolved-phase Incidents and moves them to PostMortemed as they age out. PostMortemed Incidents become deletable after postMortemRetentionDays.

SensitiveDataFinding controller

When the leak detector creates a SensitiveDataFinding:

1. The controller label-matches non-resolved Incidents for the same workload.
2. For each in Detected with onSensitiveDataActive = ManualReviewOnly, it transitions to ManualReviewRequired.
3. Findings for Incidents already past Detected are recorded but don't retroactively block.

When a finding is acknowledged or marked false-positive:

1. The controller finds Incidents in ManualReviewRequired blocked by it.
2. If no other active blocking findings remain, it transitions them back to Detected for re-evaluation.

LLMEndpoint controller

Health-checks each endpoint against its URL on a configurable period. Updates status.phase (Ready | NotReady | Unknown). See LLM endpoints for probe methods and capability discovery.

NotificationChannel controller

Validates channel configuration — secret existence, URL format, PagerDuty integration key format. It does not deliver messages; the notification dispatcher does that.

Conditions

Core conditions you can grep for on an Incident:

Condition	Meaning
Detected	Initial detection confirmed
BudgetAvailable	Budget permits investigation
NoLLMEndpoint	Configured endpoint not resolvable or not Ready
LLMEndpointUnavailable	Endpoint Ready flipped during investigation
SensitiveDataBlocking	A SensitiveDataFinding is blocking investigation
VerdictReceived	Human verdict captured
BudgetExceeded	Investigation halted due to budget
Reinvestigated	Re-investigated due to new signals

Labels

Query-friendly labels on every Incident:

Label	Value
palena.ai/workload	Workload name
palena.ai/workload-kind	Deployment, StatefulSet, etc.
palena.ai/signal-class	CrashLoop, Memory, etc.
palena.ai/severity	Low / Medium / High / Critical
palena.ai/fingerprint	Truncated hash (first 16 chars)
palena.ai/srewatch	Owning SREWatch name
palena.ai/phase	Current phase (cached; authoritative value is in status)

Annotations

Annotation	Purpose
palena.ai/dedup-full-fingerprint	Full fingerprint hash
palena.ai/first-detector	Detector identifier that created the Incident
palena.ai/reinvestigate	Set to "true" to trigger re-investigation

Idempotency and observability

Every controller action is idempotent:

• Deterministic fingerprint-derived names on Incident creation — safe to retry
• retry.RetryOnConflict on status updates
• Server-side apply with managed fields for evidence and trace ConfigMaps
• The notification dispatcher tracks last-notified phase per channel — no duplicate notifications for the same phase

The operator exports Prometheus metrics covering every aspect of the lifecycle — see the installation guide.