CRDs

Every CRD lives in API group palena.ai, version v1alpha1, and has the status subresource enabled.

CRD	Scope	Typical owner	Typical count
SREWatch	Namespaced	App team	1–10 per namespace
DetectionPolicy	Namespaced	Platform team	1–3 per namespace
InvestigationPolicy	Namespaced	Platform team	1–3 per namespace
LLMEndpoint	Namespaced	Platform team	2–5 per cluster
Incident	Namespaced	Detector (auto)	Grows over time
Recommendation	Namespaced	Investigator (auto)	0–5 per Incident
SensitiveDataFinding	Namespaced	Leak detector (auto)	0–100 per cluster
Runbook	Namespaced	Platform team	5–50 per cluster
NotificationChannel	Namespaced	App team	1–20 per namespace

A dedicated page covers LLM endpoints in more depth, and incident traces covers the IncidentTrace schema emitted for palena-eval.

SREWatch

Declares a workload selector and the policies that apply. The primary user-facing CRD.

apiVersion: palena.ai/v1alpha1
kind: SREWatch
metadata:
  name: payments-service
  namespace: payments
spec:
  selector:
    matchLabels:
      app: payments
  detectionPolicyRef:
    name: default-strict
  investigationPolicyRef:
    name: default-strict
  notificationChannels:
    - name: payments-slack
    - name: payments-pagerduty
  disabledSignals:
    - LogPatternDrift
  runbookOverrides:
    CrashLoopBackOff:
      name: payments-crashloop-runbook
status:
  observedGeneration: 1
  phase: Active
  watchedPods: 7
  openIncidents: 1
  lastReconciled: "2026-04-20T10:00:00Z"

Validation.

• The selector must match at least one resource at reconcile time (soft — warning, not error).
• detectionPolicyRef and investigationPolicyRef are optional; cluster defaults apply when omitted.
• Entries in disabledSignals must be known signal types.

DetectionPolicy

Which detectors run, their thresholds, and their windows. Every detector block is independently enable/disable-able.

apiVersion: palena.ai/v1alpha1
kind: DetectionPolicy
metadata:
  name: default-strict
  namespace: payments
spec:
  k8sEvents:
    enabled: true
    signals: [CrashLoopBackOff, OOMKilled, Evicted, FailedScheduling, ImagePullBackOff, UnhealthyProbe]
    minSeverity: Medium

  promRules:
    enabled: true
    prometheusRef:
      name: prometheus-default
    rules:
      - name: high-error-rate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (pod)
          /
          sum(rate(http_requests_total[5m])) by (pod) > 0.05
        for: 5m
        severity: High
        signalType: HighErrorRate

  metricsServer:
    enabled: true
    anomaly:
      enabled: false
      detectors:
        - target: PodMemory
          method: MAD
          window: 15m
          threshold: 4.0
          severity: Medium
          signalType: MemoryAnomaly

  anomaly:
    enabled: true
    source:
      prometheusRef:
        name: prometheus-default
    detectors:
      - metric: container_cpu_usage_seconds_total
        method: MAD
        window: 1h
        threshold: 4.0
        severity: Medium
        signalType: CpuAnomaly

  logPatterns:
    enabled: false
    source:
      lokiRef:
        name: loki-default
    streams:
      - selector: '{app="payments"}'
        method: Drain3
        windowSize: 1h
        newTemplateThreshold: 10
        signalType: LogPatternDrift

  alertReceiver:
    enabled: true
    workloadLabelKey: "pod"
    namespaceLabelKey: "namespace"

Validation.

• At least one detector block must be enabled.
• PromQL expr is parsed at admission via the Prometheus parser — invalid expressions fail the apply.
• anomaly.method is MAD or EWMA in v1. STL is deferred.
• metricsServer.anomaly.detectors[].target is one of PodCPU, PodMemory, NodeCPU, NodeMemory.
• window must be between 1m and 7d.

InvestigationPolicy

Controls when and how investigations run, which LLM endpoints are used, and the budget ceilings.

apiVersion: palena.ai/v1alpha1
kind: InvestigationPolicy
metadata:
  name: default-strict
  namespace: payments
spec:
  triage:
    enabled: true
    llmEndpointRef:
      name: gemma-triage
    maxTokens: 1000

  investigation:
    llmEndpointRef:
      name: qwen-investigator
    maxTokens: 8000
    maxToolCalls: 15

  toolsets:
    enabled:
      - kubernetes/core
      - kubernetes/logs
      - kubernetes/resource-usage
      - prometheus/query
      - loki/query
      - argocd/status
      - github/recent-prs
    disabled:
      - kubernetes/exec
      - bash

  budget:
    maxInvestigationsPerHour: 20
    maxInvestigationsPerIncident: 3
    maxTotalTokensPerHour: 500000
    minSeverityForInvestigation: Medium

  onSensitiveDataActive: ManualReviewOnly   # ManualReviewOnly | Proceed

  retention:
    closedIncidentRetentionDays: 90
    postMortemRetentionDays: 180

Validation.

• triage.enabled: true requires triage.llmEndpointRef.
• investigation.llmEndpointRef is required (no endpoint → no investigation).
• minSeverityForInvestigation is one of Low, Medium, High, Critical.
• onSensitiveDataActive is ManualReviewOnly or Proceed.

LLMEndpoint

A named, gateway-agnostic OpenAI-compatible endpoint. See LLM endpoints for the full spec, capability discovery, and health checking.

Incident

The durable artifact. Created by the detector, updated by the investigator, resolved by a human.

apiVersion: palena.ai/v1alpha1
kind: Incident
metadata:
  name: payments-crashloop-20260420-100523
  namespace: payments
  labels:
    palena.ai/workload: payments
    palena.ai/signal-class: CrashLoop
    palena.ai/severity: High
    palena.ai/fingerprint: "3a7f9c2b..."
    palena.ai/srewatch: payments-service
    palena.ai/phase: Investigating
spec:
  srewatch: { name: payments-service }
  workload:
    kind: Deployment
    name: payments
    namespace: payments
    uid: abc-123
  fingerprint: "3a7f9c2b..."
  firstSeen: "2026-04-20T10:05:23Z"
  clusterID: "<cluster-id>"
status:
  phase: Investigating
  severity: High
  signals:
    - source: K8sEvent
      signalType: CrashLoopBackOff
      firstSeen: "2026-04-20T10:05:23Z"
      lastSeen: "2026-04-20T10:07:14Z"
      count: 4
  resourceSnapshot:
    capturedAt: "2026-04-20T10:05:25Z"
    pods:
      - name: payments-abc-xyz
        cpuMillicores: 980
        memoryBytes: 2041053184
        cpuLimitMillicores: 1000
        memoryLimitBytes: 2147483648
  evidence:
    bundleHash: "sha256:abc..."
    bundleRef:
      configMapName: incident-payments-crashloop-20260420-100523-evidence
  investigation:
    startedAt: "2026-04-20T10:06:30Z"
    completedAt: "2026-04-20T10:07:45Z"
    llmEndpointUsed: qwen-investigator
    runbookUsed: payments-crashloop-runbook
    tokensIn: 2341
    tokensOut: 1080
    hypothesis: "Memory leak triggered by ConfigMap change at 10:04..."
    confidence: 0.82
  recommendations:
    - name: rollback-recent-deploy
      type: RollbackArgoCDRevision
      confidence: 0.82
      approvalRequired: true
  correlation:
    k8s:
      recentRollout: { revision: "144", deployedAt: "2026-04-20T10:04:12Z", age: "3m" }
    argocd: { application: "payments", syncRevision: "abc123def456" }
    github:
      recentPRs:
        - number: 4821
          title: "Update memory limits for workers"
          author: "alice"
          mergedAt: "2026-04-20T10:03:50Z"
  verdict:
    correctDiagnosis: true
    useful: true
    verifiedBy: alice@example.com
    verifiedAt: "2026-04-20T10:15:00Z"
  resolution:
    type: Rollback
    resolvedAt: "2026-04-20T10:12:33Z"

Phases. Detected, Suppressed, ManualReviewRequired, Investigating, Hypothesized, Resolved, PostMortemed. See Incident lifecycle for the transition rules.

Validation. spec.fingerprint is immutable after creation. Phase transitions are enforced by a validating webhook. Severity is monotonically non-decreasing. A verdict is required before Hypothesized → Resolved.

Recommendation

A proposed action, gated behind approval. Not executed in v1 — execution is a post-v1 concern.

apiVersion: palena.ai/v1alpha1
kind: Recommendation
metadata:
  name: payments-crashloop-20260420-100523-rec-1
  namespace: payments
spec:
  incidentRef:
    name: payments-crashloop-20260420-100523
  type: RollbackArgoCDRevision
  # Valid: RollbackArgoCDRevision | RollbackDeployment | RestartPods |
  #        ScaleUp | ScaleDown | UpdateResourceLimits | Custom
  params:
    argoCDApp: payments
    targetRevision: "143"
  approval:
    required: true
status:
  phase: PendingApproval
  # Valid: PendingApproval | Approved | Rejected | Executed | Failed

SensitiveDataFinding

Created by the leak detector when enabled. Consumed by the Incident controller. See Leak detection.

apiVersion: palena.ai/v1alpha1
kind: SensitiveDataFinding
metadata:
  name: payments-sdf-20260420-095500
  namespace: payments
spec:
  workload:
    kind: Deployment
    name: payments
    namespace: payments
  severity: High
  categories: [Email, CreditCard]
  firstDetected: "2026-04-20T09:55:00Z"
  source: LogStream
  sourceRef: 'loki://{app="payments"}'
  detectorEndpointRef: { name: gliner-pii-detector }
  excerpt: "User [EMAIL_REDACTED] paid with card [CC_REDACTED] successfully"
  entityPositions:
    - { category: Email, start: 5, end: 20 }
    - { category: CreditCard, start: 36, end: 50 }
status:
  phase: Active
  # Valid: Active | Acknowledged | FalsePositive | Remediated
  occurrenceCount: 42
  lastSeen: "2026-04-20T10:10:00Z"

Validation. The excerpt must not contain raw sensitive values — enforced by emit logic in the detector and a CI invariant test. Severity is immutable after creation.

Runbook

An investigation playbook for specific signal types and workloads. See Investigation.

apiVersion: palena.ai/v1alpha1
kind: Runbook
metadata:
  name: payments-crashloop-runbook
  namespace: payments
spec:
  appliesTo:
    signalTypes: [CrashLoopBackOff, OOMKilled]
    workloadSelector:
      matchLabels:
        app: payments
  systemPrompt: |
    You are an SRE for the payments service. Focus on:
    1. Recent deploys (last 30 min)
    2. Memory and CPU patterns
    3. Database connection errors
  preferredTools:
    - argocd/status
    - kubernetes/logs
    - prometheus/query
  evidenceHints:
    - "Include last 5 ArgoCD revisions"
    - "Include container_memory_working_set_bytes for 30m window"
  stopConditions:
    confidenceAbove: 0.8
    maxToolCalls: 10

NotificationChannel

Per-channel notification configuration. Exactly one of slack | teams | pagerduty | webhook | email must be set, matching type.

apiVersion: palena.ai/v1alpha1
kind: NotificationChannel
metadata:
  name: payments-slack
  namespace: payments
spec:
  type: Slack
  slack:
    webhookURLSecretRef: { name: payments-slack-secret, key: webhook-url }
    channel: "#payments-alerts"
    mention: "@oncall"
  filters:
    minSeverity: Medium
    phases: [Detected, Hypothesized, Resolved, ManualReviewRequired]
  rateLimit:
    maxPerHour: 50
  template:
    format: Markdown
    includeEvidence: false
    includeRecommendations: true

Teams.

spec:
  type: Teams
  teams:
    webhookURLSecretRef: { name: payments-teams-secret, key: webhook-url }
    webhookStyle: Workflow   # Classic | Workflow
  template:
    format: AdaptiveCard
    includeEvidence: true

PagerDuty.

spec:
  type: PagerDuty
  pagerduty:
    integrationKeySecretRef: { name: payments-pd-secret, key: integration-key }

Webhook.

spec:
  type: Webhook
  webhook:
    url: "https://ticketing.internal/api/incidents"
    auth:
      type: Bearer
      bearerTokenSecretRef: { name: ticketing-token, key: value }
    headers:
      X-Source: "palena-sre"
  template:
    format: JSON

Email.

spec:
  type: Email
  email:
    smtpConfigRef: { name: corporate-smtp }
    to: [ "sre-oncall@example.com" ]
    from: "palena-sre@example.com"
    subjectPrefix: "[PAYMENTS]"

Conventions

• Every CRD has observedGeneration in status.
• Every CRD emits standard Kubernetes conditions.
• Finalizers are used only where external cleanup requires them — palena.ai/trace-finalize on Incidents ensures IncidentTrace emission completes, palena.ai/recommendation-cleanup cleans up dependent Recommendation objects.
• Default Incident retention: 90 days after Resolved, then 90 more days after PostMortemed, then eligible for deletion.