Incident Traces

Every investigated Incident emits one or more IncidentTrace CRDs — a structured record of the full investigation lifecycle. Traces are the substrate for offline evaluation of model quality, tool usage, prompt tuning, and runbook effectiveness.

Why emit unconditionally

Traces are emitted on every investigated Incident, from the first day the operator runs. This guarantees historical data is available when a consumer (such as palena-eval) is installed later. Customers who bring up evaluation tooling weeks or months after the operator don't lose the investigation record from that window.

The operator does not require a consumer to be present. If nothing consumes traces, they simply accumulate as Kubernetes objects until the retention window elapses.

IncidentTrace schema

apiVersion: palena.ai/v1alpha1
kind: IncidentTrace
metadata:
  name: trace-payments-crashloop-20260420-100523-resolved
  namespace: payments
  labels:
    palena.ai/incident-ref: "payments-crashloop-20260420-100523"
    palena.ai/schema-version: "v1alpha1"
    palena.ai/emission-reason: "Resolved"
spec:
  schemaVersion: "v1alpha1"
  emissionReason: "Resolved"
  # Valid: Hypothesized | Resolved | ManualReviewRequired

  incidentRef:
    namespace: payments
    name: payments-crashloop-20260420-100523
    uid: "abc-123-def-456"
  clusterID: "<cluster-id>"
  operatorVersion: "0.1.0"

  detection:
    firstSeen: "2026-04-20T10:05:23Z"
    signals:
      - source: K8sEvent
        signalType: CrashLoopBackOff
        firstSeen: "2026-04-20T10:05:23Z"
        count: 4
      - source: PromRule
        signalType: HighErrorRate
        firstSeen: "2026-04-20T10:06:05Z"
    fingerprint: "3a7f9c2b..."
    severity: High
    signalClass: CrashLoop

  workload:
    kind: Deployment
    namespace: payments
    name: payments
    labelsHash: "sha256:..."

  investigation:
    runbookUsed: payments-crashloop-runbook
    llmEndpointUsed: qwen-investigator
    llmEndpointURL: "http://vllm-qwen.models.svc.cluster.local:8000/v1"
    model: "qwen/qwen2.5-7b-instruct"
    startedAt: "2026-04-20T10:06:30Z"
    completedAt: "2026-04-20T10:07:45Z"
    tokensIn: 2341
    tokensOut: 1080
    toolCalls:
      - name: kubernetes/logs
        paramsHash: "sha256:..."
        durationMs: 450
        outputTruncated: false
        outputTokens: 340
      - name: argocd/history
        paramsHash: "sha256:..."
        durationMs: 210
        outputTruncated: false
        outputTokens: 180
    traceRef:
      configMapName: "trace-payments-crashloop-20260420-100523-trace"

  evidence:
    bundleHash: "sha256:abc..."
    bundleRef:
      configMapName: "incident-payments-crashloop-20260420-100523-evidence"

  output:
    hypothesis: "Memory leak triggered by ConfigMap change..."
    confidence: 0.82
    recommendations:
      - type: RollbackArgoCDRevision
        paramsHash: "sha256:..."
        confidence: 0.82

  verdict:
    correctDiagnosis: true
    useful: true
    feedback: "Rollback fixed it"
    verifiedBy: "alice@example.com"
    verifiedAt: "2026-04-20T10:15:00Z"

  resolution:
    type: Rollback
    description: "Rolled back ArgoCD to revision 143"
    resolvedAt: "2026-04-20T10:12:33Z"

  correlation:
    hadK8sDeploy: true
    hadArgoCD: true
    hadGitHub: true

status:
  # Populated by a consumer (e.g. palena-eval); empty otherwise
  evaluatedAt: null
  evalScore: null
  evalFindings: []

Emission timing

A single Incident can produce up to three traces — one per emission reason. Each is a separate IncidentTrace object with append-only semantics.

• Hypothesized. Investigation has completed; verdict is still pending. A consumer can score hypothesis quality before the human reviews.
• Resolved. Emitted with the verdict and resolution attached. The highest-value trace for evaluation — the verdict is ground truth.
• ManualReviewRequired. Emitted once when an Incident transitions into ManualReviewRequired because of a blocking SensitiveDataFinding. A consumer can track how often this happens and which categories drive it.

Name pattern: trace-<incident-name>-<emission-reason-lowercase>.

The trace ConfigMap

traceRef.configMapName points at a ConfigMap holding the full investigation trace — every message, tool call, tool result, and raw LLM response as structured JSON. It lives as a ConfigMap rather than inline on the CRD because traces can be tens of kilobytes and etcd object size matters.

Schema of the ConfigMap content:

{
  "incident_ref": "payments/payments-crashloop-...",
  "runbook_used": "payments-crashloop-runbook",
  "llm_endpoint_used": "qwen-investigator",
  "started_at": "...",
  "completed_at": "...",
  "initial_context": { ... },
  "turns": [
    {
      "role": "assistant",
      "content": "...",
      "tool_calls": [{ "name": "...", "params": {...} }]
    },
    {
      "role": "tool",
      "tool_call_id": "...",
      "result": { "output": "...", "duration_ms": 450 }
    }
  ],
  "final_hypothesis": "...",
  "final_confidence": 0.82,
  "tokens_in": 2341,
  "tokens_out": 1080
}

Retention

Controlled by the Helm value evalExport.retentionDays (default 180). A cleanup controller deletes aged-out traces and their associated ConfigMaps.

When a consumer like palena-eval is installed, it typically copies traces to its own storage on ingestion — so the operator's retention can be kept generous without data loss. Choose a window long enough that a late-installed consumer can still pick up history.

Schema versioning

Traces are a stable, versioned schema — consumers code against them.

• Adding an optional field. Backward compatible. No version bump.
• Adding a required field. Breaking. schemaVersion bumps. Dual-emit across versions during a transition period.
• Removing a field. Deprecate, keep emitting, remove later.
• Semantic change. Version bump.

Consumer-side logic

The operator does not parse traces, score them, generate prompt improvements, or watch status. Traces are an output. A consumer's consumption logic is that consumer's concern.

What the operator does provide: a stable schemaVersion, consistent emission on every investigated Incident, and a retention window long enough for late-installed consumers to backfill.