Investigation

The investigator picks up Incidents in the Investigating phase, runs a bounded agentic loop against your configured LLM, and writes hypothesis, confidence, and recommendations back to the Incident.

Every call has a hard ceiling — token budget, tool-call count, per-hour quota. Investigation is an asynchronous, off-hot-path operation that never blocks detection or notification.

The loop

ReAct-style with explicit phases. Each phase is bounded by policy.

Phase 0: Admission
  - Read Incident
  - Check budget (hourly investigations, hourly tokens, per-incident limits)
  - Check blocking SensitiveDataFinding → route to ManualReviewRequired
  - Resolve triage + investigation LLMEndpoints
  - Resolve Runbook for signal class + workload
  - Start heartbeat goroutine

Phase 1: Evidence gathering
  - Build evidence bundle using Runbook.evidenceHints
  - Pull pod specs, events, logs, metrics, resourceSnapshot, recent deploys
  - Hash bundle; write reference onto the Incident

Phase 2: Triage  (optional, cheap model)
  - Classify severity if not already high
  - Select relevant tools from the enabled toolset list
  - Write triage summary onto the Incident

Phase 3: ReAct loop  (main model)
  - System prompt = Runbook.systemPrompt + operator framing
  - Initial user message = evidence bundle
  - Loop:
      - Call LLM endpoint with current messages + tools
      - If response has tool_calls → execute each, append results, continue
      - If response has a final answer → break
      - If toolCalls >= maxToolCalls → break
      - If tokensUsed >= maxTokens → break

Phase 4: Structuring
  - Parse final answer into hypothesis + confidence + recommendations
  - Validate recommendations against known types
  - Write the full trace to a ConfigMap

Phase 5: Emission
  - Create Recommendation CRDs (PendingApproval)
  - Transition Incident phase → Hypothesized
  - Notification dispatcher picks up the phase change separately

The investigator does not know or care whether your LLMEndpoint.spec.url points at vLLM, KServe, Ollama, a LiteLLM gateway, or anything else — OpenAI chat completions is the only protocol it speaks.

Toolsets

Toolsets are named capabilities the agent can invoke. They're loaded from the investigator's built-in set and enabled or disabled per InvestigationPolicy.spec.toolsets.

Built-in toolsets.

Name	What it does
kubernetes/core	get/describe pods, deployments, events, configmaps in the watched namespace
kubernetes/logs	Fetch pod logs (past 30m by default, tail lines bounded)
kubernetes/resource-usage	Current pod/node CPU + memory via metrics-server
prometheus/query	Instant PromQL against configured Prometheus
prometheus/range-query	Range PromQL with start, end, and step
loki/query	LogQL query against configured Loki
tempo/search	Find traces by service, duration, and status (when Tempo is configured)
argocd/status	Current sync status and health of an ArgoCD application
argocd/history	Revision history for an ArgoCD application
github/recent-prs	Merged PRs in a repo within a time window
github/pr-details	Files changed and reviewers on a specific PR

Opt-in toolsets (off by default).

Name	Why opt-in
kubernetes/exec	Mutating — requires explicit allow, emits an Event per invocation
bash	Allowlisted runbooks only — not enabled in v1

Output sizing. Tools return truncated output (default 2048-token budget). When output is truncated, the full text lands in a ConfigMap and the agent receives an OutputRef it can dereference with a follow-up tool call. Very large outputs pass through a summarization step using the Classifier or Triage role LLM endpoint before entering agent context.

Evidence bundle

The pre-agent context, curated before the agent runs.

Default contents.

• Pod specs for up to 10 pods matching the workload selector
• Last 50 Events filtered to the workload
• Last 200 log lines per pod (configurable)
• Key metrics for the last 30m: CPU, memory, request rate, error rate
• resourceSnapshot from Incident status (metrics-server, point-in-time)
• Last 5 deploys — Kubernetes rollout history always; ArgoCD history when configured
• Last 5 merged PRs touching the relevant repo — when GitHub is configured

Per-class augmentations.

Signal class	Added to bundle
Memory	container_memory_working_set_bytes at 10s resolution (last 30m)
CrashLoop	Container exit codes, last termination reasons across all pods
Scheduling	Node list with taints, namespace resource quotas
ErrorRate	Log samples matching ERROR level

Bundle hash. SHA256 of canonical JSON. Recorded on the Incident — makes cross-session comparison and re-investigation deterministic.

Storage. Canonical JSON in a ConfigMap named incident-<name>-evidence, referenced from the Incident.

Runbooks

A Runbook matches an Incident by signal type and workload selector. Resolution order:

1. SREWatch.spec.runbookOverrides[signalType]
2. Best-matching appliesTo.signalTypes + workloadSelector
3. Default runbook for the signal class
4. No runbook — agent runs with the generic system prompt only

A runbook provides:

• System prompt additions specific to the domain
• Preferred tool ordering as hints, not enforcement
• Evidence hints — what to include in the bundle beyond defaults
• Stop conditions — confidenceAbove and maxToolCalls overrides

Example:

apiVersion: palena.ai/v1alpha1
kind: Runbook
metadata:
  name: payments-crashloop-runbook
  namespace: payments
spec:
  appliesTo:
    signalTypes: [CrashLoopBackOff, OOMKilled]
    workloadSelector:
      matchLabels:
        app: payments
  systemPrompt: |
    You are an SRE for the payments service. Focus on:
    1. Recent deploys (last 30 min)
    2. Memory and CPU patterns
    3. Database connection errors
    Always check ArgoCD history before suggesting rollback.
  preferredTools:
    - argocd/status
    - kubernetes/logs
    - prometheus/query
  evidenceHints:
    - "Include last 5 ArgoCD revisions"
    - "Include container_memory_working_set_bytes for 30m window"
    - "Include pod events"
  stopConditions:
    confidenceAbove: 0.8
    maxToolCalls: 10

Investigation trace

Every investigation writes a structured trace to a ConfigMap referenced from Incident.status.investigation.traceRef. The trace captures every turn — system prompt, user messages, tool calls, tool results, final hypothesis — as structured JSON.

The same data drives the IncidentTrace CRD emitted for palena-eval. See Incident traces.

Budget enforcement

Two layers.

Operator hard gate. Before dispatching, the operator reads current counters and checks thresholds. If exceeded, the Incident stays in Detected with a BudgetExceeded condition, re-evaluated each reconcile.

Investigator soft gate. The investigator tracks tokens used; if it exceeds the per-incident cap mid-loop it stops and structures what's there. Partial traces are never silently discarded.

Counters tracked on InvestigationPolicy.status:

• currentHourInvestigations — resets at the top of each hour
• currentHourTokens — resets at the top of each hour
• Per-Incident tokensUsed and toolCallsCount

Parallelism and re-investigation

One investigation per Incident at a time. Re-investigation triggers:

• New signals append to an Incident after its previous investigation completed
• The annotation palena.ai/reinvestigate: "true" — a human escape hatch

Maximum re-investigations per Incident is InvestigationPolicy.spec.investigation.maxInvestigationsPerIncident (default 3).

Failure handling

LLM endpoint failure. Retry with exponential backoff. After retries exhausted, the Incident stays in Investigating with a LLMEndpointUnavailable condition. The operator re-dispatches when the endpoint recovers.

Tool failure. Individual tool failures don't fail the investigation. The tool result records the error and the agent can adapt.

Budget exceeded mid-investigation. Save the partial trace, transition to Hypothesized with a BudgetExceeded condition. Never silently discard work.

Investigator pod crash. The Incident is stuck in Investigating with a stale heartbeat. The operator marks it stale after five minutes and re-dispatches. On startup, the investigator checks Incident state and resumes if a partial trace exists.

Heartbeat. The investigator updates Incident.status.investigation.heartbeatAt every 30s during an active investigation.