Quick Start

This walkthrough installs palena-sre into a local kind cluster, points it at an LLM, and triggers an end-to-end investigation of a crashlooping pod. Plan for 10–15 minutes.

Prerequisites

• kubectl ≥ 1.28
• helm ≥ 3.12
• A working kind cluster, or any Kubernetes 1.28+ cluster
• An OpenAI-compatible LLM endpoint reachable from the cluster. For local testing, an Ollama instance with a tool-capable model (qwen2.5:7b-instruct, llama3.1:8b) works well.

1. Install the CRDs and the operator

helm install palena-sre oci://ghcr.io/palenaai/charts/palena-sre \
  --namespace palena-sre-system --create-namespace

The chart installs:

• All CRDs under palena.ai/v1alpha1
• The operator, detector, and investigator Deployments
• ClusterRole + ClusterRoleBinding for get/list/watch on the workload kinds the operator supervises
• A Service exposing the AlertManager webhook receiver and the /metrics endpoint

Verify:

kubectl -n palena-sre-system get pods
kubectl get crds | grep palena.ai

2. Register your LLM endpoint

Create a Secret holding any token the endpoint requires, then create an LLMEndpoint:

apiVersion: v1
kind: Secret
metadata:
  name: llm-bearer
  namespace: palena-sre-system
type: Opaque
stringData:
  token: "sk-your-token-here"
---
apiVersion: palena.ai/v1alpha1
kind: LLMEndpoint
metadata:
  name: investigator
  namespace: palena-sre-system
spec:
  url: "http://ollama.ollama.svc.cluster.local:11434/v1"
  model: "qwen2.5:7b-instruct"
  role: Investigation
  auth:
    type: Bearer
    bearerTokenSecretRef:
      name: llm-bearer
      key: token
  capabilities:
    maxContextTokens: 32768
    supportsToolCalling: true
  healthCheck:
    enabled: true
    periodSeconds: 30
    method: models
  defaults:
    temperature: 0.2
    timeoutSeconds: 60

Apply it:

kubectl apply -f llm-endpoint.yaml
kubectl -n palena-sre-system get llmendpoint investigator -o yaml | yq .status

Wait for status.phase: Ready. If it stays NotReady, check the endpoint URL and auth secret — the operator writes a specific condition explaining the failure.

See the LLM endpoints reference for the full spec, including KServe integration, custom auth headers, and corporate CA bundles.

3. Create a demo namespace and a victim workload

kubectl create namespace demo

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: crashloop-demo
  namespace: demo
  labels:
    app: crashloop-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: crashloop-demo
  template:
    metadata:
      labels:
        app: crashloop-demo
    spec:
      containers:
        - name: app
          image: busybox:1.36
          command: ["/bin/sh", "-c", "echo starting; sleep 2; exit 1"]
EOF

The pod will enter CrashLoopBackOff within about a minute.

4. Configure the operator to watch it

Three CRDs: a DetectionPolicy, an InvestigationPolicy, and an SREWatch.

apiVersion: palena.ai/v1alpha1
kind: DetectionPolicy
metadata:
  name: demo-detection
  namespace: demo
spec:
  k8sEvents:
    enabled: true
    signals:
      - CrashLoopBackOff
      - OOMKilled
      - Evicted
      - ImagePullBackOff
    minSeverity: Medium
---
apiVersion: palena.ai/v1alpha1
kind: InvestigationPolicy
metadata:
  name: demo-investigation
  namespace: demo
spec:
  investigation:
    llmEndpointRef:
      name: investigator
      namespace: palena-sre-system
    maxTokens: 4000
    maxToolCalls: 10
  toolsets:
    enabled:
      - kubernetes/core
      - kubernetes/logs
      - kubernetes/resource-usage
  budget:
    maxInvestigationsPerHour: 20
    maxInvestigationsPerIncident: 2
    maxTotalTokensPerHour: 200000
    minSeverityForInvestigation: Medium
---
apiVersion: palena.ai/v1alpha1
kind: SREWatch
metadata:
  name: crashloop-demo
  namespace: demo
spec:
  selector:
    matchLabels:
      app: crashloop-demo
  detectionPolicyRef:
    name: demo-detection
  investigationPolicyRef:
    name: demo-investigation

Apply:

kubectl apply -f demo.yaml

5. Watch the incident life cycle

kubectl -n demo get incidents -w

Within about 30 seconds an Incident appears in phase Detected. The Incident controller moves it through Investigating → Hypothesized once the LLM has run its loop. Describe it for the full detail:

kubectl -n demo describe incident

The evidence bundle lives in a sibling ConfigMap:

kubectl -n demo get configmap -l palena.ai/kind=evidence

The full investigation trace — including every tool call and every LLM response — lives in another ConfigMap referenced from status.investigation.traceRef.

6. Close the loop

Once you've reviewed the hypothesis, record your verdict:

kubectl -n demo patch incident <incident-name> --subresource=status --type=merge -p '
status:
  verdict:
    correctDiagnosis: true
    useful: true
    feedback: "It was the failing command. As expected."
    verifiedBy: "you@example.com"
    verifiedAt: "2026-04-21T12:00:00Z"
  resolution:
    type: Other
    description: "Demo workload — verdict recorded."
    resolvedAt: "2026-04-21T12:00:00Z"
'

The Incident transitions to Resolved. An IncidentTrace CRD is emitted for palena-eval consumption. After the retention window it transitions to PostMortemed and becomes eligible for deletion.

What's next

• Installation — production and air-gapped installs, Helm values, OpenShift specifics.
• Architecture — how the three components split responsibility.
• Detection — every signal source the detector can use.
• Investigation — the ReAct loop, toolsets, and runbooks.
• CRDs — the full spec of every resource.