Introduction

palena-sre is a Kubernetes operator that supervises workloads you tell it to watch, detects problems using a combination of classical signals, investigates those problems with a customer-configured LLM, and produces a durable record of what happened.

It runs entirely inside your cluster. It ships no models. It calls no services you haven't configured. It's designed for platform and SRE teams that can't hand production telemetry to a SaaS AIOps vendor.

The shape of the problem

Most Kubernetes incidents follow a familiar loop: a pod enters CrashLoopBackOff, someone pages the on-call, someone digs through logs, someone finds the recent deploy that caused it, someone rolls back. The mechanical parts of that loop — correlating a signal with a deploy, pulling logs and events, finding the offending PR — are repetitive and rarely need human judgment.

palena-sre automates the mechanical parts:

1. Detect the incident from Kubernetes events, PromQL rules, metrics-server, anomaly detection, log pattern drift, or AlertManager webhooks.
2. Deduplicate signals into a single Incident CRD per root cause.
3. Enrich the incident with recent Kubernetes rollouts, ArgoCD syncs, and GitHub PRs.
4. Investigate using an LLM that runs a ReAct loop over a toolset — kubectl, PromQL, Loki, ArgoCD, GitHub — against a runbook you write.
5. Record the evidence bundle, the full investigation trace, the hypothesis, and any recommendations on the Incident.
6. Notify on-call via Slack, Teams, PagerDuty, webhook, or email.
7. Capture the verdict a human provides, and emit an IncidentTrace CRD suitable for offline evaluation.

At no point is the LLM on a hot path. Detection fires the incident instantly; investigation runs asynchronously. If the LLM endpoint is down, detection and notification still work — the incident simply waits with a clear condition on it.

Design principles

Bring your own LLM. The operator speaks OpenAI chat completions to whatever URL you configure. Hosting is your problem — use vLLM, KServe, Ollama, LiteLLM, or a corporate proxy. The operator is gateway-agnostic.

Classical detection first, LLM last. Incidents are detected by deterministic rules. LLMs investigate. They never decide whether something is broken.

Read-only by default. The operator's RBAC is get/list/watch. Remediation requires a Recommendation CRD, human approval, and an executor that's opt-in and post-v1.

No required Palena siblings. palena-sre is standalone. It emits IncidentTrace CRDs that the palena-eval evaluator consumes if installed, but nothing breaks if it isn't.

GitOps-first configuration. Everything reconfigurable is a CRD field, not a Helm value. You manage the operator the same way you manage the rest of your cluster.

Air-gap friendly. Zero required outbound calls. The operator works on clusters with deny-all-egress network policies in place.

Scope

palena-sre is a workload-level incident operator. It's opinionated about what it does and what it doesn't.

What it does

• Detect incidents on workloads selected by SREWatch CRDs
• Investigate incidents using a customer-configured LLM endpoint
• Correlate with Kubernetes rollouts, ArgoCD syncs, GitHub PRs
• Notify on-call through standard channels
• Optionally detect PII/PHI in watched workloads' logs
• Emit auditable IncidentTrace records for every investigated incident

What it explicitly is not

• Not a model hosting platform. You host the model; the operator calls its URL.
• Not a general AIOps platform. Scope is workloads you tell it to watch.
• Not a remediation engine in v1. Recommendations are emitted; execution requires human approval and an opt-in executor.
• Not a SaaS product. No control plane external to your cluster.
• Not dependent on other Palena products. palena-eval, palena-shield, and anything else are optional.

Who it's for

Platform and SRE teams running their own Kubernetes in regulated verticals — fintech, healthtech, govtech — where data residency, air-gap requirements, BAA scope, or HIPAA/PCI/GDPR audits make SaaS AIOps a non-starter.

Also: tech-forward platform teams that already run their own LLM infrastructure and want a coherent SRE story on top of it.

Where to go next

• Quick Start — install into kind, trigger a crashloop, watch it get investigated.
• Installation — production, regulated, and air-gapped installs.
• Architecture — how the components fit together.
• CRDs — the full set of resources you create.