LLM Endpoints

palena-sre speaks OpenAI chat completions. That's the whole contract. Whatever is behind the URL — vLLM, KServe, Ollama, LiteLLM, LocalAI, a corporate proxy, or a custom shim — is your decision, expressed through LLMEndpoint CRDs.

Principles

1. Gateway-agnostic. The operator doesn't know or assume what's behind the URL.
2. OpenAI-compatible only. If your setup isn't OpenAI-compatible, put a shim in front.
3. Per-role configuration. Multiple endpoints coexist with roles Triage, Investigation, Classifier, and NER.
4. Authoritative health via probes. Bad endpoints don't silently break investigations — the LLMEndpoint controller flips them to NotReady on persistent failure.
5. No bundled models. The operator ships zero weights.

Full spec

apiVersion: palena.ai/v1alpha1
kind: LLMEndpoint
metadata:
  name: qwen-investigator
  namespace: palena-sre-system
spec:
  # Where to send requests (required — or use kserveRef)
  url: "http://vllm-qwen.models.svc.cluster.local:8000/v1"
  # KServe convenience — operator resolves the InferenceService URL
  # kserveRef:
  #   name: qwen-7b
  #   namespace: models

  model: "qwen/qwen2.5-7b-instruct"        # required; sent in the request body
  role: Investigation                      # Triage | Investigation | Classifier | NER

  auth:
    type: None                             # None | Bearer | Header
    bearerTokenSecretRef:
      name: qwen-bearer
      key: token
    # OR for an X-API-Key-style setup:
    # headerName: "X-API-Key"
    # headerValueSecretRef: { name: proxy-key, key: value }

  tls:
    insecureSkipVerify: false
    caBundleSecretRef:
      name: internal-ca
      key: ca.crt

  capabilities:
    maxContextTokens: 32768
    supportsToolCalling: true
    streaming: true

  healthCheck:
    enabled: true
    periodSeconds: 30
    timeoutSeconds: 5
    method: models                         # models | completion

  defaults:
    temperature: 0.2
    timeoutSeconds: 60

status:
  phase: Ready                             # Ready | NotReady | Unknown
  resolvedURL: "http://vllm-qwen.models.svc.cluster.local:8000/v1"
  lastHealthCheck: "2026-04-20T10:09:30Z"
  capabilitiesObserved:
    supportsToolCalling: true

Endpoint styles

All of these are "just an LLMEndpoint" from the operator's perspective.

Direct URL (most common).

spec:
  url: "http://vllm-qwen.models.svc.cluster.local:8000/v1"
  model: "qwen/qwen2.5-7b-instruct"

KServe. The operator resolves InferenceService.status.url internally. Expected convention: the InferenceService fronts an OpenAI-compatible predictor (vLLM on KServe, kserve/huggingface-server with OpenAI mode, etc.). Non-OpenAI-compatible predictors (raw Triton, sklearn, and so on) are not supported — wrap them yourself.

spec:
  kserveRef:
    name: qwen-7b
    namespace: models
  model: "qwen/qwen2.5-7b-instruct"

LiteLLM gateway. You deploy LiteLLM; the endpoint points at it. The operator doesn't know LiteLLM is there.

spec:
  url: "http://litellm.gateway.svc.cluster.local:4000/v1"
  model: "bedrock/claude-3-5-sonnet"

Ollama. Ollama exposes an OpenAI-compatible API at /v1.

spec:
  url: "http://ollama.ollama.svc.cluster.local:11434/v1"
  model: "llama3.1:8b"

Remote cloud provider through a gateway. A gateway translates cloud APIs (OpenAI, Anthropic, Bedrock) to OpenAI-compatible. Not air-gap friendly, but supported.

Authentication

auth.type is None, Bearer, or Header.

• None. No auth header sent.
• Bearer. Authorization: Bearer <token> where the token is read from the referenced Secret on each request.
• Header. Custom header name plus a value from a Secret. Use for X-API-Key-style setups.

Secrets are read on each request through a cached lister. Rotation is picked up on the next request — no operator restart required.

TLS

• insecureSkipVerify: true for dev clusters with self-signed certs. Logged as a warning on use.
• caBundleSecretRef for corporate internal CAs.
• Default: the system cert pool.

Health checking

The LLMEndpoint controller runs health checks per healthCheck.periodSeconds. Two methods:

• models (preferred). GET <url>/models with configured auth. Expect 200 + a JSON list. Cheapest possible probe.
• completion. POST <url>/chat/completions with a minimal payload ([{"role": "user", "content": "ping"}] and max_tokens: 1). Expect a valid chat completion. Use when /models isn't implemented.

Status flips NotReady after 3 consecutive failures and back to Ready after 2 consecutive successes.

Capability discovery

At startup and every 10th health check, the controller probes:

• Tool calling. Sends a minimal request with tools. Presence of a tool_calls field in the response — or an error indicating tool calling isn't supported — tells the operator whether to include tools in future requests.
• Streaming. Sends stream: true and checks for SSE chunks.
• Context size. Not reliably discoverable; the declared capabilities.maxContextTokens is used.

Observed capabilities land in status.capabilitiesObserved. If declared and observed differ, the controller emits a CapabilityMismatch condition.

Roles

The role is a label the operator uses to resolve which endpoint to call:

• Triage. Cheap model for severity classification and tool-selection triage.
• Investigation. Main-loop model.
• Classifier. Binary "contains sensitive data?" classifier used by leak detection.
• NER. Named entity extraction used by leak detection.

Role is validated against intent but not strictly enforced — the operator warns if, say, triage.llmEndpointRef points at an endpoint with role: Investigation.

Request format

POST <endpoint>/chat/completions
Authorization: Bearer <token>        (when configured)
Content-Type: application/json

{
  "model": "<spec.model>",
  "messages": [
    {"role": "system", "content": "..."},
    {"role": "user", "content": "..."},
    {"role": "assistant", "tool_calls": [...]},
    {"role": "tool", "tool_call_id": "...", "content": "..."}
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "kubernetes/logs",
        "description": "...",
        "parameters": { ... JSON schema ... }
      }
    }
  ],
  "max_tokens": 8000,
  "temperature": 0.2,
  "stream": false
}

Response: standard OpenAI chat completion. Tool calls: standard OpenAI tool_calls with id, type: "function", function.name, and function.arguments (JSON string).

Oversized context handling

When an evidence bundle would exceed capabilities.maxContextTokens, the investigator routes oversized chunks through a summarization call to a Classifier or Triage endpoint before they enter the main loop. There's no special API — it's just another ChatCompletion with a fixed summarization prompt.

Failure modes

• All replicas down. The controller flips status.phase to NotReady. The operator refuses to dispatch new investigations. Incidents stay in Detected with a NoLLMEndpoint condition. When the endpoint recovers, the operator re-dispatches.
• Slow endpoint. Per-request timeout. Timeouts retry with backoff; a persistent timeout saves the partial trace and transitions the Incident to Hypothesized with a BudgetExceeded condition.
• Bad output. The agent sees malformed tool calls and retries once with a corrective prompt. Persistent bad output degrades to an "inconclusive" hypothesis.
• Rate limited (429). Exponential backoff. Surfaced as a metric; if persistent, the operator slows its dispatch rate.
• Auth failure (401/403). Fatal for the endpoint. NotReady with a specific condition. Human intervention is required.

What's out of scope

• Model training and fine-tuning — your problem.
• Model serving — your problem.
• Prompt management via CRDs — prompts live in Runbook CRDs and in code; there is no separate PromptVersion CRD.
• Cost tracking per model — your gateway's billing handles this. The operator reports tokens used for its own budget enforcement.