Notifications

The notification dispatcher watches Incident phase changes and delivers to configured NotificationChannel targets with channel-appropriate rendering.

It's event-driven, not reconciler-driven. The dispatcher runs as a predicate-filtered controller inside the operator and delivers on every relevant phase transition, subject to per-channel filters and rate limits.

Channels

Channel	Rendering	Transport
Slack	Markdown + blocks	Incoming webhook
Microsoft Teams	Adaptive Cards (Classic or Workflow style)	Incoming webhook
PagerDuty	Events API v2 payloads	HTTPS
Generic webhook	JSON with Bearer or custom-header auth	HTTPS
Email	HTML + plaintext	SMTP

Inbound channels — Slack/Teams commands back to the operator — are a later release. Everything listed above is outbound.

Typical setup

apiVersion: palena.ai/v1alpha1
kind: NotificationChannel
metadata:
  name: payments-slack
  namespace: payments
spec:
  type: Slack
  slack:
    webhookURLSecretRef:
      name: payments-slack-secret
      key: webhook-url
    channel: "#payments-alerts"
    mention: "@oncall"
  filters:
    minSeverity: Medium
    phases: [Detected, Hypothesized, Resolved, ManualReviewRequired]
  rateLimit:
    maxPerHour: 50
  template:
    format: Markdown
    includeEvidence: false
    includeRecommendations: true

Full schemas for Teams, PagerDuty, webhook, and email variants are in the CRDs reference.

Separation of concerns

• The channel controller validates that a channel is reachable and configured correctly — secrets exist, URL formats parse, keys have the expected shape. It does not send test messages that would clutter real channels.
• The dispatcher does the work of delivery when Incidents transition.

Slack

Webhook-based. Create a Slack incoming webhook, store the URL in a Secret, and reference it from the NotificationChannel.

Thread reuse. On the first notification for an Incident, the dispatcher captures Slack's response ts and stores it as an annotation on the Incident. Subsequent phase updates post as thread replies using thread_ts.

Mentions. spec.slack.mention supports @here, @channel, @<user>, or a group ID.

Truncation. Slack block fields have a 3000-character limit. The dispatcher truncates hypothesis text with a link back to the Incident in the cluster.

Microsoft Teams

Two webhook styles are supported, selectable per channel:

• Classic — legacy connector webhooks. Straightforward card payload.
• Workflow — Teams workflow-based webhooks. The same Adaptive Card wrapped in a different envelope.

No native threading. Teams channels don't expose a thread_ts equivalent via webhook, so each phase update is a new Adaptive Card referencing the Incident name. Document this for users who are used to Slack threading.

Actions. Only Action.OpenUrl is supported from a webhook. Action.Execute requires a registered bot and is a later-release concern.

Color coding. Severity drives Adaptive Card color: Default, Accent, Good, Warning, Attention.

PagerDuty

Events API v2. The simplest integration.

Dedup key. The palena-sre Incident fingerprint (truncated) becomes the PagerDuty dedup_key, so a single incident in PagerDuty tracks the lifecycle of a single palena-sre Incident.

Severity mapping.

Incident severity	PagerDuty severity
Low	info
Medium	warning
High	error
Critical	critical

Resolution. When the Incident transitions to Resolved, the dispatcher sends a resolve event with the same dedup_key.

Generic webhook

A POST with the full Incident JSON plus a phase_change field indicating the transition. Bearer or custom-header auth.

POST <url>
Authorization: Bearer <token>
X-Source: palena-sre
Content-Type: application/json

{
  "incident": { ...Incident CRD JSON... },
  "phase_change": "Detected -> Hypothesized"
}

Useful for ServiceNow, Jira, SIEMs, or in-house alerting. You build the receiver.

Email

SMTP. Requires an SMTPConfig ConfigMap plus a Secret for credentials. Basic text + HTML rendering. One recipient list per channel. Mailing lists are handled on your side.

Filtering

Every channel can filter which phases, severities, and signal classes it cares about:

filters:
  minSeverity: Medium
  phases:
    - Hypothesized
    - Resolved
  signalClasses:
    - CrashLoop
    - Memory
  excludeSignalClasses:
    - Latency

If an Incident doesn't match, the dispatcher skips the channel silently.

Rate limits

Per channel. maxPerHour is enforced by a token bucket keyed on the channel's UID. State is in-memory with periodic checkpoints to status.

Global. A cluster-wide cap of 200 notifications per minute across all channels protects against runaway delivery during detection storms.

Default phase behavior

Transition	Notify by default?
→ Detected	Only if severity ≥ High
→ Investigating	No (internal)
→ Hypothesized	Yes — the actionable moment
→ Resolved	Yes — closes the loop
→ ManualReviewRequired	Yes — ops needs to act
→ PostMortemed	No
→ Suppressed	No

Override per channel via filters.phases.

Delivery guarantees

Best-effort with three retries and exponential backoff. After three failures, the dispatcher records an error on the channel status and emits a metric. There is no persistent queue — your SIEM or ticketing system is the system of record, and operator notifications are UX, not canonical.