# Webhooks

Source: https://docs.havilo.ai/reference/ai-engine/webhooks

The AI Engine API can deliver run lifecycle events to a `callback_url` you provide on `POST /runs`. Every webhook is signed with HMAC-SHA256 so you can verify it is genuine. This page tells you everything you need to receive webhooks safely.

## Overview

Webhooks are fire-and-forget from your perspective: include `callback_url` on a run, and Havilo POSTs a signed JSON payload to that URL when each eligible event fires. Step-level events (`step.started`, `step.completed`, etc.) are **not** delivered by webhook — fetch them via polling or SSE.

### Eligible event types

| Event type                    | Delivered        |
| ----------------------------- | ---------------- |
| `run.succeeded`               | Yes              |
| `run.failed`                  | Yes              |
| `run.created`, `run.paused`   | No (poll or SSE) |
| `run.canceled`, `run.started` | No (poll or SSE) |
| `step.*`, `artifact.*`        | No (poll or SSE) |

Only the two terminal outcomes are delivered. A run that is created, paused or canceled emits the
event on the run's own event log — read it with `GET /runs/{id}/events` or the SSE stream — but
nothing is POSTed to your callback for it.

## Receiving a webhook

Webhooks arrive as `POST` requests to your `callback_url` with the following headers:

```http
POST /your/webhook/path HTTP/1.1
Content-Type: application/json
X-Havilo-Signature: t=1714500000,v1=811900d36a6227fa2d87b81913f19cf42f25100c569b9ad405a4003b1d82a0fb
X-Havilo-Event-Id: evt_01hxbexample
X-Havilo-Api-Version: 2026-04-22
User-Agent: havilo-webhook/1
```

Body shape:

```json
{
  "event_id": "evt_01hxbexample",
  "event_type": "run.succeeded",
  "occurred_at": 1714500000,
  "api_version": "2026-04-22",
  "run_id": "550e8400-e29b-41d4-a716-446655440000",
  "workspace_id": "8b3d0c7a-4ef5-4f11-9e8a-8d3b08c29c10",
  "data": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "object": "run",
    "status": "succeeded",
    "created_at": 1714499820000,
    "completed_at": 1714500000000
  }
}
```

`occurred_at` is Unix epoch **seconds** (matches the timestamp in the signature header). `created_at` and `completed_at` inside `data` are Unix epoch **milliseconds** (consistent with the run envelope).

The `data` field is intentionally minimal. To get full run details, including artifacts, steps, and
the canonical model-call usage projection, call `GET /runs/{id}` after receiving the webhook.

## Your signing secret

Your webhook signing secret is the **SHA-256 hex digest of the API key that created the run**. There
is no separate secret to fetch or rotate: if you hold the key, you can derive it.

```ts
import { createHash } from 'crypto'

const secret = createHash('sha256').update(process.env.HAVILO_API_KEY).digest('hex')
```

```python
import hashlib, os

secret = hashlib.sha256(os.environ["HAVILO_API_KEY"].encode()).hexdigest()
```

Two consequences worth planning for:

* **Rotating your API key changes the signing secret.** When you rotate, update your webhook
  verifier in the same change as the caller. Until you do, deliveries keep arriving and fail
  verification — a loud mismatch, not a silent drop.
* **The run is signed by the key that created it.** `POST /runs/{id}/resume` continues the same run
  and keeps the original key's secret. `POST /runs/{id}/rerun-from` starts a new run and uses the
  key that asked for it.

Derive the secret once at startup rather than per request, and keep it out of logs — it is a
deterministic function of your credential.

## Verifying the signature

> **Always verify the signature before acting on a webhook**
>
> Without verification, an attacker can forge `run.succeeded` events at your endpoint.

The signature header has the form:

```
X-Havilo-Signature: t=<unix_seconds>,v1=<hex_sha256>
```

The signed payload is `${timestamp}.${raw_body}` — the unix-seconds timestamp, a literal dot, and the **raw HTTP request body** (no whitespace normalization, no JSON re-serialization).

```
HMAC-SHA256(secret, "${timestamp}.${raw_body}")
```

Use a constant-time comparison (`timingSafeEqual` in Node, `hmac.compare_digest` in Python) to prevent timing oracle attacks.

### Test vector

To confirm your verification code is correct, run it against this fixed vector:

| Input         | Value                                                              |
| ------------- | ------------------------------------------------------------------ |
| `secret`      | `whsec_test` (an arbitrary literal, not a Havilo-issued secret)    |
| `timestamp`   | `1714500000`                                                       |
| `raw_body`    | `{"event_type":"run.succeeded"}`                                   |
| Expected `v1` | `811900d36a6227fa2d87b81913f19cf42f25100c569b9ad405a4003b1d82a0fb` |

If your implementation produces a different hex string for these inputs, the bug is in your code, not in the contract.

### Node.js verification

```ts
import { createHmac, timingSafeEqual } from 'crypto'

export function verifyWebhook(rawBody: string, header: string, secret: string): boolean {
  const match = header.match(/^t=(\d+),v1=([0-9a-f]{64})$/)
  if (!match) return false

  const timestamp = parseInt(match[1], 10)
  const received = match[2]

  const now = Math.floor(Date.now() / 1000)
  if (Math.abs(now - timestamp) > 300) return false

  const expected = createHmac('sha256', secret).update(`${timestamp}.${rawBody}`).digest('hex')

  const a = Buffer.from(expected, 'utf8')
  const b = Buffer.from(received, 'utf8')
  if (a.length !== b.length) return false

  return timingSafeEqual(a, b)
}
```

### Python verification

```python
import hmac
import hashlib
import re
import time

_HEADER = re.compile(r'^t=(\d+),v1=([0-9a-f]{64})$')

def verify_webhook(raw_body: bytes, header: str, secret: str) -> bool:
    m = _HEADER.match(header)
    if not m:
        return False

    timestamp = int(m.group(1))
    received = m.group(2)

    if abs(int(time.time()) - timestamp) > 300:
        return False

    expected = hmac.new(
        secret.encode("utf-8"),
        f"{timestamp}.".encode("utf-8") + raw_body,
        hashlib.sha256,
    ).hexdigest()

    return hmac.compare_digest(expected, received)
```

In your framework of choice, capture the raw request body **before** any JSON parsing — `JSON.parse` followed by `JSON.stringify` is not idempotent across implementations, and the resulting body will fail signature verification.

## Replay protection

Reject webhooks whose `t=` timestamp differs from the current time by more than 5 minutes. The reference implementations above default to a 300-second tolerance.

This guards against:

* **Replay attacks** — an attacker who captures a valid webhook cannot replay it 6 minutes later.
* **Out-of-sync clocks** — if your server clock drifts, a strict 0-second tolerance would spuriously reject legitimate webhooks.

If your clock is more than \~30s out of sync with NTP, the 5-minute window will absorb the drift but you should still fix the clock.

## Retry behavior

Each webhook is delivered up to 3 times.

| Attempt | Delay before this attempt |
| ------- | ------------------------- |
| 1       | (immediate)               |
| 2       | 30 seconds                |
| 3       | 2 minutes                 |

Havilo retries on:

* Connection errors (DNS failure, refused, timeout)
* HTTP `408 Request Timeout`
* HTTP `429 Too Many Requests`
* HTTP `5xx` Server Error

Havilo does **not** retry on:

* HTTP `2xx` (success — no retry needed)
* HTTP `3xx` redirects (treated as a misconfiguration)
* HTTP `4xx` other than 408 / 429 (your code is rejecting the request; retrying won't help)

After 3 failed attempts, delivery is recorded as failed for support investigation. Store the `event_id` in your logs so support can correlate the failed delivery.

## Timeouts

Each delivery attempt has a hard 10-second timeout. If your handler does not return a 2xx response within 10 seconds, the request is aborted and counted as a failure.

**Best practice:** acknowledge the webhook with a 2xx response immediately, then process it asynchronously (e.g., enqueue to a background job). If processing fails, do not return a 5xx — that triggers a retry that you don't need. Instead, return 2xx and handle the failure in your background queue.

## Handling duplicates and ordering

Webhooks may arrive **more than once**. Havilo retries on transient failures, and a slow consumer that takes 11 seconds to respond will see the same event twice (once retried, once your delayed response).

Use `event_id` as an idempotency key in your consumer:

```ts
async function handleWebhook(event: WebhookEvent) {
  const seen = await db.webhookSeen.findUnique({ where: { event_id: event.event_id } })
  if (seen) return // already processed

  await processEvent(event)
  await db.webhookSeen.create({ data: { event_id: event.event_id } })
}
```

Webhooks may also arrive **out of order**. If you depend on ordering (e.g., `run.succeeded` must be processed after `run.created`), use `occurred_at` rather than the order in which webhooks arrive at your endpoint.

## Consumer checklist

Before shipping a webhook integration, confirm each item:

* Verify `X-Havilo-Signature` on every request — never skip.
* Reject if `|now - t| > 300 seconds` (replay protection).
* Use `timingSafeEqual` (or `hmac.compare_digest`) — never `==`.
* Capture the raw request body before parsing.
* Return 2xx within 10 seconds; move processing to a background queue if needed.
* Deduplicate using `event_id`.
* Order events by `occurred_at`, not arrival time.
* Log `event_id` and `request_id` so support can correlate with server-side logs.
* Have a plan for the failed-delivery case; contact support with the `event_id` if all retries fail.

## Related

* [Events](https://docs.havilo.ai/reference/ai-engine/events.md) — full event catalog and the polling/SSE alternatives to webhooks.
* [Runs](https://docs.havilo.ai/reference/ai-engine/runs.md) — `callback_url` field on `POST /runs`.
* [Reliability](https://docs.havilo.ai/reference/ai-engine/reliability.md) — overall idempotency, retry, and limits guidance.