# Reliability

Source: https://docs.havilo.ai/reference/ai-engine/reliability

The AI Engine API is designed for async operation. Treat `POST /runs` as job creation, then use polling, SSE, or webhooks to observe completion.

## Idempotency

Use `Idempotency-Key` on every `POST /runs` request that your client may retry.

```bash
curl -X POST https://havilo.ai/api/v1/runs \
  -H "Authorization: Bearer $HAVILO_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: client-job-123" \
  -d '{"input_type":"query","query":"list 3 climate-tech startups in Europe"}'
```

Behavior:

* Same key and same body returns `200` with that run's current envelope. A new create returns `202`.
* Same key and different body returns `409 idempotency_key_conflict`.
* Use a stable key for retrying one logical create request.
* Use a new key for a new logical run.

## Polling

Poll `GET /runs/{id}` until the run reaches `succeeded`, `failed`, or `canceled`.

Recommended starting interval: 2 to 5 seconds. Increase the interval for long-running jobs or when many runs are in flight.

## SSE

For responsive progress UI, create an event stream token and connect to `/runs/{id}/events/stream`. Recreate the token when it expires.

## Webhooks

Pass `callback_url` or `config.callback_url` on `POST /runs` to receive run lifecycle callbacks. If both fields are present, they must match exactly.

Webhook consumers should still be able to poll `GET /runs/{id}` because webhooks can be delayed or retried.

## Retry Policy

Retry transient errors with exponential backoff and jitter.

| Attempt | Example delay |
| ------- | ------------- |
| 1       | 1 second      |
| 2       | 2 seconds     |
| 3       | 4 seconds     |
| 4       | 8 seconds     |

Keep the same `Idempotency-Key` when retrying `POST /runs` after a network failure or transient response.

## Limits and quotas

| Limit                         | Value                     | Error code on breach           |
| ----------------------------- | ------------------------- | ------------------------------ |
| `query` length                | ≤ 4,000 characters        | `schema_validation_failed`     |
| `plan.steps` count            | 1–50 steps                | `plan_too_large`               |
| `params` size per step        | ≤ 64 KB (JSON-serialized) | `schema_validation_failed`     |
| Request body                  | ≤ 1 MB                    | `payload_too_large` (HTTP 413) |
| Concurrent runs per workspace | Tier-dependent            | `concurrent_runs_exceeded`     |
| Daily cost per workspace      | Tier-dependent            | `workspace_cost_cap_exceeded`  |
| Event log retention           | 30 days                   | (events older are unavailable) |

Concurrent-run and cost caps depend on your plan tier. Contact your customer success manager for current values.

| Code                          | Meaning                                                                                          |
| ----------------------------- | ------------------------------------------------------------------------------------------------ |
| `concurrent_runs_exceeded`    | Too many runs are currently in flight. Retry later.                                              |
| `workspace_cost_cap_exceeded` | Workspace cost cap has been reached. Retry after the cap resets or contact your workspace owner. |
| `plan_too_large`              | Submitted `plan.steps` exceeds the 50-step maximum. Split into multiple runs.                    |

For lightweight client validation, keep concurrency low and prefer polling or SSE over frequent tight-loop requests.