Make your integrations robust

Node examples use the shared Node client (retriever.mjs). Problem: network timeouts, 429, restarts. You want zero duplicates and zero lost runs.

Idempotent creation

Generate the Idempotency-Key before the call and store it with your job. If the call fails with no response, replay with the same key: you get the same run back.

import { randomUUID } from "node:crypto";
import { call } from "./retriever.mjs";

job.idempotencyKey ??= randomUUID();
await db.jobs.save(job); // persist BEFORE the call
const run = await call("/v1/runs", { method: "POST", headers: { "Idempotency-Key": job.idempotencyKey }, body: job.payload });
job.runId = run.id;
await db.jobs.save(job);

409 idempotency_conflict means a key was reused with a different body. That is a client bug: do not retry. POST /v1/workspaces, POST /v1/workspaces/{id}/tables and POST /v1/go/* have no idempotency. Do not auto-replay them after 5xx or a timeout: check whether the resource already exists first. Two exceptions: 503 server_restarting on POST /v1/go/runs started nothing, so send it again after Retry-After; and a cancel (POST /v1/go/runs/{id}/cancel) can always be sent again, a second one does nothing more. A fast service called on POST /services/{serviceName} has none either: a replay runs it again and is billed again.

Retry only what can be retried

Status	Retry?
`429`	Yes, after `Retry-After`.
`500`, `502` (including `interrupted`, but not `invalid_output`), `503`, `504` (`timeout`), network error	Yes for `GET`, `POST /v1/runs` with `Idempotency-Key`, and `POST /v1/exclusions/{kind}` and `/remove`: they are idempotent by value, so a replay leaves the same list (they ignore `Idempotency-Key`). Not for a body over the size limit: a connection reset on such a body is a `413`, see below. `POST /v1/services/jobs/batch`: yes, with care. A line whose job is still running joins it. A `getEmail` or `getPhoneNumber` person your organization got a complete answer for in the last seven days gets that answer again and is not billed (a “not found” with a skipped database does not settle that database: the same request runs again). Any other finished line, `webResearch` included, runs again and is billed again. `503 server_restarting` on `POST /v1/go/runs`: yes, after `Retry-After`. `POST /v1/go/runs/{id}/cancel`: yes. No for other `POST`.
A batch line `failed` with `error.retryable: true` (`capacity`, `timeout`, `interrupted`, `provider_error`, `provider_unavailable`, `capability_error`, `service_error`, `service_execution_failed`, `worker_lost`)	Yes, in a new batch a few minutes later: the lookup is unresolved. Its failed attempt costs nothing, except the web agent’s miss price when a contact lookup’s web agent finished its search. A call cut by a timeout can still be billed if the provider’s answer lands just after the cut. A line that ends `worker_lost` can also be billed: its call may have run, and been paid, before the worker was lost. The line’s `credits.charged` can be above 0 from earlier attempts.
`502 invalid_output` on `POST /services/{serviceName}`	No. The service answered outside its own contract, a fault on our side: the same request fails the same way. Tell us.
A batch line `failed` with `usage_limit`	Yes, after a top-up: send the failed inputs again in a new batch.
A batch line `failed` with `error.retryable: false` (`provider_rejected`, `validation_error`, `not_found`, `invalid_output`, any other code)	No. The same input gets the same result. `validation_error`: fix the value named in `message`. `not_found`: what the input names does not exist, do not send it again (no job service returns it today; treat it as not retryable if it appears). Tell us for `invalid_output`.
`409 job_shared` on a cancel	No. Another list of your organization uses that job; it keeps running for them.
`409 conversation_busy`	Yes, after the conversation’s active run finishes.
`400`, `401`, `402`, `404`, `409 idempotency_conflict`, `413 body_too_large`, `415`	No. Fix the request or account. For `413`, send less data per request: see Request size.

The call() client applies these rules, with two exceptions. It does not handle conversation_busy: handle it per your business logic. It does not replay exclusion writes, because it never replays a POST without Idempotency-Key: rerun your exclusion sync instead.

Efficient polling

Respect Retry-After (15 s for an active run). Polling faster consumes your reads budget without speeding up the run.
Send If-None-Match with the previous ETag: 304 has no body. It still counts toward the reads budget.

import { API, KEY, sleep } from "./retriever.mjs";

async function pollWithEtag(id) {
  let etag;
  let run;
  for (;;) {
    const res = await fetch(`${API}/v1/runs/${id}`, {
      headers: { Authorization: `Bearer ${KEY}`, ...(etag ? { "If-None-Match": etag } : {}) },
    });
    if (res.status === 200) {
      etag = res.headers.get("etag");
      run = await res.json();
      if (!["queued", "running", "recovering"].includes(run.status)) return run;
    } else if (res.status !== 304 && res.status !== 429 && res.status < 500) {
      throw new Error(`poll ${id}: HTTP ${res.status}`);
    }
    await sleep(Number(res.headers.get("retry-after") || 15) * 1000);
  }
}

Concurrency

By default, a key can create 30 runs per minute. The “3 concurrent” limit applies to in-flight HTTP requests, not active runs. For a large batch, spread creations (for example one every 2 s) rather than Promise.all on 100 items. For POST /v1/go/run (synchronous), limit to 3 parallel calls per key. Active runs have their own cap: your organization executes at most 5 runs at the same time by default. The others stay queued and start on their own. Do not create a queued run again, and do not time out on queued time. See Runs per organization.

Recovery after crash

Store run.id as soon as you get the 202 response. On restart, resume polling unfinished runs instead of creating new ones. A run in recovering resumes on its own on Retriever’s side, including after a deploy. A run in failed does not resume. To continue work, create a new run on the same conversation_id with a new Idempotency-Key: the agent keeps history and tables already produced; ask it to finish what is missing (see New conversation or follow-up?). run_timeout (maximum duration exceeded, 3 hours by default) is restarted the same way.

Get started

Model Context Protocol

Concepts

Recipes

Make your integrations robust

Idempotent creation

Retry only what can be retried

Efficient polling

Concurrency

Recovery after crash

​Idempotent creation

​Retry only what can be retried

​Efficient polling

​Concurrency

​Recovery after crash

Idempotent creation

Retry only what can be retried

Efficient polling

Concurrency

Recovery after crash