VI EN
System Design

Idempotency keys: safe retries for payments, APIs and agent tools

Binh 3 min read Đọc bài này bằng tiếng Việt

TL;DR

Idempotency keys turn 'resending might duplicate' into 'resending is always safe': one key per operation, and the server stores and replays the original result. Pair them with backoff and jitter, and use them on every side-effecting tool an agent calls.

A client sends "create a payment of $50", the network flakes, and no response arrives. Resend and you risk charging twice; don't resend and you don't know whether the payment went through. It's a classic distributed-systems problem, and the standard solution is the idempotency key. Stripe popularised the approach, and the IETF is standardising it as the Idempotency-Key header. For AI agents that call tools and retry, the concept matters even more.

What is idempotency?

An operation is idempotent if performing it many times has the same effect as performing it once. Per the HTTP standard, GET, PUT and DELETE are defined as idempotent; POST (create) isn't, so it needs an extra mechanism.

How idempotency keys work

  1. The client generates a unique key (usually a UUID v4) for each logical operation.
  2. It sends the header: Idempotency-Key: 5f1c….
  3. The server stores the key together with the result of the first processing.
  4. A repeated request with the same key → the server returns the stored result without processing again.
POST /v1/payments
Idempotency-Key: 5f1c2a7e-9d3b-4c61-a8e2-0b7f3c9d1e44
Content-Type: application/json

{"amount": 5000, "currency": "usd"}

Cases the server must handle

CaseSensible response
New keyProcess normally, store the result
Known key, already processedReturn the stored result
Key currently being processed (concurrent request)Report a conflict (the IETF draft suggests 409); the client retries later
Same key, different request bodyReject (the IETF draft suggests 422), since it's a client error
Key required but missing400 Bad Request

Keys should expire (Stripe keeps them for about 24 hours) so storage doesn't grow forever.

Pairing with retries

Idempotency keys are what make retries safe. Stripe's post stresses the two pieces that go with them: retry with exponential backoff and jitter (random delay) so thousands of clients don't retry at the same moment and overwhelm the server.

Why it matters for AI agents

  • Agents often re-call tools after a timeout or when they "think" the last attempt failed.
  • Tools with side effects (sending email, creating orders, moving money) must accept idempotency keys; agent frameworks should generate a key per decision to act, not per retry.

Risks and caveats

  • Generating a new key on every retry: the most common bug, which defeats the whole purpose. The key belongs to the operation, not to the send.
  • Non-atomic result storage: write the key and result in the same transaction as the main operation.
  • Storing transient errors: decide whether to store 500s or allow retries; Stripe doesn't store results when the request wasn't actually processed.

How to apply it

  • Dev: Every POST endpoint with important side effects accepts an Idempotency-Key; clients generate one key per operation.
  • Tech Lead: Make idempotency part of your API design standard and of every agent tool that takes real actions.
  • PM: For payments and orders, this is a requirement, not an optimisation.

Sources & references

  1. Designing robust and predictable APIs with idempotency — Stripe Blog
  2. The Idempotency-Key HTTP Header Field — IETF HTTPAPI WG
More on System Design