VI EN
System Design

Rate limiting with token buckets: lessons from Stripe for APIs and AI apps

Binh 3 min read Đọc bài này bằng tiếng Việt

TL;DR

A token bucket allows short bursts while keeping the average rate within limits. Stripe runs four layers of limiters; for AI apps, limit by tokens and concurrent agents, and always dark-launch before blocking for real.

One customer's buggy script sends thousands of requests per second and the system slows down for everyone. Or, with AI APIs: an agent loop that never stops burns the entire token budget overnight. Rate limiting is the first line of defence. Stripe's classic post "Scaling your API with rate limiters" shares the four kinds of limiter it runs in production, and the token bucket algorithm behind them.

How does a token bucket work?

  • Each user has a "bucket" holding at most N tokens.
  • Tokens are added at a fixed rate (say 10 per second) until the bucket is full.
  • Each request takes one token; an empty bucket means the request is rejected (HTTP 429).

The advantage: it allows short bursts (using the whole bucket at once) while keeping the average rate within limits. Stripe implements it with Redis so every server shares state.

# Token bucket pseudocode
tokens = min(capacity, tokens + (now - last) * rate)
if tokens >= 1:
    tokens -= 1; allow()
else:
    reject(429, retry_after=(1 - tokens) / rate)

Stripe's four limiters

TypeLimitsPurpose
Request rate limiterRequests per second per userThe most important; stops one user affecting everyone
Concurrent requests limiterRequests in flight at onceProtects slow, resource-heavy endpoints
Fleet usage load shedderReserves capacity for critical requestsUnder overload, drops less important requests first
Worker utilization load shedderBased on how busy workers areThe last line of defence, shedding load by priority

Applying it to AI applications

  • Limit by tokens, not just requests: one LLM request can cost a hundred times more than another. Size buckets in input/output tokens.
  • Limit agent concurrency: each user runs only a few agents at a time.
  • A layer above your provider: the Claude API has its own rate limits; your limiter should stop traffic before hitting them, and read the retry-after header on a 429.
  • Prioritise traffic: real user requests matter more than background jobs; separate the queues.

Risks and caveats

  • Turning it on abruptly breaks customers: Stripe recommends a "dark launch" (log only, don't block) to see who would be affected.
  • Vague errors: return 429 with how long to wait before retrying; document the limits.
  • Redis as a single point of failure: decide up front whether to let traffic through or block everything if Redis dies (fail open/closed).
  • Synchronised retries: clients should retry with exponential backoff and jitter, or you'll create a retry storm.

How to apply it

  • Dev: Start with a per-user request rate limiter; use an existing library rather than writing your own.
  • Tech Lead: Add token-based and concurrency limiters to every endpoint that calls an LLM; monitor the 429 rate.
  • PM: Define quotas per plan; they protect the system and double as a pricing tool.

Sources & references

  1. Scaling your API with rate limiters — Stripe Blog
More on System Design