Cost and Reliability

Build a Chatbot: From API Call to Deployed Bot

Chapter 7 · Cost, Rate Limits and Reliability

A bot that works in testing meets three problems the day real people use it. It costs money on every message, and one enthusiastic user (or one abusive script) can run up a bill. Its users send messages in bursts, and its provider has limits of its own. And the provider fails: it rate-limits you, it is overloaded, it times out, or it is simply down. This chapter gives the bot a meter, a set of limits, and a layered defence against failure: retries, a circuit breaker and a backup provider.

What was run, and what was not
Everything ran on Python 3.14.0 with the FakeProvider, fake clocks, a fake sleep and a local scripted server; the project now has 135 passing tests. No real provider was called, so nothing here measures a real service's latency or limits. The prices in the demonstrations are made-up examples, not any vendor's: copy real ones from your provider's pricing page, with the date. The provider facts come from Anthropic's rate-limit and error documentation, which I read; the jitter advice comes from an AWS Architecture Blog article on backoff, whose formulas are images I could not read, so the exact formula below is the standard "full jitter" definition as I implement it, which my own simulation then tests.

What Providers Tell You About Limits

Anthropic's rate-limit page (the one I read; other vendors differ) describes the shape of the problem:

  • Rate limits are measured in requests per minute, input tokens per minute and output tokens per minute for each model class, applied with a token bucket (capacity is continuously replenished rather than reset at fixed intervals). A limit of 60 requests a minute may be enforced as one a second, so short bursts can trigger a 429. A 429 carries a retry-after header.
  • Only uncached input tokens count toward the input limit for most models, and max_tokens does not count toward the output limit; the output limit counts tokens actually generated. (Cost is a separate matter: you pay for the tokens generated.)
  • Spend limits are different from rate limits. Each tier has a monthly cap (the page lists $500 for the Start tier, $1,000 for Build and $200,000 for Scale; check the current page). When the cap is reached, requests return a 429 with the error type rate_limit_error but no retry-after header, a detail field error_code: "enforced_spend_limit_reached", and they keep failing until access resumes. A spend limit that you set returns a 400 instead.
  • The official SDK, per the errors page, retries transient failures twice by default, honouring retry-after. This course builds the equivalent itself, because it works for any provider.

The spend-cap 429 is a trap for a naive retry loop: it looks like an ordinary rate limit but waiting will not help. The Claude provider from chapter 2 now recognises it and raises a new error, QuotaExhausted. I ran the three situations through the provider against a local server that replies in the documented shapes:

situation class retryable fallback_ok retry_after ordinary rate limit (429) RateLimited True True 4.0 monthly spend cap (429, no header) QuotaExhausted False True None your own spend limit (400) BadRequest False False None

Each error class now carries two flags: retryable (would trying the same provider again help?) and fallback_ok (would trying a different provider help?). A spent quota, a dead key and an overloaded server all justify a backup; a malformed request of ours does not, because it would fail there too. One gap remains, and I have left it visible: the 400 for your own spend limit is still a plain BadRequest. Recognising it would mean matching the wording of its message, which vendors change.

Counting Cost

You cannot control what you do not measure. After every successful call the bot records the tokens used, the model that answered, which provider served it, the user, and the cost. The price is data you supply, per million tokens, keyed by the model name the provider reports back:

from dataclasses import dataclass from bot.providers import Usage @dataclass(frozen=True) class Price: """US dollars (or any currency you like) per million tokens. Copy the numbers from your provider's pricing page, and put the date you did it in a comment next to them: prices change.""" input_per_mtok: float output_per_mtok: float def cost_of(usage: Usage, price: Price | None) -> float | None: """The cost of one call, or None if we do not know the price of the model that answered.""" if price is None: return None return (usage.input_tokens * price.input_per_mtok + usage.output_tokens * price.output_per_mtok) / 1_000_000

If the model's name is not in your price table, the cost is recorded as unknown, not zero, and an unpriced:<model> event is raised. That matters because vendors report versioned model names and a typo or a new version would otherwise make a bill look cheaper than it is. The storage and the report query:

# --- usage and cost ---------------------------------------------------------------------- def record_usage(self, user_id: str, session_id: str, served_by: str, model: str, input_tokens: int, output_tokens: int, cost: float | None, now: float) -> None: with self.db: self.db.execute("INSERT INTO usage (user_id, session_id, served_by, model, input_tokens, output_tokens, cost, at) " "VALUES (?, ?, ?, ?, ?, ?, ?, ?)", (user_id, session_id, served_by, model, input_tokens, output_tokens, cost, now)) def usage_since(self, since: float, user_id: str | None = None) -> tuple[int, float]: """(total tokens, total known cost) since a time, for one user or for everyone.""" sql = "SELECT COALESCE(SUM(input_tokens + output_tokens), 0), COALESCE(SUM(cost), 0) FROM usage WHERE at > ?" args: list = [since] if user_id is not None: sql += " AND user_id = ?" args.append(user_id) tokens, cost = self.db.execute(sql, args).fetchone() return int(tokens), float(cost) def usage_report(self, since: float) -> list[tuple]: """Rows of (served_by, model, calls, input tokens, output tokens, cost or None if any call was unpriced).""" return self.db.execute( "SELECT served_by, model, COUNT(*), SUM(input_tokens), SUM(output_tokens), " "CASE WHEN COUNT(cost) < COUNT(*) THEN NULL ELSE SUM(cost) END " "FROM usage WHERE at >= ? GROUP BY served_by, model ORDER BY served_by, model", (since,)).fetchall()

A worked example, by hand and by code, with example prices (see Exercise 1):

example-small 1200 in 300 out = 0.002700 example-large 3000 in 800 out = 0.028000 example-small 800 in 150 out = 0.001550 by hand total: 0.032250 served by model calls in tok out tok cost fake example-large 1 3000 800 0.028000 fake example-small 2 2000 450 0.004250 report total : 0.032250

Two honest limits of this meter. It uses the plain input and output token counts a provider reports, so it ignores cache pricing: Anthropic's page says cached input tokens are billed at a cache-read rate and (for most models) are not counted toward input limits, which this simple meter does not model. And the numbers are only as good as the usage the provider reports.

Limits: Rate, Per-User Budget, Global Budget

Three limits, checked in this order before any model call, each with a fixed polite reply and an event:

LimitProtects againstHow
Rate limit per userFloods, scripts, double-sendsA token bucket in memory: a burst of burst messages, then per_minute sustained. The reply says how many seconds to wait.
Daily tokens per userOne person using all of the budgetSum of the user's recorded tokens over a rolling 24 hours
Daily spend for the whole botA surprise billSum of recorded cost over a rolling 24 hours
import time from dataclasses import dataclass, field from bot.costs import Price class RateLimiter: """A token bucket per key: it holds up to `capacity` messages and refills at `refill_per_s`. Unlike a counter that resets every minute, a bucket lets a person send a short burst and then slows them smoothly. Buckets live in memory, so they reset when the program restarts, and an entry per key is kept forever; a long-running bot should prune idle keys. """ def __init__(self, capacity: float, refill_per_s: float, clock=time.monotonic) -> None: self.capacity = capacity self.refill_per_s = refill_per_s self.clock = clock self._buckets: dict[str, tuple[float, float]] = {} # key -> (tokens, last_seen) def allow(self, key: str) -> tuple[bool, float]: """(True, 0) and spend one token, or (False, seconds until a token is available).""" now = self.clock() tokens, last = self._buckets.get(key, (self.capacity, now)) tokens = min(self.capacity, tokens + (now - last) * self.refill_per_s) if tokens >= 1: self._buckets[key] = (tokens - 1, now) return True, 0.0 self._buckets[key] = (tokens, now) return False, (1 - tokens) / self.refill_per_s @dataclass(frozen=True) class Limits: """How much the bot is allowed to be used, and what it costs. None means no limit.""" burst: int = 5 # messages a person can send at once ... per_minute: float = 20.0 # ... then this many per minute sustained user_daily_tokens: int | None = None # tokens (in + out) one user may use in a rolling 24 hours global_daily_cost: float | None = None # money the whole bot may spend in a rolling 24 hours prices: dict[str, Price] = field(default_factory=dict) # keyed by the model name a Reply reports rate_reply: str = "You're sending messages quickly. Please wait {seconds:.0f} seconds and try again." user_budget_reply: str = "You've reached today's usage limit. Please come back tomorrow." global_budget_reply: str = "I've reached my daily budget and have to stop for today. Please try again tomorrow."

The checks and the metering inside Bot (the rest of the class is unchanged from chapter 6):

def _check_limits(self, session_id: str, user_id: str) -> str | None: """Rate limit, then the user's daily tokens, then the bot's daily spend. A returned string is the reply instead.""" limits, now = self.limits, self.clock() allowed, wait = self._rate.allow(user_id) if not allowed: self.events.append((session_id, "rate_limited")) return limits.rate_reply.format(seconds=wait) day_ago = now - 86400 if limits.user_daily_tokens is not None and self.store.usage_since(day_ago, user_id)[0] >= limits.user_daily_tokens: self.events.append((session_id, "user_budget")) return limits.user_budget_reply if limits.global_daily_cost is not None and self.store.usage_since(day_ago)[1] >= limits.global_daily_cost: self.events.append((session_id, "global_budget")) return limits.global_budget_reply return None def _meter(self, session_id: str, user_id: str, reply) -> None: """Record what the call used and (if the model's price is known) what it cost.""" cost = cost_of(reply.usage, self.limits.prices.get(reply.model)) if cost is None: self.events.append((session_id, "unpriced:" + reply.model)) self.store.record_usage(user_id, session_id, reply.served_by or self.provider.name, reply.model, reply.usage.input_tokens, reply.usage.output_tokens, cost, self.clock()) def report(self, hours: float = 24.0) -> str: """A plain-text usage and cost table for the last `hours`.""" rows = self.store.usage_report(self.clock() - hours * 3600) lines = [f"{'served by':<14}{'model':<14}{'calls':>6}{'in tok':>9}{'out tok':>9}{'cost':>10}"] for served_by, model, calls, tin, tout, cost in rows: shown = "unpriced" if cost is None else f"{cost:.6f}" lines.append(f"{served_by:<14}{model:<14}{calls:>6}{tin:>9}{tout:>9}{shown:>10}") return "\n".join(lines)

A simulated session (times are simulated seconds, prices are examples, the daily limits are deliberately tiny so every limit is reached):

t+ 0s ada msg 0 -> answer t+ 1s ada msg 1 -> answer t+ 2s ada msg 2 -> answer t+ 3s ada msg 3 -> You're sending messages quickly. Please wait 7 seconds and try again. t+ 14s ada later 0 -> You've reached today's usage limit. Please come back tomorrow. t+ 15s ada later 1 -> You're sending messages quickly. Please wait 5 seconds and try again. t+ 16s ada later 2 -> You're sending messages quickly. Please wait 4 seconds and try again. t+ 17s bob hello -> answer t+ 18s bob hello again -> I've reached my daily budget and have to stop for today. Please try ag served by model calls in tok out tok cost fake model-big 1 2200 300 0.011100 fake model-small 3 6400 900 0.010900 events: ['rate_limited', 'user_budget', 'rate_limited', 'rate_limited', 'global_budget'] total in the last 24 h: 9800 tokens, cost 0.0220 (daily budget 0.02)

Read it as a story. Ada's fourth message in three seconds hits the rate limit and is told to wait 7 seconds. Later she is under the rate limit but over her daily token budget. Her next two messages meet the rate limit again, because the bucket does not refill while she keeps trying. Bob is a different user with his own bucket and budget, so his first message works. His second hits the global daily spend limit, which Ada's usage and his own first message had used up. The report at the bottom is the whole day in one table.

Three caveats. The budget is checked before the call and recorded after, so a user can overshoot by one call (a strict limit would reserve the worst case in advance). The rolling window is "the last 24 hours", not a calendar day. And the rate-limit buckets live in memory: they reset when the program restarts, and a bot with several processes would need a shared store.

Retries: Backoff, Jitter and Retry-After

Many failures are brief: a rate limit that clears in a second, a server that hiccups. Retrying handles them, if it is done carefully. Retrying at once makes an overloaded service worse; retrying at fixed intervals makes every client that failed together come back together. The standard fix is exponential backoff with jitter: wait longer after each failure (doubling, up to a cap), and make each wait a random fraction of that so clients spread out. The version here is "full jitter": the wait before the next attempt is a random number between 0 and min(max_delay, base_delay × 2n−1).

import asyncio import dataclasses import random import time from bot.providers import Message, Provider, ProviderError, ProviderUnavailable, Reply class RetryingProvider: """Retry a provider's retryable failures, with exponential backoff and full jitter. The wait before attempt n+1 is a random number between 0 and min(max_delay, base_delay * 2**(n-1)). If the server says how long to wait (Retry-After), that is used instead, unless it is longer than max_delay, in which case we give up straight away instead of tying up the user for minutes. """ def __init__(self, inner: Provider, *, max_attempts: int = 3, base_delay: float = 0.5, max_delay: float = 8.0, sleep=asyncio.sleep, rng=random.random) -> None: self.inner = inner self.name = f"retry({inner.name})" self.max_attempts, self.base_delay, self.max_delay = max_attempts, base_delay, max_delay self._sleep, self._rng = sleep, rng self.retries = 0 # how many retries have been made (for logs and tests) async def complete(self, messages: list[Message], *, system: str | None = None, max_tokens: int = 1024) -> Reply: for attempt in range(1, self.max_attempts + 1): try: return await self.inner.complete(messages, system=system, max_tokens=max_tokens) except ProviderError as e: if not e.retryable or attempt == self.max_attempts: raise if e.retry_after is not None: if e.retry_after > self.max_delay: raise delay = e.retry_after else: delay = self._rng() * min(self.max_delay, self.base_delay * 2 ** (attempt - 1)) self.retries += 1 await self._sleep(delay) raise AssertionError("unreachable") class CircuitOpen(ProviderUnavailable): """Raised instead of calling a provider that has been failing: fail fast, spare the user the wait.""" class CircuitBreaker: """closed (calls go through) -> open after `failure_threshold` failures in a row -> half-open after `reset_after` seconds (exactly one trial call is allowed) -> closed again if the trial succeeds.""" def __init__(self, failure_threshold: int = 5, reset_after: float = 30.0, clock=time.monotonic) -> None: self.failure_threshold, self.reset_after, self.clock = failure_threshold, reset_after, clock self.failures = 0 self.opened_at: float | None = None self._trial_in_flight = False @property def state(self) -> str: if self.opened_at is None: return "closed" return "half-open" if self.clock() - self.opened_at >= self.reset_after else "open" def before_call(self) -> None: state = self.state if state == "open" or (state == "half-open" and self._trial_in_flight): raise CircuitOpen("circuit open: provider has been failing") if state == "half-open": self._trial_in_flight = True def release_trial(self) -> None: self._trial_in_flight = False def record_success(self) -> None: self.failures, self.opened_at, self._trial_in_flight = 0, None, False def record_failure(self) -> None: self.failures += 1 self._trial_in_flight = False if self.failures >= self.failure_threshold or self.opened_at is not None: self.opened_at = self.clock() # (re)open; a failed trial starts a new wait class BreakerProvider: """Wrap a provider in a circuit breaker. Only failures that could be the provider's fault count.""" def __init__(self, inner: Provider, breaker: CircuitBreaker | None = None) -> None: self.inner = inner self.name = f"breaker({inner.name})" self.breaker = breaker or CircuitBreaker() async def complete(self, messages: list[Message], *, system: str | None = None, max_tokens: int = 1024) -> Reply: self.breaker.before_call() try: reply = await self.inner.complete(messages, system=system, max_tokens=max_tokens) except ProviderError as e: if e.retryable: self.breaker.record_failure() else: # our own bad request is not evidence the provider is down self.breaker.release_trial() raise self.breaker.record_success() return reply class FallbackProvider: """Try providers in order; move to the next one when a failure says a backup could help.""" def __init__(self, providers: list[Provider]) -> None: self.providers = providers self.name = "fallback(" + ",".join(p.name for p in providers) + ")" self.fallbacks: list[tuple[str, str]] = [] # (provider that failed, error class), for logs async def complete(self, messages: list[Message], *, system: str | None = None, max_tokens: int = 1024) -> Reply: last: ProviderError | None = None for provider in self.providers: try: reply = await provider.complete(messages, system=system, max_tokens=max_tokens) except ProviderError as e: if not e.fallback_ok: raise self.fallbacks.append((provider.name, type(e).__name__)) last = e continue return dataclasses.replace(reply, served_by=reply.served_by or provider.name) assert last is not None raise last

Rules the code follows: only retryable errors are retried; a server's Retry-After is obeyed, unless it is longer than max_delay, in which case giving up at once beats keeping a person waiting for minutes; and the number of attempts is bounded. Five scenarios with a seeded random source (the waits are random but repeatable):

two failures, then success model calls=3 waits=[0.32, 0.03] -> answered 'ok' server says Retry-After: 3 model calls=2 waits=[3.0] -> answered 'ok' server says Retry-After: 120 (too long) model calls=1 waits=[] -> raised RateLimited our own bad request model calls=1 waits=[] -> raised BadRequest fails every time (3 attempts) model calls=3 waits=[0.32, 0.03] -> raised ServerError

Note the first line: the second wait is only 0.03 seconds. Full jitter can produce a very short wait; that is by design, since the point is to spread clients out, not to guarantee a minimum.

Does jitter really help? I simulated 200 clients that all fail at the same instant and counted how many retries land in any single 0.1-second slot (this is my own small simulation, not a measurement of a real service):

200 clients all fail at the same instant. When do their retries arrive? (peak = most retries in any single 0.1 s slot) attempt 1 no jitter peak = 200 retries in one 0.1 s slot, spread over 1 slots attempt 1 full jitter peak = 49 retries in one 0.1 s slot, spread over 5 slots attempt 2 no jitter peak = 200 retries in one 0.1 s slot, spread over 1 slots attempt 2 full jitter peak = 27 retries in one 0.1 s slot, spread over 10 slots attempt 3 no jitter peak = 200 retries in one 0.1 s slot, spread over 1 slots attempt 3 full jitter peak = 15 retries in one 0.1 s slot, spread over 20 slots

Without jitter all 200 retries arrive together, at every attempt. With full jitter the worst slot holds 49, then 27, then 15, because the cap doubles each time and the retries spread over a wider window. That is the thundering-herd effect that jitter removes, and it matches the AWS article's conclusion that jitter spreads retries into a roughly constant rate of calls.

Retries and timeouts: two cautions I did not build around
  • A timeout does not mean the request failed. The provider may have processed (and billed) it while the answer was lost. A retry can therefore cost twice. For a chat reply that is usually acceptable; for anything with side effects it is not, and later chapters on tools would need idempotency.
  • Total waiting time is not capped. Three attempts with a 60-second timeout each could keep a person waiting three minutes. For a chat, lower the provider timeout (for example to 20 seconds) or add an overall deadline. The code above does not have a deadline.

The Circuit Breaker: Stop Hammering What Is Down

If the provider is down, retrying every request wastes time for everyone: each user waits through the full retry cycle just to learn what the bot could already know. A circuit breaker remembers recent failures. After failure_threshold failures in a row it opens: further calls fail instantly (with CircuitOpen) for reset_after seconds. Then it goes half-open and lets exactly one trial call through: success closes it, failure opens it again for another wait. Only failures that could be the provider's fault count; our own bad requests do not.

Combined with a backup provider (next section), the timeline looks like this. The primary fails four times, so the breaker opens after two requests; for the next minute the primary is not even tried; at 40 seconds the one trial succeeds and the breaker closes (times are simulated):

time breaker before served by primary calls so far 0s closed backup 2 (backup answer) 5s closed backup 4 (backup answer) 10s open backup 4 (backup answer) 15s open backup 4 (backup answer) 20s open backup 4 (backup answer) 40s half-open breaker(retry(primary)) 5 (primary is back) 45s closed breaker(retry(primary)) 6 (primary is back) 50s closed breaker(retry(primary)) 7 (primary is back) fallback log: [('breaker(retry(primary))', 'ServerError'), ('breaker(retry(primary))', 'ServerError'), ('breaker(retry(primary))', 'CircuitOpen'), ('breaker(retry(primary))', 'CircuitOpen'), ('breaker(retry(primary))', 'CircuitOpen')]

Look at the "primary calls so far" column: it stops growing while the breaker is open (4 calls from 5 s to 20 s) and starts again at the trial. The fallback log at the bottom shows the first two failures were real server errors and the rest were fast CircuitOpen rejections.

A Backup Provider

The last layer is a second provider. FallbackProvider tries each provider in order and moves on when the error says a backup could help (fallback_ok); it records which provider failed and why, and stamps the reply with served_by so the meter and the logs know who answered. The usual stack wraps each provider in retry, then breaker, then puts the list in the fallback:

provider = FallbackProvider([ BreakerProvider(RetryingProvider(primary)), RetryingProvider(backup), ]) bot = Bot(provider, persona=persona, limits=limits)

A 12-request timeline for exactly that stack (Exercise 3 asks you to write it): the primary fails on its third to eighth calls and works otherwise. The backup answers during the trouble; the breaker opens, a trial at 40 s fails and reopens it, and the next trial at 60 s succeeds and hands traffic back to the primary:

request time breaker answered by primary calls 1 5s closed breaker(retry(primary)) 1 2 10s closed breaker(retry(primary)) 2 3 15s closed retry(backup) 4 4 20s closed retry(backup) 6 5 25s open retry(backup) 6 6 30s open retry(backup) 6 7 35s open retry(backup) 6 8 40s half-open retry(backup) 8 9 45s open retry(backup) 8 10 50s open retry(backup) 8 11 55s open retry(backup) 8 12 60s half-open breaker(retry(primary)) 9

The backup is a different model, so its answers may differ from the primary's in tone, length and following the persona. Run the scenarios from chapter 5 against the backup too, and give it a price entry: the meter keys prices by the model name each reply reports. A cheaper model that you route simple messages to is the same idea with a different trigger (message length, or a classifier) instead of failure; I have not built that.

Mistakes while building this
  • An off-by-one in the budget window. A test advanced the clock by exactly 24 hours and expected the user's old usage to have expired, but it had not: my query counted records with at >= since, so a record exactly 24 hours old was still inside the window. The fix was at > since. Boundary conditions deserve their own test; this one was found only because the test used the exact boundary.
  • A test that expected exactly one event. My rate-limit test asserted the event list was exactly [rate_limited], but the fake's replies named a model with no price, so "unpriced" events were in the list too. The code was right and the assertion too strict; the assertion now filters for the event it is about. Over-specific assertions break when unrelated features are added.
  • A demo that did not show what it claimed. My first breaker timeline never showed the breaker closing again, because the scripted failures lasted too long, and my first limits demo never reached the global budget. I adjusted the scripts until each run showed the whole life cycle. A demonstration should be checked against the claim it illustrates, not just run.

What Is Still Missing

  • Streaming. A reply that fails halfway has already shown the user some text; retry and metering need extra care there (chapter 8 touches this).
  • Cache-aware cost. See above: cached tokens are priced differently.
  • Reading the provider's rate-limit headers. Anthropic returns headers such as anthropic-ratelimit-requests-remaining and anthropic-ratelimit-tokens-reset. A smarter bot would slow itself down before getting a 429; this one reacts afterwards.
  • Real latency. Everything here ran against fakes and a local server, so I have no real timing numbers.

Hands-On Exercises

Exercise 1

With two example prices and three scripted calls of known token counts, work out the cost of each call and the total by hand (in code, with cost_of or plain arithmetic), then run the same calls through a Bot with Limits(prices=...) and show that bot.report() gives the same total.

📄 View solution
Exercise 2

Predict the waits RetryingProvider will make in four situations (use rng=lambda: 1.0 so the jitter always picks the longest allowed wait): five failures with the default settings, five failures with a base of 1.0 and a cap of 3, a Retry-After: 2 followed by a server error, and max_attempts=1. Then run them and compare.

📄 View solution
Exercise 3

Build FallbackProvider([BreakerProvider(RetryingProvider(primary)), RetryingProvider(backup)]) with a fake clock, make the primary fail on its 3rd to 8th calls, send 12 requests five seconds apart, and print for each: the breaker state, who answered and how many times the primary has been called. Explain every row where the primary was not tried.

📄 View solution

Chapter 7 Quick Reference

  • Provider facts (Anthropic): limits in requests, input tokens and output tokens per minute, token bucket, bursts can 429; 429 has retry-after; the monthly spend-cap 429 has none and keeps failing (enforced_spend_limit_reached); your own spend limit gives a 400
  • Errors have retryable (retry the same provider) and fallback_ok (try another); QuotaExhausted: not retryable, fallback ok
  • Cost: Price(input_per_mtok, output_per_mtok) keyed by the model name in the reply; unknown model means cost unknown plus an unpriced: event; usage table; bot.report()
  • Limits (checked before the call, recorded after): token-bucket rate limit per user, rolling-24-hour tokens per user, rolling-24-hour cost for the bot; a user can overshoot by one call
  • Retry: only retryable errors; wait = random(0, min(max_delay, base × 2^(n−1))) (full jitter); obey Retry-After unless it exceeds max_delay; bounded attempts; no overall deadline
  • Breaker: closed, then open after N failures in a row, half-open after reset_after with one trial; only the provider's faults count; CircuitOpen is a retryable unavailable error
  • Stack: FallbackProvider([BreakerProvider(RetryingProvider(primary)), RetryingProvider(backup)]); reply carries served_by; test the persona on the backup too
  • Boundary conditions (> versus >=) need tests; a timeout does not mean the request did not happen
Coming next
Build a Chatbot: From API Call to Deployed Bot 8 gives the bot its first public face: a web chat page. A FastAPI endpoint with session cookies, streaming replies to the browser with Server-Sent Events or a WebSocket (the streaming API itself is Advanced AI Workflows & the Claude API 4; the transport is covered in WebSockets & Real-Time Communication), a plain HTML and JavaScript chat page with a typing indicator, retry and stop buttons, accessibility basics, and CORS and CSRF for a chat endpoint, tested in a real browser against the fake provider.