chatbot1-3  Exercise 3: A Benchmark With a Warm-Up
====================================================

ex3.py (save in the project root):

  import asyncio
  import statistics
  from bot.compare import run_prompts
  from bot.providers import FakeProvider, Reply, Usage


  async def benchmark(label, provider, prompts, runs=3):
      await run_prompts({label: provider}, prompts[:1])           # warm-up: not counted (a real local model loads on the first call)
      rates = []
      for _ in range(runs):
          rows = await run_prompts({label: provider}, prompts)
          rates.extend(r.tokens_per_second for r in rows)
      return statistics.median(rates), min(rates), max(rates)


  async def main():
      prompts = ["Explain DNS in one sentence.", "Say hello to Ada."]
      fast = FakeProvider([Reply("a b c d", Usage(5, 4))] * 20, delay=0.05)
      slow = FakeProvider([Reply("a b c d e f g h", Usage(5, 8))] * 20, delay=0.40)
      print(f"{'provider':<10}{'median tok/s':>14}{'min':>8}{'max':>8}")
      for label, provider in (("fake-fast", fast), ("fake-slow", slow)):
          med, lo, hi = await benchmark(label, provider, prompts)
          print(f"{label:<10}{med:>14.1f}{lo:>8.1f}{hi:>8.1f}")

  asyncio.run(main())

Run with:  py ex3.py

Output (these are FAKE providers with scripted delays of 0.05 s and 0.40 s, so the numbers show the harness and say nothing about any real model; yours will differ by a few tenths):

  provider    median tok/s     min     max
  fake-fast           64.3    63.4    65.0
  fake-slow           19.8    19.6    19.8

Why the warm-up matters for a real local model: a local server loads the model into memory on the first request (and again after it has unloaded it; Ollama's docs say a model stays loaded for 5 minutes by default). That first call is therefore much
slower than a normal reply and is not what a user waiting in a conversation usually experiences. If it is counted, it drags the average down and hides how fast the model really answers. The warm-up call pays that cost first, and the median over repeated
runs then reflects steady-state speed. (If your bot will often meet a cold model, measure the cold start separately and report it as its own number.)

WHY THIS WORKS AS AN ANSWER
---------------------------
The harness times each call with a high-resolution clock and divides the output tokens by the seconds, giving tokens per second per call. benchmark() makes one uncounted call to warm up, repeats the whole prompt list three times, and reports the median
(robust against one odd run) with the minimum and maximum so you can see the spread. With the fakes the result is predictable: 4 tokens in about 0.05 s is about 64 tokens per second, and 8 tokens in about 0.40 s is about 20, which is what the output shows.
That predictability is the point of testing a harness with fakes first: when the numbers come from a real model you know the harness is not the thing you are measuring.