chatbot1-4  Exercise 3: How Far Off Is the Estimate?
======================================================

pip install tiktoken      (tested with tiktoken 0.14.0; it downloads its vocabulary the first time it is used, so it needs an internet connection once)

ex3.py (save in the project root):

  import tiktoken
  from bot.context import estimate_tokens

  enc = tiktoken.get_encoding("o200k_base")
  samples = {
      "English": "Could you tell me where the nearest railway station is, please?",
      "Hungarian": "Meg tudná mondani, hol van a legközelebbi vasútállomás, kérem szépen?",
      "Japanese": "最寄りの駅はどこですか。切符を二枚お願いします。",
      "Numbers": "Invoice 20261010-0042 total 1,234.56 GBP due 2026-11-09",
  }
  worst = 0.0
  for name, text in samples.items():
      est, real = estimate_tokens(text), len(enc.encode(text))
      ratio = real / est
      worst = max(worst, ratio)
      print(f"{name:<10} chars={len(text):>3} estimate={est:>3} o200k_base={real:>3} real/estimate={ratio:.2f}")

  SAFETY = 2.0

  def estimate_tokens_safe(text: str) -> int:
      return int(estimate_tokens(text) * SAFETY) + 1

  print()
  print(f"worst real/estimate ratio in these samples: {worst:.2f}; a multiplier of {SAFETY} would have covered every sample:",
        all(estimate_tokens_safe(t) >= len(enc.encode(t)) for t in samples.values()))

Run with:  py ex3.py

Output:

  English    chars= 63 estimate= 16 o200k_base= 13 real/estimate=0.81
  Hungarian  chars= 69 estimate= 18 o200k_base= 25 real/estimate=1.39
  Japanese   chars= 24 estimate=  6 o200k_base= 18 real/estimate=3.00
  Numbers    chars= 55 estimate= 14 o200k_base= 24 real/estimate=1.71

  worst real/estimate ratio in these samples: 3.00; a multiplier of 2.0 would have covered every sample: False

Does one factor cover all the samples? No. With a factor of 2.0 the check printed False, because the Japanese sample needed 3.0. A factor of 3 would cover these samples but would make the English budget about 3.7 times too cautious (English real/estimate was 0.81), wasting most of the window.
What I would do instead: count with something real. Options, none run against a live model here: (1) the tokeniser for the model family when one exists (tiktoken for OpenAI-family models); (2) the provider's token-counting endpoint (Anthropic documents POST /v1/messages/count_tokens);
(3) feed the usage numbers from Reply.usage back into the budget, so the bot learns the real size of its own conversations; or (4) pick the factor per language when you know the language. Whichever you use, keep a margin: this tokeniser is OpenAI's, and other models count differently.

WHY THIS WORKS AS AN ANSWER
---------------------------
The estimate is characters divided by four. English words average close to that, so English comes out a little high (ratio 0.81, safe). Hungarian words are longer and use letters a tokeniser splits more, Japanese has few characters but many tokens each, and strings of digits and identifiers
split into many short pieces, so the estimate falls short by 1.4x, 3.0x and 1.7x here. Four samples is a small sample, so these ratios show the direction and size of the problem, not exact values for any language.