chatbot1-2  Exercise 1: Running the Contract Against a New Provider
====================================================================

ex1.py (save in the project root, next to run_terminal.py):

  import unittest

  from bot.providers import AuthError, BadRequest, ProviderError, RateLimited, ServerError
  from bot.providers.fake import EchoProvider
  from tests.test_providers import ProviderContract


  class UppercaseProvider:
      """A new provider: replies with the last user message in capitals."""
      name = "upper"

      def __init__(self):
          self.sent = []
          self.next_error = None

      async def complete(self, messages, *, system=None, max_tokens=1024):
          from bot.providers import Reply, Usage
          self.sent.append((list(messages), system))
          if self.next_error:
              raise self.next_error
          return Reply(self.reply_text if hasattr(self, "reply_text") else messages[-1]["content"].upper(),
                       Usage(1, 1))


  class UppercaseProviderTests(ProviderContract, unittest.IsolatedAsyncioTestCase):
      exact_usage = False  # this provider invents its own token counts

      def setUp(self):
          self.provider = UppercaseProvider()

      def arrange_ok(self, text, input_tokens, output_tokens):
          self.provider.reply_text = text

      def arrange_error(self, kind, retry_after=None):
          cls = {"rate_limit": RateLimited, "auth": AuthError, "server": ServerError, "bad_request": BadRequest}[kind]
          self.provider.next_error = cls(kind, retry_after=retry_after)

      def sent_messages(self): return self.provider.sent[-1][0]
      def sent_system(self): return self.provider.sent[-1][1]


  if __name__ == "__main__":
      unittest.main(argv=["x", "UppercaseProviderTests", "-v"])

Run with:  py ex1.py

Output (the eight contract tests, all passing):

  test_bad_key_is_normalised ... ok
  test_bad_request_is_normalised ... ok
  test_no_system_prompt_sends_none ... ok
  test_passes_the_system_prompt ... ok
  test_rate_limit_is_normalised ... ok
  test_returns_text_and_usage ... ok
  test_sends_the_conversation_in_order ... ok
  test_server_failure_is_normalised ... ok
  Ran 8 tests in 0.322s
  OK

Which test made me decide something about token usage?  test_returns_text_and_usage. It checks that the Usage in the Reply equals the numbers the
test staged. A provider that only estimates tokens (like FakeProvider, which counts words, and this UppercaseProvider, which invents 1 and 1) cannot pass that
check honestly, so the test class sets exact_usage = False. The first time I ran the contract against this provider without that line, that one test failed
with: AssertionError: Usage(input_tokens=1, output_tokens=1) != Usage(input_tokens=12, output_tokens=6).

WHY THIS WORKS AS AN ANSWER
---------------------------
The contract is written against the Provider interface, not against any one class, so a new provider only has to say how to stage a success, how to stage each kind of failure, and how to see what it
was sent (the four hook methods). Everything else is inherited: the eight behaviours that every provider must share. The failure on token usage is the useful part. It forces you to decide whether your provider
reports real usage (then the flag stays True and the provider must return the counts the model reported) or an estimate (then you say so explicitly, in one flag, rather than special-casing the provider by name
inside the shared test, which is what my first version did).