chatbot1-3  Exercise 1: Will It Fit?
=====================================

ex1.py (save in the project root):

  from bot.sizing import table, weights_gb

  print("A 16 GB machine:")
  print(table([3, 8, 12, 14], 16))
  print()
  print(f"12B at 4-bit: {weights_gb(12, 4):.1f} GB of weights; at 8-bit: {weights_gb(12, 8):.1f} GB")
  print(f"70B at 4-bit: {weights_gb(70, 4):.1f} GB of weights")

Run with:  py ex1.py

Output:

  A 16 GB machine:
    params    16-bit     8-bit     4-bit   fits in 16 GB (lower bound)?
        3B      6.0G      3.0G      1.5G   16-bit, 8-bit, 4-bit
        8B     16.0G      8.0G      4.0G   8-bit, 4-bit
       12B     24.0G     12.0G      6.0G   8-bit, 4-bit
       14B     28.0G     14.0G      7.0G   8-bit, 4-bit

  12B at 4-bit: 6.0 GB of weights; at 8-bit: 12.0 GB
  70B at 4-bit: 35.0 GB of weights

Your own machine: get the RAM with PowerShell:  (Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory / 1GB   (this course's machine reported 29.8), then pass that number as the second argument to table().

Three things the table does not include (that you would still need memory for):
  1. The context: the conversation the model holds while answering (the "KV cache"); it grows with the context length and depends on the model's architecture.
  2. The runtime itself and its working buffers (Ollama or llama.cpp, plus the operating system's own needs).
  3. Everything else running: Windows, your browser, your editor, and the bot process itself.
(Also: real model files are somewhat larger than parameters x bits / 8 because quantised formats store extra scaling information.)

WHY THIS WORKS AS AN ANSWER
---------------------------
The sum is parameters x bits per weight / 8, so halving the bits halves the memory: 12B is 24 GB at 16-bit, 12 GB at 8-bit and 6 GB at 4-bit, which matches the printed table. On a 16 GB machine the 16-bit versions of the
8B, 12B and 14B models are out, and the 8-bit and 4-bit versions pass the weights-only test, but a 14B model at 8-bit would take 14 of the 16 GB for weights alone, which is not realistic once Windows and the context are counted. That is
why the right-hand column is labelled a lower bound: it can only rule things out, never promise that a model will run well.