Local Models

Build a Chatbot: From API Call to Deployed Bot

Chapter 3 · Running a Model Locally

Every provider so far lives on someone else's computer. A local model runs on yours: the same chat conversation, answered by software on your own machine, with no account, no key, no per-token bill and no data leaving the building. Thanks to the interface from chapter 2 it is not a new kind of bot, only a different base_url. This chapter is about the decisions around that: whether it is worth doing, what your hardware can hold, and how to check and compare a local model without fooling yourself.

What was run, and what was not

No language model was run in this chapter. There is no local model runtime (Ollama or llama.cpp) on this machine, and I did not install one or download a multi-gigabyte model without asking. What was run: a survey of this machine's real hardware, the memory arithmetic, the new code (a local provider, a model-listing check, a benchmarking harness) and its tests against the local scripted server from chapter 2 (45 tests passing in total), and a real attempt to reach a model server that is not there. The facts about Ollama and llama.cpp come from their official documentation, which I read, and the setup commands are marked "from the docs, not run here". Every comparison number below comes from fake providers with scripted delays and says so; none describes a real model.

Should You Run One Locally?

Reason	What it means	What it costs you
Privacy	Messages never leave your machine or network	You are now responsible for securing that machine
Cost	No per-token charges; the marginal cost of a message is electricity	The hardware, and the time to run it; heavy use can still outgrow a laptop
Offline and control	Works without internet; no vendor changing or retiring the model under you	You do the updating and the monitoring
Learning	The best way to see what a model does with no product around it	Time spent on setup instead of the bot

The usual trade-off, a general expectation I did not measure here, is that models small enough for a laptop are less capable and slower than large hosted ones, so a local model is a good fit for a simple, private or offline bot and a poor fit when you need the best answers. This course keeps both options open for exactly that reason, and the comparison section below shows how to find out for your own prompts rather than trusting a rule of thumb.

This Machine, as a Worked Example

Whether a model runs well depends on the hardware, so here is the real survey of the machine this course is being built on (Windows, queried with PowerShell's CIM commands):

Part	Measured
Memory (RAM)	29.8 GB
CPU	AMD Ryzen 7 8745H, 8 cores, 16 threads
Graphics	AMD Radeon 780M, an integrated GPU that shares system memory (Windows reported "2 GB", but that figure is not meaningful for an integrated GPU, so I draw no conclusion from it)
NVIDIA GPU	None (`nvidia-smi` is not installed)
Free disk	457.4 GB

What this means is partly a documentation question. Ollama's GPU page lists supported AMD cards on Windows through ROCm as discrete Radeon RX 7000-series and Radeon PRO cards; the Radeon 780M is not mentioned. The page says Vulkan is enabled by default when its backend is installed and warns that an integrated GPU can be unstable on mixed integrated/discrete systems. So on this machine the safe expectation is CPU inference, with GPU acceleration an experiment I did not try. A discrete NVIDIA card would be the straightforward case (the docs give a minimum compute capability of 5.0 and a recent driver). Your own survey takes one PowerShell command per line above (Get-CimInstance Win32_ComputerSystem, Win32_VideoController, Win32_Processor).

Memory Arithmetic

The first question is whether the model fits in memory at all. A model's weights take roughly parameters × bits per weight ÷ 8 bytes: a 7-billion-parameter model stored at 16 bits is 7 × 10⁹ × 2 bytes = 14 GB; at 4 bits (a common quantised format that trades some quality for size) it is about 3.5 GB. The small module below does the sum and prints a table for a given amount of memory. It is deliberately a lower bound: real files are somewhat larger, and the context (the conversation the model holds in memory) and the runtime need more on top, by an amount that depends on the model's architecture, so the module does not pretend to estimate them.

"""Back-of-the-envelope memory sums for running a model locally.

Only the model weights are estimated: parameters x bits per weight / 8. Real files are somewhat
bigger (quantisation formats store extra scaling data), and the context (KV cache) and the runtime
need memory on top, which depends on the model's architecture. Treat the answer as a lower bound.
"""

BITS = {"16-bit": 16, "8-bit": 8, "4-bit": 4}


def weights_gb(params_billion: float, bits: int) -> float:
    return params_billion * 1e9 * bits / 8 / 1e9  # gigabytes (10^9 bytes)


def table(sizes_billion: list[float], memory_gb: float) -> str:
    lines = [f"{'params':>8}" + "".join(f"{name:>10}" for name in BITS) + f"   fits in {memory_gb:g} GB (lower bound)?"]
    for p in sizes_billion:
        row = f"{p:>7g}B"
        fits = []
        for name, bits in BITS.items():
            gb = weights_gb(p, bits)
            row += f"{gb:>9.1f}G"
            if gb < memory_gb:
                fits.append(name)
        lines.append(row + "   " + (", ".join(fits) if fits else "none"))
    return "\n".join(lines)

Run for this machine's 29.8 GB:

Memory for the weights alone on a machine with 29.8 GB of RAM (this one): params 16-bit 8-bit 4-bit fits in 29.8 GB (lower bound)? 1B 2.0G 1.0G 0.5G 16-bit, 8-bit, 4-bit 3B 6.0G 3.0G 1.5G 16-bit, 8-bit, 4-bit 8B 16.0G 8.0G 4.0G 16-bit, 8-bit, 4-bit 14B 28.0G 14.0G 7.0G 16-bit, 8-bit, 4-bit 32B 64.0G 32.0G 16.0G 4-bit 70B 140.0G 70.0G 35.0G none

Read the right-hand column with care. "Fits" means only that the weights alone are smaller than all the machine's memory. Windows, your browser and your bot need memory too, so leave generous headroom: an 8B model at 4-bit (about 4 GB of weights) is comfortable on this machine, a 32B model at 4-bit (16 GB) is probably possible but tight, and a 70B model is out of reach at any of these precisions. The sizes in the table are round numbers for illustration, not a list of specific models.

The Two Servers You Will Meet

A local model needs a program that loads it and answers requests. Both common choices offer the OpenAI-compatible interface that OpenAICompatProvider already speaks. All of the following comes from their documentation, and none of it was run here:

	Ollama	llama.cpp (`llama-server`)
What it is	An application that downloads, stores and serves models, with a command-line tool	The reference C/C++ inference engine; you supply or download model files
Get a model	`ollama pull <name>`; models are stored under `C:\Users\<you>\.ollama\models` on Windows (change with `OLLAMA_MODELS`)	`-m <file>` for a file you have, or `-hf <user>/<model>[:quant]` to download from Hugging Face
Address	`http://localhost:11434`; binds to 127.0.0.1 only unless you set `OLLAMA_HOST`	Listens on 127.0.0.1; the README I read gives port 9931 as the default, so pass `--port` explicitly rather than rely on it
OpenAI-style base URL	`http://localhost:11434/v1/`; a key is required by some client libraries but ignored	`/v1/chat/completions` and `/v1/models` under its address; `--api-key` turns authentication on
Context size	Default depends on GPU memory: 4k tokens under 24 GiB of VRAM, 32k for 24-48 GiB, 256k for 48 GiB or more; change with `OLLAMA_CONTEXT_LENGTH`	`-c / --ctx-size`; 0 (the default) loads the size from the model
Readiness	Check `GET /v1/models`; `ollama ps` shows loaded models and the CPU/GPU split	`/health` returns 503 "Loading model" until ready, then 200
Loading behaviour	Models stay loaded 5 minutes by default (`OLLAMA_KEEP_ALIVE`); one request at a time per model by default (`OLLAMA_NUM_PARALLEL`); requests queue up to 512 then get a 503	Runs the model you start it with

Ollama's OpenAI-compatibility page lists the request fields it supports on /v1/chat/completions (including messages, max_tokens, stream, tools, temperature and seed) and the ones it does not (tool_choice, n, user and logit_bias, among others). Our provider sends only model, messages and max_tokens, so it is within the supported set. The page also lists a /v1/responses endpoint (stateless only), a reminder that the "OpenAI-compatible" surface keeps growing and differs at the edges between servers.

The commands you would run, from the documentation and not run here:

# Ollama (install it from ollama.com first)
ollama pull <a model name from Ollama's library>
ollama ps                                  # what is loaded, and the CPU/GPU split
curl http://localhost:11434/v1/models      # is the OpenAI-style endpoint up, and what does it offer?

# llama.cpp, after building or downloading llama-server
llama-server -m ./model.gguf --port 8080 -c 4096
curl http://localhost:8080/health          # 503 while loading, 200 when ready

The Code: A Local Provider and a Startup Check

A local provider is an OpenAICompatProvider with three sensible defaults: no key, Ollama's address, and a long timeout, because the first request after the server starts (or after it unloaded the model) includes loading the model into memory, which can take far longer than a normal reply. list_models asks the server what it offers, which doubles as a "is it running?" check:

import httpx

from bot.providers.base import ProviderError, ProviderUnavailable, error_from_status
from bot.providers.openai_compat import OpenAICompatProvider

OLLAMA_URL = "http://localhost:11434/v1"


def local_provider(model: str, base_url: str = OLLAMA_URL, *, timeout: float = 300.0) -> OpenAICompatProvider:
    """A provider for a model running on this machine behind an OpenAI-compatible server.

    No key is needed (Ollama ignores one). The timeout is long because the first request after
    the server starts, or after it unloaded the model, includes loading the model into memory.
    """
    return OpenAICompatProvider(base_url, model, None, name="local", timeout=timeout)


async def list_models(base_url: str = OLLAMA_URL, *, timeout: float = 3.0) -> list[str]:
    """Ask the server which models it offers (GET /models). Raises ProviderUnavailable if nothing answers."""
    try:
        async with httpx.AsyncClient(base_url=base_url.rstrip("/"), timeout=timeout) as client:
            response = await client.get("/models")
    except httpx.TransportError as e:  # includes timeouts and refused connections
        raise ProviderUnavailable(f"no model server at {base_url}: {e!r}") from e
    if response.status_code != 200:
        raise error_from_status(response.status_code, response.text)
    try:
        return [m["id"] for m in response.json()["data"]]
    except (ValueError, KeyError, TypeError) as e:
        raise ProviderError(f"unexpected /models answer: {response.text[:200]}") from e

The check against the scripted server (it answers with the OpenAI shape, {"data": [{"id": ...}]}), and against a real address where nothing is listening, which is the situation on this machine:

['llama-a', 'qwen-b'] path requested: /models

ProviderUnavailable: no model server at http://localhost:11434/v1: ConnectError('All connection attempts failed')

The second result is the useful one: with no server, the bot gets a typed ProviderUnavailable with a message saying where it looked, instead of a stack trace from deep inside an HTTP library. Checking at startup turns "the first user finds out the model server is off" into "the operator finds out before anyone is waiting". Exercise 2 builds that check.

Two design notes. First, the context-size defaults above matter for chapter 4: a conversation that is longer than the server's context window cannot all be sent to the model, so the bot must budget history by tokens against the actual limit of the running server, not an assumed one (I have not verified what Ollama does when a prompt exceeds the window; read its documentation for your version before relying on any behaviour). Second, the provider's contract tests from chapter 2 apply unchanged, because local_provider is an OpenAICompatProvider: the suite is now 45 tests, all passing.

Comparing a Local Model With a Hosted One, Honestly

"Is the local model good enough?" is only answerable with your prompts, so build a small harness once and reuse it. This one sends each prompt to every provider with no history and times the call:

import time
from dataclasses import dataclass

from bot.providers import Provider


@dataclass(frozen=True)
class Row:
    provider: str
    prompt: str
    seconds: float
    output_tokens: int
    text: str

    @property
    def tokens_per_second(self) -> float:
        return self.output_tokens / self.seconds if self.seconds > 0 else 0.0


async def run_prompts(providers: dict[str, Provider], prompts: list[str], *, system: str | None = None,
                      max_tokens: int = 256) -> list[Row]:
    """Send every prompt, one at a time and with no history, to every provider; time each call."""
    rows = []
    for prompt in prompts:
        for label, provider in providers.items():
            start = time.perf_counter()
            reply = await provider.complete([{"role": "user", "content": prompt}], system=system, max_tokens=max_tokens)
            rows.append(Row(label, prompt, time.perf_counter() - start, reply.usage.output_tokens, reply.text))
    return rows


def format_rows(rows: list[Row]) -> str:
    lines = [f"{'provider':<10}{'seconds':>8}{'out tok':>8}{'tok/s':>8}  prompt"]
    for r in rows:
        lines.append(f"{r.provider:<10}{r.seconds:>8.2f}{r.output_tokens:>8}{r.tokens_per_second:>8.1f}  {r.prompt[:40]}")
    return "\n".join(lines)

Here it is running on two fake providers with scripted delays (50 ms and 400 ms), to show the harness and the shape of its output. The labels say so deliberately; these are not the speeds of any real model:

provider seconds out tok tok/s prompt fake-fast 0.06 4 62.8 Explain DNS in one sentence. fake-slow 0.41 8 19.7 Explain DNS in one sentence. fake-fast 0.06 4 63.1 Say hello to Ada. fake-slow 0.41 8 19.6 Say hello to Ada.

When you run it for real, a few rules keep the numbers honest:

Exclude the first call. A local model's first request includes loading it, so it is much slower than the rest (the idea behind Exercise 3's warm-up).
Repeat and take the median, and look at the minimum and maximum. One run says very little.
Same prompts, same system prompt, same max_tokens for every provider.
Record the conditions: the model name and size, the quantisation, the context size, CPU or GPU, and what else was running.
Judge quality separately from speed: read the answers side by side without looking at which provider wrote which, and decide against your bot's actual job.
Tokens per second is not comparable across servers' tokenisers. Different models count tokens differently, so compare time for the same task as well.

A correction to chapter 2, and how it happened

In chapter 2 I wrote that Ollama's documentation calls its OpenAI compatibility "experimental". That came from a search result that quoted an older version of the docs. When I fetched the current compatibility page for this chapter it had no such warning, so I corrected chapter 2 (it now says an earlier version of the docs used the word, and lists the unsupported fields instead). The mistake was repeating a caveat from a secondary summary without checking the current source. The habit that avoids it: before you pass on a warning or a limitation, read the page that is current for the version you are using. The same applies to your own bot's documentation: dated advice goes stale.

Hands-On Exercises

Exercise 1

Use bot.sizing to print the table for a 16 GB machine and for your own machine's RAM (look it up with the PowerShell command in this chapter). Compute the weights for a 12B model at 4-bit and 8-bit. Then say, in your own words, three things the table does not include that you would still need memory for.

📄 View solution

Exercise 2

Write require_model(base_url, model): it asks the server for its models, exits with a helpful message if the server cannot be reached, and exits with a different helpful message if the server is up but does not have the model you want. Run it against the scripted server (model present, model missing) and against an address with no server.

📄 View solution

Exercise 3

Extend the harness into benchmark(label, provider, prompts, runs=3): do one warm-up call that is not counted, then run the prompts several times and report the median, minimum and maximum tokens per second. Run it on two fake providers with different delays. Explain why the warm-up matters for a real local model.

📄 View solution

Chapter 3 Quick Reference

A local model is the same OpenAICompatProvider with a different base_url: local_provider(model) = Ollama's http://localhost:11434/v1, no key, 300 s timeout (the first call loads the model)
Reasons for: privacy, cost per message, offline use, control, learning. Costs: capability and speed on laptop-sized models (a general expectation; measure your own), setup and operations
Weights memory ≈ parameters × bits ÷ 8: 7B is 14 GB at 16-bit and 3.5 GB at 4-bit; a lower bound, since context and runtime need more
This machine: 29.8 GB RAM, Ryzen 7 8745H, integrated Radeon 780M (not in Ollama's listed supported GPUs; expect CPU), no NVIDIA, 457 GB free
Ollama (11434, ollama pull, ollama ps, key ignored, context default 4k under 24 GiB VRAM, models unload after 5 minutes, one request at a time per model by default); llama.cpp (llama-server, -m / -hf, --api-key, /health 503 until loaded)
Check the server at startup (list_models): a clear ProviderUnavailable beats a stack trace
Honest comparison: exclude the warm-up, repeat and take the median, same prompts and settings, record the conditions, judge quality separately
Check the current docs for the version you use before repeating a warning

Coming next

Build a Chatbot: From API Call to Deployed Bot 4 gives the bot a memory that survives: sessions per user and per channel, history stored in SQLite, trimming and summarising old turns to fit a token budget (against the context limits you just saw), a small "facts about the user" memory with consent and deletion, and reset and export commands. Everything is tested with the fake provider.