Bonus dive

#Local Models: A Guided Deep Dive

A hands-on playground for running open-weight LLMs on your own machine, and discovering that running local is mostly an operations choice rather than a new API. You'll serve a model with Ollama, talk to it with the exact same OpenAI SDK from the API deep dives with one changed URL, and understand every moving part. Model sizing, quantization, serving engines, performance, local embeddings, structured output, and the real local-against-hosted tradeoff. No framework magic, just enough code to see how it works.

Here's the hook that makes this repo click. Open-weight models speak the OpenAI-compatible API. So the client you already know works against a model on your laptop by changing base_url and nothing else, and there's no API key and no per-token bill anywhere in this repo. The one offline section, the sizing calculator, runs with nothing installed at all. The rest need only a local runtime.

This repo is standalone and teaches everything it needs on its own. It's the full version of the "use a local model" footnote in the OpenAI and Prompt Engineering dives, and it's where the Fine-tuning dive points for running open weights. Its code depends on none of them.

Like its siblings, walk through it. Each section ends with something to run, and the first runs offline and free with no server needed. EXERCISES.md has a predict-then-run prompt for each section.

#0. The one big idea

An open-weight model on your machine speaks the same OpenAI API, so running local is mostly an ops choice. You trade hosted convenience for privacy, cost at volume, offline use, and control.

That's the whole repo. The first time you point the OpenAI SDK at http://localhost:11434/v1 and get an answer back with no key, the trick shows itself. The code doesn't change. The operations do. Everything below, from sizing a model to your RAM to picking a quantization, choosing a serving engine, and reading tokens per second, is about running that model well on hardware you own. And because no provider is doing the work, "will it even fit?" becomes a question you answer with arithmetic before you download a thing. Hold onto that and none of this feels complicated.

#1. Setup (5 minutes)

# 1. Create an isolated Python environment
python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate

# 2. Install dependencies (just the OpenAI SDK + helpers: no key needed)
pip install -r requirements.txt

# 3. Copy the env file (it holds settings, not secrets)
cp .env.example .env

# 4. Install a local runtime and pull a small model
#    Ollama is the easiest (https://ollama.com):
ollama pull qwen3:4b

# 5. Confirm everything is wired up (no hosted call, costs nothing)
python check_setup.py

Unlike its siblings, this repo has no API key, because the model runs on your machine. What it needs instead is a local server. Ollama is the default, and any OpenAI-compatible runtime works by setting OPENAI_BASE_URL.

The whole repo is free, and Section 2 needs nothing at all. The sizing calculator is pure arithmetic, so run it before you install a runtime. Everything else needs a local server up, and never a key or a cent.

#2. Will it fit? Sizing and quantization math

python examples/01_quant_math.py     # offline, no server needed

The most useful local-models skill is arithmetic you do before downloading anything. How much memory will this model need? One formula answers it: memory ≈ parameters × bytes-per-parameter (+ KV cache + overhead), where bytes-per-parameter is set by the quantization (fp16 = 2.0, q8 is about 1.0, q4 is about 0.5). The example prints the size of a model at every quant level, shows how the KV cache grows with context length, which eats memory without announcing itself, and answers the practical question: given my RAM, what's the best version of this model I can run? It needs no server, no model, and no key.

#3. Your first local request

python examples/02_first_local_request.py

The repo's whole idea in one script. The chat call below is the same client.chat.completions.create you'd make against OpenAI, with the same messages and the same response shape. The only difference lives in local/providers.py, where base_url points at your machine instead of api.openai.com. No key, no cost. If no server is running, the script tells you how to start one and exits cleanly. It can't charge you, because there's nothing to charge.

#4. Picking a model

python examples/03_pick_a_model.py

"Which model?" has two halves. Capability, where bigger is roughly smarter up to a point, and whether it fits, which is Section 2's math. The example lists what your server has pulled, then maps the popular small families (Llama 3.x, Qwen2.5, Phi, Mistral, Gemma) to their sizes and a fit estimate. Rules of thumb. 3B is fast and fine for simple work. 7B to 8B is the laptop sweet spot. Pick an instruct tag for assistant tasks. And a higher-quality quant of a smaller model usually beats a crushed quant of a bigger one.

#5. Quantization in practice: size, quality, speed

python examples/04_quantization_tradeoff.py

Section 2 sized quantization. This one is about what it costs. Fewer bits means a smaller file, less memory, and often faster generation, at some cost in quality. For most tasks q4 to q6 is the sweet spot: most of the quality, a fraction of the size. The example shows the tradeoff table offline, then measures real tokens per second if a server is up, so you can feel it. Start at q4 and go up to q6 or q8 if you have memory to spare.

#6. Serving engines: Ollama, llama.cpp, vLLM

python examples/05_serving_engines.py

The model is a file. A serving engine loads it and answers requests. They all expose the same OpenAI-compatible API, which is why your code never changes, and they trade off ease against control against scale. The example probes the common ports and tells you what's running. Then: Ollama for easy laptop use, llama.cpp for hand-tuning GGUF files, LM Studio for a GUI, vLLM for high-throughput GPU serving.

#7. Performance: time-to-first-token and tokens per second

python examples/06_performance.py

Local inference has two speeds, and conflating them causes most of the confusion. Time-to-first-token is the pause while the model reads your prompt, which is prompt processing rather than network. Generation speed is tokens per second once it starts. The example streams a reply, measures both, then shows that a longer prompt mostly grows TTFT while generation speed barely moves. Keep prompts tight for fast responses, and remember the first call after load is always the slowest.

#8. Embeddings, locally

python examples/07_embeddings.py

Embeddings turn text into vectors so that similar meanings sit close, and they're what search and RAG run on. They also run locally through the same endpoint, with a different model (ollama pull nomic-embed-text). No per-token embedding bill, ever. The example embeds a handful of sentences, ranks them against a query by cosine similarity in five lines of standard library with no magic, and retrieves the right one by meaning. That's the exact core of the RAG deep dive, for $0.

#9. Structured output and tool calling

python examples/08_structured_and_tools.py

"Give me JSON" and "call this function" work locally too, with rougher edges. Smaller models follow a schema less reliably and sometimes wrap JSON in prose or fences. The fix is the same defensive habit from the API and prompt dives. Ask clearly, parse forgivingly. The example requests JSON, using response_format when the server supports it, and parses it defensively. Then it describes one tool and lets the model choose to call it, reporting honestly when a weaker model just answers in text. Reliability tracks model size, and capable small models like qwen3:8b and gemma3:4b are good at both.

#10. When local beats hosted (and when it doesn't)

python examples/09_local_vs_hosted.py

Local isn't better in the abstract. It's a set of tradeoffs, and this lays the decision out so you make it on purpose. Local wins on privacy and data control, cost at volume, offline use, and no rate limits. Hosted wins on peak quality, zero ops, elastic scale, and day-one access to the newest models. The example prints the scorecard and measures your own local latency if a server is up, so the speed column isn't hypothetical. The best answer is often both: a local small model for the common, private, high-volume path, falling back to a hosted frontier model for the hard cases.

#11. Run the rest of the series locally, for $0

python examples/10_run_the_series_locally.py

This is what "local speaks the OpenAI API" buys you. The sibling dives on prompt engineering, RAG, agents, and evals were all built on the OpenAI SDK, so they run against your local model with an env change and no code edits. The example runs a tiny RAG loop, retrieve then ground then answer, entirely on the local server. To point any sibling repo here, set this in its own .env:

OPENAI_API_KEY=local                          # any non-empty string; ignored locally
OPENAI_BASE_URL=http://localhost:11434/v1
MODEL=qwen3:4b

That's the whole change. The whole learning series, now running at zero cost.

#The capstone: local_chat.py

Everything assembled into a tool you'd actually use. A streaming, multi-turn chat assistant running entirely on your machine, with no key, no bill, and nothing leaving the laptop. It shows a live tokens-per-second readout from Section 7, remembers the conversation, and can print an offline fit estimate before you start, from Section 2.

# interactive chat (Ctrl-D or "quit" to exit):
python hands_on/local_chat.py

# one-shot question, then exit:
python hands_on/local_chat.py "Explain quantization in one sentence."

# use a specific model you've pulled:
python hands_on/local_chat.py --model qwen3:8b

# offline: estimate memory for an 8B model before running anything:
python hands_on/local_chat.py --fit 8

Read hands_on/local_chat.py. It's the library, providers.stream plus sizing, wired to a CLI. Suggested exercise: pull a second model with ollama pull qwen3:8b and chat with both via --model. You'll feel the size, speed, and quality tradeoff from Sections 4 and 5 in your own hands.

#Where to go next

You've run a model end to end on your own hardware. What comes next is more control and more scale.

• Fine-tune the open weights you're running. LoRA and PEFT with transformers, peft, and trl. The Fine-tuning dive explains the concepts, and this repo runs the result.
• GPU serving with vLLM. Continuous batching and high throughput for many concurrent users.
• Bigger models through more memory. A quantized 70B on a 64 GB machine, or multi-GPU.
• Speculative decoding and draft models. A small model drafts, a big one verifies, and generation gets faster.
• Embeddings and reranking models locally. A fully local RAG stack with no hosted calls at all.

#From teaching code to production

The teaching shortcuts here are what you'd harden once a local model serves real traffic.

The general ops machinery (observability, cost, reliability, caching, guardrails, prompt versioning, eval gates) is built from scratch and wired into one running app in Production (#8 in the series), which runs offline on a mock provider.

#File map

check_setup.py              ← run first: Python, packages, and is a server up?
README.md                   ← this guide
EXERCISES.md                ← predict-then-run prompts, one per section
local/                      ← the tiny from-scratch library (read it!)
  providers.py              ← THE trick: the OpenAI SDK pointed at localhost
  sizing.py                 ← offline calculator: params × bits → memory, will it fit?
hands_on/
  local_chat.py             ← capstone: a streaming, fully-local chat CLI
examples/
  01_quant_math.py          ← will it fit? sizing & quantization (offline, no server)
  02_first_local_request.py ← the same SDK, one changed base_url
  03_pick_a_model.py        ← params, families, and what fits your machine
  04_quantization_tradeoff.py ← size vs. quality vs. speed
  05_serving_engines.py     ← Ollama vs. llama.cpp vs. vLLM (probes what's up)
  06_performance.py         ← time-to-first-token vs. tokens/sec
  07_embeddings.py          ← local embeddings + cosine similarity
  08_structured_and_tools.py ← JSON output & tool calling (and their caveats)
  09_local_vs_hosted.py     ← the decision, with your own latency measured
  10_run_the_series_locally.py ← point the other repos here; a local RAG loop

(out/ is created by some examples and is git-ignored.)

#Troubleshooting

Run python check_setup.py first; it catches most problems. Then, by symptom:

Still stuck? Every file is small and self-contained. Open it, read the docstring at the top, and run it. local/providers.py is the whole story: the OpenAI SDK, one changed URL.

#The series

This is one of the standalone, hands-on deep dives into building with LLM APIs. Eight core dives, plus the bonus ones listed below. Each one stands on its own, with its own setup, examples, and capstone, and they all share one house style. Provider-agnostic where it makes sense, built from scratch with no frameworks, offline-first examples, and a real capstone at the end. Do them in any order. This sequence builds naturally.

1. OpenAI API: the API from zero
2. Claude API: the same ideas, the Anthropic way
3. Prompt Engineering: shape model behavior with better prompts
4. RAG: answer questions over your own documents
5. Evals: measure whether a change actually helps
6. Agents: give a model tools and a loop so it can act
7. Prompt Injection & Guardrails: attack and defend all of the above
8. Production: operate one app end to end

Bonus dives, standalone and slotting in where they're most useful:

• Context Engineering: manage what's in the window, with memory, compaction, and assembly
• AI Data Engineering: the corpus behind the index, with versions, lineage, ACLs, and deletes
• Multimodal: images and audio as well as text
• Fine-tuning: teach a model new behavior by example
• MCP: serve tools, data, and prompts to any LLM over a standard protocol
• Local Models: run open-weight models on your own machine
• Agent Harnesses: build on the loop, adding hooks, permissions, sandboxing, and subagents
• Realtime Voice: low-latency speech-to-speech agents
• Observability: watch a running app over time, covering drift, quality, alerting, and the feedback loop
• Architecture: the seams between the components, each decision measured rather than asserted
• GenAI Security: treat the model as an untrusted principal, and put identity, supply chain, isolation, budgets, and release gates around it
• Inference Platform Engineering: turn finite GPU memory and a request queue into latency, throughput, and a fleet size you can defend
• Testing & Delivery: decide whether a build is fit to promote, using evidence, gates, staged rollout, and rollback
• Professional Tools: rebuild each hand-written piece with the tool professionals reach for, and measure both

And the whole series lands in one codebase in the capstone: a codebase Q&A tool built step by step, one tag per dive.

Local Models is a bonus dive in the series. It pairs naturally with the two API dives (#1-2), since it's their code with one changed base_url, and with Fine-tuning, which explains training the open weights you run here. Section 11 lets you run the whole series locally at zero cost.