#Exercises: make the learning stick

memory ≈ parameters × bytes-per-parameter (plus KV cache + overhead). fp16 ≈ 2.0, q8 ≈ 1.0, q4 ≈ 0.5 bytes per parameter. So a 7B model is ~14 GB at fp16 but ~3.5 GB of weights at q4, which is why q4 is the laptop default.

It grows a lot. The KV cache scales linearly with context length, and at 32k it can rival or exceed the weights themselves. The weights are fixed; the context is the lever. The example prints this curve.

Only the client's base_url (pointed at http://localhost:11434/v1), and the api_key, which is required by the SDK but ignored by the local server, so any string works. The messages, parameters, and response shape are identical. That's the entire repo in one sentence.

Something else: providers.ensure_server() probes the port first, prints start-up instructions, and exits cleanly (code 0). No stack trace, no charge because there's nothing to charge. Graceful degradation is the local analogue of the sibling repos' "missing API key" guard.

Usually the 7B at q6. A higher-quality quant of a smaller model tends to beat a heavily-crushed quant of a bigger one. Below ~q4 quality drops off, and q3 of a 13B may be noticeably degraded. Rule of thumb: smaller model + q4/q6 over bigger model + q2/q3.

Size and memory roughly halve; generation often gets a bit faster (less data to move); quality drops a little. For most tasks q4 keeps most of the quality at a quarter of fp16's size, which is why it's the default. examples/04_... shows the table and (with a server) measures tokens/sec.

vLLM, which is built for serving at scale (continuous batching, high throughput). Ollama is fantastic for one user on a laptop but isn't aimed at many-user GPU serving. Crucially, your code is identical either way; only the base_url changes.

Time-to-first-token (TTFT, the model reading/prefilling your prompt) and generation speed (tokens/sec once it starts). A longer prompt mainly grows TTFT; generation speed barely changes. And the very first call after the model loads is always slowest.

Because the extra time is prompt processing (reading all those tokens), which is separate from generation. More input → longer TTFT; the per-token generation rate is set by the model and hardware, not the prompt length.

Just the model name (an embedding model like nomic-embed-text) on the same local endpoint. The recurring cost is zero: no per-token embedding bill. This is the core of local RAG.

Not a failure. Small models drift more than hosted ones. The example strips the fences before json.loads, the same defensive parsing you learned for hosted models. Lower temperature, a tighter prompt, or a stronger model (qwen2.5, llama3.1) all help reliability.

Local wins: privacy/data control, cost at volume, offline use, no rate limits. Hosted wins: peak answer quality (the biggest models don't fit locally), zero ops, elastic scale, day-one access to new models. The best design is often both using local for the common/private/high-volume path and hosted for the hard cases.

OPENAI_BASE_URL=http://localhost:11434/v1, MODEL=<your local tag>, and OPENAI_API_KEY=local. The API key is the no-op: the local server ignores it, but the SDK insists on a non-empty string. No code changes; the prompt-engineering, RAG, agents, and evals repos all run locally for $0.

--fit 8 printed an offline memory estimate for an 8B model (weights + KV + overhead) and whether it fits 16 GB, which is Section 2 without a server. The tokens/sec line is the generation speed (Section 7) of your machine on each reply, plus the time-to-first-token.