Small language models

What counts as small

There is no official cutoff. In practice the term covers models from well under a billion up to roughly ten billion parameters, the range that runs on a laptop, a phone or a single modest GPU. They use the same transformer design as large models; the difference is scale.

Small models have improved mostly through better data rather than new tricks. Training on carefully filtered text and on examples produced by larger models, a process called distillation, lets a compact model handle tasks that once needed a much bigger one.

Most open-weight families include small sizes: Phi from Microsoft, Gemma from Google, the smaller Qwen and Llama models, compact Mistral releases and SmolLM from Hugging Face. Lesson 17 showed how to run them with Ollama, and the same endpoint works here.

Size alone does not predict quality. A well-trained small model can beat a larger, older one on a narrow task, and two models of the same size can differ a lot, so the only reliable comparison is your own eval set.

Why choose a small model

Speed is the first reason. A small model starts answering sooner and produces tokens faster, which matters for autocomplete, live grading and anything a user waits on.

Cost and privacy come next. A small model runs on hardware you already own, so heavy use does not grow a bill per token, and notes processed on the device never travel to a server. It also keeps working on a train with no signal.

Small models are also cheaper to fine-tune, so the techniques from lesson 19 become practical on a single rented GPU or even a strong laptop.

Running on a user's device brings its own limits. The model file is a large download, inference drains the battery and warms the device, and the memory is shared with every other open app. Download the model separately from the app, after the user agrees, and offer a smaller option for older hardware.

• Classifying text, such as tagging a note by subject or spotting a question type.
• Extracting fields into JSON from short, well-formed input.
• Summarising or rewriting a paragraph the user just wrote.
• Checking a learner's answer against a known correct answer.
• Deciding which tool or which larger model should handle a request.

Quantisation and memory

Model weights are numbers, normally stored with 16 bits each. Quantisation stores them with fewer bits, such as 8 or 4, so the model takes less memory and often runs faster, at the price of some precision.

The arithmetic is simple. Multiply the parameter count by the bytes per weight: a 3-billion-parameter model needs about 6 GB at 16 bits, about 3 GB at 8 bits and roughly 2 GB at 4 bits. Add memory for the context, which grows with every token the model keeps in view, and for everything else running on the machine.

That context memory surprises many people. A long RAG prompt with several chunks can need a noticeable share of the memory the weights take, so on small devices keep prompts short and retrieve fewer, tighter passages.

Quality usually holds up well at 8 bits and stays acceptable for many tasks at 4 bits, while very low settings degrade answers noticeably. Do not guess: run your eval set on two quantisation levels of the same model and compare.

You will meet a few formats and runtimes. GGUF files are used by llama.cpp and Ollama, MLX targets Apple silicon, and ONNX Runtime runs models across Windows, Linux, macOS and mobile. In the browser, projects such as WebLLM and Transformers.js run small models on the user's own GPU or CPU.

Where small models fail

Fewer parameters mean less stored knowledge. Ask a small model about a minor historical figure and it is more likely to invent details, so pair it with retrieval whenever facts matter.

They also struggle with long, multi-part instructions, multi-step reasoning and long documents, and their tool calling and JSON output can be less reliable. Support for languages other than English varies widely between families, so test in the languages your learners use.

You can narrow the gap. Give one clear task per call, keep prompts short, show an example of the exact output, validate the result with code and retry once if it fails. A small model doing one well-defined job often beats a large model given a vague one.

Safety tuning also differs between small models. Run the refusal cases from your eval set on each one instead of assuming it behaves like the larger model you tested before.

Routing: small first, large when needed

Many real apps use both sizes. The simplest router decides by task type: grading, tagging and short rewrites go to the small model; explanations, plans and anything open-ended go to the larger one.

A cascade goes one step further. The small model tries first, your code checks the result, for example valid JSON or an answer from the allowed set, and only failures go to the large model. Most requests then finish quickly and cheaply, while hard cases still get a strong answer.

Measure before you route. Run your eval set on both models, split by task type, and send each task type to the smallest model that passes it reliably.

A local small model also makes a good fallback. When the network or the cloud provider fails, Study Buddy can still check answers and run flashcards offline, and tell the learner that full explanations will return when the connection does.