You will be able to
- Tell open-weight, fully open source and closed models apart
- Name the main open-weight families and what to check before choosing one
- Read a model licence for the points that matter to a real project
- Run a model locally with Ollama and call it through its OpenAI-compatible endpoint
- Decide when a local model is the better choice for an app
Open weights, open source and closed models
A closed model lives behind a provider's API. You send text and get text back, but you never see the weights, and the provider decides when the model changes or disappears.
An open-weight model publishes the trained parameters for download. You can run it anywhere the hardware allows, fine-tune it and keep using the exact version you tested for as long as you like.
Open source in the strict sense asks for more than weights: the training code, details of the training data and the freedom to use and change all of it. Few models meet that bar; the OLMo models from the Allen Institute for AI are one example. Most popular models are open-weight rather than open source, which is why this course uses that term.
The main families
Each family below ships in several sizes, usually as a base model for further training and an instruct or chat version tuned to follow requests. For apps, pick the instruct version.
On top of these sit thousands of community fine-tunes for particular languages, coding, medicine or maths, published on Hugging Face. A good derivative for your exact domain can beat a larger general model, but check who made it and how it was evaluated.
Some families use a mixture-of-experts design: the model holds many expert sub-networks but activates only a few of them for each token. Such a model answers faster than its total size suggests, yet all its weights still have to fit in memory.
To choose, start from your own eval set instead of public leaderboards. Pick the largest size your hardware handles comfortably, try two or three families on the same cases, and keep the one that passes most often at an acceptable speed.
- Llama from Meta: a widely supported family with many sizes and a large ecosystem of derivatives.
- Mistral from Mistral AI: efficient models, some released under the permissive Apache 2.0 licence and others only under commercial terms.
- Qwen from Alibaba: general, coding and multilingual variants across a wide range of sizes.
- Gemma from Google: lightweight models built to run on modest hardware.
- DeepSeek: large models known for reasoning and coding, plus smaller distilled versions.
- Phi from Microsoft and OLMo from the Allen Institute: small models, the latter with fully open training data and code.
Read the licence before you ship
Downloadable does not mean free for any use. Some models use standard permissive licences such as Apache 2.0 or MIT. Others come with the maker's own licence, which may add an acceptable use policy, naming or attribution rules, or extra conditions for very large services.
Licences can differ between versions and sizes of the same family, so check the one attached to the exact files you download. Write the model name, version and licence into your project's README, and read it again whenever you upgrade.
- Is commercial use allowed, and under what conditions?
- Which uses does the acceptable use policy forbid?
- Must the product name, documentation or output carry an attribution?
- May you use the model's output to train other models?
- Can you redistribute a fine-tuned version, and under which terms?
Running a model locally with Ollama
Ollama is a free tool for macOS, Windows and Linux that downloads models, runs them and serves them over HTTP. After installing it, ollama pull followed by a tag from its model library downloads a model, ollama run with the same tag opens a quick chat in the terminal, and ollama list shows what you have.
The useful part for this course is that Ollama also exposes an OpenAI-compatible API at localhost:11434/v1. Point LLM_BASE_URL there, set LLM_API_KEY to any placeholder text and LLM_MODEL to the tag you pulled. Every Study Buddy script you wrote keeps working unchanged.
Memory decides what you can run. A rough rule: a model needs about its parameter count times the bytes per parameter, plus room for the context. Ollama models are usually quantised to around 4 bits, so a 7-billion-parameter model fits in a few gigabytes, while a 70-billion one needs a workstation. Lesson 20 explains quantisation in detail.
Watch the context length too. Ollama may run a model with a shorter context window than the model supports, so if long RAG prompts seem to lose their beginning, raise the context setting in Ollama's options.
Ollama is not the only option. LM Studio offers a desktop app with a local server, llama.cpp is the lightweight engine underneath many such tools, and vLLM is built for serving open models to many users on GPU servers. Most of them speak the same OpenAI-compatible API.
When a local or open-weight model is the right call
Open weights are a strong choice when data must not leave your machine or your country, when the app has to work offline, when you want a version that never changes under you, or when you plan to fine-tune. Steady high volume can also be cheaper on your own hardware, though you pay for that hardware, power and maintenance.
The trade-offs are real. The largest closed models often still lead on hard reasoning, you are responsible for updates and security of the server, and safety filtering that a provider would apply is now your job.
There is a middle path. Many hosting providers and routers such as OpenRouter serve open-weight models through an OpenAI-compatible API, so you can test a family in the cloud before buying hardware, or keep the model choice open while someone else runs the servers.
Example
Call a local model through the OpenAI SDK
import os, timefrom openai import OpenAI # For Ollama on your machine:# LLM_BASE_URL=http://localhost:11434/v1# LLM_API_KEY=ollama (any non-empty text; Ollama ignores it)# LLM_MODEL=the tag you pulled, exactly as `ollama list` shows itclient = OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"]) print("Models on this server:", [m.id for m in client.models.list().data]) start = time.perf_counter()reply = client.chat.completions.create( model=os.environ["LLM_MODEL"], messages=[ {"role": "system", "content": "You are Study Buddy. Answer in two short sentences."}, {"role": "user", "content": "What is the difference between weather and climate?"}, ],)print(reply.choices[0].message.content)print(f"{time.perf_counter() - start:.1f} s on {os.environ['LLM_MODEL']}")The same script works against any OpenAI-compatible endpoint, so only the environment variables decide whether it runs locally or in the cloud. The first call after starting Ollama is slower because the model has to load into memory.
Practice
Run Study Buddy on your own machine
- Install Ollama, pull a small instruct model from its library and test it once with
ollama runin the terminal. - Run
ollama listand copy the exact tag, then set the three environment variables for the local server in a separate file such as.env.local. - Run the example and check that your model appears in the printed list and answers the question.
- Run your lesson 15 eval script against the local model and then against your usual cloud model, keeping both logs.
- Turn off your internet connection and ask Study Buddy one question from your notes with the local settings loaded.
Done when: Study Buddy answers with the network switched off, and you have two eval logs that show the pass count and seconds per answer for the local and the cloud model.
Common mistakes
- Assuming every downloadable model may be used commercially without reading its licence.
- Pulling a model too large for your memory and blaming the code when it crawls or crashes.
- Changing the code to talk to a local model when only the three environment variables needed to change.
- Judging a local model by one chat instead of running the same eval set you used for the cloud model.
Check yourself
Pick an answer to see whether it is right and why.
From the founder
Run the same eval set on the local and the cloud model before you form an opinion about either. Then write the model name, version and licence into your Study Buddy README.