Building chat applications

The model is stateless, your app is not

A chat completion endpoint has no memory of earlier requests. Each call receives a list of messages, produces one new message and then forgets the whole exchange. What feels like a conversation is your program sending the growing transcript again every time the learner presses Enter.

Every message in that list carries a role. The system message sets the rules for the whole session and usually sits first. User messages hold what the person typed. Assistant messages hold what the model answered earlier, and you add them back yourself so the model can see its own previous words.

Later in the course you will meet a fourth role, tool, which carries the result of a function your code ran. For now three roles are enough to build a working tutor.

Because you own the transcript, you also own its risks. Whatever you store can be read by anyone who gets access to your logs or database, so treat chat history like any other personal data. Give users a visible way to clear it and keep it only as long as the product really needs.

Designing a system prompt that holds up

The system prompt is the one message the learner never sees, and it shapes every reply. A vague line such as asking the model to be helpful gives you a generic chatbot. A short, specific brief gives you a product with a recognisable voice and clear limits.

Write it as a job description, not a wish list. Order matters less than clarity, but most good system prompts answer the same handful of questions.

Test the prompt against awkward inputs, not only friendly ones. Ask Study Buddy something off-topic, something it cannot know, and something rude, then read how it copes. Edit the system prompt, rerun the same inputs and compare.

Keep the prompt in its own file or constant and treat it like code. When a change makes answers worse, you want to see exactly which line you edited and roll it back in seconds.

• Who the assistant is and who it serves, for example a tutor for one adult learner
• What it should do and what is out of scope, such as writing graded homework for the learner
• The shape of a good answer: length, steps, a check question at the end
• What to do when unsure: say so, ask a clarifying question, or suggest where to check
• Tone and language, including whether to answer in the language the learner used
• One or two short examples of an ideal reply, if the format is unusual

Keeping the history inside the context window

Every model has a context window, a maximum number of tokens it can read in one request. Resending the full transcript works for a few dozen turns, then the request grows slow, expensive and eventually too large to accept. A chat app needs a policy for what to drop.

The simplest policy keeps only the last few exchanges. It is predictable and cheap, and it suits a tutor where the current topic matters most. Trim whole pairs, so the list after the system message still begins with a user turn; some providers reject a history that opens with an assistant message.

A token budget is a finer tool. Count the tokens in the history, using your provider's tokenizer or a rough estimate of four characters per token for English, and remove the oldest pairs until you are under the limit you chose.

For long sessions you can summarise instead of deleting. When the history gets big, ask the model to compress the oldest turns into a short note of facts and decisions, then keep that note as an extra system message. You lose detail but keep the thread.

Some facts must survive any trimming: the learner's name, level and current goal. Store them separately and insert them into the system prompt on each call rather than hoping they stay in the transcript.

Streaming replies

Without streaming, the learner stares at an empty screen until the whole answer is ready. With stream=True the SDK returns an iterator of small chunks, and each chunk holds a delta with a few new characters. Printing them as they arrive makes even a slow model feel responsive.

Streaming changes your bookkeeping. You must join the pieces into one string yourself, and only append that string to the history once the stream ends. Guard against chunks with no choices or an empty delta, because providers differ in what they send at the start and the end.

A stream can also break halfway, for example when the network drops. Decide what the user sees in that case: keep the partial text with a clear note that it was cut off, and offer a retry button that resends the same history.

Small details that make a chat feel trustworthy

A good chat interface lets the user steer. Offer a new chat button that clears the history, a way to regenerate the last answer and an easy way to copy text. Let people ask follow-up questions such as asking for a simpler explanation without retyping the topic.

Accessibility is part of quality, not an extra. Make sure the chat works with the keyboard alone, that new messages are announced to screen readers, and that text can be resized without breaking the layout.

Once real people use it, watch a few signals: how long replies take, how often calls fail, and how often users press a thumbs-down or rephrase the same question. These numbers tell you where to improve the prompt or the trimming policy long before anyone writes a complaint.

Ambiguous questions deserve a question back. If the learner types only the word photosynthesis, a tutor that asks whether they want a definition, the chemical equation or a quiz is more useful than one that guesses and writes five paragraphs.