Building a text generation app

From prompt to program

A text generation app takes some input, wraps it in a prompt your code controls, sends it to a model and does something useful with the reply. Summarisers, quiz makers, rewriting tools and email drafters all follow this shape.

Unlike a command-line tool with fixed commands, the input can be natural language in any phrasing. That flexibility is the appeal, and it is also the risk: your code has to cope with replies that are long, short, oddly formatted or occasionally nonsense.

The work splits cleanly. The model writes text; your code owns the template, checks the input, sets the parameters, validates the output and decides what the user sees when something fails. Most of the reliability of an AI app lives in that ordinary code.

You can build all of this with the plain openai SDK. Frameworks such as LangChain or LlamaIndex add abstractions on top, which help in bigger projects but hide the mechanics. Learn the direct calls first; the frameworks will make more sense afterwards.

Parameters that matter

Every request names a model and a list of messages. Beyond those, a few parameters shape the reply, and setting them deliberately is part of designing the feature.

Pick values per feature and write down why. A quiz maker wants enough variety that repeated runs give fresh questions, so a moderate temperature suits it, while a feature that marks answers should run cold. The length cap should fit the largest valid reply with a margin, not the smallest.

Some providers accept extra options such as n for several alternative replies in one call or seed for more repeatable sampling. They are not universal, so keep the core of your app working without them and treat them as optional improvements.

• temperature: low for predictable, factual output; higher for variety.
• max_tokens: a cap on the length of the reply, which also caps its cost.
• stop: one or more strings that end generation when the model writes them.
• timeout and max_retries on the client: how long to wait and how often the SDK retries.

Reading the reply properly

Check response.choices[0].finish_reason when output looks wrong. A value of stop means the model ended normally; length means it hit your token cap mid-answer. A truncated JSON object is not a model bug, it is a limit set too low.

Parameter support varies. Some reasoning models reject temperature or expect max_completion_tokens instead of max_tokens, and some providers ignore options they do not implement. When a request fails on a parameter, read the error message; it usually names the argument it dislikes.

Record the token usage of each call while you develop. It tells you what a feature costs per use and warns you early if a template or a history has grown much longer than you meant.

Treat empty content as a real possibility. A reply can come back with no text, for example after a safety filter steps in, so write content or "" before calling string methods, as the example does, and decide what the user should see in that case.

Templates with variables

Keep the fixed part of a prompt in a template and insert user input into placeholders. Python's str.format works well for this, but literal braces in the template, such as a JSON example, must be doubled as {{ and }} so they are not treated as placeholders.

Validate input before it reaches the template. Trim whitespace, cap the length, and restrict values that should come from a fixed list, such as the number of questions or a difficulty level. The model should never receive a request for ten thousand questions because a user typed an extra zero.

Place user text where it is clearly data, inside quotes or a labelled block, and keep your instructions outside it. That will not stop every attack, but it is the first defence against users who type instructions instead of a topic, a problem lesson 14 covers in depth.

Templates make behaviour testable. Because the wording is fixed, you can run the same template against your test set whenever you change the model or the parameters and compare results directly.

Structured output and failures

When code consumes the answer, ask for JSON with exactly the keys you need and show their shape. Many providers also offer a JSON mode, set with response_format={"type": "json_object"}, and some support a strict JSON Schema option. Support differs between services, so test it on yours.

JSON mode guarantees valid syntax at best, not correct content. Always parse with json.loads and check what your code depends on: required keys, list lengths, value ranges. Models also like to wrap JSON in Markdown fences, so strip those before parsing.

When validation fails, retry once with a short note explaining the problem, then give up cleanly with a message the user understands. Endless retries waste tokens and hide a prompt that needs fixing.

Network failures are a separate matter. The SDK already retries connection errors, rate limits and server errors with backoff; you control how many times with max_retries. Do not retry authentication or bad-request errors, because they will fail the same way every time; catch them and show a clear message instead.

Set a timeout that matches the feature. A quiz can wait a minute, but an autocomplete suggestion that arrives after ten seconds is useless, and a short timeout with a polite fallback gives a better experience than a spinner that never stops.