Using generative AI responsibly

What can go wrong

Harms from generative AI rarely look dramatic in development. They show up as a wrong date given to a student, a stereotype in an example, or a private detail repeated back to the wrong person. Naming the categories in advance makes them easier to spot.

For a study app the most likely harm is the quiet one: a confident wrong fact that a learner memorises. The model will not flag it, so your design has to.

Harms also depend on who your users are. A tool for adults studying for a professional exam carries different risks from one used by twelve-year-olds, and the same answer can be fine for one group and inappropriate for the other. Write down your audience before you write any safeguard.

• Hallucination: invented facts, sources or quotes presented as true.
• Harmful content: instructions for hurting people, hateful text, sexual content where it does not belong.
• Bias: answers that treat groups unequally or repeat stereotypes from the training data.
• Privacy: personal data sent to a provider, stored in logs or leaked in answers.
• Misuse: people using your app for cheating, spam or deception.
• Over-reliance: users trusting the tool more than its accuracy deserves.

Measure: test for harm before users find it

In ordinary software you test what users are expected to do. Here you also test what they might do. Write a list of prompts that probe each harm: factual questions whose answers you can check, questions about things that do not exist, requests for personal details, requests that should be refused and questions about different groups of people.

Include the messy reality of real users: typos, slang, other languages and very long or very short inputs. A safeguard that only works for tidy English questions will fail on the first day.

Read the answers yourself and give each one a verdict. Automated scorers help later, but at this stage a human reading every row catches problems that no checklist anticipated. Keep the list and rerun it after every prompt or model change.

Give each prompt a category label so you can see patterns. If most failures sit in one category, such as questions about events that never happened, you know where to put your effort first instead of tweaking everything at once.

A small, honest list beats a large one you never finish. Twelve well-chosen prompts reviewed with care are a real start; you can grow the set every time a user reports a problem, turning each report into a permanent test.

Mitigate in layers

No single safeguard is enough, because each one misses some cases. Combine several, so that what slips through one layer is caught by the next.

Start with the model and platform. Choose a model whose provider runs safety systems you trust, and use a moderation endpoint or content filter where your provider offers one. A smaller model limited to a narrow task also gives fewer surprises than a general one.

Then shape behaviour with the system prompt: state the app's purpose, what it must refuse and that it should admit uncertainty. Ground factual answers in text you supply, such as the learner's own notes, instead of the model's memory.

Finally use the interface. Limit input length, show the sources an answer came from, label generated text clearly and give users a one-click way to report a bad answer. Interface choices often prevent more harm than clever prompts.

Check outputs as well as inputs. Simple code can catch some problems after the model replies, for example an answer that contains an email address, a phone number or a phrase from a blocklist, and replace it with a polite fallback before the user sees it.

Human review and disclosure

Where output feeds a decision with real consequences, a person should check it before it takes effect. In education that means grades, feedback on assessed work and anything about a student's wellbeing; in other fields it covers health, legal and financial advice.

Tell people when they are talking to an AI and when content was generated. Say what the app stores, for how long and who can see it. Some laws now include disclosure duties for certain AI systems, so check the rules where your users live.

Honest disclosure also sets expectations. A short line such as Study Buddy can make mistakes; check important facts in your textbook reduces over-reliance more effectively than a long policy nobody reads.

Make it easy to reach a person. A visible way to contact a teacher, a moderator or the app's owner turns a bad answer from a dead end into something that can be corrected, and it gives you the reports you need to improve the test set.

Operate: keep watching after launch

Responsible use continues once real users arrive. Log enough to investigate problems, such as prompt version, model name and a user's report, without storing personal data you do not need. Decide in advance how long logs are kept.

Plan for incidents before they happen. Know how to switch a feature off, roll back a prompt and contact affected users. Add rate limits so one person cannot turn your app into a spam machine.

Send providers only the data a request needs, and read their terms on retention and training use. For minors or sensitive subjects, prefer a provider or a self-hosted model whose data handling you can explain to a parent in one sentence.

Review your safety list on a schedule, not only after incidents. New features, new user groups and model updates all change the risks, and a quarterly rerun of the same prompts is a cheap way to notice drift before your users do.