Skip to content
  • Official
  • Beginner
  • English
  • 9 steps · 2 quizzes

Spotting AI Hallucinations: A Fact-Checking Workshop

AI hallucination training for teams: check a support reply, sales brief, policy summary and citations claim by claim, then build your team's checklist.

  • Free
  • No account needed
  • Stop any time, pick up where you left off
Workshopy.io

WorkshopyOfficial

78 lessons · used by 2 teachers

For teachers

Teaching this? Run it live with your class.

  • Students join with a code — no accounts
  • See who’s done and who’s stuck, step by step
  • It’s your copy — change anything before class

Sign in to copy it to your workspace. It’s free.

Your path

9 steps · 2 quizzes · open any step, or read it here

  1. Spotting AI hallucinations: what this workshop practisesStart hereIn this fact-checking workshop you practise spotting AI hallucinations: statements in an AI answer that sound sure but that no source supports.InteractiveHands-on
    Read this step

    In this fact-checking workshop you practise spotting AI hallucinations: statements in an AI answer that sound sure but that no source supports. It is AI hallucination training for everyday work, so you check real-looking drafts (a support reply, a sales brief, a policy summary, a report's citations) one claim at a time.

    Here is the whole idea in one draft. Watch the checker go through it claim by claim.

    Interactive · One AI draft, checked claim by claim — try it in the lesson

    In this lesson you'll:

    • see why a model writes things that aren't true, and why it sounds just as sure when it does,
    • find the claims in an answer that need checking: names, numbers, dates, quotes, policies, links,
    • check each claim against a source and call it supported, contradicted or missing,
    • check citations, decide how much checking a piece of work needs, and build your team's checklist.

    It takes about an hour. No AI account is needed: every AI answer here is a written example.

    Takeaway: a draft can be fluent from start to finish and still be wrong in the middle. Checking is done claim by claim.

  2. Why models invent: likely words, not checked factsA language model writes by predicting the next word, again and again. It picks words that are likely after the words so far. Nothing in that step looks the fact up.InteractiveHands-on
    Read this step

    A language model writes by predicting the next word, again and again. It picks words that are likely after the words so far. Nothing in that step looks the fact up.

    When the model has seen a fact many times in its training text, the likely word and the true word are usually the same. When it has never seen the fact (your company's support hours, a client's revenue, last week's meeting), many words are about equally likely, and it still writes one of them, in the same confident sentence.

    That is what people call a hallucination: not a glitch, but the normal way of writing, applied to something the model doesn't know.

    Try it: pick each prompt and press Generate five times. Watch how many different answers you get, and how sure each one sounds.

    Interactive · Generate the same sentence again and again — try it in the lesson

    What changed between the two prompts: the bars, not the sentences. Both sentences read as facts. You can't hear the difference; you can only check.

    Models trained to answer helpfully tend to answer rather than say "I don't know", and newer tools that search the web or read your files make fewer of these slips. Fewer is not none, so the habits in this lesson apply to every tool.

    Takeaway: a model writes what is likely, and it sounds the same whether it knows or not. The facts it is least likely to know are your company's own.

  3. Where hallucinations hide: names, numbers, policies and linksMost of an AI answer is safe to read: greetings, structure, ordinary advice. The risky parts are short and specific: a number (45 days, $15), a name (a person, a product, a policy), a date, a quote, a link, a rule ("you can…", "we never…").InteractiveHands-on
    Read this step

    Most of an AI answer is safe to read: greetings, structure, ordinary advice. The risky parts are short and specific: a number (45 days, $15), a name (a person, a product, a policy), a date, a quote, a link, a rule ("you can…", "we never…").

    These are the parts a reader acts on, and they are the parts a model fills in when it doesn't know. So the first habit is to find them before you send anything.

    Try it: this reply was drafted by an assistant for a support agent (it's invented). Tap each part a customer could act on. There are four.

    Interactive · Mark the claims in a support reply — try it in the lesson

    Notice what you didn't mark: the friendly opening and closing. Those are the parts people usually polish, and the parts that never need a source.

    Takeaway: before you check anything, list what can be checked: names, numbers, dates, quotes, links and rules.

  4. Break an answer into claims you can checkA paragraph is hard to check. A list of single claims is easy: each one is true or not, and each one has a place where you can look. So the second habit is to split the answer. One claim per line, its kind, and where you would check it.InteractiveHands-on
    Read this step

    A paragraph is hard to check. A list of single claims is easy: each one is true or not, and each one has a place where you can look.

    So the second habit is to split the answer. One claim per line, its kind, and where you would check it. Some lines turn out to be opinion or filler: nothing to check, but nothing to rely on either.

    Try it: here is a sales brief an assistant wrote about a prospect (company and people invented). Press Split into claims, then open each card to see where you would check it.

    Interactive · Split a sales brief into claims — try it in the lesson

    A brief like this is where invented details do the most damage: a salesperson who opens a call with the wrong CEO's name or a quote nobody said loses the room in a sentence.

    You don't have to do the split by hand every time. You can ask the assistant to "list every factual claim in this answer, one per line". Then you check the list, not the prose.

    Takeaway: split an answer into single claims, note the kind of each, and name where you'd check it. Opinions have nothing to check and nothing to rely on.

  5. Check: why models invent and what to checkQuiz · 4 questions
    Read the questions

    Question 1. Why can a language model state something false in a fluent, confident sentence?

    • It lies on purpose when it doesn't like the question
    • It writes likely next words, and nothing in that step looks the fact up
    • It copies the first website it finds, right or wrong
    • It only happens when the model is overloaded

    Question 2. An assistant drafts: "Thanks for asking. Our Premium plan includes 24/7 phone support and a 99.9% uptime guarantee." Which part most needs checking?

    • "Thanks for asking"
    • "24/7 phone support and a 99.9% uptime guarantee"
    • The friendly tone
    • The word "Our"

    Question 3. An answer is detailed and sounds certain. What does that tell you about whether it's right?

    • It is probably right: models hedge when they are unsure
    • It is probably wrong: detail is a sign of invention
    • Nothing: a model sounds the same whether it is right or wrong

    Question 4. A brief says the prospect's CEO "said in a 2023 interview that fuel is our biggest cost". What is the best check?

    • Ask the assistant where it found the quote
    • Trust it, because it is in quotation marks
    • Find the interview itself and the sentence in it
    • Search the quote and accept any page that repeats it
  6. Compare with the source: supported, contradicted or missingOnce you have a claim, you need a source that is not the AI: the policy itself, the price list, the contract, the database, the person who owns the fact. Read the claim against the source and give it one of three verdicts: Supported: the source says it.InteractiveHands-on
    Read this step

    Once you have a claim, you need a source that is not the AI: the policy itself, the price list, the contract, the database, the person who owns the fact.

    Read the claim against the source and give it one of three verdicts:

    • Supported: the source says it.
    • Contradicted: the source says something else. Fix it.
    • Not in the source: the source doesn't say it either way. Cut it, or ask whoever owns the fact. Never keep it because it sounds reasonable.

    The third verdict is the one people miss. A missing claim feels like a small extra, but it is exactly what a model adds when it fills a gap.

    Try it: an assistant summarised a travel policy for new hires (both invented). Judge each line of the summary; the policy line it is about lights up when you answer.

    Interactive · Judge a policy summary against the policy — try it in the lesson

    Look at line 3. It had the right number and a wrong detail attached. Partly-true claims are the hardest to catch, because the part you check first is the part that's right. Read the whole source line, not just the number.

    Takeaway: every claim gets a verdict against a source that isn't the AI. "Not in the source" means cut it or ask, never keep it.

  7. Citations: does the source exist, and does it say that?A citation looks like proof, which is why invented ones are dangerous. A model can produce an author, a title, a year and a journal that fit together perfectly and don't exist. This happened in a real court case. In Mata v.InteractiveHands-on
    Read this step

    A citation looks like proof, which is why invented ones are dangerous. A model can produce an author, a title, a year and a journal that fit together perfectly and don't exist.

    This happened in a real court case. In Mata v. Avianca (New York, 2023), lawyers filed a brief citing court decisions that ChatGPT had made up. One lawyer had asked ChatGPT whether one of the cases was real; it said yes. In June 2023 the judge fined the lawyers and their firm 5,000 dollars, and wrote that using AI is not wrong in itself, but that the lawyers are responsible for what they file.

    The lesson for any report: asking the same assistant "is this real?" is not a check. It writes a likely answer to that question too. Check outside it, three ways:

    1. It exists: you find it in a catalog, database or site you trust, not only in the AI's answer.
    2. It says that: you open it and find the sentence or number the report relies on.
    3. The details match: author, year and title are the ones in the reference.

    Try it: a report on remote work cites four sources (all invented for this exercise, including the catalog). For each, ask the AI and search the catalog, then give your verdict.

    Interactive · Check the citations in a report — try it in the lesson

    The "doesn't say that" kind is the most common in real reports, and the hardest: the source is real, so it passes a quick look. Only opening it catches the gap between what it found and what the sentence claims.

    Takeaway: a citation is checked when you have found it outside the AI, opened it, and seen it say what the sentence says. "The AI says it's real" counts for nothing.

  8. Ask for answers you can checkYou can't stop a model from inventing, but you can ask in a way that makes inventions easier to catch: Give it the source. Paste the policy, the data or the document and say "use only this".InteractiveHands-on
    Read this step

    You can't stop a model from inventing, but you can ask in a way that makes inventions easier to catch:

    • Give it the source. Paste the policy, the data or the document and say "use only this". The answer then has something to agree with, and you have something to compare against.
    • Ask for the line behind each claim. "After each point, give the section number it comes from." A claim without a reference stands out.
    • Allow "I don't know". "If the text doesn't say, write that it doesn't say." Without that line, a model tends to fill the gap.

    None of these makes an answer true. A model can still misread the text, or quote a section that doesn't say what it claims. They make checking faster, because each claim now points at the place to check.

    Try it: turn the three switches on one at a time and watch the prompt and the answer change. The policy is the travel policy from step 5.

    Interactive · Turn a prompt into one you can check — try it in the lesson

    Notice the taxi line. With the policy pasted, the answer got the numbers right but kept the taxi claim. Only the "allow it doesn't say" line turned it into an honest gap.

    Takeaway: give the model the source, ask for the reference behind each point, and allow "it doesn't say". Then still open the references.

  9. Match the checking to the riskYou can't check every word of every AI answer, and you don't need to. Two questions decide how much checking a piece of work needs: Does it leave the team? A customer, a client, the public or senior leadership will read it and act on it.InteractiveHands-on
    Read this step

    You can't check every word of every AI answer, and you don't need to. Two questions decide how much checking a piece of work needs:

    1. Does it leave the team? A customer, a client, the public or senior leadership will read it and act on it.
    2. Could a mistake cost something? Money, someone's safety, a legal or compliance problem, a broken promise.

    No to both: read it once. Yes to one: check every fact in it against a source. Yes to both: check every claim, and a second person reads it before it goes out.

    Try it: choose the level for each piece of work.

    Interactive · How much checking does it need? — try it in the lesson

    The rule is yours to change. A regulated team may put everything that touches a customer at the top level; a design team may put most drafts at the bottom. What matters is that the team agrees on it, so nobody has to guess.

    Takeaway: two questions (does it leave the team, could a mistake cost) decide how much checking a piece of work gets.

  10. Your team's fact-checking checklistHabits stick when they are written down where the work happens. Build a short checklist from what you practised: pick the checks your team needs, add one of your own, and keep it beside the tools you use.InteractiveHands-on
    Read this step

    Habits stick when they are written down where the work happens. Build a short checklist from what you practised: pick the checks your team needs, add one of your own, and keep it beside the tools you use.

    Try it: tap checks to add them, add one of your own, then press Show as text to copy the list into your team's wiki or chat.

    Interactive · Build your team's fact-checking checklist — try it in the lesson

    A good team checklist is short (five to eight lines), uses your own words, and names your own sources: "prices come from the price list", "policy answers link to the policy page". Agree on it together, put it where people draft, and look at it again after the first mistake it didn't catch.

    Takeaway: the checklist turns this lesson into a habit. Keep it short, in your words, next to the tools.

Comments

You’ll sign in with Google before it’s posted.Comments are public — don’t name your students.
0 / 1,500

No comments yet. Be the first to say how it went.

Using it

Workshopy shares this lesson under CC BY 4.0. Anyone may use and change it, with credit to you. Anyone can learn it here, free. A copy in your workspace is yours to change and run live; your students never need an account.

Versions

  1. 1

    Version 1Current

    · First published

Copies are made from the current version. A copy you already have doesn’t change.

More from Workshopy

All 78 lessons
All 21 Data & AI lessons