Keep decisions clear
Human in the loop AI: decide what the system does and what a person checks
Human in the loop AI means a person participates in a process that uses AI. In client work, that may mean approving a draft, resolving contradictory information or deciding whether an action should happen. The useful question is where the person takes part, what they can inspect and what decision they are responsible for.
A review step is useful when a person has enough evidence, authority and time to make the decision the workflow needs.
Choose the kind of human involvement you need
The term is also used for people labelling data or helping train a model. This guide focuses on people participating while a business workflow runs. Training-time involvement and approval of today’s client email are different activities.
| Pattern | When the person acts | What the workflow needs |
|---|---|---|
| Approval before action | A person checks the output before it is sent or executed. | A clear hold state and an authorised release. |
| Exception handling | A person takes over when an input is missing or the situation falls outside agreed rules. | A recognisable exception and enough context to resolve it. |
| Review after bounded action | A person samples or checks actions already taken within agreed limits. | A record of what happened, appropriate limits and a way to stop or correct the process. |
Choose a pattern for a particular action. A system might collect permitted records independently while a person still decides whether to make a promise to a client. One approval rule for “the AI” is too vague to describe that split.
Use a real approval point as a starting example
Our journalist-request system sorts incoming requests, matches them to clients and prepares drafts. At launch, people approved every outgoing message. Automated sorting and drafting did not give the system permission to send those messages on its own.
That split protects a business decision: whether this response is appropriate for this client and opportunity. It also means the review queue is part of the delivery process. If nobody has time to review it, more drafts do not produce more useful replies.
Read the journalist-request case study
When estimating the return for your own workflow, include review time, exceptions and corrections. You may release capacity, but that becomes a realised benefit only when the business can use it.
Give the person something they can actually check
Said describes a rule in our current knowledge build: an answer must carry evidence with a traceable or deterministic basis that a person can query. If the system cannot supply that basis, it is forced to say it does not know.
Records also preserve origin and validity information. When sources disagree, those details help a person see whether one statement is older, concerns a different period or needs confirmation. A confidence score can help direct attention; it does not settle the business decision by itself.
Design the review view around that job. Show the proposed answer or action, the relevant source passages, the dates and any unresolved disagreement. Asking someone to approve a fluent paragraph while hiding those details leaves them doing detective work.
Make the route through review explicit
System
Prepare an answer from permitted records
Preserve the origin and dates of the records used.
Software checks
Check the evidence package
An absent required source sends the work to the unknown route.
Named reviewer
Resolve uncertainty or approve the answer
Inspect disagreement and decide what the business can say or do.
Workflow
Record the accepted decision
Keep the approved version and return the decision to the business information used later.
This is a teaching design. Recording reviewed information for later use is different from automatically training a model.
Say who can approve and what approval covers. Permission to release a report should not accidentally become permission to change a campaign budget. If the reviewer changes a draft, keep the accepted version rather than continuing with the earlier one.
Write down the responsibility map
Worked example
A request for a client-facing answer
Invented teaching example. No client outcome is claimed.
PREPARATION
The system retrieves the client’s current permitted records and drafts a response.
EVIDENCE
Each factual part carries the source needed for review.
The relevant validity and collection dates remain visible.
DECISION
A reviewer resolves a disagreement before the response is released.
Missing required evidence produces an unknown and a next question.
AFTER REVIEW
The accepted answer and the business decision are recorded.
A rejected draft is not sent while the correction is being prepared.
Test the human part as carefully as the model
Give a reviewer a normal answer, an answer with an irrelevant citation and a conflict between sources. Watch whether they can tell the difference and find the records without leaving the workflow to search through several tools.
Check the awkward operational cases too: someone is away, an approval is delayed, or the same request is submitted twice. Decide what should happen before opening the workflow to more users.
Further reading
Written by Said Lopez, founder of LeverageAI and author of Lean AI. Our guides draw on the method in the book and our work with clients.