The rule, and why it is drawn there #
What those three have in common is not risk in the abstract. It is that the damage happens at the moment of the action and cannot be undone by noticing afterwards. A wrong draft sitting in a queue costs nothing. The same draft in a client's inbox has already done whatever it was going to do, and the rest is apology.
Everything before that point is reversible by definition. Reading, finding, matching, sorting, extracting, summarising, preparing. Those can run unattended all night, and for most businesses that is where the hours actually are, which is the point made in which jobs are worth automating.
This is our own rule, not a general disclaimer #
We hold this line in the work we do, and we say it early because it changes what a project looks like. It means we will not build the thing that emails your customers on its own, however confident anyone is about the quality of the drafts, and however much more impressive the demo would be.
The reason is in how these systems fail. They produce plausible output whether or not it is correct, so nothing in the artefact tells you which kind you have. That is set out with the regulator's own wording in what happens when AI gets it wrong. A rule that depends on spotting bad output is not a rule. A rule that puts a person in front of the irreversible step works whether or not anybody spots anything.
Where the line falls in practice #
Runs unattended
- Reading the inbox and sorting what arrived
- Pulling the figures together and preparing an invoice
- Spotting that a certificate or deadline is due
- Drafting the note, the summary or the letter
- Flagging records that look duplicated or stale
- Ranking a list by whatever rule you agreed
Waits for a person
- Replying to any of it
- Issuing it, and anything that takes payment
- Telling the customer their appointment has moved
- Sending it, filing it as the record, or acting on it
- Deleting or merging any of them
- Rejecting a person, or telling them they were rejected
The last line on each side is a different kind of case from the rest, and worth separating out.
The category the law treats separately #
Most of this page is judgement. One part of it is not. Where a decision is based solely on automated processing and produces legal effects or similarly significantly affects someone, UK GDPR gives the individual the right not to be subject to it, permits it only in narrow circumstances, and requires safeguards.
The ICO describes those safeguards as including the ability to obtain human intervention, express a point of view, and obtain an explanation of the decision and challenge it. A data protection impact assessment is expected as well.
In a business this catches fewer things than people fear and more than they expect. Deciding who gets an interview, who is declined, what someone is charged, or whether an application proceeds all have the right shape. Drafting a reply about any of those does not. Whether your own process is caught is a judgement your business has to make and be able to show, and the assessment question is covered in do you need a DPIA.
A check that people actually perform #
Four things keep a check real.
-
Show the working
The draft next to the record it came from, so checking means comparing rather than trusting.
-
Keep the volume readable
If the only way through the queue is bulk approval, the design is wrong, not the person.
-
Make rejection cheap
If sending something back is fiddly and approving is one click, you have built a machine that produces approvals.
-
Give it to someone by name
Shared responsibility for a check reliably becomes nobody's.
One more, less obvious: watch the rejection rate. A queue where nothing is ever sent back is either genuinely excellent or no longer being read, and those two look identical from the outside. Knowing which is worth a deliberate spot check now and then.
What this costs you, honestly #
It costs the fantasy version, where the business runs itself overnight and everyone arrives to find the work done. That version is what gets sold and it is not what gets delivered well.
What it does not cost is the benefit. The expensive part of most repeated admin is the gathering: opening five systems, finding the right record, working out what stage something is at, assembling the facts. Approving prepared work is a fundamentally different task from producing it, even though both end in a click. The cost of admin time calculator counts the hours currently going into that gathering, which is the number worth knowing before anyone designs anything.