These systems fail differently from software #
Ordinary software fails loudly. A field is empty, a total does not add up, a job stops and somebody gets an error. You find out because the failure announces itself.
A language model does not have that property. It produces the most plausible continuation of what it was given, and a wrong answer is manufactured by exactly the same process as a right one. The result is fluent, correctly formatted, appropriately detailed, and sometimes entirely invented.
The ICO is unusually direct about the consequence. It describes statistical accuracy as how often a system guesses the correct answer against correctly labelled test data, and says plainly that a system does not need to be one hundred per cent statistically accurate to comply with the accuracy principle. It advises that records should show outputs for what they are.
statistically informed guesses rather than facts
A regulator telling you to write down that the output is a guess is a regulator describing the failure mode accurately.
The four ways it actually goes wrong in a business #
| Failure | What it looks like | How it usually surfaces |
|---|---|---|
| Invention | A detail that was never in the source: a date, a reference, a clause, a name | The recipient queries it, which means it has already gone out |
| Misreading | The right fact taken from the wrong record, or the wrong one of two similar records | Rarely. It is internally consistent, so it reads as correct to everyone |
| Confident omission | A summary that quietly drops the exception, the caveat or the one line that mattered | Later, when somebody relies on the summary rather than the original |
| Drift | Output that was fine for months and is now subtly different in tone or shape | Only if someone is still looking, which by month six usually nobody is |
Notice that only the first of those is the failure people worry about, and it is the one most likely to be caught. The middle two are the expensive ones, because they produce work that passes a glance.
Who carries the consequence #
Three things are worth being blunt about.
-
A message sent in your name is yours
Your client, your customer, your regulator. The chain of how it was produced is of no interest to the person who received it.
-
Professional and statutory duties do not move
Supervision, record keeping and data protection obligations sit with the business and its people. A supplier arrangement allocates work, not accountability.
-
Some decisions have their own rules
Where a decision is made solely by automated means and has a legal or similarly significant effect on someone, UK GDPR restricts it outright, permits it only in defined circumstances, and requires safeguards including the ability for the person to obtain human intervention, express a view and challenge the outcome. Screening, scoring, pricing and declining people are the shapes to watch for.
When a mistake becomes a reportable one #
Most errors are just errors: an awkward email, a correction, a slightly embarrassing phone call. A smaller set is different, because personal data went somewhere it should not have. That is a personal data breach whether the cause was a person, a spreadsheet or a model.
The ICO's position is that you assess the likely risk to people's rights and freedoms, and where that risk is likely you notify as soon as possible and where feasible within 72 hours. Where the risk to people is high, they have to be told without undue delay too. The ICO explicitly advises reporting early and updating later rather than waiting until you have the full picture.
The practical consequence for a business is that the 72 hours starts when you become aware, so a process that only notices problems at the end of the month is not a process that can meet it.
What reduces this, without pretending to eliminate it #
Nobody can promise a system that does not make mistakes. What is available is a system whose mistakes are cheap.
-
Keep the source next to the output
A draft that arrives beside the record it was made from can be checked in seconds. One that arrives alone has to be trusted, and it will be.
-
Make the check a real job with a name against it
A review step that belongs to everybody belongs to nobody, and becomes a formality by week three.
-
Log what went in and what came out
Without that you cannot answer the only question that matters after something goes wrong, which is how many others were affected.
-
Know what an error costs before you choose where to use it
Drafting an internal note and issuing a statutory notice are different risk propositions, and the same underlying technology is appropriate for one and not the other.
The part that tells you where to start #
If you are deciding whether to automate a job at all, the consequence of an error is one of the two questions that settles it, alongside how repeatable the job is. Both are covered in which jobs are worth automating, and the regulator's wider position is summarised in what the ICO actually says about AI.
The honest summary is that these systems are useful and unreliable at the same time, and that combination is workable as long as nobody designs around the first half and forgets the second.