How we stop AI making things up

During testing, our AI reviewed a RAMS document that it never received.

The submission had come in through a subcontractor submission page, the review ran, and back came a full verdict: non-compliant, every one of the 19 assessment criteria flagged as a critical gap, with citations to the Health and Safety at Work Act 1974, the Management Regulations 1999 and CDM 2015. It read like a thorough, damning safety audit. On a live project that verdict would have been written into the risk register and would have created follow-up actions for real people.

But the AI had never seen the document.

A fault earlier in the process meant the content never arrived. The model was handed an empty submission and, instead of saying so, it produced a plausible-looking answer anyway.

A safety audit of nothing, delivered with complete confidence.

We caught it in testing, worked out why it happened, and built the control that stops that failure now. That's the pattern behind everything in this article.

I want to walk through what we've learnt building PlanOps about why AI makes things up, and the fairly unglamorous engineering that stops those failures reaching a project record. Not because our situation is special, but because if you're putting AI anywhere near project information, these are the sorts of failures you're also likely to meet.

This isn't a model problem you can wait out

The comfortable and common assumption is that hallucination is mainly a maturity issue. Models are continually improving, so perhaps you wait a year and the problem largely goes away.

That's not a safe assumption.

It's not as simple that hallucination disappears as models improve. Language models generate responses from learned statistical patterns rather than retrieving a guaranteed set of facts. When the evidence they need is present, they can be extremely capable. When information is missing or ambiguous, they can still produce an answer that sounds entirely plausible.

Newer models are getting better and often hallucinate less often. But the remaining failures can still be remarkably convincing.

For a project record, "less often" just isn't good enough.

So the answer isn't simply a better model. The answer is process: engineering around the model so that situations where it could make things up either can't proceed or can't survive into a project record.

The industry already knows this move. We don't make site work safe by hiring people who never make mistakes. We use permits, hold points, inspections and test plans so that we do all we can to prevent mistakes and where they occur they are caught as early as possible.

The same thinking applies to AI.

The controls, one by one

Every AI task in PlanOps runs as a structured task rather than an open conversation.

That's the foundation the other controls bolt onto.

A structured task is a defined sequence of steps. Each step has a specific job, defined inputs and a defined output.

Fetch these documents. Extract these items. Check them against this standard. Write the result in this format.

The AI doesn't freewheel across an entire problem. It does one bounded thing at a time, and the process around it knows what each step was supposed to receive and produce.

That structure makes the controls possible. There are five of them:

1. Verify the evidence exists before judgement starts. No content, no verdict - the task stops and tells someone instead.
2. Restrict choices to things that actually exist. The model picks identifiers from a list the system supplies, and every choice is validated before it's used.
3. Use systems for facts and models for judgement. Lookups go to code. Only interpretation goes to the model.
4. Show the evidence boundary. Every result reports what it read - and what relevant material it didn't.
5. Validate the output and put people at the hold points. Automated checks on every output, and a human review before anything consequential becomes a record.

Each one below came out of a real failure. Here they are in turn.

1. Verify the evidence exists before judgement starts

The RAMS story above had a simple root cause.

The step that assembled the document content had a gap for one type of submission, so the review step received nothing. Nothing between those two steps checked that there was actually something to review.

The model filled the silence.

We fixed it by adding a content check gate.

Before any step that makes a judgement, the process checks that the information being assessed actually exists and is readable. Not simply "did the step run?" but "is there real content here?"

If the check fails, the task stops and reports why. Someone gets informed. No verdict is produced.

Once we knew what that failure looked like, we looked, found and fixed variations of it elsewhere:

• A document review task could run with no documents selected and generate queries out of thin air.
• A tender compliance review - one of the highest-consequence verdicts in the system - also had no equivalent gate.
• A drawing comparison could assess a drawing against a predecessor that didn't exist and produce a meaningless revision analysis on a first upload.

Each got the same treatment.

An assessment of nothing must be impossible.

Not unlikely. Impossible.

2. Restrict choices to things that actually exist

A construction platform is full of identifiers: document numbers, template references, classification codes, works package IDs and item IDs.

Ask a language model to provide one and sooner or later, it will produce something that looks perfectly valid but doesn't exist.

So we don't let the model be the source of an identifier.

When an AI task needs to select something - a template to apply, a works package to attach material components to, a classification to use - it happens in three steps:

1. The process fetches the real options from the system.
2. The model chooses from that list.
3. The process validates the choice against the same source.

If the model selects something that isn't there, the selection is rejected. Depending on the task, the process either falls back to a safe default or stops.

Classification codes got the same treatment. We found the model generating Uniclass codes as free text. A code that's nearly right can be worse than no code at all because it files information in the wrong place while looking perfectly tidy.

Now every code is checked against the published Uniclass tables. If it doesn't exist in the real table, it doesn't get stored.

We also have an additional check, the information from the model, is also checked for identifiers the model was never given.

If one appears, that's a flag it has been invented, and the output doesn't pass.

3. Use systems for facts and models for judgement

Some of our early hallucinations weren't really the model failing at its job.

They were there because we were giving models the wrong job.

We once asked the model to provide organisation logo URLs.

It produced plausible-looking URLs, and plenty of them were dead or wrong.

Knowing a URL is a lookup problem, not a reasoning problem.

The fix wasn't a better prompt. We replaced that step with a service that fetches the real logo.

The model kept the work that required interpretation and judgement. A traditional code system took the work that required a factual lookup.

The subtler version of the same mistake is asking for analysis the supplied evidence can't support.

An early monthly-report task asked the model to comment on progress against a previous month, but only supplied the current month's progress.

The model filled the gap with fluent, generic language about trends it had never seen.

On a client-facing document, that isn't acceptable.

The fix was to ensure that where we require an analysis, from factual information, that data is always provided.

The rule is simple: if you want the AI to talk about something, give it the evidence.

If you can't give it the evidence, don't let it make the claim.

We learnt another version of this lesson from our project chat assistant.

In one early test it answered a how-to question by referring the user to a project called Bluebell Fields.

There was no project called Bluebell Fields. It didn't exist in our code, our data or our configuration.

The model invented it on the first turn.

Because conversation history remains within a project, that fabrication then turned-up in later answers as though it were established fact.

That led to another rule:

The AI's previous answer is not evidence.

Conversation history can provide useful context. It cannot be treated as a source of truth.

Every answer has to be linked back to retrieved project data or another verified source, even when the model sounds confident it already knows the answer.

4. Show the evidence boundary

Hallucination has a quieter sibling: silent omission.

An answer built on half the picture can look identical to an answer built on all of it.

We learnt this during a job-setup test.

The package assessment for a steel-framed building with a swimming pool produced no structural steelwork package and no pool package.

Nothing had been invented.

Important things were simply missing.

The cause was upstream. The task had read only the documents a person had manually selected, and those selections didn't adequately cover the whole project.

Two changes came out of that:

1. Assessments now retrieve from the full indexed project document set, with documents manually selected by a user acting as a steer rather than the only information.
2. Every assistant task includes in the output report the detail of what information was referenced: which project documents were read and which weren't.

That second control is one I'd encourage anyone building or buying an AI system to look for.

Every result should tell you not only what evidence it used, but what relevant evidence it did not use. It stops the black box problem, providing clear detail of what information was used to establish the outputs.

If a reviewer can see that the structural steelwork specification wasn't read, they can immediately judge the result accordingly.

It's the AI equivalent of knowing what was actually tested on a test certificate rather than a test certificate that only tells you the test passed.

There's a related rule across the platform: a failed check must never present as an empty result.

"No issues found" and "the check didn't run" are completely different statements.

An error that ends up as a clean pass, or a skipped assessment displayed as a pass, gets treated as an issue and the user is given that information.

5. Validate the output and put people at the hold points

Everything above is automatic.

The final layer is deliberate friction in the right places.

Outputs that matter carry references to the evidence they were based on, down to the actual document and the revision, so the reviewer can check a claim rather than simply trust it.

Automated checks run against every output, we don't just check against a sample of outputs.

And where an output feeds a consequential decision, a responsible person sits at a defined hold point and reviews it before it becomes part of the project record or reaches a client.

Human review is intentionally considered and added to our workflows, not just at vague points across the process.

It sits where the consequences are.

I won't pretend we've published every detail of how those checks work, and some of the machinery is proprietary and what makes PlanOps the tool it is.

But the categories aren't secret, and they shouldn't be.

They translate into straightforward questions you can put to any AI system:

• What stops it judging missing content?
• What stops it inventing references?
• How does it show what evidence it actually used?
• How do you know what it didn't read?
• What happens if a check fails?
• Where does a person have to approve the result before it becomes a record?

The ITP for AI

If this all sounds familiar, it should.

It's an inspection and test plan.

Nobody on site treats an ITP as an insult to the tradesperson.

Hold points, witness points and checks are how the industry makes quality outcomes repeatable and bad ones catchable, using people whose work is mostly good but occasionally isn't.

AI deserves the same respect and the same scepticism.

The process engineering above is essentially hold points and inspections applied to a very fast, very confident colleague that sometimes fills gaps without telling you.

What we still can't prevent

This is also where it's important to be clear about the limits, because most AI marketing gets noticeably quieter here.

What we can do:

• Engineer the workflow so that a judgement cannot proceed without verified content.
• Reject references that don't exist before they reach the project record.
• Show which evidence an assessment did and didn't use.

What we cannot do is guarantee that a model will read a genuine document and always draw the correct conclusion from it.

Within the space of real inputs and valid references, the model can still be wrong.

It can misinterpret wording. It can give too much weight to one piece of evidence. It can overlook an implication. And it can phrase a poor conclusion extremely well.

That is why the five controls end where they do, with a person at the hold point.

That risk is managed, not eliminated.

And it's managed much the same way we manage it for people: sources on show, evidence coverage visible, automated checks where they work and human review at the hold points that matter.

Anyone claiming they've engineered the risk to zero hasn't understood the problem.

The questions to ask

So finally, whether you ever use PlanOps or not.

When someone offers you AI for work that ends up in a project record, don't ask whether it hallucinates.

Everything built on these models can.

Ask what they've done so that making things up doesn't become a project record:

• What happens when the input is missing?
• Is the model allowed to create IDs and codes?
• If and how can you see what evidence it used?
• How would you know what it didn't read?

A specific answer to those questions is worth more than any claimed accuracy assessment.

Hand waving and vague answers tell you everything.

We write about what we're learning building AI for UK construction - the failures included - in the Construction AI Brief. If this was useful, you can get it at planops.ai/insights/construction-ai-brief.

And if you want to see what structured AI tasks look like in practice, there's a plain-English tour at planops.ai/workflows.

Ian Yeo is the founder of PlanOps, an AI-native planning operations platform for UK construction.