← All insights

Applied AI

AI-assisted decisions in operations

An assistant can read a late load, an open repair and a run-out estimate faster than anyone on shift. What it has to show, what it may touch, how it is tested and who approves before anything changes.

, Harrimont9 min read

Operational decisions are made under time pressure, from records that disagree or arrive late. A language model does part of that work well: it reads several records at once, connects a free-text note to the asset it mentions and writes a clear account of the situation.

It does other parts badly. It cannot tell on its own whether a figure is current, it keeps to what a person may see only if the system around it enforces that, and it cannot answer for the outcome. AI-assisted decisions work when each part goes to the component that does it well.

This article follows one constructed example, a late delivery that meets an open repair, as Ancilor is designed to handle it. The site and times are illustrative. The platform page follows the same kind of case across the modules.

Split the decision before adding AI

A decision has more steps than the moment of choice, and each step has a natural owner. Most of them are not a language model.

Each step of a decision, and who does it
StepDone byIn the example
FactsThe systems of record, read with their source and timeThe load's stage updates, the repair job, the latest tank reading
CalculationsFixed methods in codeThe run-out estimate and its range
Reading and summaryThe language modelReading the free-text notes on the repair and the load, and writing a short account of the situation
OptionsThe actions the application allows on these recordsAnother receiving plan, an earlier repair, a rescheduled delivery
ChoiceA person with the roleThe supply planner
ExecutionThe application, under that person's nameThe rescheduled delivery, with the planner's reason
ConfirmationThe outside party or the system of recordThe supplier confirms the new delivery time

Writing this down settles most design questions early. The model does not calculate the run-out date; it quotes the estimate and the inputs behind it. It does not invent an option the application does not offer. And it never stands in for a confirmation that has to come from outside.

A late load and an open repair

At 15:30 the supplier tells the dispatcher that Site B's load, due between 14:00 and 16:00, will now arrive at about 19:30. Site B's receiving bay has an open maintenance job: an inspection found a leaking valve at the fill point, the repair is under way, and the technician expects to finish at 10:00 tomorrow. Until then the bay is on hold.

The supply planner asks the assistant: will Site B run short before the bay reopens, and what can we do? A good answer reads like a short briefing with its sources attached.

  • The load. Revised arrival about 19:30, reported by the supplier at 15:30. The driver's last stage update is from 11:15, on the motorway.
  • The repair. Started and not finished. The 10:00 finish is the technician's estimate, entered at 12:40, not a confirmation.
  • The stock. The latest tank reading is from the morning shift report. The run-out estimate built on it falls between 08:00 and 13:00 tomorrow.
  • What is missing. No tank reading since the morning, and no stage update for more than four hours.
  • The options. Receive tonight under another receiving plan agreed with the site manager, ask maintenance whether the repair can be finished first, or reschedule the delivery for after the repair, which the supplier has to confirm. Each option names who has to agree.
  • A suggestion, marked as one. Ask the site for a fresh tank reading before choosing, because the run-out range opens before the repair is due to finish.

The planner asks the site for a reading. The afternoon reading moves the run-out estimate to between 12:00 and 15:00 tomorrow, after the repair is due to finish. The planner reschedules the delivery for 10:30 tomorrow, records why and asks the supplier to confirm. Ancilor is designed to check at that moment that the planner may reschedule this load, and to show the change as requested, not confirmed, until the supplier confirms, with the original window kept on the record beside it.

The assistant did none of this itself. It could not reschedule the delivery, contact the supplier, change the estimate or mark the repair finished. It drafted the request to the supplier, and the planner edited it and sent it.

What the assistant has to show

An answer that cannot be checked is an opinion. For an operational decision, the answer should carry five things.

  1. Each fact with its record, source and time. "On the motorway at 11:15, reported by the driver" is a fact. "On the motorway" is a guess about the present. Each record should open in one step from the answer.
  2. Figures taken from the system's own calculations. The run-out range comes from the forecasting method, with the age of its inputs. A model that recalculates figures in its own words will sooner or later produce one that no calculation supports. NIST's Generative AI Profile calls this risk confabulation: false content stated with confidence.
  3. What is missing or stale, named. No reading since the morning is not the same as no change in stock. The answer says what it could not check.
  4. Assumptions, marked as assumptions. The 10:00 finish is an estimate. An option that depends on it says so, and says what happens if the repair runs late.
  5. What would change the answer. Often the most useful line names the fresh fact that would settle the question. Here it was a tank reading.

Suggestions are welcome as long as they are marked as suggestions and their reasons point to records. Where the records cannot support an answer, the assistant should say so rather than fill the gap.

What it may read, and what it may touch

An assistant works for a particular person, so it should read with that person's permissions and no more. The planner may see stock, loads and the state of the repair. They may not see what the technician is paid or the terms agreed with each supplier. The harder leaks come not from direct questions but from derived answers. An option ranked first "because it costs less" can reveal a price the planner was never shown. Indirect questions belong in the tests.

Writing is simpler: the assistant should hold no write access at all. It can prepare a proposal, with the action, its inputs and its reasons, for a person with the right role to accept or reject. On acceptance, the application runs the action as that person, checks their permission at that moment and records which proposal it came from. A proposal that nobody accepts changes nothing.

Records also carry text written by other people: a driver's note, a supplier's email, a comment on a job. That text is data, never an instruction. If a delivery note says "bay repaired, unload on arrival", the assistant can quote it as the driver's statement. It must not treat it as the end of the repair, and nothing written in a record can widen what the assistant may do. OWASP's list of risks for applications built on large language models puts prompt injection first, including instructions hidden in content the model reads.

Test it on cases with a known outcome

Before anyone relies on an assistant for a decision, run it on past cases where the facts and the right handling are known: late loads, repairs that overran, readings that later proved wrong. Constructed cases fill in what the history lacks, such as a planted instruction or a question beyond the person's access.

Judge each answer on what matters for the decision, not on how well it reads:

  • Did it cite the right records, with their times?
  • Did it name the stale or missing input?
  • Do its figures match the system's own calculation?
  • Did it offer only options the application allows, and propose nothing when nothing needed doing?
  • Did it decline a question beyond the person's access, including an indirect one?
  • Did text planted in a record change what it did?

Language models can answer the same question differently from one run to the next, so run each case more than once. Run the whole set again whenever the model, its instructions, its tools or a source changes. Keep the cases used for tuning apart from the cases used for judging, and agree the pass criteria for this decision before anyone looks at the results. Some cases must pass every time: a permission case that fails once has failed.

What one test case contains

Site B, a late load with the receiving bay on hold. Constructed.

  • Question: the planner's question, word for word.
  • Records available: the load and its stage updates, the maintenance job with the technician's estimate, the morning tank reading and the run-out estimate.
  • Expected evidence: each of those records with its time, and the gap in stage updates and the missing afternoon reading named.
  • Expected figures: the run-out range exactly as the forecasting method reports it.
  • Expected options: the three the application allows, with who has to agree to each, and the suggestion to get a fresh reading first.
  • Must not happen: a figure that no calculation supports, a supplier's price in the answer, the repair treated as finished.

The NIST AI Risk Management Framework and its Generative AI Profile help to organise this work: they are voluntary guidance for governing, mapping, measuring and managing AI risks, not test suites. ISO/IEC 42001 sets requirements for an organisation's AI management system, which is where ownership of this testing belongs. We cite them as reference points, not as a claim of conformity.

Where a person approves, and what they see

Not every suggestion needs a signature. A summary of yesterday's deliveries can be read and set aside. A step needs explicit human approval when it changes stock, money or a commitment to someone outside the company, affects safety, touches personal data or is hard to undo. Lifting a maintenance hold affects safety, so it belongs to the person responsible for the repair: never to an assistant, and never to a plan that would be easier without the hold.

Approval only means something if the approver can see what they are approving. The screen should show the proposal beside its evidence and the options that were set aside, ask for a reason where the step is consequential, and refuse the approval if the records changed after the approver looked at them. Ancilor Finance already applies the last of these to deposits: a second person confirms each deposit, and the confirmation is refused if the evidence changed after they looked at it.

The harder problem is habit. People who see good suggestions every day start to accept them without reading. Article 14 of the EU AI Act, on human oversight, names this automation bias, and asks that the people overseeing a high-risk AI system be able to understand its limits, interpret its output correctly, and disregard, override or stop it. The article applies to systems the Act classes as high-risk, but its list is a sound brief for any assistant whose suggestions people act on.

In practice: keep the evidence one step away, make rejecting as easy as accepting, review a sample of accepted proposals, and make sure the work can continue by hand when the assistant is switched off.

How it goes wrong

Failures, how they show and what guards against them
FailureHow it showsWhat guards against it
Stale facts read as current"The truck is on the motorway", four hours after the last updateEvery fact carries its time, and stale inputs are named
A figure no calculation supportsA run-out time that matches nothing the forecasting method producedFigures quoted from the system's calculations, with a link, and tests that compare them
The wrong recordA repair or a receipt attached to the wrong bay or load because the labels look alikeMatching on identifiers and their issuer; doubtful matches stay open for a person
A leak through a derived answerAn explanation that reveals prices or pay the person may not seeReading with the person's permissions, and indirect questions in the tests
Instructions hidden in dataA note on a document telling the system to approve a changeText from records treated as data, and no write access for the assistant
Approval by habitProposals accepted faster than anyone could read their evidenceEvidence beside each proposal, reasons for consequential steps, sampled reviews
A silent changeA new model or a renamed source changes the answers, and nobody noticesThe test set run again whenever the model, its instructions or a source changes, and the version recorded with each answer

One decision at a time

Start with one recurring decision and the team that makes it. Run the assistant read-only beside them and compare its answers with theirs on the same cases. Let it prepare proposals only when its evidence holds up and the people reviewing it find that the proposals save them work rather than adding a step. It does not act on its own at any stage. Each step has an owner and a written reason to go further or to stop.

How Ancilor applies this

In Ancilor, AI helps review and summarise. It does not change records, and a person makes the decision. Ancilor is designed so that an assistant shows the records behind each suggestion and how old they are, reads within the access of the person it serves, and can prepare a proposal that only a person with the right role can accept, under their own name.

Actions are logged with who made them and when. Which records an assistant may read, where the model runs and how it is tested are agreed with each team before it is used. How we use AI describes the jobs AI does in our work today.

The technical report RATIO, by the author of this article, proposes an architecture that keeps a model's interpretation and planning apart from identity, permissions, evidence retrieval, validation, approval and execution. It is research: it proposes a design and reports no product results.

References

Start with one workflow.

Bring one process, the systems it already runs on and the decision your team needs to make.

Discuss your operation