From an AI demo that impresses to a system that holds up.
We test your pilot against an agreed pass mark, then harden what passes and cut what doesn't.
Evaluation bars climbing past a pass-mark line, then a gate labeled promote to production opening.
Overview
Most AI pilots work on the examples somebody picked and fail on the ones nobody did. We turn your pilot into a measured system: a test set built from real cases, a pass mark you sign off, and the guardrails, logging and fallbacks a production service needs. A promotion gate decides when the pilot is ready, based on evidence rather than a good feeling.
A good fit if
- Your demo works on the examples someone picked.
- Nobody can say how often it's wrong, or what each answer costs.
- Real customers or sensitive data come next.
- A model upgrade means a week of testing by hand.
Not the right start if
- There's no business measure of "good" yet. Start with Discovery & MVP
- The core idea hasn't been tried yet. Build a cheap prototype first, then come back.
- The data it needs isn't in order. Start with AI Agents & RAG
Release
A release has to pass every gate
Pick a problem and see where the release stops. Each gate is a check we agree with you before launch.
Try a release that...
- Test set Stopped
- Attack tests Stopped
- Cost per answer Stopped
- Response time Stopped
- Shadow run Stopped
- 5% of users Stopped
- Everyone Stopped
Released to everyone once the 5% stage held its numbers. Rolling back stays one step away.
Released to everyone once the 5% stage held its numbers. Rolling back stays one step away. Stopped at the test set: 84% right against a mark of 90%. The failures are grouped, the biggest group is fixed first, and the run repeats. Stopped at cost: the agent makes too many model calls per task. We cap calls per task or send easy questions to a smaller model. Stopped at attack tests: an instruction hidden in a document got the assistant to reveal another customer's data. Fixed with stricter search permissions and an output check, then tested again. Stopped at the shadow run: on real requests, it disagreed with your team on a case the test set didn't have. That case joins the test set.Every case
- passes every check
- Released to everyone once the 5% stage held its numbers. Rolling back stays one step away.
- misses the accuracy mark
- Stopped at the test set: 84% right against a mark of 90%. The failures are grouped, the biggest group is fixed first, and the run repeats.
- goes over the cost budget
- Stopped at cost: the agent makes too many model calls per task. We cap calls per task or send easy questions to a smaller model.
- leaks data under attack
- Stopped at attack tests: an instruction hidden in a document got the assistant to reveal another customer's data. Fixed with stricter search permissions and an output check, then tested again.
- fails on real traffic
- Stopped at the shadow run: on real requests, it disagreed with your team on a case the test set didn't have. That case joins the test set.
Sample values. The real marks are agreed with you in step 3.
Process
How it runs
You keep
- 01An architecture review and a risk list.
- 02A failure list with counts, and a named quality owner on your side.
- 03A versioned test set and a baseline score.
- 04An attack report with fixes.
- 05Guardrails and fallback paths.
- 06Tracing, plus cost and speed dashboards with alerts.
- 07The gate settings and a rollback runbook.
- 08A test set that grows, and a playbook for model upgrades.
-
01 of 08
Audit the pilot.
We read the code, the prompts and at least 100 real traces.
You keep An architecture review and a risk list.
-
02 of 08
Find the failures.
Traces are labeled by hand and grouped, so you see which failure costs most.
You keep A failure list with counts, and a named quality owner on your side.
-
03 of 08
Build the test set.
Real cases plus a check for each failure, with automated graders checked against human labels.
You keep A versioned test set and a baseline score.
-
04 of 08
Attack it.
Prompt injection, data leaks, runaway actions and runaway spend, tried on purpose.
You keep An attack report with fixes.
-
05 of 08
Fence it.
Input and output checks, spend limits, and a hand-off to a person when the model is unsure.
You keep Guardrails and fallback paths.
-
06 of 08
Trace it.
Every request recorded with its cost and response time.
You keep Tracing, plus cost and speed dashboards with alerts.
-
07 of 08
Gate the release.
Every change runs the checks, then a shadow run and 5% of users before everyone.
You keep The gate settings and a rollback runbook.
-
08 of 08
Run and learn.
New failures from live traffic become new test cases.
You keep A test set that grows, and a playbook for model upgrades.
Risks
Pilots stall between the demo and the customer
-
Tested on picked examples.
It shines on the cases someone chose and fails on the ones nobody did.
Prevented in step 3 -
No ceiling on cost.
Agents multiply model calls, and the bill shows up a month later.
Prevented in step 5 -
No way back.
A bad release reaches every user at once.
Prevented in step 7
4 of 33
Of every 33 AI proofs of concept, 4 reached production.
10%
No more than 10% of companies were scaling AI agents in any one business function.
Deliverables
What you keep
-
Evaluation set
Real cases from your work, labeled and versioned, so quality is a number rather than an opinion.
-
Pass mark and promotion gate
A written threshold the pilot must clear before it reaches users.
-
Guardrails and fallbacks
Input checks, output checks and a safe answer when the model is unsure.
-
Tracing and logging
Every request recorded, so a bad answer can be found and fixed.
-
Cost and latency budget
Targets for spend and speed per request, with alerts when they slip.
Done means
- Every change runs the test set automatically.
- A release is blocked if quality, cost or speed misses its mark.
- Every request can be traced from question to answer.
- Spend has a ceiling and an alert.
- People are told they're talking to AI.
- Rolling back takes one step.
Sample
Promotion gate report
| Test set, right answers | 92%, mark 90%: passed |
|---|---|
| Attack tests, open issues | 0: passed |
| Cost per answer | inside budget: passed |
| Response time, slowest 5% | 2.1 s, mark 3 s: passed |
| Shadow run disagreements | 3, all reviewed: passed |
| Decision | Promote to 5% of users |
FAQ
Questions about AI Pilot to Production
Still deciding?
Ask the people who would build it. We reply within one business day.
Because a demo runs on chosen inputs. Production brings messy data, people trying to break it and cost at volume, and none of those show up in a demo.
Then you decide whether we change the approach or stop. That is the point of agreeing the mark first: you find out before you have built the whole product, and you keep what we built.
Not always. We start with an audit of what you have. If it is sound, we harden it. If it isn't, we tell you which parts to replace and why.
Yes. We check code quality, security and infrastructure first, then propose a plan to stabilize and extend it. Some parts usually need rewriting before real users arrive.
No, and be wary of anyone who does. We measure how often it happens, cut it down, send unsure cases to a person and log everything, so a bad answer can be found and fixed.
Some of it already does. Since 2 August 2026, people must be told when they're talking to an AI system, and stricter duties for high-risk uses start on 2 December 2027. We'll tell you which apply to your use, as engineers rather than lawyers.
Yes. The test set, prompts, code and infrastructure are yours from the first commit, so any team can rerun them later.
Contact
Have an idea or a stuck project? Let's talk.
We reply within one business day. If we're not the right team, we'll say so on the first call.
On the first call
- What the pilot does today, and who uses it.
- How you judge a good answer now.
- What would make you pull it from users.