Evaluation bars climbing past a pass-mark line, then a gate labeled promote to production opening.

Most AI pilots work on the examples somebody picked and fail on the ones nobody did. We turn your pilot into a measured system: a test set built from real cases, a pass mark you sign off, and the guardrails, logging and fallbacks a production service needs. A promotion gate decides when the pilot is ready, based on evidence rather than a good feeling.

A good fit if

  • Your demo works on the examples someone picked.
  • Nobody can say how often it's wrong, or what each answer costs.
  • Real customers or sensitive data come next.
  • A model upgrade means a week of testing by hand.

Not the right start if

A release has to pass every gate

Pick a problem and see where the release stops. Each gate is a check we agree with you before launch.

  1. Test set Stopped
  2. Attack tests Stopped
  3. Cost per answer Stopped
  4. Response time Stopped
  5. Shadow run Stopped
  6. 5% of users Stopped
  7. Everyone Stopped

Released to everyone once the 5% stage held its numbers. Rolling back stays one step away.

Every case

passes every check
Released to everyone once the 5% stage held its numbers. Rolling back stays one step away.
misses the accuracy mark
Stopped at the test set: 84% right against a mark of 90%. The failures are grouped, the biggest group is fixed first, and the run repeats.
goes over the cost budget
Stopped at cost: the agent makes too many model calls per task. We cap calls per task or send easy questions to a smaller model.
leaks data under attack
Stopped at attack tests: an instruction hidden in a document got the assistant to reveal another customer's data. Fixed with stricter search permissions and an output check, then tested again.
fails on real traffic
Stopped at the shadow run: on real requests, it disagreed with your team on a case the test set didn't have. That case joins the test set.

Sample values. The real marks are agreed with you in step 3.

How it runs

  1. 01 of 08

    Audit the pilot.

    We read the code, the prompts and at least 100 real traces.

    You keep An architecture review and a risk list.

  2. 02 of 08

    Find the failures.

    Traces are labeled by hand and grouped, so you see which failure costs most.

    You keep A failure list with counts, and a named quality owner on your side.

  3. 03 of 08

    Build the test set.

    Real cases plus a check for each failure, with automated graders checked against human labels.

    You keep A versioned test set and a baseline score.

  4. 04 of 08

    Attack it.

    Prompt injection, data leaks, runaway actions and runaway spend, tried on purpose.

    You keep An attack report with fixes.

  5. 05 of 08

    Fence it.

    Input and output checks, spend limits, and a hand-off to a person when the model is unsure.

    You keep Guardrails and fallback paths.

  6. 06 of 08

    Trace it.

    Every request recorded with its cost and response time.

    You keep Tracing, plus cost and speed dashboards with alerts.

  7. 07 of 08

    Gate the release.

    Every change runs the checks, then a shadow run and 5% of users before everyone.

    You keep The gate settings and a rollback runbook.

  8. 08 of 08

    Run and learn.

    New failures from live traffic become new test cases.

    You keep A test set that grows, and a playbook for model upgrades.

Pilots stall between the demo and the customer

  1. Tested on picked examples.

    It shines on the cases someone chose and fails on the ones nobody did.

    Prevented in step 3
  2. No ceiling on cost.

    Agents multiply model calls, and the bill shows up a month later.

    Prevented in step 5
  3. No way back.

    A bad release reaches every user at once.

    Prevented in step 7

4 of 33

Of every 33 AI proofs of concept, 4 reached production.

IDC with Lenovo, 2025Checked Nov 2026

10%

No more than 10% of companies were scaling AI agents in any one business function.

McKinsey, 2025Checked Nov 2026

What you keep

  • Evaluation set

    Real cases from your work, labeled and versioned, so quality is a number rather than an opinion.

  • Pass mark and promotion gate

    A written threshold the pilot must clear before it reaches users.

  • Guardrails and fallbacks

    Input checks, output checks and a safe answer when the model is unsure.

  • Tracing and logging

    Every request recorded, so a bad answer can be found and fixed.

  • Cost and latency budget

    Targets for spend and speed per request, with alerts when they slip.

Done means

  • Every change runs the test set automatically.
  • A release is blocked if quality, cost or speed misses its mark.
  • Every request can be traced from question to answer.
  • Spend has a ceiling and an alert.
  • People are told they're talking to AI.
  • Rolling back takes one step.

Sample

Promotion gate report

Test set, right answers92%, mark 90%: passed
Attack tests, open issues0: passed
Cost per answerinside budget: passed
Response time, slowest 5%2.1 s, mark 3 s: passed
Shadow run disagreements3, all reviewed: passed
DecisionPromote to 5% of users
Illustrative values, not a client result.

Questions about AI Pilot to Production

Still deciding?

Ask the people who would build it. We reply within one business day.

Because a demo runs on chosen inputs. Production brings messy data, people trying to break it and cost at volume, and none of those show up in a demo.

Then you decide whether we change the approach or stop. That is the point of agreeing the mark first: you find out before you have built the whole product, and you keep what we built.

Not always. We start with an audit of what you have. If it is sound, we harden it. If it isn't, we tell you which parts to replace and why.

Yes. We check code quality, security and infrastructure first, then propose a plan to stabilize and extend it. Some parts usually need rewriting before real users arrive.

No, and be wary of anyone who does. We measure how often it happens, cut it down, send unsure cases to a person and log everything, so a bad answer can be found and fixed.

Some of it already does. Since 2 August 2026, people must be told when they're talking to an AI system, and stricter duties for high-risk uses start on 2 December 2027. We'll tell you which apply to your use, as engineers rather than lawyers.

Yes. The test set, prompts, code and infrastructure are yours from the first commit, so any team can rerun them later.

Have an idea or a stuck project? Let's talk.

We reply within one business day. If we're not the right team, we'll say so on the first call.

On the first call

  • What the pilot does today, and who uses it.
  • How you judge a good answer now.
  • What would make you pull it from users.