AI

How to run an AI proof of concept that reaches production

Developer typing on a laptop showing source code, with a second monitor behind it

In July 2024, Gartner predicted that at least 30% of generative AI projects would be abandoned after the proof of concept stage by the end of 2025. The reasons it listed were poor data quality, inadequate risk controls, escalating costs and unclear business value.

A year later, MIT’s NANDA initiative published The GenAI Divide: State of AI in Business 2025. As reported by Fortune, it found that about 5% of the AI pilots it studied achieved rapid revenue acceleration, while the large majority showed little or no measurable effect on profit and loss.

Look again at the reasons Gartner gave: data, risk, cost, value. Every one of them can be checked during a proof of concept, and most PoCs are not set up to check them.

Write success and kill criteria before any code

Before building anything, agree on one business metric, the result that would justify a production build and the result that would end the project. Put both thresholds in writing and get the budget owner to sign off. A PoC without a kill criterion can only end in a vague maybe, and vague maybes are how pilots drift for a year.

A usable criterion looks like this: the system drafts replies for tier-one support tickets, and agents send at least 70% of drafts with light edits or none, at a cost per ticket below the agent time it saves. The matching kill criterion: if fewer than half of drafts are usable after two rounds of improvement, we stop.

The numbers will differ for you. The important part is that they are specific, measurable on real work and agreed before anyone sees a demo. Demos are persuasive in a way that makes people move the goalposts afterwards.

Build an evaluation set from real examples

Collect 50 to 200 real cases from the actual process, each paired with the answer a good employee would give. Include the awkward ones: incomplete inputs, edge cases, requests that should be refused. Score every version of the system against this set. It turns opinions about quality into numbers you can compare.

The evaluation set is the most valuable thing a PoC produces. It outlives the first model you try and the first prompt you write. When a better model comes out next quarter, you can measure whether it is better for your work in an afternoon.

Building it takes time from people who know the process, usually a subject expert for a few days. That is often the hardest resource to secure, so ask for it in week one.

Start with the simplest approach that could work

Try the approaches in order of complexity and stop at the first one that meets your success criteria. That order is ordinary code and rules, then a single model call with a careful prompt, then retrieval over your documents, then an agent that uses tools. Each step up adds cost, latency and new ways to fail.

Agents get the most attention right now. In June 2025 Gartner predicted that over 40% of agentic AI projects will be canceled by the end of 2027, citing costs, unclear business value and inadequate risk controls. It also warned about “agent washing”, where existing products are rebranded as agents without real agentic capability.

Plenty of problems do need an agent. Many more are solved by retrieval-augmented generation or a well-structured prompt plus a few lines of validation code. The evaluation set tells you which.

Use your real data, with its real mess

A PoC built on clean sample data proves very little. Use the documents, tickets or records the production system will see, with redaction where privacy requires it. Messy formats, missing fields and contradictory sources are exactly what the PoC needs to surface while the stakes are low.

Data access is usually the slowest part of the first two weeks. Approvals, exports and privacy reviews take longer than building the first prototype. Start those requests on day one, and treat any data you cannot get during the PoC as a risk to the production plan.

Measure cost per task

Track what each completed task costs: model usage, retrieval and hosting, plus the human review time the output still needs. Compare that with what the task costs today. A system that is accurate but more expensive per task than the current process is unfinished, however good the demo looked.

Run the numbers at production volume. A cost that looks trivial for 40 test cases can look different at 40,000 a month. Model prices also move, so record your assumptions and recheck them before the production decision.

Plan the integration while the PoC runs

The MIT report pointed to integration as a central problem. According to Fortune’s summary, the tools that failed did not learn from feedback or adapt to how people work, so they stayed outside the workflow. The report also found that purchased or partnered solutions succeeded about 67% of the time, while internal builds succeeded about a third as often.

We read that second finding as a reason to check whether a good product already solves your problem before building one. Sometimes the right outcome of a PoC is a purchase.

When building is the right call, decide during the PoC where the output will appear, ideally inside a tool people already use. Decide who approves or corrects it, and how those corrections get captured so the system improves. The same report noted that the clearest returns came from back-office automation, which is less visible than sales and marketing projects but easier to measure.

A four to six week shape

A focused PoC fits in four to six weeks when the criteria and data access are settled early. The point of the schedule is a clear decision at the end, with evidence behind it. Stopping is a valid outcome, and much cheaper than finding out after a production build.

  • Week 1: success and kill criteria signed off, data access requested, evaluation set started.
  • Weeks 2 and 3: a baseline using the simplest viable approach, scored against the evaluation set.
  • Weeks 4 and 5: improvements targeted at the failures, cost per task measured, a thin integration tested with a few real users.
  • Week 6: a written recommendation to build, change direction or stop, with the scores and costs that support it.

This is how we run our AI proof of concept engagements. If you have a use case in mind and want help turning it into criteria you can test, get in touch.

Sources

AI
Previous PostNext Post