--- title: "The pilot worked. That's the problem." description: "A successful AI pilot proves the model can do the task. It says nothing about production cost, data, ownership or model churn. Four questions to ask first." url: "https://www.lmtek.com/source-code/the-pilot-worked-thats-the-problem" locale: "en" updated: "2026-10-11" --- # The pilot worked. That's the problem. By Fedja Hvastija · Marketing Director, LM TEK · Published 11 Oct 2026 ## Key points - A successful AI pilot on a rented API proves one thing: the model can do the task. That was never the hard question. - It says nothing about what production will cost, where your data goes, who runs it after the pilot team moves on, or what happens when the model changes. - Before you call a pilot a success, write the production memo first. The demo is going well. The pilot takes a stack of supplier contracts and summarises them in the time it takes to pour a coffee. The summaries are good. Better than good: the head of procurement leans forward, which in procurement is the equivalent of a standing ovation. Then the CFO asks the question everyone was hoping for. *How fast can we roll this out to everyone?* Hold that moment, because it's the most dangerous one in the whole project. The pilot worked. That's the problem. ## A pilot tests the one thing you were never worried about Piloting on a rented API is the right way to start. It's fast, it's cheap, and nobody has to sign a capital expenditure form to find out whether an idea has legs. I'd do it the same way. The mistake isn't in running the pilot. It's in what we conclude from it. A pilot answers one question: *can a model do this task well enough?* In 2026 the answer is very often yes. Models are good. That's the part of the project that was least likely to fail, and it's the part the pilot proved. Everything that decides whether the project survives production sits outside the pilot's scope. Success doesn't answer those questions. Success makes them invisible, because the room is too busy being pleased. ## The Pilot Illusion: four questions a pilot never answers **The bill at full volume.** The pilot ran for one team, on a few weeks of requests. Production runs for the whole company, every day, for years. The pilot's bill was a rounding error. The production bill is a line item, and the per-token price on that line isn't yours to set. Nobody in the demo room knows what the line will say. **Where the data went.** To summarise those contracts, the pilot sent them somewhere. Which systems did they pass through, under whose terms, and stored for how long? For a test on sample documents, nobody minds much. For every contract the company signs, someone will. The pilot didn't check, because the pilot wasn't asked to. **Who runs it on day two.** Pilot teams are built to move on. That's their job. When they do, someone has to own the thing: watch it, fix it, answer for it when it gives a wrong answer to a supplier. In the demo, that person doesn't exist yet. Usually nobody notices until the first bad week. **What happens when the model changes.** The prompts, the tests and the outputs the team learned to trust were all tuned to one version of one model. Providers publish schedules for retiring model versions and recommend replacements. ([OpenAI](https://developers.openai.com/api/docs/deprecations), [Anthropic](https://platform.claude.com/docs/en/about-claude/model-deprecations)) When yours is retired, the workflow moves with it. You can plan for that. You can't decline it. Cost, data, ownership, change. A pilot that passes on capability and stays silent on all four isn't a success. It's a demo with good manners. That's the Pilot Illusion: the test passes, so the project looks finished, at the exact moment the real work starts. ## When the pilot really is enough Some pilots should go straight to production on the rented API. Low volume, nothing sensitive in the data, a task that isn't core to how the business makes money: rent it, run it, move on. That's often the sensible call, and it's the call I'd make. But you can only make that call on purpose if the pilot was designed to answer the four questions. Otherwise you're not choosing to rent in production. You're drifting into it, because the demo went well and nobody went back to the question. That's how core work ends up rented by default, which is the argument of [*Renting intelligence is the new outsourcing mistake*](/source-code/renting-intelligence-is-the-new-outsourcing-mistake). ## Before you say "success", write the production memo Here's the assignment. Before anyone calls a pilot a success, write one page. Four lines: 1. **Monthly cost at full production volume**, and who sets the price. 2. **The data path:** where prompts, documents and outputs go, and under whose terms. 3. **The named owner** after the pilot team moves on. 4. **The fallback** when the model you tuned for is retired. If you can fill in all four, you have a production plan, and you can decide calmly whether to keep renting or bring the work in-house. If you can't, the pilot didn't fail. It just answered a different question from the one the CFO asked. So next time the demo goes well, enjoy it for a minute. Then ask for the memo. ## Frequently asked questions ### Why do successful AI pilots fail in production? Because a pilot usually tests only whether a model can do the task. It rarely tests the cost at full volume, where the data goes, who will own the system after the pilot team moves on, or what happens when the provider changes or retires the model. ### Should an AI pilot run on a rented API? Usually yes. A rented API is the fastest and cheapest way to test whether a use case works. The mistake is treating a successful pilot as a production decision without answering the production questions. ### What should I check before moving an AI pilot to production? Write down the monthly cost at full volume and who sets that price, the path your data takes and under whose terms, the person who will own the system, and the fallback if the model you tuned for is retired. ## About the author Fedja Hvastija is Marketing Director at LM TEK. He has spent more than twenty years in B2B marketing and sales strategy, including senior roles at Corsair and NiceLabel. ## Read next - [Renting intelligence is the new outsourcing mistake](https://www.lmtek.com/source-code/renting-intelligence-is-the-new-outsourcing-mistake): Why core, steady AI work shouldn't stay rented by default. - [Do this before you buy an AI server](https://www.lmtek.com/source-code/do-this-before-you-buy-an-ai-server): Four things to have in place before the purchase order. Free to cite with attribution: Hvastija, F. (2026). The pilot worked. That's the problem. LM TEK Source Code. https://www.lmtek.com/source-code/the-pilot-worked-thats-the-problem