AI Pilot to Production: The Questions a Demo Never Answers

The demo had worked.

The scan was processed, the vision-language model returned the required information in a structured format, and the client could immediately see the product opportunity. For a brief moment, everyone on the call was smiling.

Then the client asked one question:

“Could we offer this as a fixed monthly subscription?”

The question sounded straightforward. It was not.

Our demo had proved that the model could perform the task. But every scan processed by the system created an additional cost. A fixed subscription would generate predictable revenue, while the cost of delivering the service would continue changing with customer usage, retries, input quality, and the models required to handle difficult cases.

That was the moment the conversation changed.

We were no longer discussing whether the AI could work. We were trying to determine whether it could become a product the client could price, operate, and scale without losing control of its margins.

The Gap Between Demo Conditions and Real Usage 

The initial evaluation confirmed that a vision-language model could perform the core task and produce commercially useful outputs. The next step was to assess how reliably the system would perform across a broader range of users, input conditions, request volumes, and operational workflows. 

Why a Flat Subscription Required Further Evaluation

The client initially expected the platform to follow a conventional monthly subscription model. However, AI-powered products typically have a more variable cost structure than traditional software.

Operating costs would depend on the number of active users, how frequently they accessed AI-powered features, the complexity of their requests, and the models required to process them. A customer with a small number of occasional users would create a very different cost profile from an enterprise with hundreds of users generating requests throughout the day.

Model selection also had a direct impact on service delivery costs. Routine requests could be handled by a smaller, more efficient model, while complex, low-confidence, or poor-quality inputs might require additional processing, repeated model calls, escalation to a more capable model, or manual review.

This created an important commercial challenge. A flat subscription would simplify pricing, but it might not reflect the actual cost of serving customers with significantly different usage patterns. Two customers could pay the same monthly fee while consuming very different levels of AI processing capacity, infrastructure, storage, and support.

A sustainable subscription model therefore needed to consider:

• Number of active users
• Expected AI usage
• Request complexity
• Access to higher-capability models
• Usage beyond agreed limits

Before recommending a final pricing structure, we needed clearer estimates for the expected user base, usage frequency, peak demand, and the proportion of requests likely to require more resource-intensive processing.

This would allow the client to offer predictable subscription plans while keeping pricing aligned with the actual cost of operating the platform.

The Failure Patterns Behind Production Risk

Broader input testing revealed three recurring failure patterns: glare obscuring printed text, dense small print affecting field extraction, and overlapping stamps disrupting the expected layout.

These were not isolated edge cases. Left unaddressed, this is often where ai project fails once it moves from testing into real customer use. They represented the type of input variability the system would need to handle consistently during regular customer use.

What We Needed Before Making a Commitment?

Before we could commit to a model, a price, or a timeline, we needed the client to define the product requirements more precisely: 

• Expected monthly usage
• Peak request volume
• Acceptable response time 
• Required output quality
• Tolerance for manual review
• Maximum sustainable cost per completed transaction. 

These requirements would let us compare the available production approaches on the same basis.

The Four Production Paths We Evaluated

The vision-language model used in the demonstration remained one possible production option, but using the strongest model for every request could increase processing cost and response time unnecessarily. We laid out four options side by side.

This comparison presents the expected relative performance of each option rather than confirmed production measurements. The final outcome would depend on input quality, request complexity, model configuration, and production usage patterns.

A smaller model might handle standard scans more quickly and at a lower cost, but it might struggle with difficult inputs or require additional training to reach the required quality. Targeted training or fine-tuning could provide a better long-term balance, but that path would require representative data, a clear evaluation method, and additional engineering effort. 

A routed architecture, where standard requests go through a lower-cost model and only failed or low-confidence cases escalate, could reduce average processing cost while preserving stronger performance where it was genuinely required.

The difficult part was deciding when to escalate a request. That decision needed to rely on measurable signals, not judgment calls made case by case.

The Metrics Behind the Production Decision

A single accuracy percentage would not tell us whether the system was ready for the intended workflow. We needed to know whether required fields were extracted correctly, whether the returned structure was valid, whether the model introduced unsupported information, and how often someone would need to correct the result manually.

●       Field-level exact-match rate: whether each required field matches the correct value.

●       Schema-validation pass rate: whether the output structure is usable without correction.

●       Unsupported or hallucinated field rate: how often the model returns information the input didn't support.

●       Manual-review rate: how often a human has to step in before the result can be trusted.

●       End-to-end successful transaction rate: the share of requests that reach a usable result without intervention.

Latency needed the same precision. Model latency alone would not tell us whether the product was fast enough; the relevant measure was the total time between submitting a scan and receiving a result the user could trust. 

The ratings shown are close estimates based on Folio3 AI’s internal testing and observed performance. Exact results are not disclosed due to client confidentiality and NDA obligations.

The routing decision itself needed to follow a defined sequence rather than a judgment call: run input-quality validation on the incoming scan, send valid standard cases to the lower-cost model, validate the response against the expected schema and business rules, escalate failed or low-confidence cases to the stronger model, send unresolved cases for manual review, and store failure categories for future evaluation and training.

The Point Where Assumptions Needed Evidence

The early results created clear momentum, but several commercial and technical decisions still depended on evidence that had not yet been collected.

The first estimate for this engagement assumed a baseline of around 500 scans per customer per month. Early conversations suggested larger accounts could run several times higher. At that range, the demo model could not support a single flat subscription without a defined usage tier, and we did not yet have the retry and fallback data to say where that tier should sit.

We were confident that the use case was worth pursuing as part of the client's broader AI transformation. We were not yet in a position to guarantee production performance or confirm that the proposed subscription would work at scale, and saying otherwise would have made a harder correction later.

What We Recommended Before Production

We recommended a focused production evaluation before any pricing or model commitment. The evaluation needed a broader set of scans than the demo had used, roughly 2,000 samples spanning standard inputs, poor-quality images, incomplete information, unfamiliar layouts, and the known edge cases already surfacing.

Five lessons for moving an AI pilot to production

●      Capability is only the first milestone. Production decisions require representative data on quality, usage, cost, latency, and operational performance.

●       Price the product on measured usage, not assumed usage. A flat subscription against a variable per-request cost is a bet until the usage data says otherwise.

●       Define quality in measurable terms. "Accuracy" is not one number; field-level, schema, and manual-review rates each answer a different question.

●       Budget latency across the whole transaction, not just the model call. Retries and fallbacks are part of the time a user actually waits.

●       Route by evidence, not instinct. Escalation decisions need defined signals, or the expensive model ends up handling requests it didn't need to.