Moving an AI prototype to production: the decisions a demo cannot answer

Turn the promising behavior into a product contract

A prototype establishes that an approach can work on selected examples. Production introduces unfamiliar inputs, concurrent users, unavailable services, and actions with lasting consequences. Write down the exact responsibility of the feature before expanding it: accepted inputs, permitted outputs, available tools, excluded actions, and the person who handles exceptions.

For a hypothetical purchasing assistant, ‘prepare a purchase request for review’ defines a different product from ‘place an order.’ Specify whether it may read supplier records, suggest a quantity, change a budget, or contact a vendor. Keep the interface consistent with that authority. A button labeled ‘Create draft’ should not silently perform the final transaction.

Assign a maintainer who owns source information, quality decisions, and incident response.

Evaluate the workflow before optimizing the prompt

Build an evaluation set that represents the intended work. Include normal cases, ambiguous requests, missing evidence, conflicting sources, and requests the system should decline or escalate. OpenAI's evaluation guidance recommends task-specific testing, realistic distributions, and continued evaluation as systems change. Its central lesson is useful beyond any particular evaluation platform: an impressive individual answer is weak evidence of dependable behavior.

Score separate properties. Did retrieval find the right record? Are important claims supported? Is the output structurally valid? Did the application choose an allowed action? How much editing did the reviewer need? A single overall score can hide a serious weakness behind fluent wording.

Agree on release criteria with the workflow owner. Keep a held-out set for assessing changes and add new failure cases as they appear. If model-based grading helps scale review, compare it with human judgments and investigate disagreement. Passing a finite test set supports a release decision; it cannot prove that every future input will work.

Keep permissions outside the model's discretion

Authenticate the user and authorize each data lookup and action in application services. Scope records to the correct account or organization, validate tool arguments, and limit which operations the feature can invoke. A model's output should be treated as a proposal that must pass ordinary business rules.

Retrieved documents, webpages, and customer messages can contain instructions aimed at changing the system's behavior. OpenAI's agent-safety guidance describes this prompt-injection risk and recommends constraining data flow and protecting consequential tool use. A useful test is to place a hostile instruction in a document the assistant needs to read, then verify that it cannot expand its access or execute an unauthorized operation.

Structured output can help validate shape and permitted values. It does not establish that an amount is correct, a source is trustworthy, or a user has permission. Preserve those checks in deterministic code.

Make human review a usable part of the workflow

Review only helps when the reviewer can understand the proposed result. Show the source evidence, meaningful differences, unresolved questions, and the action that approval will trigger. Give the person a way to edit or reject the proposal without losing their work. Record the decision so later troubleshooting can distinguish a generated suggestion from an approved action.

Match the review burden to the consequence. A suggested internal label and an externally sent commitment need different controls. NIST's framework includes defining human oversight and assessing whether a system meets its intended purpose. Apply that thinking to your actual workflow rather than adding a confirmation dialog to every harmless interaction.

Measure reviewer effort during the pilot. If checking takes longer than completing the task manually, investigate the presentation, evidence, or task choice before expanding deployment.

Design what happens when a request cannot complete

Set timeouts and bounded retries. A failed model call should not create an endless spinner or repeatedly trigger a paid operation. Where a workflow can cause side effects, use stable request identifiers and application-level idempotency so a retry cannot accidentally duplicate the completed action.

Keep useful work recoverable. Preserve the user's input, explain what completed, and offer a fallback such as manual editing, a source-only view, or a queued retry. Test provider errors, rate limits, partial uploads, stale records, and the user closing the page halfway through.

Observe quality and cost at the task level

Record enough operational context to reconstruct a failure: request identifier, model and prompt version, retrieval version or source references, tool outcomes, latency, and completion state. Minimize sensitive content in logs, restrict access, and set a retention policy. Useful diagnostics do not require copying every private document into a general analytics system.

Measure cost per useful completed task, including retries, retrieval, review, and infrastructure. OpenAI's cost and latency guidance discusses tradeoffs such as using a suitable smaller model, generating less unnecessary output, and avoiding redundant requests. Test each optimization against the same quality criteria. A cheaper call that causes more rework may increase the cost of delivering the outcome.

LAYIQ's published technical details include versioned recommendation artifacts, prediction-parity checks, holdout evaluation, and rollback for its LightGBM system, plus Sentry and PostHog for operational visibility. Those are concrete examples of the supporting engineering around an intelligent feature; the appropriate controls for a different application depend on its task.

Release with a fallback and a way back

Start with a bounded group and an owner who can review actual outcomes. Version the model configuration, prompts, retrieval settings, and relevant application code together. Re-run evaluations after meaningful changes, including changes to source data. Keep the previous working configuration available and define who can disable the feature.

Before launch, demonstrate a successful task, an authorized refusal, an unavailable dependency, and a rollback. Document who receives alerts and how an affected user continues their work. These demonstrations give an operating team evidence for launch and a plan for ongoing support.