APilotWithoutaLearningQuestionIsaDemoWithaBudget

Most AI pilots begin with a capability question.

Can the system summarize a policy? Can it prefill an application? Can it classify an inbox? Can it draft a renewal email? Can it prepare a producer before a call?

Those questions are easy to demonstrate. They are not enough to guide an operating decision.

A useful pilot asks something harder:

What does the agency need to learn before it can trust this workflow at greater volume?

That question changes the cases selected, the evidence collected, the role of the reviewer, and the decision leadership makes at the end.

The commercial submission trap

Consider an agency testing AI-assisted submission preparation.

The demonstration file is complete. The applications are current. The loss runs are named correctly. The exposure schedule follows a consistent format. The system extracts the information, maps it into the submission structure, and produces an impressive package.

The pilot is declared successful because the technology performed the task.

But the agency's real problem was never the clean file.

The real work includes outdated applications, conflicting revenue figures, missing schedules, carrier-specific questions, last-minute producer notes, and risk details buried in correspondence. Experienced marketers know which gap requires clarification, which can be documented as pending, and which makes the submission unsafe to release.

If the pilot avoids those cases, it has demonstrated technical capability without testing operating usefulness.

Begin with an assumption

Every pilot should name the operating assumption it is testing.

For example:

Submission preparation is slow primarily because account teams spend time locating information, checking completeness, and rebuilding the same risk for different carrier requirements.

That statement may be right. It may also hide the true constraint.

Perhaps the delay comes from unclear ownership. Perhaps producers submit incomplete intake. Perhaps carrier appetite changes faster than the guidance is maintained. Perhaps the final quality depends on an experienced marketer's unwritten judgment.

The purpose of the pilot is to make those assumptions visible before the agency builds around them.

Seven useful learning questions

1. What context is consistently missing?

Do not only count whether the system completed the task. Record which facts were unavailable, stale, conflicting, or stored outside the approved sources.

2. Which corrections recur?

A recurring correction is not merely a model-quality problem. It may reveal a missing workflow rule, source priority, review standard, or training need.

3. Which cases require escalation?

The agency needs to know what the system cannot safely resolve and who should receive the work next.

4. Where does human judgment change the outcome?

Identify the moment where an experienced employee does more than check accuracy. That is an authority or relationship boundary, not a clerical review step.

5. Does the assistance reduce the complete pile?

A system may save preparation time while adding review, reconciliation, data cleanup, or exception management elsewhere.

6. Do practitioners trust the output for the right reasons?

Trust should come from visible evidence and a reliable boundary, not from a polished answer or a run of easy cases.

7. What would justify expansion?

Define the evidence before the pilot begins. Otherwise enthusiasm becomes the success criterion.

Test ordinary and difficult cases

A pilot should include straightforward work, common exceptions, and cases the team expects to be difficult.

The objective is not to create a failure showcase. It is to understand the operating boundary while the volume is small and the decisions remain reversible.

If the agency plans to use assistance on small commercial renewals, test accounts with clean documentation and accounts with midterm exposure changes. If the agency plans to classify service requests, include ordinary certificate requests and messages that appear routine but contain coverage questions.

The safest time to discover ambiguity is before the system handles it at scale.

Define the human boundary before the test

Do not let the pilot quietly expand from preparation into authority.

For a submission workflow, the system may retrieve approved data, identify missing fields, prepare applications, and organize the package. A licensed or accountable person still decides how the risk is represented, which markets receive it, what requires clarification, and whether the submission is ready to release.

The boundary should be written in language the team can use while working.

The end of the pilot is a decision

A pilot has four legitimate outcomes:

• Stop because the assumption was wrong or the value is too small.
• Revise because the workflow, data, or boundary needs to change.
• Repeat because the evidence is not yet strong enough.
• Expand because the operating evidence supports more volume or another bounded task.

Scaling is not the reward for completing the pilot.

Sometimes the most valuable result is learning that a platform decision is premature, a source cannot be trusted, or a human checkpoint must remain exactly where it is.

The operating principle

The objective is not to prove AI works. The objective is to discover the conditions under which this workflow works better.

A pilot without a learning question is a demo with a budget.

Related framework: Regesta Board, Extend Deliberately.