What a Small Call Pilot Can—and Cannot—Tell You

Decide what the pilot is meant to answer

Write the pilot question before delivery begins. Questions such as whether the intended destination rings, whether Spanish coverage is available and whether records reconcile can be investigated directly. Questions about stable acquisition cost or long-term policy performance need more outcomes and time. Combining everything into a single pass or fail label hides that distinction. A campaign can pass its delivery check while leaving its economics unresolved, or generate a sale while revealing a receiving process that needs repair.

Assign an observable result to each operational question. Record delivery, language handling, follow-up ownership and reconciliation separately. For economic questions, specify which outcome will eventually count and when it can be observed. An application submitted during the pilot is not automatically an issued policy or received commission. The first-call preparation guide covers the practical setup. Use the pilot question as a boundary: it should prevent the team from treating an encouraging anecdote as an answer to every business question.

Understand how much one outcome changes the rate

In a hypothetical sample of 5 calls, 1 outcome represents 20 percentage points. If 1 policy is issued, the observed proportion is 20%; if 2 are issued, it is 40%. Those calculations describe the sample correctly, but the difference does not demonstrate that the underlying process has doubled in quality. A small denominator makes the observed rate highly sensitive to each result. Report the counts beside the percentage so that a reader can see how little evidence supports it.

NIST's discussion of confidence limits for a binomial proportion explains why estimated proportions have uncertainty. A formal interval also depends on assumptions about the observations and the method chosen. Calls handled by different people, at different times or under changing criteria may not behave like identical independent trials. Do not present a narrow-looking statistical calculation as a cure for inconsistent data. The practical lesson is to resist projecting a tiny observed rate across a large future budget, even when the arithmetic itself is correct.

Sources: NIST: Confidence limits for a binomial proportion

Keep unresolved outcomes in the record

Use explicit statuses for pending applications and follow-up. Suppose 5 calls have arrived, but some outcomes are still unresolved. Reporting only completed cases can make the apparent rate depend on which cases resolve fastest. Keep the original sample visible and identify the pending portion. Set a follow-up checkpoint based on the actual operational process, rather than marking unknown cases as successes or failures to complete a dashboard. Later updates should preserve the original cohort so the team can understand what matured.

Also distinguish the delivery date from the outcome date. A policy issued later can belong to an earlier call cohort. Moving it into whichever week makes the report look better breaks the relationship between spend and results. The cost-per-issued-policy guide explains why both the outcome definition and the cost boundary matter. Your pilot report should state its cutoff date and remaining uncertainty in plain language. That is more useful than a polished return figure whose inputs silently change between meetings.

Investigate patterns without inventing explanations

Review individual calls for operational evidence, using the agency's authorized access and data practices. Repeated wrong destinations, unavailable language coverage or missing dispositions can justify changes even in a small sample because the issue is directly observed. A claim that a particular hour or caller group always converts better requires a different level of evidence. Avoid attributing a result to personal characteristics, market stereotypes or agent quality when the sample does not support that explanation.

When a problem is confirmed, change the relevant part of the process and record the change. Do not silently pool the before and after observations as though conditions were identical. For example, calls received before a destination repair and those received afterward answer different delivery questions. You can describe the whole pilot operationally while keeping those segments distinguishable. This makes the next decision more honest: the team knows whether it is evaluating a corrected process or averaging across a failure that no longer applies.

Choose a bounded next step

A reasonable next step may be another limited observation period, a pause for operational repair or no further purchase. It should not be an automatic commitment to scale because one early call went well. Define the budget boundary, staffed coverage and evidence you want from the next period. Increasing the sample only helps when the collection process remains interpretable. More records with inconsistent definitions can create a larger spreadsheet without improving the agency's understanding of its performance.

At Valtier, the stated trial is 5 calls funded by a $300 wallet top-up, with a stated cost of $60 per billable call. These commercial inputs describe the offer; they do not promise a conversion outcome or define how much evidence is sufficient for every agency. Confirm the current terms before starting. Treat the trial as an opportunity to examine fit and execution, then choose any continuation using your own budget and operational constraints. Avoid translating the trial price into a predicted commission amount.

Write a report that separates observation from expectation

A useful pilot summary contains the original question, call count, relevant costs, completed outcomes, pending outcomes, observed operating issues and proposed next action. Label projections explicitly as scenarios. If you illustrate what a larger volume might produce, explain that you are holding assumptions constant rather than forecasting results. Keep the actual pilot data and the hypothetical scenario visually distinct so that someone reading the summary later cannot mistake one for the other.

Include a short explanation of what remains unknown. This could be outcome maturity, staffing under heavier demand or whether an early pattern repeats. That statement is not an argument against testing; it identifies the purpose of the next decision. A small pilot is most valuable when it reduces specific operational uncertainty and prevents an agency from making a much larger commitment on the strength of a few vivid conversations.