We revised a chatbot’s source material: results from the same 30 questions

More precise source material helped in this test, but it did not remove every error. With the same 30 questions, we classified 13 answers as complete and correct against the reference requirements using the original document and 27 using the revised version. After a supplementary check for support in the supplied source, 10 of those original answers and 27 revised answers passed both checks. Two revised answers still omitted a required qualification, and one made an overly definite dispatch statement.

We ran the experiment through a real model API on 21 October 2026. The shop, Severka, is fictional; no customer account or customer data was involved. This is a laboratory demonstration of how supplied information can affect answers. It is not a production accuracy benchmark for Breezaro, a workload study or evidence of increased sales.

What we held constant

Before making the requests, we prepared two Czech documents, 30 Czech questions and a criterion for each answer. The revised document defines the fictional shop's fixed reference policy. The original deliberately contains an obsolete free-shipping threshold and missing conditions. It is not a randomly selected example of a typical company's knowledge base.

The instruction stayed the same: use only the supplied material, acknowledge missing information and never invent completed actions. All 60 responses finished with completed status. We did not fine-tune the model. The only variable changed between conditions was the document supplied with the question.

This English article reports the same Czech experiment. The English explanations of examples below are translations or paraphrases, not a second English-language run.

We placed the entire document directly in context. We did not test website ingestion, document chunking, retrieval or Breezaro's production conversation pipeline. In a real installation, a well-written fact still has to be imported and retrieved successfully.

What we changed in the documents

The original included statements such as “we dispatch quickly” and “delivery usually takes two to three days.” The discount code had no exclusions or expiry. The backpack description offered only a laptop screen-size limit. The free-shipping threshold was also wrong against the reference policy.

The revision supplied:

• a CZK 1,990 free-shipping threshold, calculated after discounts and excluding shipping charges;
• received payment before noon as a condition of same-working-day dispatch;
• a distinction between carrier handover and estimated arrival;
• discount eligibility, exclusions and an expiry with a time zone;
• laptop-pocket dimensions instead of relying on screen size;
• boundaries for requests requiring a person's decision or access to a connected system.

Some changes corrected stale information; others added facts that had been absent. The result cannot be attributed solely to clearer formatting or better headings. All shop policies in the experiment are fictional test material, not a recommended legal policy for your store.

Results against the reference requirements

A safe acknowledgement of missing information counted as incomplete when the reference policy contained the requested fact. That is preferable to making something up, but it still leaves the customer without the answer. For a live-stock question without inventory access, acknowledging that boundary was the intended successful outcome.

The questions covered three groups:

A separate AI assistant reviewed the responses against the published criteria. This was not an independent human evaluation or a blinded study. The review file explains every decision and the ambiguous cases. For example, the grading strictly required calendar days and the starting point of the delivery estimate where the criteria specified them.

The original rubric contained an ambiguity: it prohibited invented claims, but its category definitions were based on the reference policy. During review, we therefore separated correctness against the reference from support in the document actually supplied. Three original answers matched the reference despite making a definite assertion not sufficiently established by the original source. Ten original answers and 27 revised answers passed both checks. This supplementary grounding audit was added during review, not registered as an original metric. It is reported separately in the download, and the original input remains unchanged.

Two differences worth inspecting

Shipping after a discount, question 3. The basket contained CZK 2,100 of goods with a ten-percent code. The original answer promised free shipping and added an unsupported rule that the threshold used the price before discount. The revised answer calculated CZK 1,890 and correctly added CZK 89 for shipping.

Laptop dimensions, question 14. The customer supplied dimensions of 35 × 23 × 1.8 cm. The original answer said the laptop would fit based only on a “up to 15-inch” description. With the actual pocket dimensions of 34 × 24 × 2 cm available, the model said it would not fit and referred to the dimensional limit.

The first example shows the cost of an unspecified calculation rule. The second shows why a precise specification is more useful than a broad compatibility label.

What still failed after the revision

In question 4, the answer repeated payment before noon but omitted that the money must have been received. In question 6, it correctly excluded weekends but omitted that the delivery estimate starts at carrier handover. Both were classified as incomplete.

In question 17, the model correctly distinguished sending a bank transfer from receiving funds, then stated, in translation: “If the money arrives tomorrow, dispatch will start tomorrow.” It did not know the arrival time or whether tomorrow was a working day. Under the strict interpretation used in the review, that is an unsupported promise. A looser reading of the original Czech as “not before tomorrow” would change the classification; the review explicitly records that uncertainty.

The remaining failures matter. Adding a condition to a document does not guarantee the model will carry it into every relevant answer.

How to run a useful version for your own business

Start with repeated customer questions, add conditional cases and include requests where the assistant should acknowledge a limit. Write the expected outcomes before testing. Revise the source, keep the model and questions constant, and retain failures in the record.

Then test the whole workflow in Breezaro: source import, refresh, the actual widget answer and any handover to an operator. Use the 30-question testing checklist and the wrong-answer troubleshooting guide as starting points.

This single run demonstrates an improvement on a deliberately prepared set. It does not predict performance for another shop, language or model. Temperature zero does not guarantee identical responses on a repeat run, and the questions were constructed around the reference policy rather than sampled from live customer traffic.

Download the evidence and repeat the experiment

• Both documents, instructions, questions and original criteria
• All 60 unedited responses and run metadata
• Classifications, rationales and judgment caveats
• The experiment runner

The runner needs your own API key in OPENAI_API_KEY, input.json in the same folder, and the arguments --run-live --output=/absolute/path/to/new-results.json. It makes 60 paid requests and refuses to overwrite an existing output file. The input hash stored with the results lets you check which exact source material was used.