Prepare a library you can actually score.
A useful technical-library test begins with document identity. Retain the title, identifier, revision, date and applicability information for each file. Give the reviewer the source pack before asking a model to answer questions. A model-generated answer cannot serve as its own ground truth.
The starter pack uses plain text and invented library-administration rules. It contains no equipment instructions. It makes revision and citation checks easy to inspect without representing a real military manual or a measured application demonstration. A later corpus should add rights-cleared PDFs, tables, diagrams and scanned pages that reflect the intended task.
Four questions reveal different failure modes.
Revision B explicitly supersedes A. Compare Section 2, Table 1, row 3: the review-owner field changes from optional to required. Section 3 changes both the review schedule and the responsible role. Ask the evaluator to identify each change and cite the relevant revision. A citation to the right title but wrong revision should not pass.
The table gives an illustrative reference key. No product run has been scored here. Record the actual response separately and inspect whether the cited words support every material statement.
| Task | Hand-authored expected finding | Where to check |
|---|---|---|
| Present answer | The reference librarian owns the current catalog review. | Revision B, Section 3 |
| Cross-document change | Owner field optional → required; quarterly → monthly; coordinator → librarian. | Both revisions, Section 2 Table 1 row 3 and Section 3 |
| Conflicting version | Quarterly review belongs to superseded revision A. | Headers and Section 3 in both revisions |
| Missing answer | The remote archive address is not stated. | Both revisions, Section 4 |
Judge the answer and its evidence together.
Check that the cited file exists, the cited section contains the claimed passage and the selected revision is applicable. Then look for omitted conditions and contrary passages. A link can be real while its passage fails to support the answer.
Keep separate columns for answer correctness, source support, citation completeness, version correctness, appropriate abstention and reviewer corrections. Do not collapse them into a single confidence number. For an incomplete answer, retain which fact was missing and how the reviewer resolved it.
Extend the test to the documents that matter.
Plain-text success would not establish performance on a scanned table, a diagram or a long technical PDF. Add a small representative pack and record the file support actually available in the candidate package. Check extraction, page mapping and source navigation before scoring an answer.
Include a question requiring two documents, an unreadable page, an ambiguous table heading and a genuinely unsupported question. When a diagram cannot be interpreted reliably, the output should expose that limitation for the reviewer. Generated text must never replace the applicable approved procedure.
Measure the complete reviewed task.
Record the Mac, operating system, package and model versions, model settings, document set, question wording and repeated-run conditions. Measure time spent loading, asking, opening citations and correcting. Keep latency separate from task acceptance; a quick response with the wrong revision is still a failure.
If disconnected use is a requirement, test the complete agreed sequence under the intended network rules. Local generation alone does not establish startup, import, licensing, update or recovery readiness. Bring the observed gaps into the deployment review.
Evaluate one approved technical library.
Start with a nonsensitive description of the document types, languages, device profile and question set. Agree on corpus rights, review ownership and handling before sharing files. The outcome should be a repeatable acceptance decision, with observed results and unresolved requirements kept together.
Sources and further reading
- Qwerty product and selected-file workflow · Reviewed 2026-10-16
- Qwerty deployment and supported environment · Reviewed 2026-10-16
- Qwerty evaluation methods · Reviewed 2026-10-16