Use five separate checks.
Begin with the claim you want to rely on. Break a sentence with several factual assertions into smaller statements so you can judge each one. A citation at the end of a paragraph does not necessarily support every claim in it.
Check the source itself rather than the answer’s description of it. Record the result of each check, including missing information. This makes an incomplete source trail visible and helps the reviewer distinguish an inaccurate answer from a correct answer with inadequate evidence.
| Check | Question to ask | What to record |
|---|---|---|
| Existence | Does the cited document and location exist? | File identity and exact section, clause or page. |
| Support | Does this passage establish this particular claim? | The supporting words and any mismatch. |
| Completeness | Are material conditions or contradictory passages omitted? | Exceptions, dependencies and missing sources. |
| Version | Is this the applicable revision for the question? | Date, revision, supersession and amendment. |
| Absent evidence | Does the answer say when support is missing? | What cannot be established from the packet. |
A real passage can still be the wrong revision.
In the synthetic training-library files, revision A Section 3 says catalog review is quarterly and belongs to the training coordinator. Revision B explicitly supersedes A and changes both details. If an answer to “Who owns the current review?” cites A, the source may exist and the quotation may be accurate while the answer still fails the version check.
Open revision B Section 3 and compare both headers. The hand-authored reference finding is the reference librarian with monthly review. Section 2 Table 1 row 3 adds another distinct change: review owner becomes a required field. A summary that reports only the timing change is incomplete for a question asking what changed.
A clause may have been amended.
The fictional agreement C1 states a 45-day invoice review period. A1 in the supplied amendment replaces it with 30 days. The invented playbook P1 flags periods longer than 30. A comparison that cites C1 and reports a flag without considering A1 misses supplied evidence.
This demonstrates a document relationship, not a legal conclusion. The fixtures and expected findings have not received practitioner review. In a real legal workflow, a qualified reviewer establishes the applicable documents, comparison criteria and interpretation. A source link supports inspection; it does not replace that responsibility.
Preserve a question the documents cannot answer.
The synthetic manual Section 4 does not provide a remote archive address. The policy fixture Section 4 provides no disclosure rule. The agreement C4 supplies no renewal provision. An answer should identify the absent support rather than complete the document from a plausible pattern.
A useful response names the source boundary: “The remote archive address is not stated in the supplied files.” That is more inspectable than guessing. Where the packet may be incomplete, the reviewer can decide whether to request another document. Absence from a selected packet does not prove the fact does not exist elsewhere.
Keep a small, repeatable review record.
For each question, retain the question wording, document identifiers and versions, candidate answer, cited locations, claim-by-claim checks, corrections and final reviewer decision. Record the package, model, Mac and settings alongside the task set. Use permitted records and avoid putting confidential document text into marketing analytics.
Count answer correctness, source support, completeness, version errors and appropriate abstention separately. Retain failed and unanswered tasks. If several reviewers score the same pack, examine disagreements before describing a score as reliable. The small fixtures here are editorial references, not measured Qwerty output or a reliability study.
- Define expected source-backed findings before the run.
- Inspect the cited passage in its document context.
- Check revisions, amendments, exceptions and missing evidence.
- Record actual errors and correction time.
- Accept the work only after the assigned reviewer decides.
Evaluate citations on the files your team uses.
Begin with a small authorized corpus and familiar questions. Add cross-document questions, conflicts, a table or scan where supported, and an unanswered question. Confirm the actual file and source-navigation support in the candidate configuration. The purpose is a source trail a person can inspect and a task whose result they can accept.
Sources and further reading
- Qwerty product and selected-file workflow · Reviewed 2026-10-16
- Qwerty deployment and supported environment · Reviewed 2026-10-16
- Qwerty evaluation methods · Reviewed 2026-10-16