Answer quality: the scaffold benchmark

Can you just paste a good prompt into a raw LLM and get the same compliance answers? On 2026-09-02 we ran a pre-registered benchmark to answer that for the chat product: the full ISMS Copilot stack versus the exact same base model (GLM 5.3-Flash, identical serving configuration) wearing no scaffolding at all. This page records the method, the numbers, and the losses. It is our own internal eval with the sample sizes we actually ran; read the caveats as part of the results.

What was compared

Six arms, identical model, reasoning effort, temperature, output budget, and provider pin. The only variable is what surrounds the request:

Twenty frozen tasks: sixteen standard questions across nine frameworks, and four traps (a nonexistent ISO 27001 control, superseded 2013 numbering, a nonexistent GDPR article, a wrong DORA date and scope premise). Nine tasks use well-known frameworks (ISO 27001, GDPR, SOC 2) and eleven use less common ones (ISO 42001, Australia's ISM, DORA, NIS 2, TISAX, Singapore's MTCS). Two tasks include a fictional contract or design document that answers must cite. Every answer was scored by a scripted judge model from a different vendor family, blind to which arm produced it, plus a deterministic checker that verifies every cited control or article identifier against the real framework registries.

The pre-registered headline, stated plainly

The primary endpoint was frozen before the run: the full product versus the best of the two DIY prompts. The full product scored 92.1% against 76.0%, a difference of +16.1 points with a 95% confidence interval of [3.4, 30.2]. Our pre-registered tie band was 18.2 points, derived from how much the judge's scores wobble when the same answer is re-judged. The difference did not clear that band, so the pre-registered verdict is a TIE, and we report it as a tie. The confidence interval excludes zero, so the direction is positive; we did not weaken the band after seeing the numbers.

What the layers show

• The knowledge module is the engine. A plain consultant prompt plus the injected framework module scored 96.9%, the highest of the six arms on this task set. That module is the same curated reference our system injects when your question names a framework.
• The product's edge concentrates on less common frameworks. On ISO 27001, GDPR and SOC 2, a good DIY prompt is statistically at parity with the product (88.0 vs 88.9). On ISO 42001, the Australian ISM, DORA, NIS 2, TISAX and MTCS, the product scores 94.7% against 66.3% for the best DIY prompt. The base model simply does not carry reliable structure for frameworks this young or this regional.
• The deterministic citation check agrees. Outside the trap battery, the naked model fabricated 8 control or article identifiers, including an invented ISO 42001 Annex A structure. The full product fabricated zero.

The losses we are publishing

• The knowledge-plus-documents arm outscored the product. It scored 98.7% against the product's 92.1%, with five task cells differing. Prompt A supplied with the same knowledge module, current date, and relevant documents outscored the production assembly on this single-question task set. The benchmark did not test the value of multi-turn state, automatic detection, workspace handling, or other workflow features.
• The product failed one of the four traps. Asked to explain ISO 27001 control A.8.35, which does not exist, the product confidently described it as secure coding, which is actually A.8.28. Its injected control table in the same prompt ended at A.8.34, and both knowledge-armed DIY configurations refused the false premise. One candidate mechanism is that the production prompt's action-bias directives affected refusal behavior, but the product arm differs from those DIY arms in several components, so this benchmark cannot attribute the failure to that mechanism. It is a reproducible product defect, and it is on the fix list.
• Detection is name-level. One task described a personal data breach without ever naming GDPR; the product injected nothing and still answered well on base-model knowledge. That is the documented behavior, not a surprise: pin the framework when you know it.

Method record and caveats

Single day (2026-09-02), one sample per arm per task, scripted judge, no human calibration yet. Judge scores on identical answers can differ by double digits on retest, which is why the tie band is wide and why the headline verdict is a tie. Twenty tasks is enough to show the layer effects above, not enough for fine-grained per-task claims. The frozen task set, rubrics, pre-registration, raw judgments, and the full report are preserved in the engineering record; the harness and task keys were adversarially reviewed by a second model before the freeze. This is not a comparison against Claude or any frontier model, it is not an audit opinion, and it supports no claim that hallucinations are eliminated.

Related: the API aliases have their own evaluation record on the model quality and grounding page.