Benchmark: Claude Code with and without ISMS Copilot

We measured one question: when you do GRC work in Claude Code, what does a correct answer cost in Claude usage if Claude Code researches it alone, and what does it cost if Claude Code hands the question to ISMS Copilot over MCP? This page gives the method, every headline number, the run where delegation did not help, the limits of the test, and what the results do and do not support.

What this supports

On the tested items, Claude Code delegating to ISMS Copilot matched the accuracy of Claude Code researching alone with web search: the pre-registered accuracy verdict was a tie in every scenario. It spent less Claude API money per correct answer on lookups and quick advice, and about the same on a long mixed coding and compliance session. The frameworks tested on 2026-09-28 are ISO/IEC 27001:2022 Annex A, ISO/IEC 42001:2023 Annex A, CMMC 2.0 Level 2, DORA RTS (EU) 2024/1774 and NIS2 IR (EU) 2024/2690, and only those. The 2026-09-29 round added 10 held-out items from NIST SP 800-171 Rev. 2 and the Cyber Resilience Act. Delegated turns count against your ISMS Copilot plan, which is not in the dollar figures. Read the limits below before relying on any of these numbers.

Limits of this test

• Accuracy is at ceiling. In v3, both setups scored 100% on the lookup, long-session and quick-advice scenarios, and in the 2026-09-29 round (which did not rerun the long session) both scored 100% on lookups and quick advice. With these items and the pre-registered rule, the test could only detect an accuracy gap of 10 percentage points or more. "Matched" on this page means that no difference of that size was found, not that the two setups are proven equally accurate.
• The two setups did not get the same instructions. In v3, the held-out check and the first 2026-09-29 round, Claude Code alone had no instruction file. The delegating setup had the routing rule as its CLAUDE.md, and that rule tells it not to re-check answers ISMS Copilot confirmed, so part of the cost difference could have come from that instruction rather than from delegation. The control has now been run, on 2026-09-29: Claude Code alone, told to verify on the web only what it is unsure of, used about as many Claude tokens per correct answer as with no instruction on lookups and quick advice, and the delegating setup still used fewer Claude tokens per correct answer than it in every scenario (1.6 to 3.2 times fewer). See the section "Round of 2026-09-29 (control)" below. The control tested one thrift instruction, written by us; a better one could narrow the gap.
• One account, with its memories and, from 2026-09-29, its company profile. Every delegated turn ran on one ISMS Copilot account, our own (the founder's), on its paid plan and with its account memories. Since 2026-09-29, ISMS Copilot adds the company profile as background to MCP turns for accounts that have one, and this account has one, so the 2026-09-29 round includes it. The 2026-09-28 runs did not. Results on an account with a different plan, different memories or no profile may differ.
• The answer key was written by an AI model. A Claude model wrote the key from the official sources listed below. The DORA RTS and NIS2 IR items were spot-checked against the official text. A review of the key by a human expert is pending, and its result will be added here with its date.

Setup

The full method (design, answer key and scorer) is internal and available on request: write to support@ismscopilot.com.

The two setups get identical task prompts, and the prompts never mention ISMS Copilot:

Both setups have web search and fetch, because a real user has them. The routing rule decides what gets delegated. The other account tools (company context, workspaces, memories, documents) were not allowed. ISMS Copilot's server-side account memories could still shape its replies, and from 2026-09-29 so could the account's company profile, which ISMS Copilot adds on its side (see Limits of this test, above).

Scenarios

• S1, lookups. 20 requirement themes, and the task is to give the exact identifier for each: 4 DORA RTS articles, 4 NIS2 IR annex points, 4 ISO/IEC 42001:2023 Annex A controls, 4 CMMC Level 2 practices and 4 ISO/IEC 27001:2022 Annex A controls. The answer is a JSON file.
• S3, long session. One Claude Code session of 8 prompts on a small Python repository: 4 code edits (checked by hidden tests) alternating with 4 GRC questions with keyed answers. The GRC answers are scored; the code checks are reported separately.
• S4, quick advice. 5 yes or no questions, each with the identifier that decides it and a one-sentence reason. Correct only if both the yes or no and the identifier match.
• Held-out. 20 new items from DORA RTS and NIS2 IR, in the S1 format, that were never in the answer key before (see "The story" below).

All fixtures are fictional. No customer data was used.

How the answer key was built

Every keyed item has the canonical identifier, the requirement in one sentence, a citation and a quote from or pointer to the source. The key was built from official sources only, and never from ISMS Copilot, because that would be circular:

• DORA RTS (EU) 2024/1774 and NIS2 IR (EU) 2024/2690: the official text from the EU Publications Office (CELEX 32024R1774 and 32024R2690), cited to the EUR-Lex addresses.
• CMMC 2.0 Level 2: the DoD CMMC Level 2 Assessment Guide, with the NIST publication its practice numbers come from.
• ISO/IEC 27001:2022 and ISO/IEC 42001:2023 Annex A: ISO/IEC 27001 and 42001 items use control identifiers, described in our own words.

The key was frozen before the first run, and every run records a digest of it. It did not change during the benchmark.

Scoring

Scoring is mechanical: no model judges any answer. A script reads the identifier field of each answer and normalizes it by kind, then compares it with the key:

• DORA RTS: "Article 7", "Art. 7(4)" and "Article 7(4) of ..." all count as Article 7.
• NIS2 IR: the answer must be a real annex point and must be in the item's pre-registered accept list (the operative point and its heading). A sibling point is wrong.
• ISO/IEC 27001:2022 and ISO/IEC 42001:2023: the Annex A number must match exactly. 2013 numbering, or an Annex B reference for 42001, is wrong.
• CMMC Level 2: the practice number must match ("SC.L2-3.13.11" and "3.13.11" both count).

A missing file, invalid JSON or an unparseable identifier counts as wrong. Runs that hit a cap would have been scored as they were, and never rerun.

Metric

The primary metric is Claude API dollars per correct answer: the Claude cost of the 3 runs of a scenario divided by the correct answers across those 3 runs. It is the list price that Claude Code reports, which is what a Claude subscriber pays for extra usage beyond the plan. It includes the web search sub-calls and search fees.

ISMS Copilot plan usage is not in this figure, and it is not free. Delegated turns count against your ISMS Copilot plan (see Where ISMS Copilot usage is billed). Leaving it out favours the delegating setup on this metric.

Accuracy, harness tokens, turns and time are reported alongside. Tokens and dollars differ: repeated context read from the prompt cache is billed at a fraction of the input price, so a setup can use more tokens and still cost fewer dollars.

Pre-registration and the logged amendment

The design, answer key, scorer and win rules were committed before any run, with a commitment to publish the results whatever they showed. The pre-registered win rules: accuracy wins at 10 percentage points or more, dollars per correct answer at 1.25 times or more; anything closer is a tie. The v3 rerun and the held-out check were each pre-registered the same way before their first run.

One amendment was made during the first run (v2). A per-run cap of 2 million harness tokens, meant as a runaway guard, stopped the benchmark after the first pair, when Claude Code alone finished a lookup run normally at 2.38 million tokens (all 20 correct). The cap was raised to 6 million for both setups, the change was logged with its time, and nothing was rerun. It was made after that first result had been seen. Without it, no scenario would have a verdict. Dollar caps and the overall spend ceiling did not change.

The story: v2, the fix, v3 and the held-out check

v2 found a knowledge gap. In the first fair run, ISMS Copilot matched Claude Code alone on every ISO/IEC 27001, ISO/IEC 42001 and CMMC item, but missed most DORA RTS article numbers and NIS2 IR annex points. Its replies marked those identifiers as unconfirmed.

The fix. We added DORA RTS and NIS2 IR knowledge to ISMS Copilot at article and annex-point level, built from the official EU texts, and added a line to the routing rule: verify any answer ISMS Copilot marks as unconfirmed. Then we reran the same scenarios with the same key and scorer.

v3 (same key, same scorer, both setups rerun):

Pre-registered verdicts: accuracy is a tie in all three scenarios. Dollars per correct answer favour the delegating setup on S1 and S4, and S3 is a tie. On S3 every code check passed in both setups.

The gain came from ISMS Copilot's own answers. A diagnostic added after the runs (no verdict attached) found that its replies carried the right DORA RTS and NIS2 IR identifier on 24 of 24 S1 items before any web check by Claude Code. No v3 reply marked an identifier as unconfirmed, so the new verification line was not exercised. Claude Code still double-checked on the web in S1 and S3 without being asked, which changed no answer and raised the delegating setup's cost.

The held-out check (20 new items). Because the knowledge fix was scoped from v2's misses, we checked for teaching to the test: 10 DORA RTS articles and 10 NIS2 IR annex points that the key never contained, built from the official texts the same way, in the S1 format.

Pre-registered verdicts: accuracy is a tie (the difference is under 10 points), and dollars per correct answer favour the delegating setup (1.34 times). The delegating setup used more tokens here because Claude Code re-checked ISMS Copilot's answers on the web in two of three runs. In the run where it did not, it wrote ISMS Copilot's answer to the file and scored 20 of 20 for $0.15.

Round of 2026-09-29

A scoped rerun after a production release that tightened the Claude Code routing rule and started adding the company profile to MCP turns (the two changes cannot be separated in these numbers). Same pinned Claude Code and model, same key and scorer, N=3 per scenario per setup. It ran S1 and S4, and 10 fresh held-out items (5 NIST SP 800-171 Rev. 2, 5 Cyber Resilience Act) that were never in the key. S3 was not rerun. The v3 and held-out tables above keep their original numbers.

Pre-registered verdicts: accuracy is a tie in all three. Dollars per correct answer favour the delegating setup on S1 and S4, and are a tie on the new items. The one miss came from a delegating-setup run that never called ISMS Copilot: the routing rule's framework list did not then name NIST SP 800-171 or the CRA, so in 2 of 3 runs Claude Code researched those items on the web instead. The one run that did delegate scored 10 of 10. The rule was updated on 2026-09-29 to name both frameworks. The same limits apply: accuracy is at ceiling, the instructions differ between setups, and this round includes the account's company profile.

Round of 2026-09-29 (control)

This round answers the question the second limit above leaves open: does the saving come from delegating to ISMS Copilot, or only from the routing rule telling Claude Code not to re-check what it is confident about? It adds a control setup with a similar instruction and no delegation. The design, win rules and every item were committed before the first run, with the same commitment to publish the results whatever they showed. It ran after the routing rule update of 2026-09-29 that names NIST SP 800-171 and the CRA.

Same pinned Claude Code 2.1.283 and claude-sonnet-5, same scorer, N=3 per scenario per setup, and the five setups of each scenario ran at the same time. Scenarios: S1 lookups (20 items), S4 quick advice (5 questions) and 16 fresh held-out items that were never in the key (6 NIST SP 800-171 Rev. 2, 6 Cyber Resilience Act and 4 DORA RTS), built from the official texts. S3 was not rerun.

The metric here is Claude tokens per correct answer: input, cache read, cache creation and output tokens across all models (web search sub-calls included), summed over the 3 runs and divided by the correct answers. It is a proxy for how fast a Claude subscription window burns, not the meter itself: Anthropic does not publish how cached tokens count against subscription limits.

Pre-registered verdicts against the control (a win needs 1.25 times or more): A2 used 3.23 times the Claude tokens per correct answer of B on lookups, 2.44 times on quick advice, 1.59 times on the fresh items and 2.64 times pooled, so B wins in every scenario. B also used fewer Claude API dollars per correct answer than A2 in every scenario. The thrift instruction alone barely moved Claude Code on lookups and quick advice (A2 against A: 1.02 and 1.01, ties) and helped on the fresh items. The saving comes from delegating, not from telling Claude Code to be thrifty. A0 needs few tokens because it answers from memory, and it is often wrong (see accuracy), so its figure is not a usable result.

Accuracy, pooled: A 99.2%, A2 98.4%, A0 60.2%, B 100%, B-fast 100%. A, A2, B and B-fast are ties under the 10-point rule in every scenario; A0 lost 20 to 46 points in every scenario. The ceiling caveat above stands: ties are ties by rule, not proof of equal accuracy.

ISMS Copilot plan usage per correct answer. Delegated turns draw on your ISMS Copilot plan's 4-hour window. The table gives ISMS Copilot tokens (input and output) per correct answer as a share of one window, using the production plan limits of 2026-09-29. Essential is a grandfathered plan. Unlimited has no token cap in the window.

The Free plan also allows only 10 messages per 4-hour window. B sent one ISMS Copilot message per run. In fast mode Claude Code split the work into up to 5 messages per run, and on quick advice that made the message limit bind first: about 7.3% of the Free message window per correct answer, against 4.8% of the token window, so about 3 quick-advice deliverables of this shape fill a Free window. In the other cells the token window fills as fast as the message window or slightly faster. B-fast only simulates fast mode on a paid account.

Caveats for this round: N=3 per scenario per setup; one harness version and one model; one thrift instruction, written by us, and a better one could narrow the gap (this round shows that this reasonable one does not); every delegated turn ran on our own (the founder's) paid account, with its memories and its company profile added to MCP turns; the token metric is a proxy for subscription usage; the answer key was written by an AI model; accuracy is at ceiling.

Where delegation does not help

The first benchmark we ran tested three small ISO 27001 tasks that Claude already knows: a gap check of four short fictional policies, an access-control policy draft and Statement of Applicability entries for 10 controls. Each task wrote its deliverable to a file, and no lookups were needed. Neither setup had web tools, and answers were checked for completeness only, not correctness.

Delegation used 1.6 to 2.1 times more Claude money here. The delegating runs took more turns (loading the MCP tools, waiting for the reply), Claude Code still read every local file, and the full deliverable came back through Claude Code, which then wrote it to disk. When a task is small, Claude already knows the answer and nothing needs looking up, keep it in your agent. See what to delegate and what to keep local.

Caveats

• Small N. 3 runs per scenario per setup, on 2026-09-28, plus two scoped rounds on 2026-09-29.
• One harness. Claude Code only. Routing rules for Codex, Cursor, OpenCode and Grok are documented, but those harnesses were not measured.
• One model. claude-sonnet-5. Another model, or a later version of Claude Code, may behave differently.
• One account. All delegated turns ran on one production ISMS Copilot account, on its paid plan, whose server-side memories, and from 2026-09-29 its company profile, may shape replies.
• Fictional fixtures. Invented companies, policies and code.
• Document review not tested. The MCP has no document upload path. A 40-page policy review ran in v2 and tied, but Claude Code did the review itself in both setups, so it says nothing about ISMS Copilot reviewing your documents.
• Key author. The items were written by a Claude model, the same family as the harness, from the official texts. The DORA RTS and NIS2 IR items were spot-checked against the official text, and the held-out items were checked against it before the runs. A human expert review of the key is pending.
• What is published. This page carries the method and every headline number. The raw transcripts stay private: they contain production conversation identifiers and verbatim model output. The task prompts, answer key, scorer and scored outputs are kept in our internal repository and are not published with this page.

What we claim and what we do not

We claim:

• On the tested items in the five frameworks above, Claude Code with ISMS Copilot matched the accuracy of Claude Code researching alone with web search (an accuracy tie in every scenario, under the pre-registered rule).
• On those lookups and quick-advice questions, it spent less Claude API money per correct answer (about half in the v3 run, about three quarters on the 2026-09-28 held-out check), and about the same on a long mixed session.

We do not claim:

• That ISMS Copilot is cheaper than Claude, or any percentage saving on your bill.
• Better answers than Claude. The accuracy results are ties.
• Anything about frameworks we did not test, other harnesses or models, or document review.
• That delegation saves Claude usage on small tasks Claude already knows. It cost more there.
• That delegated turns are free. They count against your ISMS Copilot plan.