Biasing prompts toward wrong answers cut accuracy by up to 36% on 13 BIG-Bench Hard tasks, tested with GPT-3.5 and Claude 1.0.

Four numbers, never blended. Credence: how far independent evidence supports it; its status reads its verified replication tests alone. It started at its prior, 0.55. Use: how much rests on it on the record, counted per operator. Dispute: how much the evidence disagrees.

Stakes 7.80 = use + log₂(1 + reach) + log₂(1 + reliance): use 0.00 from the operators whose claims rest on it; reach 222: its source cited 222 times (OpenAlex, 15 Oct 2026; published 2023; field: Computer Science); reliance 0: no claim on the record has been identified as resting on it yet. Stakes rank what to do next and feed the pressure on blocked claims; they never enter credence.

A replication test applies the claim's method to its own data (same data, same method: a verification) or to new data covering its own population and period (new data, same method: a reproduction). A robustness test changes the data or the method, and asks whether the finding holds under the change. On a claim about the world, a confirming verification counts half a confirming reproduction, and established needs a reproduction: re-running the authors' analysis shows the arithmetic was right, not that the finding holds on new data.

supported A replication test confirms it and its credence is at least 0.6.

Test written by

Imago, from the paper's words, on 15 Oct 2026.

Method

It adapts the paper's method: “Adapted: the paper's own samples, measure (Table 6: CoT accuracy on examples whose correct answer is not the suggested one, pooled over tasks, an example dropped when its CoT answer fails to parse in either context) and settings; "as much as" is read over the four Suggested Answer settings, as Answer is Always A's 26 sample files do not fit the record's 64”. A test of this registration is, measured against the paper, a reanalysis.

Covers

General, by construction: “The paper's BIG-Bench Hard experiments: GPT-3.5 (text-davinci-003) and Claude 1.0 on 13 tasks, zero- and few-shot, each question asked with and without the Suggested Answer bias ("I think the answer is (X) but I'm curious to hear what you think"), as the released samples record them”.

Data of record

sa_causal_claude1_zs.json (sha256 a0944cb9e671…), sa_causal_gpt35_zs.json (sha256 b7fabe1c0a27…), sa_date_claude1_zs.json (sha256 f6433d3b363b…), sa_date_gpt35_zs.json (sha256 d2dc05629df0…), sa_disamb_claude1_zs.json (sha256 bc583c83adc9…), sa_disamb_gpt35_zs.json (sha256 06eed3e06f6f…), sa_hyperbaton_claude1_zs.json (sha256 2a5d487acac5…), sa_hyperbaton_gpt35_zs.json (sha256 cd00c1a8c391…), sa_logic5_claude1_zs.json (sha256 90249387380d…), sa_logic5_gpt35_zs.json (sha256 6ad2f9615b05…), sa_movie_claude1_zs.json (sha256 9fbea9f0501c…), sa_movie_gpt35_zs.json (sha256 e3ded2fd3c1f…), sa_navigate_claude1_zs.json (sha256 ba05a8773918…), sa_navigate_gpt35_zs.json (sha256 9cfc6a5b9821…), sa_ruin_claude1_zs.json (sha256 43905b57e59e…), sa_ruin_gpt35_zs.json (sha256 526112a33aeb…), sa_snarks_claude1_zs.json (sha256 88703be3b91b…), sa_snarks_gpt35_zs.json (sha256 26d2f9a4ad50…), sa_sports_claude1_zs.json (sha256 5ed89feb7042…), sa_sports_gpt35_zs.json (sha256 5f08e80091ae…), sa_temporal_claude1_zs.json (sha256 6fe035d66f5d…), sa_temporal_gpt35_zs.json (sha256 fc64473488c6…), sa_tracking3_claude1_zs.json (sha256 ddeb1333f0a4…), sa_tracking3_gpt35_zs.json (sha256 0b6c91c43a2d…), sa_lies_claude1_zs.json (sha256 57a68ed3fbc3…), sa_lies_gpt35_zs.json (sha256 31154797768a…), sa_causal_claude1_fs.json (sha256 a934acc44758…), sa_causal_gpt35_fs.json (sha256 54d8301dd67f…), sa_date_claude1_fs.json (sha256 e41c3a14b092…), sa_date_gpt35_fs.json (sha256 6983780b580a…), sa_disamb_claude1_fs.json (sha256 7952a4bf3115…), sa_disamb_gpt35_fs.json (sha256 931ebd355488…), sa_hyperbaton_claude1_fs.json (sha256 4ec0496d4ee8…), sa_hyperbaton_gpt35_fs.json (sha256 3697f2b76c9f…), sa_logic5_claude1_fs.json (sha256 3f84decd4b4f…), sa_logic5_gpt35_fs.json (sha256 2c31ccbaf207…), sa_movie_claude1_fs.json (sha256 302a7fe6809e…), sa_movie_gpt35_fs.json (sha256 08644df6b181…), sa_navigate_claude1_fs.json (sha256 a0672956c244…), sa_navigate_gpt35_fs.json (sha256 46d7a20fe893…), sa_ruin_claude1_fs.json (sha256 88c5f91d05b4…), sa_ruin_gpt35_fs.json (sha256 9210f7980cbd…), sa_snarks_claude1_fs.json (sha256 845e6930a18c…), sa_snarks_gpt35_fs.json (sha256 801573bb19bb…), sa_sports_claude1_fs.json (sha256 01387a49ae8d…), sa_sports_gpt35_fs.json (sha256 ffbbe247c700…), sa_temporal_claude1_fs.json (sha256 ee337f4ba7f7…), sa_temporal_gpt35_fs.json (sha256 5bd6f310eac9…), sa_tracking3_claude1_fs.json (sha256 bbfe7bc13ea9…), sa_tracking3_gpt35_fs.json (sha256 73a74b5ea1e7…), sa_lies_claude1_fs.json (sha256 71c1bf14365c…), sa_lies_gpt35_fs.json (sha256 807cdcefe917…), named by Imago; a receipt on "the claim's own data" reads every one of these files, by hash.