The paper reports that sensitivity to the order of few-shot examples appears across model sizes and sample subsets, and good orderings do not carry over between models.

The paper

The paper's details are OpenAlex's; the citation count is OpenAlex's, 16 Oct 2026. The line on the paper is machine-written, as noted under Why it matters.

Why it matters

Written by Claude (claude-sonnet-5-5) on 16 Oct 2026 from the paper's abstract (as arXiv publishes it) and its OpenAlex record. Machine-written context to help a reader: it is not evidence, it moves no number, and it may be wrong. The quoted sentence is the claim; where it stands is computed from the record. If it misreads the paper, tell the stewards.

The story so far

1. 1

   What the authors did

   The authors tested how the order of a handful of training samples in a prompt affects pretrained language models such as GPT-3, across model sizes and sample sets. They then built a method that selects orderings using an artificial development set.

   Machine-written from the paper's abstract, as noted under Why it matters.

2. 2

   What they found

   • The order of samples in the prompt can make the difference between near state-of-the-art and random-guess performance.
   • Order sensitivity is present across model sizes, is not related to a specific subset of samples, and good permutations are not transferable between models.
   • Selecting orderings using entropy statistics on a model-generated artificial development set gives a 13% relative improvement for GPT-family models across eleven text classification tasks.

   Machine-written from the paper's abstract, as noted under Why it matters.

3. 3

   What has been checked on Ecdysis

   Exuvia registered the claim on 16 October 2026, with a test written from the paper. No check has been filed yet.

What would check it

How far it has been checked

1. 1
   Same data, same methodverification · not yet

   Not yet: re-run the paper's analysis on its own data, where the authors have published it.

2. 2
   New data, same methodreproduction · not yet

   Not yet: the same method on new data covering the claim's population and period. Established needs one.

3. 3
   The designrobustness tests and arguments · not yet

   Nothing yet: change the method or the data and see whether it holds (a robustness test), or argue that the method does not test what the claim says.

How sure is the record?

Share this finding

Ready-made posts, written from the record. You post them yourself, from your own account; nothing is ever posted for anyone.

Click a post's text to select all of it. Both posts give the claim's standing on the record, and the longer one says what the checks show and what they do not; the wording changes when the record does. The longer post quotes the paper first, then gives the machine-written headline, marked as such; edit it as you like. To cite the claim, see Cite this claim.

What would prove it wrong

Refuted if, for any model size ≥ 175B, the maximum performance difference between best and worst prompt permutations on a held‑out few‑shot task is ≤5%; or if the top‑k permutations for one model achieve >20% of the best permutation’s accuracy on another model; or if the set of top permutations differs by more than 30% when restricted to any specific subset of samples.

The test as Exuvia registered it on 16 Oct 2026, written from the paper's words.

The wider literature

Other claims from the same paper

• uncheckedThe authors' method for choosing the order of few-shot examples gave a 13% relative improvement for GPT-family models on eleven text classification tasks.“Our method yields a 13% relative improvement for GPT-family models across eleven different established text classification tasks.”

Headlines are machine-written from the paper's abstract, or from the quote and the paper's title where no abstract is open; each claim's own words are quoted beneath its headline.