Language models with far fewer parameters than GPT-3, by several orders of magnitude, can reach similar few-shot performance, the paper reports.

The paper

The paper's details are OpenAlex's; the citation count is OpenAlex's, 15 Oct 2026. The line on the paper is machine-written, as noted under Why it matters.

Why it matters

Written by Claude (claude-sonnet-5-5) on 15 Oct 2026 from the paper's abstract (as arXiv publishes it) and its OpenAlex record. Machine-written context to help a reader: it is not evidence, it moves no number, and it may be wrong. The quoted sentence is the claim; where it stands is computed from the record. If it misreads the paper, tell the stewards.

The story so far

1. 1

   What the authors did

   The authors converted textual inputs into cloze questions containing a task description and trained small language models with gradient-based optimisation, also using unlabelled data. They compared the results with GPT-3.

   Machine-written from the paper's abstract, as noted under Why it matters.

2. 2

   What they found

   • Performance similar to GPT-3 can be obtained with models whose parameter count is several orders of magnitude smaller.
   • This is achieved by turning inputs into cloze questions that contain a task description, combined with gradient-based optimisation.
   • Using unlabelled data gives further improvements, and the authors identify key factors for successful language understanding with small models.

   Machine-written from the paper's abstract, as noted under Why it matters.

3. 3

   What has been checked on Ecdysis

   Exuvia registered the claim on 15 October 2026, with a test written from the paper. No check has been filed yet.

What would check it

How far it has been checked

1. 1
   Same data, same methodverification · not yet

   Not yet: re-run the paper's analysis on its own data, where the authors have published it.

2. 2
   New data, same methodreproduction · not yet

   Not yet: the same method on new data covering the claim's population and period. Established needs one.

3. 3
   The designrobustness tests and arguments · not yet

   Nothing yet: change the method or the data and see whether it holds (a robustness test), or argue that the method does not test what the claim says.

How sure is the record?

Share this finding

Ready-made posts, written from the record. You post them yourself, from your own account; nothing is ever posted for anyone.

Click a post's text to select all of it. Both posts give the claim's standing on the record, and the longer one says what the checks show and what they do not; the wording changes when the record does. The longer post quotes the paper first, then gives the machine-written headline, marked as such; edit it as you like. To cite the claim, see Cite this claim.

What would prove it wrong

Refuted if no language model with parameter count at least 100× smaller than GPT‑3 achieves a few‑shot accuracy within 5% of GPT‑3 on the same benchmark tasks (e.g., LAMBADA, SuperGLUE) using identical prompt formats and evaluation protocols.

The test as Exuvia registered it on 15 Oct 2026, written from the paper's words.

The wider literature

Earlier work it rests on, as the citing paper says

• uncheckedUnlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers.takes its method from, as the citing paper says · from the literature · identified by Exuvia: “Our modified version of P ET uses masked language models (Devlin et al., 2019) to assign probabilities to sequences of text; this is similar to using them in a…”