Renee Jia, Di Mu · R2M AI · Transactions on Machine Learning Research, 2026

📄 Paper · 🤗 Dataset · 📊 Findings · ⚡ Quickstart · 🔬 Reproduce · 🗺️ Experiment map

When a language model writes a chain of thought, does that reasoning actually constrain its answer?

It matters because chain-of-thought is widely proposed as a way to monitor what models are doing — and a monitor that reads the chain is only informative if the chain drives the output.

We test it directly: change one step mid-chain, cut the trace there, and force the model to continue. What it writes next shows whether it ignores the injected error, catches and repairs it, or carries it straight into the answer.

The answer depends on difficulty. On easy tasks the written chain is close to decorative. On hard ones it is load-bearing.

_{Pooled Gemma results under the production judge (n = 21,238). A and B both land on the correct answer and differ only in whether the model visibly notices the error — a distinction sensitive to the judge prompt. C does not, and is prompt-invariant. The protocol tests the CoT → answer link, not whether the trace mirrors the model's internal computation.}

Error propagation climbs 3.9% → 22.3% → 40.9% across GSM8K, MMLU and BBH as Gemma's own accuracy falls from 86.5% to 57.9%. The cleanest cut is inside MMLU, where the question format and every perturbation strategy are held fixed and the rate still spans 7.5% to 53.7% across 29 subjects.

_{Error propagation, with baseline accuracy underneath. Difficulty here is relative to the model's own competence, not an absolute property of the task.}

Hold the perturbation type fixed at numerical and propagation still rises 16× from GSM8K to BBH multistep arithmetic (3.9% → 64.5%). Text edits on GSM8K land at 6.5%. A regression over all 28,584 continuations assigns 98.8% of explained deviance to task difficulty against 0.8% to perturbation type. → analysis

Llama-3.1-8B-Instruct reproduces the gradient at a lower accuracy level. DeepSeek-R1-Distill propagates far fewer errors and shows a compressed gradient — consistent with reasoning-specific post-training playing a role, though our setup can't isolate its causal effect.

Probes hit 75.3% three-class and 86.2% error-propagation accuracy under question-grouped 5-fold CV. Steering along those same directions is a null result across 11,556 generations. An eight-direction intervention moves just 24.6% of Type C cases at its strongest setting (14/57; 95% CI 15.1–37.1%).

Two blind annotators agree at κ = 1.00 on the C-vs-non-C split and κ ≈ 0.93 on the full scheme. Against the paper's production labels: 96.2% agreement on the binary call, but only 76.0% three-way — the A/B boundary is genuinely prompt-sensitive. → human-study materials

The paper has the full methods, uncertainty estimates and limitations. The experiment map links every analysis to its code and released artifacts.

Measure whether a model's chain-of-thought is causally load-bearing. Python 3.10+, in an environment supported by vLLM:

git clone https://github.com/r2m-ai/load-bearing-cot.git
cd load-bearing-cot
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

python evaluate.py --model deepseek-r1-distill-qwen-7b --dataset gsm8k \
  --output-dir runs/deepseek-gsm8k

The default run samples 100 questions with a fixed seed, uses greedy decoding, and tests early, middle and late interventions with a 4,096-token budget per generation. Use --all for a full dataset run. No external judge API is called.

# other datasets, tasks and strategies
python evaluate.py --model gemma-2-9b-it --dataset mmlu

python evaluate.py --model llama-3.1-8b-instruct --dataset bbh \
  --task multistep_arithmetic_two --strategy arithmetic_change

python evaluate.py --model gemma-2-9b-it --dataset mmlu --strategy wrong_elimination

# any vLLM-compatible model, by Hugging Face ID or local path
python evaluate.py --model Qwen/Qwen2.5-7B-Instruct --dataset gsm8k

A model needs vLLM inference, a chat template, and continuation from an assistant prefix. Accept the model license and authenticate with Hugging Face before loading gated weights. The paper ran on a single NVIDIA H100 80 GB. See python evaluate.py --help for revision pins, local JSONL input and parallelism.

⚠️ Implementation status. load-bearing-v0.2 has offline tests for scoring, continuation setup and output handling, but real GPU runs of this entry point are not yet validated. The published Gemma, Llama and DeepSeek results come from the paper pipeline. Qwen is an example of how to plug in another model, not a reported result.

The evaluator keeps correctly answered baselines with at least three visible reasoning lines, edits one line, truncates there, and generates both a perturbed continuation and an unedited-prefix control.

Error propagation rate is the fraction of parseable perturbed continuations with an incorrect final answer — conditional on baseline correctness, eligibility and the chosen intervention. Low propagation may mean either bypass or correction, and high propagation is not a measure of reasoning quality. The evaluator reports the correctness-based C-vs-non-C proxy only; confirming that an error was genuinely taken up means reading the trace.

Unparseable answers stay unscored and undefined rates are null. Output directories are never overwritten; --resume RUN_DIR continues an interrupted run under the same settings. When comparing models, report coverage and eligibility alongside the rate and keep the intervention and parsing settings fixed. The protocol spec defines the estimand and how it differs from the paper experiments.

🤗 The paper's full record set is published as a Hugging Face dataset, ReneeJia/cot-load-bearingness: every labeled continuation from all three models, the matched perturbation controls, every steered generation, four-variant judge labels and the paper's aggregate tables. The headline numbers are recomputable from those rows.

Git keeps the generated records out of this repository, so clone the dataset rather than this repository if you want the rows themselves. Hidden-state arrays and probe checkpoints are in neither — the reproduction guide lists what ships and what has to be regenerated. See Contributing for adding models, datasets or results.

@article{jia2026loadbearing,
  title   = {From Decorative to Load-Bearing: Task Difficulty Shapes the Causal Role of Chain-of-Thought},
  author  = {Renee Jia and Di Mu},
  journal = {Transactions on Machine Learning Research},
  year    = {2026}
}

Please report the protocol version and evaluation settings alongside any number you publish.

Code: MIT · Paper text and figures © the authors · Datasets and model weights keep their own terms.

Questions and contributions welcome — Renee Jia, reneejia@r2m.ai · r2m.ai