LiberCoders/FeatureBench

FeatureBench combines a feature-level coding benchmark with a fully automated, Universal Coding Data Factory. With one CLI, it continuously discovers GitHub repositories, builds reproducible environments across languages, and generates validated coding tasks for training and evaluation through extensible Adapters.

🏭 2026.10.30: We launched a Universal Coding Data Factory—fully automated, language-agnostic, and Adapter-driven—to continuously turn GitHub repositories into validated training data for any coding task format.

📈 2026.10.30: In internal mixed-task RL experiments using data from our Universal Coding Data Factory, Qwen3.8-27B improved SWE-bench Verified Pass@1 from 70.2% to a peak of 79.8% and FeatureBench 1.1 Tests Passed from 50.3% to a peak of 60.3%.

📦 2026.08.24: We released FeatureBench Dataset V1.1, primarily correcting inconsistencies and omissions in task problem statements.

Full interactive leaderboard with tabs, filters, and sorting (Dataset V1.0).

Prerequisites:

• Python 3.12 or later
• Docker for reproducible builds and evaluation
• uv when using uv or installing from source

# Install from PyPI
pip install featurebench

# Or install the CLI with uv
uv tool install featurebench

# Install from source
git clone https://github.com/LiberCoders/FeatureBench.git
cd FeatureBench
uv sync
source .venv/bin/activate

Create config.toml from config_example.toml. From a source checkout:

cp config_example.toml config.toml

Optionally pre-pull the benchmark images:

fb pull --mode fast

Run inference:

fb infer \
    --config-path config.toml \
    --data-version v1.1 \
    --agent mini_swe_agent \
    --model openai/qwen3-coder-480b-a35b-instruct \
    --split fast

Evaluate the generated patches:

fb eval \
    -p runs/<timestamp>/output.jsonl \
    --split fast

The factory automatically discovers GitHub repositories, builds reproducible environments, and generates validated coding tasks. Adapters define the task format; FeatureBench is included by default. This workflow is optional for benchmark users.

Put provider credentials in .env. From a source checkout:

cp .env.example .env

The default settings work out of the box and can be customized through configs/default/*.yaml or --config.

fb discover
fb build outputs/discover-YYYYMMDD-HHMMSS/repositories.jsonl
fb generate outputs/build-YYYYMMDD-HHMMSS/environments.jsonl \
  --adapter featurebench

Benchmark:

• fb infer runs coding agents on FeatureBench tasks (Documentation).
• fb eval evaluates generated patches (Documentation).
• fb pull pre-pulls benchmark Docker images (Documentation).

Data factory:

• fb discover finds candidate GitHub repositories (Documentation).
• fb build creates reusable, validated environments (Documentation).
• fb generate generates tasks through an Adapter (Documentation).
• fb configure edits an inactive run before resuming it (Documentation).
• fb --resume continues an interrupted run (Documentation).

If you found FeatureBench useful, please cite us as:

@article{zhou2026featurebench,
  title={FeatureBench: Benchmarking Agentic Coding for Complex Feature Development},
  author={Zhou, Qixing and Zhang, Jiacheng and Wang, Haiyang and Hao, Rui and Wang, Jiahe and Han, Minghao and Yang, Yuxue and Wu, Shuzhe and Pan, Feiyang and Fan, Lue and others},
  journal={arXiv preprint arXiv:2602.10975},
  year={2026}
}

If you have any questions, feel free to contact qixing.zhou@smail.nju.edu.cn or zjcheng2022@gmail.com.