Pushing the Limits of Audio-Visual Joint Reasoning

OmniReasoningBench accuracy (%). Our model is initialized by Qwen3-Omni-30B-A3B-Thinking.

Model	Modality	Reasoning over video Multiple-choice	Reasoning over video Open-ended	Reasoning beyond video Multiple-choice	Reasoning beyond video Open-ended	Overall
Proprietary models
⁠ ⁠Gemini‑3.7‑Flash	Omni	57.6	47.7	72.0	58.0	57.6
⁠ ⁠Gemini‑3.8‑Flash	Omni	55.2	49.6	69.6	60.0	57.1
⁠ ⁠Gemini‑3.5‑Flash	Omni	53.5	47.2	66.4	51.3	54.0
⁠ ⁠Qwen3.8‑Omni‑Flash	Omni	50.4	42.1	69.2	57.3	52.7
⁠ ⁠Gemini‑3.1‑Pro	Omni	51.9	47.2	64.0	49.3	52.7
⁠ ⁠Doubao‑2.0‑Lite	Omni	54.7	47.2	57.2	40.7	51.0
⁠ ⁠Qwen3.5‑Omni‑Plus	Omni	42.9	42.2	60.4	40.7	46.2
⁠ ⁠Muse‑Spark‑1.2	Omni	38.1	36.0	60.0	58.0	44.8
⁠ ⁠Muse‑Spark‑1.1	Omni	36.4	35.9	56.0	46.0	41.8
Open-source models
⁠ ⁠MiMo‑V2.5	Omni	42.7	40.6	54.0	44.0	44.6
⁠ ⁠Kimi‑K3	Visual	21.4	11.7	50.8	35.3	26.5
⁠ ⁠Qwen3.8‑Max	Visual	19.7	10.5	46.8	34.7	24.6
⁠ ⁠Qwen3.5‑Plus	Visual	17.2	8.7	46.0	30.0	22.5
⁠ ⁠Nemotron‑3‑Nano‑Omni	Omni	25.6	22.9	36.0	13.3	25.4
⁠ ⁠Qwen3‑Omni‑30B‑A3B‑Thinking	Omni	31.9	25.9	46.4	32.0	33.2
Ours
⁠ ⁠OmniReasoning‑30B‑A3B	Omni	48.0	30.1	58.8	32.7	42.5

Model	OmniVideoBench	LVOmniBench	Video‑MMMU	Video‑MME‑v2
Proprietary models
⁠ ⁠Gemini‑3.5‑Flash	66.5	66.9	86.1	67.6
⁠ ⁠Gemini‑3.1‑Pro	60.7	60.9	84.3	53.4
⁠ ⁠Qwen3.5‑Omni‑Plus	53.8	53.2	84.6	47.9
⁠ ⁠Muse‑Spark‑1.1	55.8	52.3	85.2	59.5
⁠ ⁠MiMo‑V2.5	51.6	52.8	79.1	51.8
Open-source models
⁠ ⁠Nemotron‑3‑Nano‑Omni	42.1	32.1	73.1	38.4
⁠ ⁠Qwen2.5‑Omni‑7B	29.3	32.0	53.9	25.3
⁠ ⁠Qwen3‑Omni‑30B‑A3B‑Thinking	37.2	37.7	68.3	36.5
Ours
⁠ ⁠OmniReasoning‑30B‑A3B	50.0	44.7	74.4	44.0

Music, Sound, and Speech are audio types; duration intervals are in minutes.

Model	Music	Sound	Speech	(0,1] minutes	(1,5] minutes	(5,10] minutes	(10,30] minutes	Average
Proprietary models
⁠ ⁠Gemini‑3.5‑Flash	56.9	69.3	70.1	69.7	68.0	63.9	64.8	66.5
⁠ ⁠Gemini‑3.1‑Pro	51.1	65.8	61.9	67.3	64.2	59.5	53.0	60.7
⁠ ⁠Gemini‑3‑Pro	56.2	54.1	55.7	61.0	56.4	52.9	52.5	55.5
⁠ ⁠Qwen3.5‑Omni‑Plus	42.3	64.9	53.3	55.8	55.2	50.7	53.4	53.8
⁠ ⁠MiMo‑V2.5	46.2	47.6	53.0	56.6	57.4	47.6	44.3	51.6
⁠ ⁠Gemini‑2.0‑Flash	29.7	40.3	43.2	49.4	43.2	41.1	34.9	41.5
Open-source models
⁠ ⁠Nemotron‑3‑Nano‑Omni	33.0	42.9	43.0	48.8	41.7	41.5	38.9	42.1
⁠ ⁠Qwen2.5‑Omni‑7B	23.1	25.3	30.7	41.6	27.4	25.3	26.7	29.3
⁠ ⁠Qwen3‑Omni‑30B‑A3B‑Thinking	26.4	37.2	38.5	46.8	35.6	35.5	35.2	37.2
Ours
⁠ ⁠OmniReasoning‑30B‑A3B	39.6	46.3	52.0	52.4	49.9	49.8	48.9	50.0

Run all commands from the repository root. Requires Python 3.10+ and ffmpeg / ffprobe.

python -m pip install '.[engine]'

This installs the omniqa and omnireasoning commands and the data-engine dependencies. For local scene detection with --plan-segmenter scenedetect, install '.[engine,scenes]' instead. Install ffmpeg / ffprobe separately; they are not Python dependencies.

Place source videos in data/videos/.

omniqa caption --data-path data/videos --output-dir data/caption/videos \
  --gemini-model gemini-3.1-pro-preview \
  --skip-upload --max-inline-video-mb 20 --max-events 64

Each route writes outputs/run/generation/accepted.jsonl with question, options, answer and a reasoning_steps dependency chain.

Reasoning over video (ROV): task selection, question/answer/chain generation, structural and grounding checks, a caption-conditioned solver, and a question-only (priors) blind gate.

omniqa generate --caption-raw-dir data/caption/videos/raw \
  --output-dir outputs/run/generation \
  --openai-api-base "$OPENAI_API_BASE" --openai-model deepseek-v4-flash \
  --questions-per-video 1

Reasoning beyond video (RBY), including generated question figures:

omniqa adapt --caption-dir data/caption/videos/raw --outdir outputs/run/generation --n 0
omniqa bridge-adaptation --outdir outputs/run/generation \
  --output outputs/run/generation/accepted.jsonl

Direct video authoring (DYNAMIC): the author watches the video; subtype weights must first be mined from benchmark result files with omniqa mine-queries (see --help).

omniqa dynamic --caption-raw-dir data/caption/videos/raw --output-dir outputs/run/generation

# 1. Verify timestamped clues against the actual clips (DashScope SDK).
omniqa verify --input outputs/run/generation/accepted.jsonl \
  --output-dir outputs/run/verification
# 2. Repair or reject questions against the video (DashScope SDK).
omniqa refine --input outputs/run/verification/accepted_verified.jsonl \
  --output-dir outputs/run/refinement
# 3. Modality shortcut screening.
omniqa gate --input outputs/run/refinement/refined_accepted.jsonl \
  --caption-raw-dir data/caption/videos/raw --video-root data/videos \
  --output-dir outputs/run/gate

omniqa think --input outputs/run/refinement/refined_accepted.jsonl \
  --caption-root data/caption --output-dir outputs/run/thinking

Only reasoning with a closing conclusion matching the verified answer enters sft.jsonl. Cached rows are checked again during export; inconsistent rows are retried on resume.

Place the two benchmark files at data/benchmark/over_video.jsonl (750) and data/benchmark/beyond_video.jsonl (400), with videos and figures under data/media/. Start an audiovisual SGLang / OpenAI-compatible server and export SGLANG_*.

for s in over_video beyond_video; do
  omnireasoning infer --benchmark $s \
    --data_json_file data/benchmark/$s.jsonl --video_dir data/media \
    --output_file outputs/$s.jsonl --model "$SGLANG_MODEL"
done

Defaults: 2 fps, at most 512 frames, audio kept, --max_tokens 32768 --temperature 0.6 --top_p 0.95 --top_k 20.

Multiple-choice answers are scored by rule-based option-letter matching (no LLM fallback; an output without a parseable letter is wrong). Open-ended answers are judged by deepseek-v4-flash. Scoring automatically uses the answer after </think> and never reads option letters from the thinking block. An unfinished thinking block counts as a missing answer. Every benchmark question stays in the denominator: missing outputs, service errors and judge failures count as wrong.

for s in over_video beyond_video; do
  omnireasoning score-judge --predictions outputs/$s.jsonl \
    --references data/benchmark/$s.jsonl --out_dir outputs/${s}_judge
done

# Table row: over-video MCQ / OE, beyond-video MCQ / OE, Overall (pooled counts).
omnireasoning report-table \
  --over_video_pred outputs/over_video_judge/predictions.jsonl --over_video_ref data/benchmark/over_video.jsonl \
  --beyond_video_pred outputs/beyond_video_judge/predictions.jsonl --beyond_video_ref data/benchmark/beyond_video.jsonl

If you find OmniReasoning useful, please cite:

@misc{lin2026omnireasoningpushinglimitsaudiovisual,
      title={OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning}, 
      author={Junming Lin and Yuxuan Wang and Zhenxin Lei and Yuxin Liu and Ruixun Liu and Yinsong Yan and Ling Wang and Minghao Han and Yunfei Chu and Shun Lei and Xueyao Zhang and Qize Yang and Jin Xu and Yiwu Zhong},
      year={2026},
      eprint={2609.39490},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.39490}, 
}

The OmniReasoning data is released for non-commercial research use under CC BY-NC 4.0.