InteractBench Benchmarking LLMs on Competitive Programming under Unrevealed Information
Jiaze Li1,2,3 Aocheng Shen1 Bing Liu1 Boyu Zhang1 Xiaoxuan Fan1 Qiankun Zhang1,2,3 Xianjun Deng1,3
- 1 School of Cyber Science and Engineering, Huazhong University of Science and Technology, Wuhan, China
- 2 Key Laboratory of Cyberspace Security, Ministry of Education, Zhengzhou, China
- 3 Hubei Key Laboratory of Distributed System Security, Wuhan, China
Abstract
Competitive programming is increasingly being used to evaluate the algorithmic reasoning capabilities of large language models (LLMs). However, existing benchmarks primarily focus on full-information tasks where all problem inputs are provided upfront. This overlooks a critical dimension of algorithmic reasoning: the ability of generated programs to operate when key information is not revealed upfront. Interactive problems, a distinctive component of competitive programming, embody this challenge. These problems require programs to engage in multi-round interaction with an interactor (a judge program) under strict protocol constraints and limited query budgets, with new information revealed only in response to queries. To address this gap, we introduce InteractBench, a benchmark comprising 322 high-quality interactive problems curated from Codeforces, AtCoder, IOI, and ICPC. Each problem is packaged with executable local interactors, enabling fully offline evaluation. Unlike existing benchmarks, InteractBench assesses whether model-generated code can acquire information and track state dynamically. Our evaluation reveals a significant interaction gap: even the most advanced reasoning models achieve limited success on interactive problems. Beyond success rates, we propose a fine-grained failure taxonomy to diagnose the root causes of these deficiencies. Although algorithmic logic errors remain dominant, protocol violations and query-budget overruns are frequent.
Why this problem matters
Most existing competitive-programming benchmarks primarily focus on batch-style tasks where all problem inputs are provided upfront. This overlooks a critical dimension of algorithmic reasoning: the ability of generated programs to operate when key information is not revealed upfront.
Interactive problems embody this challenge. These problems require programs to engage in multi-round interaction with an interactor under strict protocol constraints and limited query budgets, with new information revealed only in response to queries.
What an interactive problem looks like
Find the smallest index x such that A[1..x] contains k zeros, with n = 8 and k = 3. In the batch setting the whole array is given upfront; in the interactive setting it is hidden, and the solver must recover the answer through range-sum queries under a fixed budget.
What this work provides
A benchmark of interactive problems
We curate 322 interactive problems from Codeforces, AtCoder, IOI, and ICPC. Each problem is annotated with consolidated categories and difficulty tiers.
A self-contained evaluation harness
Each problem is packaged with an executable local interactor, enabling fully offline evaluation without external judge submission.
Interaction-aware diagnostics
Beyond pass@k, our harness checks protocol compliance, enforces query budgets, and reports a failure taxonomy that separates algorithmic errors from interaction-specific failures.
Where interactive evaluation is missing
Existing benchmarks focus on batch-style problems. Some rely on external online judges, limiting fully offline evaluation. Even when interactive problems are included, they typically constitute a small subset.
How each problem is built and verified
Select
We select interactive tasks with multi-round protocols and explicit query budgets from Codeforces, AtCoder, IOI, and ICPC.
Propose-Validate-Adjudicate
Proposer models draft candidate generators and interactors. Candidates are validated against a per-task verification pool of submissions with known official verdicts.
Audit
As a supplementary sanity check, we conduct a one-time post-hoc online agreement audit.
Interactive tasks expose a persistent gap
- Hard interactive tasks remain unsolved and strongly discriminative.
- Gaps between pass@1 and pass@5 quantify the benefit of resampling.
- Explicit reasoning improves pass rates, but does not remove the Hard barrier.
Separating algorithmic errors from interaction failures
Algorithmic logic
- WA protocol-compliant but incorrect answer
Interaction-specific
- PE protocol or format violations
- QLE query-budget overrun
- IDLE blocked I/O or deadlock terminated by the wall-clock cap
Execution
- CE compile error
- RE runtime error
- TLE time limit exceeded
- MLE memory limit exceeded
What goes wrong, and how often
- Algorithmic logic errors (WA) dominate, but interaction-specific failures occur at different stages.
- Query efficiency limits frontier models as well.
- Runtime and resource-limit failures are present but secondary.
Takeaways
- 01
A significant interaction gap: even the most advanced reasoning models achieve limited success, with consistent degradation as difficulty increases.
- 02
Failures are dominated by algorithmic logic errors, yet protocol violations and query-budget overruns remain frequent even for frontier reasoning models.
- 03
Looking forward, we plan to develop prompting and training strategies that directly target interactive querying, state maintenance, and protocol adherence.
Frequently asked questions
Cite this work
@inproceedings{li2026interactbench,
title = {InteractBench: Benchmarking {LLM}s on Competitive
Programming under Unrevealed Information},
author = {Jiaze Li and Aocheng Shen and Bing Liu and
Boyu Zhang and Xiaoxuan Fan and Qiankun Zhang and
Xianjun Deng},
booktitle = {Forty-third International Conference on
Machine Learning},
year = {2026},
url = {https://openreview.net/forum?id=Y4T4w0Tj0l}
}