PRISM uses step-level reward models to guide deep reasoning refinement, achieving state-of-the-art math competition performance.
DEEPTHINK methods improve reasoning by generating, refining, and aggregating populations of candidate solutions, which enables strong performance on complex mathematical and scientific tasks. However, existing frameworks often lack reliable correctness signals during inference, which creates a population-enhancement bottleneck where deeper deliberation amplifies errors, suppresses correct minority solutions, and yields weak returns to additional compute. In this paper, we introduce a functional decomposition of DEEPTHINK systems and propose PRISM, a Process Reward Model (PRM)-guided inference algorithm that uses step-level verification to guide both population refinement and solution aggregation. During refinement, PRISM treats candidate solutions as particles in a PRM-defined energy landscape and reshapes the population through score-guided resampling and stochastic refinement, which concentrates probability mass on higher-quality reasoning while preserving diversity. Across mathematics and science benchmarks, PRISM is competitive with or outperforms existing DEEPTHINK methods, reaching 90.0%, 75.4%, and 71.4% with gpt-oss-20b on AIME25, HMMT25, and GPQA Diamond, respectively, while matching or exceeding gpt-oss-120b. Additionally, our analysis shows that PRISM produces consistent net-directional correction during refinement, remains reliable when the initial population contains few correct candidates, and often lies on the compute-accuracy Pareto frontier.
PRISM uses Process Reward Models to guide solution refinement by scoring reasoning steps, enabling directional error correction instead of stochastic rewriting.
Simple parallel sampling with majority voting matches sophisticated refinement methods, indicating initial population diversity matters more than iterative refinement.
PRISM achieves 90.0% on AIME25 with gpt-oss-20b, matching or exceeding gpt-oss-120b while mitigating majority dilution failure modes.
Method requires training accurate Process Reward Models, which may not generalize across diverse reasoning domains or problem types.
Approach relies on step-level correctness signals that may be difficult to obtain for open-ended or subjective reasoning tasks.
Shaobo Wang, Xuan Ouyang +10
As high-quality public text approaches exhaustion, a phenomenon known as the Data Wall, pre-training is shifting from more tokens to better tokens. However, existing methods either rely on heuristic s...
Zhongzhi Li, Xuansheng Wu +3
The diversity of post-training data is critical for effective downstream performance in large language models (LLMs). Many existing approaches to constructing post-training data quantify diversity usi...
Zixuan Huang, Xin Xia +12
Recent advancements in large reasoning models (LRMs) have greatly improved their capabilities on complex reasoning tasks through Long Chains of Thought (CoTs). However, this approach often results in ...
Tengyue Xu, Zhuoyang Qian +17
Autonomous scientific discovery with large language model (LLM)-based agents has recently made substantial progress, demonstrating the ability to automate end-to-end research workflows. However, exist...
Zhiyuan Hu, Yucheng Wang +8
Reinforcement learning (RL) has become a central paradigm for post-training large language models (LLMs), particularly for complex reasoning tasks, yet it often suffers from exploration collapse: poli...
