LLM 에이전트에서 멀티턴 강화 학습은 실제로 무엇을 개선하는가?
이 논문은 multi-turn agentic reinforcement learning (RL)의 성능 향상이 interaction depth T와 sampling breadth K 등 예산 제약에 따라 크게 달라진다는 것을 Pass@(K,T) 지표를 통해 규명하였다. 연구 결과, agentic RL은 단일 단계의 의사결정 품질을 균일하게 높이기보다는 error recovery와 long-horizon resilience에 강점을 가짐을 확인하였다. 이러한 예산 의존성으로 인한 배포 리스크를 완화하기 위해 학습 중 interaction budget을 다양하게 변화시키는 BudgetMix RL 기법을 제안하였다.
While multi-turn agentic reinforcement learning (RL) excels on long-horizon tasks, its performance gains remain highly sensitive to interaction constraints. Evaluating across sampling breadth K and interaction depth T via Pass@(K,T), we show that trajectory-level RL advantages depend on sufficient turn budgets, diminishing significantly when budgets tighten. Besides, agentic RL does not uniformly elevate single-step decision quality; rather, its strength concentrates in error recovery and long-horizon resilience. To mitigate the deployment risks caused by this budget dependence, we provide a preliminary discussion on the benefits and limitations of process supervision signals and introduce BudgetMix RL as an alternative that varies interaction budgets during training. Experiments demonstrate that BudgetMix RL achieves robust performance across diverse execution settings. Project Page: https://rethinking-agentic-rl.github.io/