{"id":166,"date":"2026-01-12T07:28:34","date_gmt":"2026-01-12T07:28:34","guid":{"rendered":"https:\/\/wordpress.cs.vt.edu\/rylai\/?page_id=166"},"modified":"2026-01-12T07:28:34","modified_gmt":"2026-01-12T07:28:34","slug":"smart","status":"publish","type":"page","link":"https:\/\/wordpress.cs.vt.edu\/rylai\/smart\/","title":{"rendered":"SMART"},"content":{"rendered":"\n

Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories.<\/strong><\/p>\n\n\n\n

\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/figure>\n\n\n\n

Despite the remarkable capabilities of large language models, current training paradigms inadvertently foster sycophancy, i.e., the tendency of a model to agree with or reinforce user-provided information even when it\u2019s factually incorrect.<\/p>\n\n\n\n

To address this challenge, we introduce SMART (Sycophancy Mitigation through Adaptive Reasoning Trajectories), which reframes sycophancy as a reasoning optimization problem rather than an output alignment issue. SMART is a two-stage framework comprising: (1) Uncertainty-Aware Adaptive Monte Carlo Tree Search (UA-MCTS), which dynamically adjusts model exploration based onstate-level uncertainty to collect high-quality, diverse reasoning trajectories alongside both stepwise progress and final outcome rewards; and (2) progress-based reinforcement learning, which fine-tunes the model using the collected trajectories and reward signals to reinforce effective reasoning patterns.<\/p>\n\n\n\n

Through extensive experiments, we show that SMART significantly reduces sycophantic behavior while preserving strong performance on out-of-distribution inputs and maintaining general capabilities. These results underscore the importance of optimizing internal reasoning mechanisms to build more truthful and aligned AI assistants.<\/p>\n\n\n\n

Publications<\/h2>\n\n\n\n

Beigi, Mohammad, Ying Shen, Parshin Shojaee, Qifan Wang, Zichao Wang, Chandan K. Reddy, Ming Jin, and Lifu Huang. “Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories.” In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 13090-13103. 2025.<\/p>\n\n\n\n

Paper<\/a><\/p>\n\n\n\n

Code<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories. Despite the remarkable capabilities of large language models, current training paradigms inadvertently foster sycophancy, i.e., the tendency of a model to agree with or reinforce user-provided information even when it\u2019s factually incorrect. To address this challenge, we introduce SMART (Sycophancy Mitigation through Adaptive Reasoning Trajectories), […]<\/p>\n","protected":false},"author":501,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-166","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/wordpress.cs.vt.edu\/rylai\/wp-json\/wp\/v2\/pages\/166","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.cs.vt.edu\/rylai\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/wordpress.cs.vt.edu\/rylai\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/rylai\/wp-json\/wp\/v2\/users\/501"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/rylai\/wp-json\/wp\/v2\/comments?post=166"}],"version-history":[{"count":1,"href":"https:\/\/wordpress.cs.vt.edu\/rylai\/wp-json\/wp\/v2\/pages\/166\/revisions"}],"predecessor-version":[{"id":171,"href":"https:\/\/wordpress.cs.vt.edu\/rylai\/wp-json\/wp\/v2\/pages\/166\/revisions\/171"}],"wp:attachment":[{"href":"https:\/\/wordpress.cs.vt.edu\/rylai\/wp-json\/wp\/v2\/media?parent=166"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}