† Deterministic-policy FQE estimates carry residual out-of-distribution optimism: the mean approaches the analytic plausibility bound V_MC + Δ_oracle ≈ 6.2 (Why the learned policy shifts as it does), individual seeds exceed it, cross-seed variance is an order of magnitude larger than for the stochastic deployment, and the FQE TD error is unstable during fitting (rising from 0.001 to 0.018, versus a stable 0.002 for the stochastic policy). These estimates are therefore flagged rather than used as evidence. MC: Monte Carlo; n.r.: not reported.
During the preparation of this work, the authors used Claude (Anthropic) to assist with language editing and manuscript preparation. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.
Author contributions
BZ: Conceptualization, Writing—original draft, Writing—review & editing, Investigation, Visualization. RZ: Project administration, Formal analysis, Writing—review & editing. Both authors read and approved the submitted version.
Conflicts of interest
The authors declare that there are no conflicts of interest.
Ethical approval
Not required.
Consent to participate
Not required.
Consent to publication
Not applicable to identifiable individuals; the study uses a public anonymized corpus.
Availability of data and materials
The ESConv corpus is publicly available; our canonical splits, code, trained-model evaluation scripts, rater guide, and analysis protocols are available from the corresponding author upon reasonable request.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Open Exploration maintains a neutral stance on jurisdictional claims in published institutional affiliations and maps. All opinions expressed in this article are the personal views of the author(s) and do not represent the stance of the editorial team or the publisher.
Patel V, Saxena S, Lund C, Thornicroft G, Baingana F, Bolton P, et al. The Lancet Commission on global mental health and sustainable development.Lancet. 2018;392:1553–98. [DOI] [PubMed]
Abd-Alrazaq AA, Alajlani M, Ali N, Denecke K, Bewick BM, Househ M. Perceptions and Opinions of Patients About Mental Health Chatbots: Scoping Review.J Med Internet Res. 2021;23:e17828. [DOI] [PubMed] [PMC]
Vaidyam AN, Wisniewski H, Halamka JD, Keshavan MS, Torous JB. Chatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape.Can J Psychiatry. 2019;64:456–64. [DOI] [PubMed] [PMC]
Inkster B, Sarda S, Subramanian V. An Empathy-Driven, Conversational Artificial Intelligence Agent (Wysa) for Digital Mental Well-Being: Real-World Data Evaluation Mixed-Methods Study.JMIR mHealth uHealth. 2018;6:e12106. [DOI] [PubMed] [PMC]
Fitzpatrick KK, Darcy A, Vierhile M. Delivering Cognitive Behavior Therapy to Young Adults With Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial.JMIR Ment Health. 2017;4:e19. [DOI] [PubMed] [PMC]
Grodniewicz JP, Hohol M. Waiting for a digital therapist: three challenges on the path to psychotherapy delivered by artificial intelligence.Front Psychiatry. 2023;14:1190084. [DOI] [PubMed] [PMC]
Sedlakova J, Trachsel M. Conversational Artificial Intelligence in Psychotherapy: A New Therapeutic Tool or Agent?Am J Bioeth. 2023;23:4–13. [DOI] [PubMed]
Hill CE. Helping Skills: Facilitating Exploration, Insight, and Action. 4th ed. Washington, DC: American Psychological Association; 2014.
Liu S, Zheng C, Demasi O, Sabour S, Li Y, Yu Z, et al. Towards emotional support dialog systems. In: Proceedings of ACL. 2021. pp. 3469–83.
Nickel M, Kiela D. Poincaré embeddings for learning hierarchical representations. In: NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017. pp. 6341–50.
Levine S, Kumar A, Tucker G, Fu J. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.arXiv:2005.01643 [Preprint]. 2020 [cited 2026 Jul 27]. Available from: https://arxiv.org/abs/2005.01643
Kostrikov I, Nair A, Levine S. Offline Reinforcement Learning with Implicit Q-Learning.arXiv:2110.06169 [Preprint]. 2021 [cited 2026 Jul 27]. Available from: https://arxiv.org/abs/2110.06169
Kumar A, Zhou A, Tucker G, Levine S. Conservative Q-Learning for Offline Reinforcement Learning.arXiv:2006.04779 [Preprint]. 2020 [cited 2026 Jul 27]. Available from: https://arxiv.org/abs/2006.04779
Rashkin H, Smith EM, Li M, Boureau YL. Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics; 2019. pp. 5370–81. [DOI]
Lin Z, Madotto A, Shin J, Xu P, Fung P. MoEL: Mixture of Empathetic Listeners. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics; 2019. pp. 121–32. [DOI]
Cheng Y, Liu W, Li W, Wang J, Zhao R, Liu B, et al. Improving Multi-turn Emotional Support Dialogue Generation with Lookahead Strategy Planning. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2022. pp. 3014–26. [DOI]
Tu Q, Li Y, Cui J, Wang B, Wen JR, Yan R. MISC: A Mixed Strategy-Aware Model integrating COMET for Emotional Support Conversation. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics; 2022. pp. 308–19. [DOI]
Peng W, Hu Y, Xing L, Xie Y, Sun Y, Li Y. Control Globally, Understand Locally: A Global-to-Local Hierarchical Graph Network for Emotional Support Conversation.Proc Thirty-First Int Jt Conf Artif Intell. 2022:4324–30. [DOI]
Zheng C, Sabour S, Wen J, Zhang Z, Huang M. AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation.arXiv:2202.13047 [Preprint]. 2023 [cited 2026 Jul 27]. Available from: https://arxiv.org/abs/2202.13047
Kang D, Kim S, Kwon T, Moon S, Cho H, Yu Y, et al. Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversation. In: Proceedings of ACL. 2024. pp. 15232–61.
Nickel M, Kiela D. Learning continuous hierarchies in the Lorentz model of hyperbolic geometry.arXiv:1806.03417 [Preprint]. 2018 [cited 2026 Jul 27]. Available from: https://arxiv.org/abs/1806.03417
Chami I, Ying Z, Ré C, Leskovec J. Hyperbolic graph convolutional neural networks.arXiv:1910.12933 [Preprint]. 2019 [cited 2026 Jul 27]. Available from: https://arxiv.org/abs/1910.12933
Tifrea A, Bécigneul G, Ganea OE. Poincaré GloVe: Hyperbolic word embeddings.arXiv:1810.06546 [Preprint]. 2019 [cited 2026 Jul 27]. Available from: https://arxiv.org/abs/1810.06546
Khrulkov V, Mirvakhabova L, Ustinova E, Oseledets I, Lempitsky V. Hyperbolic Image Embeddings. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020. pp. 6417–27. [DOI]
Fujimoto S, Meger D, Precup D. Off-policy deep reinforcement learning without exploration. In: International Conference on Machine Learning. 2019. pp. 2052–62.
Wu Y, Tucker G, Nachum O. Behavior regularized offline reinforcement learning.arXiv:1911.11361 [Preprint]. 2019 [cited 2026 Jul 27]. Available from: https://arxiv.org/abs/1911.11361
Raghu A, Komorowski M, Ahmed I, Celi L, Szolovits P, Ghassemi M. Deep Reinforcement Learning for Sepsis Treatment.arXiv:1711.09602 [Preprint]. 2017 [cited 2026 Jul 27]. Available from: https://arxiv.org/abs/1711.09602
Komorowski M, Celi LA, Badawi O, Gordon AC, Faisal AA. The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care.Nat Med. 2018;24:1716–20. [DOI] [PubMed]
Fox I, Lee J, Pop-Busui R, Wiens J. Deep reinforcement learning for closed-loop blood glucose control. In: Machine Learning for Healthcare Conference. 2020. pp. 508–36.
Snell C, Kostrikov I, Su Y, Yang M, Levine S. Offline RL for Natural Language Generation with Implicit Language Q Learning.arXiv:2206.11871 [Preprint]. 2023 [cited 2026 Jul 27]. Available from: https://arxiv.org/abs/2206.11871
Le HM, Voloshin C, Yue Y. Batch policy learning under constraints. In: Proceedings of the 36th International Conference on Machine Learning (ICML). 2019. pp. 3703–12.
Precup D, Sutton RS, Singh SP. Eligibility traces for off-policy policy evaluation. In: Proceedings of the 17th International Conference on Machine Learning (ICML). 2000. pp. 759–66.
GLM Team.ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. arXiv:2406.12793 [Preprint]. 2024 [cited 2026 Jul 27]. Available from: https://arxiv.org/abs/2406.12793
Fleiss JL. Measuring nominal scale agreement among many raters.Psychol Bull. 1971;76:378–82. [DOI]
Krippendorff K. Content Analysis: An Introduction to Its Methodology. 2nd ed. Thousand Oaks, CA: Sage; 2004.
