Image by PeopleImages.com - Yuri A/Shutterstock

Researchers at University of Tsukuba have demonstrated the potential of emotion recognition technology that combines noncontact sensors with AI to accurately detect patients' emotions and support physicians in providing appropriate empathetic responses. This technology helps physicians better understand patients' emotions while objectively reducing empathy fatigue. It is expected to serve as a new support tool for physicians in clinical settings.

Tsukuba, Japan—In clinical settings, accurately understanding patients' emotions and responding appropriately plays a critical role in improving treatment outcomes and patient satisfaction. In this study, researchers developed a noncontact "multimodal emotion recognition" framework that combines multiple types of information, including patients' voices, conversations with physicians, and physiological responses.

This technology acquires physiological data such as heart rate and breathing without any physical contact, integrates it with voice and speech content, and analyzes patients' emotions accurately. Researchers conducted consultations between experienced physicians and simulated patients trained to replicate cancer treatment scenarios. They compared the accuracy of emotion recognition by physicians and the AI system against patients' self-reported emotions. The AI system scored higher than physicians in terms of how accurately emotions were captured, demonstrating that it can recognize emotions more accurately.

Conventionally, physicians are regarded as having high-level empathy. However, this study shows that AI has the potential to surpass specialists in emotion recognition by integrating and processing diverse information sources. Another significant advantage is that noncontact technology reduces patients' physical and psychological burden while enabling emotion data collection in a natural conversational environment.

These findings highlight the potential of this technology as a new support tool for physicians, helping them accurately understand patients' emotions and reduce the risk of "empathy fatigue" among healthcare professionals. The research team plans to conduct further verification and improvements to implement the system into real-world medical settings and expand its applications to areas such as elderly care and mental health support.

###
This work was supported by JSPS KAKENHI under Grant 22H03693 and Grant 23K24948.

Original Paper

Title of original paper:

Framework for Emotion Recognition Using Cross-Modal Transformers With Non-Contact Multimodal Signals Aiming Clinical Service Support

Journal:

IEEE Access

DOI:

10.1109/ACCESS.2025.3573648

Correspondence

Associate Professor ZEMPO Keiichi
Institute of Systems and Information Engineering, University of Tsukuba

Related Link

Institute of Systems and Information Engineering