There is growing consensus in human-computer interaction (HCI) and affective computing that affect — users’ emotional and psychological states — is a critical contextual dimension for truly adaptive XR systems. While traditional context-aware systems primarily adapt to users’ tasks and environmental conditions, affect-aware systems should extend this capability by continuously inferring users’ affective states, reasoning about their causes within a given context, and delivering adaptive responses that help users mitigate unwanted emotions and successfully progress through their tasks. One example of where this might be particularly relevant is education, where a student’s emotions strongly influence key learning processes such as attention, memory, reasoning, and decision-making. In this context, intelligent XR (iXR) systems should aim to recognize and adapt to users’ affective states in real time, enabling experiences that respond dynamically to how users feel and behave.
Prior research demonstrates that affective states can be inferred by AI models analyzing physiological, behavioral, and cognitive signals, many of which are now accessible through modern head-worn displays (HWDs), such as gaze data. However, integrating real-time, AI-driven emotion recognition into XR systems remains both technically and conceptually challenging. In the realm of intelligent adaptations, existing adaptive systems typically employ rule-based mechanisms, such as modifying game difficulty based on detected stress or anxiety levels. While these methods can be effective for targeted use cases, they under-exploit the broader potential of AI to enable nuanced, context-aware adaptations.
Our project aims to pioneer the integration of continuous, real-time affect inference within interactive XR environments, enabling meaningful adaptations tailored to users’ emotional states as they occur. As a proof-of-concept, an initial study will gather both self-reported and gaze data and preliminary insights into how adaptive XR environments can detect and respond specifically to boredom. By immersing participants in a simulated educational environment, this study aims to provide valuable insights into how intelligent XR technology could be applied in relevant real-world settings, where emotions play a critical role in performance and learning outcomes.
