Authors:
Ka Hei Carrie Lau、Efe Bozkir、Hong Gao、Enkelejda Kasneci
Paper:
https://arxiv.org/abs/2408.09285
Introduction
In recent years, the integration of digital technologies in education has revolutionized learning paradigms, making them more immersive, interactive, and personalized. Virtual Reality (VR) has demonstrated its capacity to create highly immersive environments that simulate real-life experiences or historical events. Similarly, advancements in Large Language Models (LLMs) enable the generation of human-like responses, providing interactive learning experiences. However, the combination of VR and LLMs to preserve cultural heritage remains largely unexplored. This paper aims to evaluate user perception and the educational impact of integrating these technologies in virtual environments for cultural learning, specifically focusing on traditional Scottish curling presented in the game “Scottish Bonspiel VR”.
Related Work
Preserving Cultural Heritage in Digital Humanities
Cultural spaces have increasingly adopted digital technologies such as digital storytelling, serious games, and interactive displays to engage users. VR, in particular, has shown promise in Digital Humanities (DH) for enhancing cultural heritage experiences by enriching user interactions and improving educational outcomes. However, these implementations often focus on individual tools rather than the holistic integration of technologies. This study explores the integration of VR and LLMs to improve cultural heritage accessibility and interactivity, leveraging LLM chatbots and web-based VR environments to enhance dynamic user engagement beyond traditional conversational chatbots and audio guides.
The Potentials of Emerging Technologies on New Learning Paradigms
LLM-based chatbots have significantly improved the delivery of historical content in museums by leveraging Natural Language Processing (NLP) and Machine Learning (ML) for enhanced visitor interactions. However, these technological solutions often face scalability challenges beyond the confines of physical institutions, restricting their broader applicability in global cultural heritage education. Recent advancements in Human-Centered LLM, particularly through explainable LLM (XLLM) and GPT-4, promise new horizons for cultural heritage accessibility. This research integrates an LLM-based chatbot within a VR environment to scale cultural heritage experiences beyond museum spaces, leveraging LLM’s advanced conversational abilities to provide more meaningful and accessible cultural heritage interactions.
Research Methodology
This study aimed to enhance user experience within cultural heritage applications by integrating VR and LLMs. The researchers developed the “Scottish Bonspiel VR” game, incorporating an LLM-based chatbot as a cultural ambassador to illustrate how traditional Scottish curling was played. This exploratory research compared the performance of ChatGPT with a predefined chatbot commonly used in digital cultural heritage settings. The predefined chatbot, featuring scripted responses within Scottish cultural heritage contexts, served as a baseline for comparison. Without further fine-tuning of ChatGPT and the prescripted chatbot, this setup allowed for a direct comparison with the dynamic capabilities of the LLM-based chatbot, highlighting potential improvements in usability, engagement, and learning outcomes.
Experimental Design
Participants
The study recruited 36 participants, each compensated with a 10 Euro Amazon voucher. Eligibility criteria included being at least 18 years old, having normal or corrected-to-normal vision, and fluency in English. Participants were academically diverse, including bachelor’s, master’s, and doctoral students, with a mix of genders and varied experiences with VR, gaming, and prior awareness of Scottish curling.
Apparatus and Tools
The Oculus Quest 2 was used for the immersive VR experience. The VR scenes were self-designed and developed using Vite and A-frame, hosted on Vercel. The OpenLLM API was employed for the interactive chatbot, utilizing the GPT-4 model for its advanced conversational capabilities. To allow participants to communicate verbally with the chatbot, the OpenLLM API was integrated with Whisper for Speech-to-Text (STT) functionality, and chatbot responses were relayed using Azure Text-to-Speech (TTS).
Experimental Procedure
The study employed a between-subjects experimental design, with participants randomly allocated into two groups: a control group interacting with a predefined chatbot and a treatment group engaging with ChatGPT within a VR environment themed around Scottish curling. The primary independent variable was the type of chatbot: predefined responses versus ChatGPT-generated responses. The dependent variables included usability feedback, user engagement, learning outcomes, and cognitive task load.
Participants engaged in two game levels of the VR experiment:
– Level 1 “Traditional Scottish Curling on a Frozen Lake”: Participants experienced an outdoor VR environment simulating the Grand Match, learning about the sport’s history, cultural significance, and environmental impact, and virtually shooting a curling stone on a frozen lake.
– Level 2 “Indoor Scottish Curling”: This level highlighted the sport’s shift to indoor settings due to climate change. Participants engaged in a Q&A session with either ChatGPT or a predefined chatbot, building on Level 1 to deepen their understanding.
Measures
A mixed-methods approach was employed to collect data, focusing on usability, user engagement, and learning outcomes. Usability was addressed through a customized usability survey based on the System Usability Scale (SUS) and the AttrakDiff questionnaire. User engagement was quantified through VR system data, including conversation logs with the chatbot and in-game performance metrics. Cultural heritage learning outcomes were measured using pre- and post-assessment surveys and tests. Cognitive load was evaluated using the NASA Task Load Index (NASA TLX).
Results and Analysis
Usability Metrics
The Shapiro-Wilk test was performed to verify the normality assumption. For non-normal distributions, the Mann-Whitney U test was used. Significant differences were found between the CGPT and PD conditions across several usability items, with CGPT scoring higher in accuracy, communication, intuitiveness, overall experience, and satisfaction.
User Engagement
User engagement was assessed through conversation logs and in-game metrics. The CGPT group had significantly higher prompting frequency and longer total playtime compared to the PD group, suggesting higher engagement levels with the LLM-based chatbot.
Learning Outcomes
Learning outcomes were evaluated by comparing changes in participants’ knowledge before and after the VR experience. Although no significant differences were found between the CGPT and PD conditions, both groups demonstrated an increased understanding of the impact of climate change and the importance of preserving cultural heritage.
Cognitive Load Assessment
Cognitive load was evaluated using the NASA TLX across six dimensions. No significant differences were found between the ChatGPT and Predefined conditions, although trends towards significance were observed in performance and effort, with lower scores in the ChatGPT condition.
Overall Conclusion
This study demonstrates the effectiveness of integrating VR and LLM technologies in enhancing the preservation and understanding of intangible cultural heritage through the “Scottish Bonspiel VR”. The findings highlight the capacity of LLM-powered chatbots to create immersive and educational experiences, significantly improving usability and engagement over traditional predefined chatbots. The research contributes to the Digital Humanities community by showcasing the potential of VR and LLM technologies to enhance accessibility and preservation of cultural heritage. Future research will refine methodologies and expand the scope to encompass a broader array of cultural contexts, leveraging technology for both educational and conservation purposes.
