A team of marine scientists employ extended reality (XR) to revolutionize people experiences in marine education. This UGC-funded project is led by Prof. Cindy Lam (Department of Ocean Science) and Prof. Sai-Kit Yeung (Division of Integrative Systems & Design) at HKUST, with joint collaboration of Prof. Apple Chui (School of Life Sciences) from CUHK, Prof. Jianwen Qiu and Dr. Veronica Chan (Department of Biology) from HKBU, and Prof. Cedric Tan (Environmental and Geographical Sciences) from the University of Nottingham Malaysia.
The project aims to design a game app with immersive Virtual Reality (VR) eLearning tools with AI machine-learning platform “MarineGPT”. The platform contains marine-specific data and information which requires players to learn right before the game starts. Our MarineGPT not only pushes the boundaries of marine understanding to the general public but also offers a standard protocol for adapting a general-purpose assistant to downstream domain-specific experts. We pave the way for a wide range of marine applications while setting valuable data and pre-trained models for future research in both academic and industrial communities.
The marine VR game offers immersive learning environment to transport students into realistic underwater scenarios that would be impossible or impractical to experience in real life. The game framework is on level-based, the player must complete certain objectives, such as defeating enemies or solving scenario-based challenges in order to progress to the next level (Figure 1). The game has 4 levels in total, starting from intertidal habitats such as mangroves and mudflats, boulder shore (Level 1) to subtidal coral reefs (Level 2) in Hong Kong and then to the deep sea (Level 3) in South China Sea. After completion of Level 3, players will step into the next level to learn similar coastal marine habitats in Malaysia (Level 4). The game offers multi-learning objectives and highlights: (i) game interactivity - students can manipulate different marine habitats and ecosystems, observe the consequences of their actions (like marine pollution or conservation efforts), and learn through active participation rather than passive observation; (ii) adaptive learning paths – our game design and setting uses adaptive learning technologies to personalize the educational experience. It can adjust the difficulty of tasks and the information presented based on the user’s performance and learning speed; (iii) collaborative features – students would work together with the VR environment, fostering teamwork and communication skills; (iv) sustainability and conservation focus – the game content and scenario-based challenges are focus on marine biodiversity and conservation, sustainable practices, aligning with growing educational trends towards environmental stewardship. The built-in assessment tools in the game would provide immediate feedback to students and teachers, helping to track progress and identify areas needs further assistance and improvement.
Figure 1. The framework of the level-based game.
In the game, 3D models of marine organisms inhabiting different marine habitats are developed (Figure 2), as well as the game scene of their living habitats in real life. To prepare for the real scene, we have recruited a 3D artist to develop high-quality 2D/3D characters and visualization of marine habitats above ground and underwater (Figures 3 &4). He has adopted advanced 3D mapping technologies to map the ocean floor at different depths. Using air-drone, he can map the overall topography and surface of intertidal habitats (Figure 5); while using underwater drones, he can create detailed maps that reveal underwater landscapes, geological features and marine ecosystems (Figure 6). We make 3D artist so that we can do different styles easily with the same 3D maps, and 3D mapping by AI and human complement each other to give a complete 3D maps with details (e.g. objects present) by our state-of-the-art 3D mapping technology Photo-SLAM. After mapping, the real scene of marine habitat has been reconstructed with some quizzes or tasks by exploring different marine organisms underwater. The topography and distinctive environmental features of the boulder shore (e.g. large bed rocks), as well as marine organisms inhabiting at high, middle and low tide levels will be incorporated in the game. These 3D maps are invaluable for scientific research and development of immersive learning virtual environment in marine XR.
Including MarineGPT, the technology adopted in our XR project is one of the most advanced in the world. We hope our Extended Reality eLearning platform can inspire marine research community to further explore the unknown ocean and increase awareness of the general public on marine biodiversity and conservation.
