The integration of technology into our physical world has become commonplace with the rise of mobile phones, smartwatches, and earbuds. However, researchers and companies are now exploring ways to assimilate ourselves into a virtual environment, where technology becomes an integral part of our existence. Assistant Professor Xiong Zehui from the Singapore University of Technology and Design (SUTD) is at the forefront of this research, collaborating with researchers from the Nanyang Technological University and the Guangdong University of Technology to explore the concept of the metaverse—a virtual reality universe where users control avatars to interact with a virtual environment.
The joint effort resulted in a preprint titled, ‘Vision-based semantic communications for metaverse services: A contest theoretic approach’. This research, which will be presented at the IEEE Global Communications Conference in December 2023, aims to address a key challenge in the mainstream adoption of metaverse services—real-time synchronization between human actions and avatar responses.
Real-time synchronization is crucial for creating an immersive metaverse experience, but it places significant demands on the rendering resource allocation scheme of the metaverse service provider (MSP). The MSP must handle massive amounts of data to relay information between users and the server. Fast actions, such as running or jumping, pose a particular challenge as the MSP struggles to update avatars in real-time, leading to a lapse in smoothness.
Traditionally, one solution to this problem is to restrict the number of users in a virtual environment to ensure the MSP has sufficient resources to simulate all users’ avatars, regardless of their activity levels. However, this approach is highly inefficient as it allocates excess resources to users who are standing still, resulting in wastage. Asst Prof Xiong and his team proposed a novel framework to optimize resource allocation in MSPs, with the goal of providing a smooth and immersive experience for all users.
Related Post
Their approach involves using a semantic communication technique called human pose estimation (HPE) to reduce the information payload for users. By identifying humans in images and highlighting only the skeletal joints, the algorithm can reconstruct a simplified stickman-like model that guides the MSP in modeling avatar actions. This technique reduces the data overhead by a million-fold, improving bandwidth efficiency significantly.
Once the information is efficiently encoded to the MSPs, the team focused on modeling interactions between the MSPs and users using contest theory. In this approach, users and their devices become competitors vying for the MSP’s resources. The algorithm aims to minimize latency across all users by optimizing the allocation of available resources, considering each device’s update rate based on the user’s actions.
To test for lag, the algorithm measures discrepancies between the avatar’s position and the HPE stickman at different update rates. Users experiencing lag will have significant differences between their stickman and avatar positions. The MSP’s resources act as awards for competitors who perform well without lag.
To accurately deduce the right amount of resources to request from the MSP, each user needs to employ machine learning techniques. Asst Prof Xiong’s team implemented a neural network called the deep Q-network (DQN) to optimize resource distribution. The adoption of DQN resulted in a 66% improvement in lag across all users compared to traditional methods.
Asst Prof Xiong is optimistic about the future of the metaverse, foreseeing its application in healthcare, education, and marketing. He envisions integrating cutting-edge technologies like generative AI and VR into the metaverse, as well as the growth of global, digital, and virtual economies. These advancements have the potential to shape the future of the metaverse, offering exciting possibilities for immersive and interactive experiences.
In conclusion, the research conducted by Asst Prof Xiong and his team at SUTD, in collaboration with other universities, brings us closer to achieving a seamless integration of humans into the metaverse. Their innovative framework optimizes resource allocation in MSPs, reducing lag and improving the overall immersive experience. As technology continues to advance, the possibilities for the metaverse are expanding, opening doors for various industries to leverage its potential.
