For more details, see previous post “Mentoring Students in Social VR”

The VRTogether project conducts ground-breaking research on Social VR, which invites multiple users to share a virtual space and interact with each other’s virtual representation. Apart from the main research activities, this project mentors master’s students in this area, forming a new generation of graduates that can, in the future, reshape the media landscape. CWI has been actively offering master’s thesis around the core topics of the project since 2018.

In 2018, two students graduated at TU Delft with the theses “Multi-Camera Registration for VR: A Flexible, Feature-based Approach” (Qian Qinzhuan) and “User Experience in Social Virtual Reality” (Yiping Kong). The latter resulted as a top-quality paper, “Measuring and understanding photo sharing experiences in social Virtual Reality” at ACM CHI 2019. In 2019, another two students graduated: Guo Chen (TU Delft) with the thesis “Designing and Evaluating a Social VR Clinic for Knee Replacement Surgery”, and Jelmer Mulder (VU Amsterdam) the thesis “Temporal Interpolation of Dynamic Point Clouds using Convolutional Neural Networks”. Guo’s work was published as an ACM CHI 2020 late-breaking-work and an improved demo of Guo’s work was published at ACM IMX 2020, which received the best demo award.  Jelmer’s work was published as an invited paper in the 2019 IEEE Conference on Artificial Intelligence and Virtual Reality (AIVR).

In 2020, again two students graduated: Yanni Mei at TU Delft with the thesis “Cake VR: Design a Social VR Tool for Remote Co-Design of Customized Cakes” and Ignacio Reimat at Universitat Politecnica de Catalunya with the thesis “Temporal Interpolation of Human Point Clouds Using Neural Networks and Body Part Segmentation”. Both students graduated with score 9 (out of 10). We are currently working on publishing the students’ work to prestigious conferences.

Yanni’s thesis explored novel use cases in the pastry design domain for Social VR, developing and evaluating a prototype of a social VR tool for clients to co-design cakes with pastry chefs. Exploring such use cases can help better understanding the exploitation opportunities for social VR. Ignacio’s thesis investigated how body part segmentation can aid in performing temporal interpolation for dynamic point clouds. Body part segmentation can be used to improve the accuracy of the neural network in dealing with parts moving at different speed. Moreover, the same principle can be applied in improving compression efficiency for point cloud transmission, by adopting a different level of detail based on salient parts.

Yanni Mei successfully defended her master’s thesis on August 20^th, 2020. The motivation behind this project was to support clients with limited design skills and pastry knowledge to better communicate and co-design their dreamed cakes with pastry chefs for special celebrations. The cake VR tool provides 3D visualization and facilitates intuitive gestural manipulation of the size and decoration of the virtual cakes, which can be used for both onsite and remote communication (Figure 1 and Figure 2). The project started with a series of ethnographic studies with five pastry chefs and four clients who had experiences in purchasing customized cakes. Then, based on the requirements gathered from the ethnographic studies, we designed and implemented a social VR prototype, which allows two users collaboratively design cakes in a shared virtual space, wearing head-mounted displays (HMDs). We also performed a user validation test with six clients and 3 pastry chefs to see to what extent the cake VR prototype meets the requirements. The prototype successfully meets the 7 (out of 10) requirements, and clients are able to design cakes and communicate the size, decoration and theme of the celebration of the cakes with pastry chefs.

Ignacio Reimat successfully defended his thesis on June 30^th 2020. The motivation behind the project was to reduce the bandwidth requirements for transmission of dynamic point cloud. Reducing the temporal resolution of the sequence to be transmitted, results in notable bitrate gains; however, it comes at the expense of perceived quality. Performing temporal interpolation at the receiver side allows to cope with bandwidth requirements, while maintaining the appearance of smooth motion. In his thesis, Ignacio provided two main contributions: a) the creation of point cloud contents with annotated body parts, from a publicly available 3D mesh dataset, and b) an architecture capable of performing temporal interpolation of point cloud contents representing body parts. Results show that that applying body part segmentation and predicting the interpolation of individual body parts can improve the accuracy of point cloud temporal interpolation systems.

---

 Yanni Mei, Design a SocialVR Tool for the Remote Co-Design of Customized Cakes (2020, TU Delft): https://repository.tudelft.nl/islandora/object/uuid%3A78a1147b-e97b-418f-a5e6-3ce944df4f49

Nacho Reimat, Temporal Interpolation of Human Point Clouds Using Neural Networks and Body Part Segmentation (2020, Universitat Politecnica de Catalunya): https://www.dis.cwi.nl/downloads/Masters-2020-Nacho.pdf

VRTogether project partner CWI participated in the 2020 ACM International Conference on Interactive Media Experiences (ACM IMX 2020), held virtually from June 17-19, 2020. On the first day of the conference, CWI’s team presented the social VR clinic, one of the VRTogether exploitation activities targeting the healthcare domain. The demo is entitled “A Social VR Clinic for Knee Arthritis Patients with Haptics” and was presented using a social virtual reality (VR) platform called Mozilla Hubs. This demo is co-authored by Tong Xue (CWI, BIT), Jie Li (CWI), Guo Chen (IBM Research, Beijing) and Pablo Cesar (CWI). The demo presentation was held in a virtual dome surrounded by virtual Barcelona cultural heritage.

The demo showcases a social VR clinic that allows patients to consult a nurse represented as a virtual avatar. It offers a “walk-in” virtual surgery room, enables patients to interact with animated virtual 3D artifacts, and trains the patient to use an injection tool while wearing a pair of mechanical VR gloves that provide haptic feedback (in collaboration with the Dutch SME SenseGlove). The demo shows the potential of social VR as a new tool to help patients receive remote personalized medical care. The virtual demo presentation showed a video of the prototype and two posters explaining the research process. The whole session was interactive and immersive with many questions raised from the audience. The conference organizing committee and the audience all found the demo well done and gave positive feedback towards the future work.

This demo was eventually awarded the Best Demo award at IMX 2020. IMX is the leading international conference for presentation and discussion of research into interactive media experiences. The conference brought together international researchers and practitioners from a wide range of disciplines, ranging from human-computer interaction, multimedia engineering and design to media studies, media psychology and sociology.

The social VR clinic demo is one of the VRTogether project exploitation activities explored by CWI. Apart from personalized healthcare, where we are as well exploring how 3D medical imaging can help in creating photorealistic models for remote treatment of patients, other domains include:

• Creative industries: interactive immersive museum experiences in social VR for both remote and on-site visitors, together with the Netherlands Institute for Sound and Vision
• Retail sector: collaborative design and prototyping in social VR for the retail sector, like bakeries and other food services

We will keep you updated with the future development of the social VR use cases.

Jie Li and Pablo Cesar, from the Distributed Interactive Systems group (DIS) at Centrum Wiskunde & Informatica (CWI), have successfully organized the ACM CHI2020 Social VR Workshop on a social VR platform called Mozilla Hubs. This workshop was co-organized together with David A. Shamma (FXPAL Laboratory), Vinoba Vinayagamoorthy (BBC R&D), Raz Schwartz (Facebook AR/VR), and Wijnand Ijsselsteijn (Eindhoven University of Technology). Due to the COVID-19, the event, scheduled on April 25 at the Hawaii Convention Center, was cancelled. However, the co-organizers decided to make use of the quarantine as a unique opportunity to explore how a social VR platform could be used to run the workshop. The workshop, “Social VR: ​A New Medium for Remote Communication & Collaboration”, was sponsored by the EU H2020 VRTogether project, and received technical support from Mozilla and FXPAL laboratory.

The workshop attracted mass attention in the ACM SIGCHI community, and on social media. For example, one of the twitter posts received over 13,000 impressions and 1200 engagements. The organizers also received hundreds of requests about participating in the workshop. In the end, the organizers reviewed and selected 20 participants worldwide based on their submitted high-quality position papers (position papers are available at https://www.socialvr-ws.com). The time zones of the participants ranged from GMT-7 (e.g., California, US) to GMT+9 (Tokyo, Japan), but they all managed to meet in this virtual workshop.

The 5-hour virtual workshop was intense, fun and engaging. The virtual workshop space had a main hall with two big screens where the presentation slides or videos could be displayed. Linked with the main hall (Figure 2), there were three breakout virtual rooms for group discussions on assigned topics: user representation & ethics, evaluation methods, and interaction techniques. The breakout rooms ensured that the audio of each group discussion was separated, and would not influence each other. The participants selected their own custom avatars to represent themselves.

The organizers designed an interactive and engaging workshop program, which started with a keynote talk from Professor Blair MacIntyre about his experience of organizing the IEEEVR 2020 conference in Mozilla Hubs (Figure 3a). Then, the big screen showed the slides prepared by participants to introduce themselves and their position paper, and each participant was invited to “fly” next to the big screen to give a 2-minute pitch about their paper. Next, they were divided into three discussion groups based on their preferences for the three pre-defined social VR topics by the organizers (Figure 3b). Finally, the organizers brought all the participants back to the main hall to present the discussion results.

The whole workshop went smoothly. Many participants reported that the activities of the virtual workshop were as engaging as the real physical workshop. It was easy and natural for them to follow the keynote talks and to contribute to the discussions. Even during the break, many participants chose to stay and explore the virtual space (e.g., dive into the virtual sea, see Figure 3c). After obtaining consent from the participants, the organizers managed to collect relevant data about the position, movement, and interaction of the participants in the virtual environment and with each other. Currently, the organizers are continuing the exploration of the possibilities offered by Social VR platforms through an online survey, and semi-structured interviews with the participants.

The workshop ended with fruitful discussions about the future of social VR technology. The participants and the organizers foresee the full potential of social VR as a new medium to connect people across the world, and engage them to communicate and collaborate in a virtual space for hours. Many participants shared their engaging and meaningful virtual conferencing experience on social media, and expressed their appreciation towards the organizers through emails. A full report on how well social VR platforms may support remote scientific events will follow.

The VRTogether project conducts ground-breaking research on Social VR. In addition, the project mentors master’s students in this area, forming a new generation of graduates that can, in the future, reshape the media landscape. CWI has been active, offering master’s thesis around the core topics of the project. Last year, two students graduated at TU Delft with the theses “Multi-Camera Registration for VR: A Flexible, Feature-based Approach” (Qian Qinzhuan) and “User Experience in Social Virtual Reality” (Yiping Kong). The latter resulting in a top quality paper, “Measuring and understanding photo sharing experiences in social Virtual Reality”, presented at ACM CHI 2019.

This year, again two students have graduated acquiring deep knowledge about the fundamentals of Social VR. Guo Chen (TU Delft) has written the thesis “Designing and Evaluating a Social VR Clinic for Knee Replacement Surgery” and Jelmer Mulder (VU Amsterdam) the thesis “Temporal Interpolation of Dynamic Point Clouds using Convolutional Neural Networks”. The first one explored novel use cases in the healthcare domain for Social VR, developing and evaluating a prototype of a Social VR clinic. Such exploration can help better understanding the exploitation opportunities. The second one proposed a temporal interpolation architecture capable of increasing, at the receiving side, the temporal resolution of dynamic point clouds. The results have consequences in the core architecture of the system, since it may allow for providing the same quality of experience of users, while using less bandwidth for delivering them.

Guo Chen successfully defended her master’s thesis, which explores how Social VR could be used in a hospital setting. The motivation behind this project was to support patients with limited physical mobility to travel fewer times to the hospital, but still communicate well with doctors and nurses. Patients with knee osteoarthritis were the target group of this work. The final goal was to build a Social VR clinic that simulates the real consultation room and facilities in the hospital, in which, patients can interact with the doctors or nurses with visualized information, such as surgery preparation procedures, 3D anatomy models and a tour in the surgery room. A series of ethnographic studies at the Reinier de Graaf hospital in Delft lead to a better understanding of the complete patient journey, and to the identification of the requirements for the design and prototyping of a Social VR solution. It supports the three main identified activities within the patent journey: visualization of the intervention process, “walking into” a 3D virtual surgery room to “meet” the medical staff and to get familiar with the equipment, and interacting with an animated virtual 3D knee anatomy model and a virtual knee prosthesis model to see what the differences are before and after the surgery. Twelve users were recruited to evaluate the Social VR clinic, who indicated that the Social VR clinic consultation is comparable with the face-to-face, but does not require patients with knee problems to go to the hospital.

Jelmer Mulder as well successfully defended his master’s thesis, which explores how machine learning techniques can help in optimizing the distribution of dense point clouds. Point clouds are a data structure that models volumetric visual data as a set of individual points in space, which it is used in VRTogether for highly realistic representing users in a Social VR session. But point clouds are voluminous in size, and thus require high bandwidth to transmit. In practice this means that concessions have to be made either in spatial or temporal resolution. In his thesis he proposes an architecture capable of increasing the temporal resolution of dynamic point clouds. With this technique, dynamic point clouds can be transmitted in a lower temporal resolution, after which a higher temporal resolution can be obtained by performing the interpolation on the receiving side. The interpolation architecture works by first downsampling the point clouds to a lower spatial resolution, then estimating scene flow, and finally upsampling the result back to the original spatial resolution. To improve the smoothness of the interpolation result, a novel technique called neighbour snapping is applied, and in order to estimate the scene flow, a newly designed neural network architecture is used. The architecture was evaluated through objective metrics and based on a small-scale user study. Existing objective quality metrics for point clouds are known to have poor correlation with user perception and the findings confirm this: the metrics correlate poorly with themselves and with the results from the user study. The user study shows that on average, participants prefer the temporally interpolated sequences generated by our architecture over current state-of-the-art or sequences that have not been interpolated.

Congratulations to the recently graduated students!

VR-Together was present at IEEE VR 2019, the 26th IEEE Conference on Virtual Reality (March 23rd -27th, 2019) in Osaka, Japan and will be present at ACM CHI 2019, the ACM Conference on Human Factors in Computing Systems (May 4th-9th) in Glasgow, United Kingdom. The goal is to present the successful results on user experience and quality of experience for social VR. In particular, the protocol for users’ QoE assessment in social VR application, developed in the framework of the project, and the user studies performed by CWI during the first year of the project, will be presented.

At IEEE VR, CWI researcher Jie Li presented the poster “Watching videos together in social Virtual Reality: an experimental study on user’s QoE”, showcasing the application of the methodology developed by CWI for users’ QoE assessment in social VR in a comparative study where a set of users used three social VR platforms to watch videos together. Find out more on the outcome of our experiment by reading our IEEE VR short paper, here.