This project has been funded by the European Commission as part of the H2020 program, under the grant agreement 762111.

The post VRTogether wins the Best Demo Award at MMM2019 appeared first on VRTogether.

This article focuses on Point Clouds and MPEG-MORE. The first part of this article is covering all the other technologies.

Point Cloud Compression

MPEG-I targets future immersive applications. Part 5 of this standard specifies Point Cloud Compression (PCC).

A point cloud is defined as a set of points in the 3D space. Each point is identified by its cartesian coordinates (x,y,z), referred to as spatial attributes, as well as other attributes, such as a color, a normal, a reflectance value, etc. There are no restrictions on the attributes associated with each point.

Point clouds allow representing volumetric signals. Because of their simplicity and versatility, they are important for emerging AR and VR applications. Point clouds are usually captured using multiple RGB plus depth sensors. A point cloud can contain millions of points in order to create a photorealistic reconstruction of an object. Compression of point clouds is essential to efficiently store and transmit volumetric data for applications such as tele-immersive video and free-viewpoint sports replays, as well as for innovative medical and robotic applications.

MPEG has a separate activity on point cloud compression: in April 2017 MPEG issued a Call for Proposals (CfP) on PCC technologies, seeking compression proposals in three categories:

1. Static frames
2. Dynamic sequences
3. Dynamically acquired/fused point clouds

Leading technology companies responded to the CfP, and the proposals were assessed in October 2017. In addition to objective metrics, each proposal was also evaluated through subjective tests, performed at GBTech and CWI. The winning projects were selected as “Test Models” for the next step of the standardization activity.

For the compression of dynamic sequences, it was found that compression performance can be significantly improved by leveraging existing video codecs after performing a 3D to 2D conversion using a suitable mapping scheme. This also allows the use of hardware acceleration of existing video codecs, which is supported by many current generation GPUs. Thus, synergies with existing hardware and software infrastructure can allow rapid deployment of new immersive experiences.

Figure 1: An example of a perspective view of a point cloud: the original version on the left, and two views of compressed versions of the same point cloud in the middle and on the right.

After the selection of the Test Models that combine the best performing technologies, the activity focused on the identification and investigation of methods to optimize the Test Models, by performing “Core Experiments”. Examples of Core Experiments include the comparison of different schemes for mapping the texture information from 3D to 2D, the analysis of hybrid codecs that combine 3D geometry compression techniques with traditional block-based video compression strategies, and the use of motion field coding. These Core Experiments are still ongoing.

At the last MPEG meetings, the PCC activity has been particularly crowded, attracting attention from many industrial partners. The main activities of the group focused on cross-checking test models and reviewing the results of the core experiments. In addition, some new datasets created by commercial companies (8i, Owlii, Samsung, Fraunhofer) were presented, and a proposal to merge two of the Test Models was presented; the goal was to take advantage of the 2D compression technics (HEVC and successors). A preliminary contribution also explored the delivery and transmission of point clouds based on this approach.

In the VRTogether project, CWI is providing a solution for lossy compression of dynamic point clouds, based on the open source software for Point Cloud Compression developed at CWI (available at https://github.com/cwi-dis/cwi-pcl-codec). This solution will not be competing in the standardization race, but it serves as an open source tool to benchmark different solutions and experiment research ideas. Currently, it is being integrated into the VRTogether DASH-based point cloud communication pipeline that will allow multiple users to see each other point cloud representation, captured in real time, and rendered in the same virtual environment. Part of CWI research within the VRTogether project will also focus on the design of new objective quality metrics for evaluating point clouds, based on the study of human perception of volumetric signals.

MPEG-MORE

MPEG‘s Media Orchestration standard (also known as MORE: MPEG-B part 13- https://mpeg.chiariglione.org/standards/mpeg-b/media-orchestration) has been finalized by the committee and the final edited version has been submitted to MPEG’s parent body and ISO for one more yes/no ballot followed by publication. This last step is a mere formality.  It is a bit hard to estimate when the specification will be published by ISO since it requires some secretariat work and this can take quite a few months. The work on Reference Content and SW continues. This work intends to make content available with MORE metadata so that (potential) users of the MORE specification can understand how the specification works and are assisted in creating implementations.

In the meantime, Social VR has become more important in MPEG, and it looks like some of the requirements can be fulfilled by the MORE specification. This notably applies to the simpler forms of Social VR, where images of users are composited into a VR360 experience. This requires both temporal synchronization (multiple users should experience the same scene simultaneously) and spatial coordination (the composited images for all users need to have consistent location and size for the experience to be perceived as realistic and compelling). MORE defines the necessary metadata and protocols for this.

About MPEG

MPEG is the Moving Picture Experts Group, a group from IEC and ISO which created some of the foundations of the multimedia industry: the MPEG-2 Transport Stream, and the MP4 file format, a series of successful codecs both in video (MPEG-2 Video, AVC/H264) and audio (MP3, AAC). A new generation (MPEG-H) emerged in 2013 with MPEG 3D Audio, HEVC and MMT, and other activities in MPEG-I like Point Cloud, MPEG Orchestration.

Who we are

Motion Spell is an SME specialized in audio-visual media technologies. Motion Spell was created in 2013 in Paris, France. On a conceptual and technical level, Motion Spell will focus on the development of transmission open tools. Furthermore, Motion Spell plans to explore encoding requirements for VR to participate in the current standardization efforts and first implementations. Finally we will also assist on the playback side to ensure the end-to-end workflow is covered.

Come and follow us in this VR journey with i2CAT, CWI, TNO, CERTH, Artanim, Viaccess-Orca, Entropy Studio.

The post VR strikes back at MPEG appeared first on VRTogether.

In this experiment, the users are equipped with a motion capture suit and reflective markers to track their movements in real time with a Vicon optical motion capture system. They also wear Manus VR gloves for fingers tracking and an Oculus HMD. At each trial, the face (eye gaze and mouth), fingers and the avatar’s upper and lower bodies are manipulated with different degree of animation fidelity, such as no animation, procedural animation and motion capture. Each time, users have to execute a number of tasks (walk, grab an object, speak in front of a mirror) and to evaluate if they are in control of their body. Users start with the simplest setting and, according to the judged priority, improve features of the avatar animation until they are satisfied with the experience of control.

Using the order in which users improve the movement features, we can assert on the most valuable animation features to the users. With this experiment, we want to confront the relative importance of animation features with the costs of adoption (monetary and effort) to provide software and use guidelines for live 3D rigged character mesh animation based on affordable hardware. This outcome will be useful to better define what makes a compelling social VR experience.

Who we are

Artanim is a non-profit foundation founded in 2011 and located in Geneva, Switzerland. The foundation carries out research activities in the field of computer graphics according to two strategic axes of research linked to motion capture: virtual reality, mainly the creation and animation of digital avatars and the development of interactive VR and AR applications; and medical research (joint biomechanics, orthopaedics, sports medicine).

Come and follow us in this VR journey with Motion Spell, i2CAT, CWI, TNO, CERTH, Viaccess-Orca, Entropy Studio

This project has been funded by the European Commission as part of the H2020 program, under the grant agreement 762111.

Text and pictures: Henrique Galvan Debarba, Caecilia Charbonnier – Artanim

The post Your body feels good appeared first on VRTogether.

• Shed light on changing behaviours and associated expectations in audio-visual consumption.
• Assess the market potential of the solutions developed within VRTogether group both regarding enhancing existing formats and creating new types of content.
• Identify technological trends, similar potential solutions as well as new partners.
• Identify the targets (B2C, B2B, …) of the solution and how they could successfully be addressed.

The scope of the study includes the head-mounted displays (HMD), the capture systems, the rendering engines and the VR softwares expected to be used by various industries.

The study provides an estimation of the forecasted revenues, an overview of the current usages, a VR SWOT (strengths, weaknesses, opportunities and Threats) and the VR applications being used or trialled nowadays in the different industries. With the eruption of virtual reality content, the user experience has been very lonesome. As a consequence, some developers have begun to include a social dimension to their content in various industries: Games, Social Networks, Sports, Music & Arts, Education, Tourism and of course business applications. The VR market remains relatively niche, but a continued moderate growth of the headset adoption is expected over the next few years: hardware prices drop, technology improves and content becomes more appealing.

Furthermore, the VR value chain is presented in the study and the associated technologies are benchmarked to highlight the best suited to the VRTogether project. The analysis will lead to the proposal of a value proposition associated with a best-of-breed ecosystem. The study will also provide the targeted customers (businesses and consumers) and the possible business models.

We will keep you informed of the publication of the document!

Who we are

As a leading global provider of content protection, delivery, and discovery solutions, Viaccess-Orca is shaping the ultimate content experience. VO will contribute to the different WP of the project VRTogether, we will focus on Platform Design and architecture and leads work package 5 on Innovation, Dissemination and exploitation. We will also contribute to the evaluation of the solution compared to market available product.

Come and follow us in this VR journey with i2CAT, CWI, TNO, CERTH, Artanim, Motion Spell, Entropy Studio.

This project has been funded by the European Commission as part of the H2020 program, under the grant agreement 762111.

The post VRTogether has launched a market research study covering VR trends and technologies appeared first on VRTogether.

In the issues analysed he addresses the topic of inference by giving different examples in which our brain’s assumptions are far from reality and how VR can use this in its favour to design the next level of immersive experiences. Moreover, he refers to open issues such as eye tracking, hand tracking, focus in VR displays and essentially the importance of coordination of real humans movements within the virtual space.

Looking a bit further around we discovered a market research study indicating how people intend to use VR technologies. We notice a high expectation of VR usage among the users who own the necessary hardware but most importantly the importance of social engagement in VR.

We recently had a great experience by trying out the platform that Oculus experiment with, Oculus Venues, by showing selected games from the World Cup to viewers based in the USA. After joining the experience and starting the application, we found ourselves represented by an avatar and seated in a virtual part of the football stadium where the game was taking place. Besides the football game and the score, the most interesting part was the rest of the people who were around us. It was possible to interact verbally and also use the Rift controllers for moving the hands of the avatar. It was really interesting to see people joining such an experience given the airing time of the events (05:00 a.m.) and its popularity as a time assigned to sleep. Communicating with the other end-users and talking about the sociality of this VR experience was very easy and a lot of this excitement was shared among the crowd of the participants.

VR-Together will be addressing this kind of social experiences in its core by assembling an innovative end-to-end pipeline for the delivery of photorealistic immersive content to multiple users. VR-Together will be going beyond the state-of-the-art experiences mentioned above. In Pilot 1, end-users will be represented by their real-life point clouds, time-varying meshes or 2D images, compared to the avatars used in Oculus Venues.  End-users will be able to interact and communicate with each other in real-time, with a special functionality for HMD removal, within a virtual interrogation room (3D scene) while also being able to interact with the environment itself (in Pilot 3). VR-Together will also conduct some experiments to evaluate the level of immersiveness, the feeling of togetherness and various other aspects that define what constitutes a “truly” social VR experience. An overview of the project, its objectives and envisioned scenarios can be found in the publication presented during the VR workshop in ACM TVX 2018^[1]. Below you can find a list of the metrics VR-Together will be using to evaluate the VR experience in development.

We are excited and proud to present our Pilot 1 experience in September 2018. Make sure to follow our Twitter account and find out about our next demonstration!

Stay tuned and see you soon!

^[1] M. Montagud, J. A. Núñez, T. Karavellas, I. Jurado, S. Fernández, “Convergence between TV and VR: Enabling Truly Immersive and Social Experiences”, Workshop on Virtual Reality, co-located with ACM TVX 2018, Seoul (South Korea), June 2018

Who we are

The i2CAT Foundation is a non-profit research and technology centre that promotes R&D activities in the fields of the Internet and advanced digital technologies. It has pioneered an innovation model based on collaboration between companies, public administrations, academia and users. i2CAT coordinates the VRTogether project

Come and follow us in this VR journey with Motion Spell, CWI, TNO, CERTH, Artanim, Viaccess-Orca, Entropy Studio

This project has been funded by the European Commission as part of the H2020 program, under the grant agreement 762111.

Text: Themistoklis Karavellas – i2CAT

The post VRTogether and the future of Social VR appeared first on VRTogether.

The first paper is titled “Virtual Reality Conferencing: Multi-user immersive VR experiences on the web”. The VRTogether project tackles the problem of the apparent discrepancy between the physical separation of wearing a head mounted display and the human need for sharing experiences. The project has developed a VR framework that allows creating VR experiences that are social and to consume them with off-the-shelf hardware. In the demo, the focus was on a communication use case, where a maximum of three people, sited around a round table could communicate within VR. To do so, users were recorded, and before the transmission, their background was replaced with a uniform green colour, which was removed later, leaving a transparent image showing just the user. In this experience, the three people sat around the table share the same view (the other users are on the opposite side of the table), and there is a video playing on the top of the table. The project team held a 1-day experiment trial, where they collected feedback from 54 participants, who communicated and watched a video with this system.

The second paper is entitled “Experiencing Virtual Reality Together: Social VR Use Case Study”. When it comes to Social Virtual Reality, users are always represented as artificial avatars. Even though this might be beneficial for some use cases, this might not be appropriate for many communication settings such as business meetings, or sharing experiences with family or friends. It is still unclear which use cases are relevant to the different methods users can be represented. Thus, more research is necessary to understand better Social VR requirements. As a first step to close the gap, the VRTogether team conducted a study where participants tried a photo-realistic Social VR experience followed by a questionnaire and informal discussion. In the VR environment, users sit beside each other on a couch in a 360-degree 2D VR environment and consume a 2D video, while being able to hear and see each other as photo-realistic video streams. The main contribution of this paper is the study of use cases in Social VR.

Both the MMSys and the ACM TVX2018 are significant dates in the media research agenda. They represent a great opportunity to disseminate the VRTogether objectives and advances and to get to know other ongoing projects in this field.

The post From Amsterdam to Seoul: disseminating the project around the globe appeared first on VRTogether.