Login(userName, userPassword) 
Logout()

GetScenarios()
GetScenarioInfo(scenarioId)
GetScenarioInstanceInfo(scenarioId)

AddSession(sessionName, sessionDescription, scenarioId, [canBeMaster])
DeleteSession(sessionId)
GetSessions()
GetSessionInfo()
JoinSession(sessionId, [canBeMaster])
LeaveSession()

GetRooms()
GetRoomInfo()
JoinRoom(roomId)
LeaveRoom()

StartScenario()
RestartScenario()
StopScenario()

GetUsers()
GetUserInfo([userId])

GetUserData([userId])
UpdateUserData(userDataKey, userDataValue)
UpdateUserDataArray(userDataArray)
UpdateUserDataJson(userDataJson)
ClearUserData()

SendMessageToAll(message)
SendMessage(userId, message)

SendSceneEventToMaster(sceneEventData)
SendSceneEventToUser(userId, sceneEventData)
SendSceneEventToAllUsers(sceneEventData)
SendSceneEventToUserDirect(userId, sceneEventData)
SendSceneEventToAllUsersDirect(sceneEventData)

GetSfuInfo()
GetSfuData()

GetLivePresenterInfo()
GetLivePresenterData()

GetNTPTime()

DeclareDataStream(dataStreamKind, dataStreamDescription)
RemoveDataStream(dataStreamKind)
RemoveAllDataStreams()

SendData(dataStreamKind, data)

GetAvailableDataStreams(dataStreamUserId)

RegisterForDataStream(dataStreamUserId, dataStreamKind)
UnregisterFromDataStream(dataStreamUserId, dataStreamKind)
UnregisterFromAllDataStreams()
GetRegisteredDataStreams()

PushAudio(audioData)

This project has been funded by the European Commission as part of the H2020 program, under the grant agreement 762111.

The post Towards a generic orchestrator: The VRTogether project experience<br><h4>Part 3/3: API Detail </h4> appeared first on VRTogether.

This project has been funded by the European Commission as part of the H2020 program, under the grant agreement 762111.

The post Towards a generic orchestrator: The VRTogether project experience<br><h4>Part 2/3: technological choices </h4> appeared first on VRTogether.

This project has been funded by the European Commission as part of the H2020 program, under the grant agreement 762111.

The post Towards a generic orchestrator: The VRTogether project experience<br><h4>Part 1/3: overview</h4> appeared first on VRTogether.

The IBC (International Broadcasting Convention) taking place at Amsterdam in September is still a major milestone for any company in the video technological industry and also for individual who wants to take part in the future of media. This show still appears to be the world’s most influential media & technology show. This year (2019), the quantity and high quality of focused exhibitors drove a large number of attendees (over 56 000) which is the largest of all previous years. 

VRTogether is a world class consortium of research entities and companies looking at evaluating the “togetherness” in VR. The results have been so promising that IBC granted the consortium a free booth in 2018 and one of the largest booth of the Future Zone in 2019. 

This article first presents VR Together’s demonstrations and summarizes the most interesting developments and innovations we experienced at IBC that will impact Social VR in the coming months and years.

VRTogether at IBC

The Future Zone brought together the very latest ideas, innovations and concept technologies from international industry and academia, and showcases them in a single specially curated area where innovators meet. Along with booths, This zone offered an open access to the IABM Future Trends Theater with an interesting program. Major trends this year included 8K, 5G, and immersive media (VR/AR/XR). The VR Together booth was ideally located in the center of the Zone and high visibility, generating a lot of attention.

The VRT project focused on the real-time photo-realistic 3D capture and the integration of a live-avatar into a several minutes scenario. The booth attendance was high and qualified and VRT project got lots of feedback from industry leaders, researchers, and tech wanderers who immediately perceived the potential of such a media for the future. Along with an ecological trend, from healthcare to manufacturing, from education to sport, from training to social gaming, all industries plan to use VR.

Beside this booth, other VRT partner presents a paper introducing some other outcome: a web-based social VR framework, that allows to rapidly develop, test and evaluate social VR experiences.

Based on this framework, the paper presents an evaluation of six user experiences in both 360-degree and 3D volumetric VR. The paper is accessible here:

Everyday photo-realistic social VR: Communicate and collaborate with an enhanced co-presence and immersion

Top star VR innovating technologies

Besides the VRT booth, there were many others relevant booths around VR. Some of them caught our attention. 

VR reconstruction for sport replays by canon

The Canon Group develops cutting-edge technologies to create a unique immersive experience placing the spectator in the middle of the action : the Free Viewpoint Video System. This system provides a video experience like never before using Canon’s unrivaled imaging technologies (https://global.canon/en/technology/frontier18.html) 

You can view the same scene from various angles, changing to the perspective of an athlete on the field or any number of alternate viewpoints. Additionally, viewers can control both viewpoint and game time at will. This revolutionary technology dramatically changes how sports will be viewed. 

OMAF4CLOUD: Standards-enabled 360° video creation as a service

Omnidirectional Media Format (OMAF) is a media standard for 360° media content developed by the Moving Picture Experts Group (MPEG). In order to address the needs of advanced media processing and delivery services, MPEG is developing a new standard called Network based Media Processing (NBMP), a standard that aims at increased media processing efficiency, faster and lower cost by leveraging the public, private or hybrid cloud services. MPEG presents a very interesting paper that covers both OMAF and NBMP standards. It also exposes an end-to-end design and proof of concept enabling immersive virtual reality experience to the end users: https://show.ibc.org/__media/Files/Tech%20Papers%202019/G3-201-Yu-You.pdf 

Nokia: Real-time decoding and ar playback of the emerging MPEG video-based point cloud compression standard

Nokia presented the world’s first implementation of the upcoming MPEG standard for video-based point cloud compression (V-PCC) on today’s mobile hardware.  

As ISO/IEC 23090-5 is about to be published as an international standard, this first V-PCC implementation is an important asset to prove its relevancy to the public.

Nokia, that won an award, describes in a paper all their works on this topic : 

https://show.ibc.org/__media/Files/Tech%20Papers%202019/Sebastian-Schwarz.pdf

ImAc : Immersive Accessibility

The goal of Immersive Accessibility (ImAc), funded by the EU as part of the H2020 framework, is to explore how accessibility tools and access services can be integrated into immersive media and in particular 360-degree content. It is not acceptable that accessibility is regarded as an afterthought; rather it should be considered throughout the design, production and delivery process.

What are the next challenges?

Some professionals of the video industry see VR as a decreasing trend. From VRT project point of view, we don’t and we are not alone with this vision. Head mounted displays are cumbersome and this is the main reason why AR and VR crossed into XR. 

One of the common denominator of many VR projects are their ability to handle 3D volumetric data, particularly to manage the compression of dynamic 3D point cloud data. The main philosophy behind V-PCC is to leverage existing video codecs for compressing the geometry and texture information of dynamic point clouds. This is essentially achieved by converting the  point cloud into a set of different video sequences. In particular, three video sequences, one that captures the geometry information, one that captures the texture information of the point cloud data, and another that describes the occupancy in 3D space, are generated and compressed using existing video codecs, such as MPEG-4 AVC, HEVC, AV1, or similar. 

V-PCC has seduced at IBC for several reasons: 

1) The video based approach rely on existing video codecs (HEVC) and can benefit from existing hardware accelerations. MPEG provides a reference software.

2) The standardization part (codec and transport) will be soon finalized, by the end of 2019.

3) The compression ratio is incredible. The lack of material makes bandwidth predictions complex but we are talking of a few Mbps for a realistic human representation.

4) This approach is AR/VR compatible.

The only drawback at this stage is about the capture. Either the capture takes time, or it is expensive, or it is low quality. The main future challenge is actually to be able to capture, in real time, high quality 3D photo realistic volumetric data. We are actively looking for a solution. Don’t hesitate to share your thoughts with us!

This project has been funded by the European Commission as part of the H2020 program, under the grant agreement 762111.

The post IBC 2019 report: VR still leads the innovation appeared first on VRTogether.

Video

In the Video coding area, the activities are quite intense since MPEG is currently developing specifications for 3 standards: MPEG-H, MPEG-I VVC and MPEG-5.

• MPEG-H: In Part 2 – High Efficiency Video Coding, the 4th edition specifies a new profile of HEVC which enables to encode single color plan video with some restriction on bits per sample, and which includes additional Supplemental Enhancement Information (SEI) messages.
• MPEG-I: In Part 3 – Versatile Video Coding, jointly developed with VCEG, MPEG is working on the new video compression standard after HEVC. This new standard that appears to be perceptually 50% better than HEVC, provides a high-level syntax allowing, for example, to access sub-parts of picture which is needed for future applications with scalability, GDR (gradual-decoding refresh) widely used for game streaming. Many other functionalities will be also provided like: coded picture regions, header info, parameters sets, access mechanisms, reference picture signaling, buffer management, capability signaling, sub profile signaling. A Comity Draft is expected in July 2019 and VVC standardization could reach FDIS stage in July 2010 for the core compression engine.
• MPEG-5: this set of technologies are still being heavily discussed, but MPEG has already obtained all technologies necessary to develop standards with the intended functionalities and performance from the Calls for Proposals (CfP).
  • Part 1 – Essential Video Coding (EVC) will specify a video codec with two layers: layer 1 significantly improves over AVC but performs significantly less than HEVC, and layer 2 significantly improves over HEVC but performs significantly less than VVC. A working draft (WD) was submitted during the MPEG Meeting. Some evaluation of the reference software AV1 versus EVC has been started: EVC seems to be a bit faster for the same quality.
  • Part 2 – Low Complexity Video Coding Enhancements (mainly pushed by V-Nova with improvements from Divideon) will specify a data stream structure defined by two component streams: stream 1 is decodable by a hardware decoder, and stream 2 can be decoded in software with sustainable power consumption. Stream 2 provides new features such as compression capability extension to existing codecs, lower encoding and decoding complexity, for on demand and live streaming applications. This new Low Complexity Enhancement Video Coding (LCEVC) standard is aimed at bridging the gaps between two successive generations of codecs by providing a codec-agile extension to existing video codecs that improves coding efficiency and can be readily deployed via software upgrade and with sustainable power consumption. The planned timeline foresees a CD in October 2019 and a FDIS a year after.

Since mid-2017 MPEG has started to work on MPEG-I (Coded Representation of Immersive Media) that targets future immersive applications. The goal of this new standard is to enable various forms of audio-visual immersion, including panoramic video with 2D and 3D audio, with various degrees of true 3D visual perception (leaning toward 6 degrees of freedom). MPEG evaluates responses to the Call for Proposals and starts a new project on Metadata for Immersive Video linked with the Three Degrees of Freedom Plus feature (3DoF+). This aspect might be very relevant to follow for the VRT Project.

Support for 360-degree video, also called omnidirectional video, has been standardized in the MPEG-I Part 2: Omnidirectional Media Format (OMAF; ISO/IEC 23090-2) and Supplemental Enhancement Information (SEI) messages for High Efficiency Video Coding (HEVC; ISO/IEC 23008-2). These standards can be used for delivering immersive visual content. However, rendering flat 360-degree video may generate visual discomfort when objects close to the viewer are rendered. The interactive parallax feature of Three Degrees of Freedom Plus (3DoF+) will provide viewers with visual content that more closely mimics natural vision, but within a limited range of viewer motion. A typical 3DoF+ use case is a user sitting on a chair (or similar position) looking at stereoscopic omnidirectional virtual reality (VR) content on a head mounted display (HMD) with the capability to move her head in any direction.

At its 126th meeting, MPEG received five responses to the Call for Proposals (CfP) on 3DoF+ Visual. Subjective evaluations showed that adding the interactive motion parallax to 360-degree video will be possible. Based on the subjective and objective evaluation, a new project was launched, which will be named Metadata for Immersive Video. A first version of a Working Draft (WD) and corresponding Test Model (TM) were designed to combine technical aspects from multiple responses to the call. The current schedule for the project anticipates Final Draft International Standard (FDIS) of ISO/IEC 23090-7 Immersive Metadata in July 2020.

Beside this MPEG standardization process 3DoF+ Visual, one can mention a collaboration between Facebook and RED Digital Cinema started last year that lead to the first studio-ready camera system for immersive 3DoF+ storytelling based on a end-to-end solution for 3D and 360 video capture with the Manifold camera. Manifold is a single product that redefines immersive cinematography with an all-in-one capture and distribution framework, giving creative professionals complete ownership of their 3D video projects, from conception to curtain call. What this means for audiences is total narrative immersion in anything shot on the new camera system and viewed through 3DoF VR headsets. The schema below shows the process pipeline put in place first studio-ready camera system for immersive 3DoF+ storytelling :

Next MPEG meeting

The next meeting (127th) will be held on July 8-12, 2019 in Gothenburg, Sweden.

Come and follow us in this VR journey with i2CAT, CWI, TNO, CERTH, Artanim, Viaccess-Orca, Entropy Studio and Motion Spell.

This project has been funded by the European Commission as part of the H2020 program, under the grant agreement 762111.

The post MPEG Meeting #126: Virtual Reality has the wind in its sails appeared first on VRTogether.

This article focuses on Point Clouds and MPEG-MORE. The first part of this article is covering all the other technologies.

Point Cloud Compression

MPEG-I targets future immersive applications. Part 5 of this standard specifies Point Cloud Compression (PCC).

A point cloud is defined as a set of points in the 3D space. Each point is identified by its cartesian coordinates (x,y,z), referred to as spatial attributes, as well as other attributes, such as a color, a normal, a reflectance value, etc. There are no restrictions on the attributes associated with each point.

Point clouds allow representing volumetric signals. Because of their simplicity and versatility, they are important for emerging AR and VR applications. Point clouds are usually captured using multiple RGB plus depth sensors. A point cloud can contain millions of points in order to create a photorealistic reconstruction of an object. Compression of point clouds is essential to efficiently store and transmit volumetric data for applications such as tele-immersive video and free-viewpoint sports replays, as well as for innovative medical and robotic applications.

MPEG has a separate activity on point cloud compression: in April 2017 MPEG issued a Call for Proposals (CfP) on PCC technologies, seeking compression proposals in three categories:

1. Static frames
2. Dynamic sequences
3. Dynamically acquired/fused point clouds

Leading technology companies responded to the CfP, and the proposals were assessed in October 2017. In addition to objective metrics, each proposal was also evaluated through subjective tests, performed at GBTech and CWI. The winning projects were selected as “Test Models” for the next step of the standardization activity.

For the compression of dynamic sequences, it was found that compression performance can be significantly improved by leveraging existing video codecs after performing a 3D to 2D conversion using a suitable mapping scheme. This also allows the use of hardware acceleration of existing video codecs, which is supported by many current generation GPUs. Thus, synergies with existing hardware and software infrastructure can allow rapid deployment of new immersive experiences.

Figure 1: An example of a perspective view of a point cloud: the original version on the left, and two views of compressed versions of the same point cloud in the middle and on the right.

After the selection of the Test Models that combine the best performing technologies, the activity focused on the identification and investigation of methods to optimize the Test Models, by performing “Core Experiments”. Examples of Core Experiments include the comparison of different schemes for mapping the texture information from 3D to 2D, the analysis of hybrid codecs that combine 3D geometry compression techniques with traditional block-based video compression strategies, and the use of motion field coding. These Core Experiments are still ongoing.

At the last MPEG meetings, the PCC activity has been particularly crowded, attracting attention from many industrial partners. The main activities of the group focused on cross-checking test models and reviewing the results of the core experiments. In addition, some new datasets created by commercial companies (8i, Owlii, Samsung, Fraunhofer) were presented, and a proposal to merge two of the Test Models was presented; the goal was to take advantage of the 2D compression technics (HEVC and successors). A preliminary contribution also explored the delivery and transmission of point clouds based on this approach.

In the VRTogether project, CWI is providing a solution for lossy compression of dynamic point clouds, based on the open source software for Point Cloud Compression developed at CWI (available at https://github.com/cwi-dis/cwi-pcl-codec). This solution will not be competing in the standardization race, but it serves as an open source tool to benchmark different solutions and experiment research ideas. Currently, it is being integrated into the VRTogether DASH-based point cloud communication pipeline that will allow multiple users to see each other point cloud representation, captured in real time, and rendered in the same virtual environment. Part of CWI research within the VRTogether project will also focus on the design of new objective quality metrics for evaluating point clouds, based on the study of human perception of volumetric signals.

MPEG-MORE

MPEG‘s Media Orchestration standard (also known as MORE: MPEG-B part 13- https://mpeg.chiariglione.org/standards/mpeg-b/media-orchestration) has been finalized by the committee and the final edited version has been submitted to MPEG’s parent body and ISO for one more yes/no ballot followed by publication. This last step is a mere formality.  It is a bit hard to estimate when the specification will be published by ISO since it requires some secretariat work and this can take quite a few months. The work on Reference Content and SW continues. This work intends to make content available with MORE metadata so that (potential) users of the MORE specification can understand how the specification works and are assisted in creating implementations.

In the meantime, Social VR has become more important in MPEG, and it looks like some of the requirements can be fulfilled by the MORE specification. This notably applies to the simpler forms of Social VR, where images of users are composited into a VR360 experience. This requires both temporal synchronization (multiple users should experience the same scene simultaneously) and spatial coordination (the composited images for all users need to have consistent location and size for the experience to be perceived as realistic and compelling). MORE defines the necessary metadata and protocols for this.

About MPEG

MPEG is the Moving Picture Experts Group, a group from IEC and ISO which created some of the foundations of the multimedia industry: the MPEG-2 Transport Stream, and the MP4 file format, a series of successful codecs both in video (MPEG-2 Video, AVC/H264) and audio (MP3, AAC). A new generation (MPEG-H) emerged in 2013 with MPEG 3D Audio, HEVC and MMT, and other activities in MPEG-I like Point Cloud, MPEG Orchestration.

Who we are

Motion Spell is an SME specialized in audio-visual media technologies. Motion Spell was created in 2013 in Paris, France. On a conceptual and technical level, Motion Spell will focus on the development of transmission open tools. Furthermore, Motion Spell plans to explore encoding requirements for VR to participate in the current standardization efforts and first implementations. Finally we will also assist on the playback side to ensure the end-to-end workflow is covered.

Come and follow us in this VR journey with i2CAT, CWI, TNO, CERTH, Artanim, Viaccess-Orca, Entropy Studio.

The post VR strikes back at MPEG appeared first on VRTogether.

MPEG

MPEG is the Motion Picture Expert Group, a group from IEC and ISO that created some of the foundations of the multimedia industry: MPEG-2 TS and the MP4 file format, a series of successful codecs both in video (MPEG-2 Video, AVC/H264) and audio (MP3, AAC). A new generation (MPEG-H) emerged in 2013 with MPEG 3D Audio, HEVC and MMT and other activities like MPEG-I (more below).

The GPAC team and its commercial arm (GPAC Licensing), which is led by Motion Spell, are active contributors at MPEG.

MPEG meetings are organized as a set of thematic meeting rooms that represent different working groups. Each working group follows its way from requirements to a working draft and then to an international standard. Each MPEG meeting gathers around 500 participants from all over the world.

MPEG-I: Coded Representation of Immersive Media

Since mid-2017, MPEG has started to work on MPEG-I. MPEG-I targets future immersive applications. The goal of this new standard is to enable various forms of audio-visual immersion, including panoramic video with 2D and 3D audio, with various degrees of true 3D visual perception (leaning toward 6 degrees of freedom). This full standard has already reached a relevant state that forces us to take it into account in VRTogether.

MPEG-I is a set of standards defining the future of media, which currently comprises eight parts:
• Part 1: Requirements – Technical Report on Immersive Media
• Part 2: OMAF – Omnidirectional Media Format
• Part 3: Versatile Video Coding
• Part 4: Immersive Audio Coding
• Part 5: Point Cloud Compression
• Part 6: Immersive Media Metrics
• Part 7: Immersive Media Metadata
• Part 8: NBMP – Network-Based Media Processing

In this article, we will focus on parts 1, 2, 3 and 8.

Architecture

MPEG-I exposes a set of architectures rather than just one:

Part 1: Requirements

MPEG-I requirements are divided into phases:

• Phase 1a: Captured image based multi-view encoding/decoding (currently finalizing standardization)
• Phase 1b: Video-based multi-view encoding/decoding. Mostly 3DoF and some 3DoF+. VR and interactivity activities are likely to be split.
• Phase 2: Video + additional data (depth, point cloud) based multi-view encoding/decoding. This phase allows to take into account 6DoF with limited freedom (Omnidirectional 6DoF, Windowed 6DoF) and to Synthesize points of view from fixed cameras

Part 2: OMAF

OMAF is a profile explanation on how to use the MPEG tools with omnidirectional media. OMAF kicked-off its activity towards a 2nd edition enabling support for 3DoF+ and social VR with the plan going to Committee Draft (CD) in October 2018.

Additionally, there is a test framework proposed which allows assessing the performance of various CMAF tools. Its main focus is on video, but MPEG’s audio subgroup has a similar framework to enable subjective testing. It could be interesting seeing these two frameworks combined in one way or the other.

OMAF implies new ISOBMFF/MP4 boxes and models. This is very interesting to follow for VRTogether, but it might be too complex to implement at this stage of specification. In addition, OMAF targets the newest technologies while VRTogether wants to deploy with existing ones. There is room for MPEG contributions in this area.

OMAF also specifies some support for timed metadata (like timed text for subtitle) that has to be followed as well. Our current workflow implies many static parameters on the capture or rendering sides that may become dynamic.

Part 3: Versatile Video Coding

This part focuses on immersive video coding which will be a successor of HEVC. The name VVC (Versatile Video Coding) was hand voted at the meeting. Current experiments show that VVC codec can outperform HEVC by 40%. The release of VVC coding is planned for October 2020.

Part 8: Network-Based Media Processing

Network-Based Media Processing (NBMP) is a framework that allows service providers and end-users to describe media processing operations that are to be performed by the network. NBMP describes the composition of network-based media processing services out of a set of network-based media processing functions and makes these NBMP services accessible through Application Programming Interfaces (APIs).

Motion Spell, partner of the VRTogether project, has decided to attend some sessions on NBMP during the 122 MPEG meeting since this new activity that allows building media workflows has generally been ignored so far. One of the main use-case covered by the output of the previous MPEG meeting in Gwangju included the ingest of media for distribution. Unified Streaming, co-chaired, wants to standardize ingest. Indeed, this is very exciting for VRTogether; Motion Spell will be interested in implementing an ingest component. For example, it could be useful to compare our low latency ingest implementation with the status of this standardization effort by the end of the project.

TNO, also a partner of the VRTogether project, proposed a contribution that puts the focus on extending scene description for 3D environment in the scope of NBMP. This contribution also exposed a tentative list of NBMP functions that could be useful for Social VR:

• Background removal.
• User detection and scaling.
• Room composition without users.
• Room composition with users.
• Low-latency 3DOF encoding.
• Network-based media synchronization.
• 3D audio mixing functionality.
• Lip-sync compensation.

Who we are

Motion Spell is an SME specialized in audio-visual media technologies. Motion Spell was created in 2013 in Paris, France. On a conceptual and technical level, Motion Spell will focus on the development of transmission open tools. Furthermore, Motion Spell plans to explore encoding requirements for VR to participate in the current standardization efforts and first implementations. Finally we will also assist on the playback side to ensure the end-to-end workflow is covered.

Come and follow us in this VR journey with i2CAT, CWI, TNO, CERTH, Artanim, Viaccess-Orca, Entropy Studio.

This project has been funded by the European Commission as part of the H2020 program, under the grant agreement 762111.

The post The comeback of VR at MPEG appeared first on VRTogether.

No actual innovations but many improvements

Regarding VR and 360 aspects, we can mention several interesting booths where improvements have been made since last year. The Chinese company Kandao Technology exposed its caption solution “VR Obsidian” with many enhancements from the previous version. Indeed, we discovered an update of the Kandao live software enable to handle 8k 360 in live, and their new 6-Dof volumetric solution based on their deep map generated with their 3D 360 camera. This last solution could be interested to follow for the VRTogether project since it allows to rebuilt 3D from any viewpoint, which pushes the VR experience to a higher level.

Regarding 360 caption, we can also talk about the Fraunhofer Heinrich Hertz Institute (HHI) current innovations in the field of immersive imaging technologies. They presented their enhanced Omnicam-360 that enables 10K video 360-degree capturing. The live encoding is tile-based and use the Fraunhofer HHI HEVC encoder compliant with the MPEG-OMAF Viewport-Dependent Media Profile.

We can also put in light the 8ball product from HEAR 360+. It’s a new immersive audio technology included patented omni-binaural microphone and companion software that provides a seamless workflow from on-set recording to content delivery. The 8ball solves the challenges that cinematic VR filmmakers face when attempting to create truly immersive spatial audio.

The general feeling is that most of the VR innovation was about united experiences, including immersion with audio, low latency, precision, wide field-of-views.

This feeling converges with VRtogether’s vision: VR, AR and Immersive Audio is going to fundamentally change how people communicate and create an intense emotional impact.

Future Park zone

We can also report exciting things into the future park zone. Indeed, two projects on the Futures Park, ImmersiaTV and ConvergenceTV are linked with some partners of the VRTogether project.
Indeed, ImmersiaTV features i2CAT, and ConvergenceTV (which won the NAB Innovation Award 2018) features Motion Spell.

Regarding the ConvergenceTV, there was a nice demonstration of MPEG-DASH ultra-low latency using the Motion Spell / GPAC Licensing Signals platforms. Professional players are stable at around 500ms.

Who we are

Motion Spell is an SME specialized in audio-visual media technologies. Motion Spell was created in 2013 in Paris, France. On a conceptual and technical level, Motion Spell will focus on the development of transmission open tools. Furthermore, Motion Spell plans to explore encoding requirements for VR to participate in the current standardization efforts and first implementations. Finally we will also assist on the playback side to ensure the end-to-end workflow is covered.

Come and follow us in this VR journey with i2CAT, CWI, TNO, CERTH, Artanim, Viaccess-Orca, Entropy Studio.

This project has been funded by the European Commission as part of the H2020 program, under the grant agreement 762111.

The post VR at NAB show: the calm before the storm? appeared first on VRTogether.

Although VR has been a buzz word for the past few years, VR experience delivery is still at its infancy. There are some products starting to use tiles and other techniques to broadcast VR videos but real deployments still stream flat equirectangular videos. There are no standards able to stream and merge 3D models (e.g. “3D videos” or persons) in a unified social VR experience. VRTogether comes to the rescue.

VR Together

VRTogether is a consortium about social virtual reality. Our goal is to provide a social virtual reality experience that mixes up natural and artificial (e.g. a movie and 3D-reconstructed persons). In one of the content samples developed, users are able to reenact themselves an experience which looks like one they could have seen in a thriller.

Market status

The world of VR has acted strange lately. After the sudden boom (from 2014’s Oculus Rift acquisition by Facebook for $2bn to 2017’s reality check from the market), new HMDs have been presented at the CES 2018, solving real issues (increased field-of-view, better displays, less motion-sickness, autonomous). However while problems are getting solved, the news about the solutions receive little coverage.

Social VR has been teased a lot during this VR frenzy. Who doesn’t remember this Facebook/Oculus conference with a room full of people wearing a Rift?

Beyond the awe that provoked this image on many people, there is an undeniable fact: social VR is one of the futures of VR.

Social VR

Social VR is one of the futures of VR.

To get a full experience of social VR, you need your friends to be represented in full 3D with a feeling of interaction with them. There are some technical catches:

• As of now, there are no standards to stream 3D videos. The two main technologies that compete are Meshes and Point Clouds. Both approaches are at different stages of standardization. Without a specification handled by a serious body (MPEG, W3C, etc.), device and browser integrations are absent. These are key points as of 2018 and we are working on it.
• Bandwidth limitation. 3D representation of the users needs to be sent in real time, which requires a lot of bandwidth at the moment to get a smooth experience. We are talking about 50mbps for a realistic representation.
• Latency. We have to ensure that, to be able to interact convincingly with your friends, you’ll be able to see yourself and your friends moving instantly, without any perceptible delay. A self-representation should show almost zero-delay.
• Synchronization: the users should not perceive strange artifacts (e.g. lip sync, freezes, zombies, etc.).

The dust is settling down, the VR mania is down, and now, we can start to get really serious about it.

Who we are

Motion Spell, the company under GPAC Licensing, will support that objective at NAB 2018, and at MPEG 122 in San Diego.

The technical objectives of the consortium are to provide more reliable technological bricks to support packaging and delivery of such experiences. As such, Motion Spell will deliver a first prototype as part of the VRTogether Pilot one, which will be available this semester.

Come and follow us in this VR journey with i2Cat, CWI, TNO, Future Lighthouse, CERTH, Artanim, Viaccess Orca, Entropy Studio.

This project has been funded by the European Commission as part of the H2020 program, under the grant agreement 762111.

VRTogether: website, Twitter.

Text and pictures: Rodolphe Fouquet – Motion Spell

The post Delivering your new VR social experience: our commitment in VRTogether appeared first on VRTogether.