Platforms
VRTogether aims at being experienced by users with different devices and infrastructure.
The consortium has then developed two distinct platforms to provide efficient solutions in a variety of scenarios, depending on the available resources and objectives.
The Web VRTogether platform is lightweight, does not require installations at the client side, makes use of simple capturing systems, and makes use of fully web based technologies.
The Native VRTogether platform requires the installation of a client application, and is able to support different end-user’s representation formats, including volumetric ones (TVMs, and Point Clouds), more advanced interaction features, more advanced capturing systems, media experiences seamlessly integrating heterogeneous media content, and higher scalability.
The figures below provide the hardware and software architectures of these platforms.
Hardware Architecture
Software Architecture
Web platform
The Web Pipeline offers a lightweight Social VR service as a real-time end-to-end communication system. Based on web technologies, such as HTML5 and JavaScript as well as WebRTC and WebGL APIs, combined with video coding and distribution standards like DASH, it provides an efficient VR based conferencing system. The setup is light and affordable, as the capturing component uses a single consumer-grade depth camera for the real-time capture system. It can be easily installed and configured at the end-user level. Combined with other components developed in the project (stream encoding and transport, orchestration …) this lightweight Social VR platform is particularly adapted for the deployment of VR conferencing, training and shared VR experiences.
In detail we can separate the Web Platform into 3 individual web server instances, namely:
- TogetherVR web service: The TogetherVR web service is the main entry point for a client. It offers a web page to be accessible for a client to download all necessary resources and logic to create, display and facilitate the VR experience to the user. This also includes the connection to the orchestrator and Signalmaster.
- TogetherVR web orchestrator: The orchestrator is mainly a message forwarding entity that allows for the client to get all session related information and metadata necessary to execute and maintain a running VR communication experience.
- WebRTC Signalmaster: The Signalmaster is a standard WebRTC component (open source and not developed in the context of VRTogether) but necessary to initiate the WebRTC connections between clients (including NAT traversal and video protocol agreement).
Components
Native platform
The native pipeline integrates all the components required to build the collaborative platform which, in the end, allows users to participate in the VRTogether experience. End-users are captured in real-time and are represented in realistic and even volumetric formats, in a fully operational virtual environment. The native pipeline scenario allows to consume video in different formats (traditional 2D, stereoscopic, 360) which are delivered On-Demand or Live, which for this last one offers the possibility for the users to interact with the actors.
The Native Platform supports a variety of end-users’ representation formats:
- TVM: Real-time 3D Time Varying Meshes (TVMs) integrated inside a graphics environment like Unity.
- PC: Volumetric Point Clouds (PCs) via geometry-based (G-PCC) encoding.
- 3D Avatar
- 2D representations from the webcam.
- No visual representation (with or without audio interaction capabilities)
