Description: Volumetric Capture (VolCap) is a multi – RGB-D sensor 3D capturing, streaming and recording application. The toolset is designed as a distributed system where a number of processing units each manage and collect data from a single sensor using a headless application. A set of sensors is orchestrated by a centralized UI application that is also the delivery point of the connected sensor streams. Volumetric Reconstruction (VolReco) uses the data provided by the former application, in order to create locally, at first, a mesh and, eventually, transmit it.

Licenses: 

• Capturing:
  • Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt
  • Kinect SDK v2 License: https://download.microsoft.com/download/0/D/C/0DC5308E-36A7-4DCD-B299-B01CDFC8E345/Kinect-SDK2.0-EULA_en-US.pdf
• Reconstruction:
  • CUDA Eula license: http://docs.nvidia.com/cuda/eula/index.html 
  • Boost license: http://www.boost.org/LICENSE_1_0.txt
  • MIT License (rendering): https://opensource.org/licenses/MIT
  • OpenCV BSD license: https://opencv.org/license.html
  • Flann BSD license: https://github.com/mariusmuja/flann/blob/master/COPYING

Links: 

• Capturing: https://github.com/VCL3D/VolumetricCapture
• Reconstruction: Not available

Description

This component offers live capture and reconstruction of 3D point clouds using IntelRealSense D400 devices. The component can run with zero or more sensors. If no sensors are found a synthetic point cloud is generated, if multiple sensors are found the point clouds from each sensor are transformed and merged together 

Links:

Github

Licenses:

To be Announced

Description:

The encoding component is responsible for the compression and decompression of the transmitted data. Over the years, CERTH has experimented with a variety of techniques and algorithms for this purpose (OpenCTM, Draco, Corto), but eventually ended up using Draco for performance reasons. Regarding the transmission, CERTH is using RabbitMQ, an open source message broker software. It accepts messages from producers, and delivers them to consumers. It acts like a middleman which can be used to reduce loads and delivery times taken by web application servers.

Licenses: 

• Encoding:
  • Libjpeg API library and associated programs: IJG (Independent JPEG Group) License (https://spdx.org/licenses/IJG.html)
  • TurboJPEG API library and  associated programs: Modified (3-clause) BSD License (https://opensource.org/licenses/BSD-3-Clause)
  • Libjpeg-turbo SIMD extensions: zlib License (https://opensource.org/licenses/Zlib)
  • Draco license: Apache License, Version 2.0 (https://www.apache.org/licenses/LICENSE-2.0.txt)
  • Corto license: GNU LESSER GENERAL PUBLIC LICENSE (http://www.gnu.org/licenses/lgpl-3.0.en.html)

• Transmission:
  • RabbitMQ License: https://www.rabbitmq.com/mpl.html 
  • Boost license: http://www.boost.org/LICENSE_1_0.txt
  • Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt

Description:

The VRT orchestrator, as a centralised server component, is primarily in charge of the management of a set of end-user profiles which are connected into a platform session. The orchestrator is able to control the various connection events in order to grant a unified experience and ensure that whole media pipelines are properly switched across the various connected nodes. It is also the reference point used for the synchronisation of the media pipelines. Then, from the backend viewpoint, the orchestrator is in charge of managing and supervising the transmission and the delivery of all of the media stream pipelines.

Licenses: 

• Core: MIT
• UMTS: To be announced
• LRTS: To be announced

Description

Software component that connects GStreamer with Unity. It is able to play any media URI provided by GStreamer (1.X) pipelines into Unity 3D textures

Licenses:

Open-Source

Links:

Github

Description

bin2dash is an open API allowing to package any volumetric data in the industry-standard MP4and then stream it using MPEG-DASH.

License:

Proprietary

Links:

Source Code Location: https://baltig.viaccess-orca.com:8443/VRT/nativeclient-group/EncodingEncapsulation

Description

Evanescent is a server component that allows to receive any kind of DASH stream and simply forward them to other receiver clients. In this sense, Evanescent is an SFU (Stream Forward Unit). More technically, it receives low latency dash chunks and serves them back to a low latency dash receiver using http chunked transfers. Evanescent is implemented in C++ based on Signals sub-modules.

License:

Proprietary

Links:

Source Code Location: https://git.gpac-licensing.com/alaiwans/Evanescent

Documentation: LD_LIBRARY_PATH=$DIR $DIR/evanescent.exe [–tls] [–port port_number]

Installation guide: just copy the files, see details at https://baltig.viaccess-orca.com:8443/VRT/deliverymcu-group/DeliveryMCU/blob/master/README.md.

Description

This component is the receiving end of the pipeline. It allows to unpackage any standard MPEG-DASH streams into raw streams to be used by the receiver client. It also contains a preliminary algorithm with respect to tiling support. It has been extended to read files and RTMP streams for debugging and interoperability purposes.

License:

This component is written in C++. It has dependencies on Motion Spell’s Signals (C++, IP-owned, proprietary license), GPAC (C, LGPLv2+ license), FFmpeg (C, LGPLv2+ license), cURL (C, MIT/X license) 

Links:

Installation Guide: https://baltig.viaccess-orca.com:8443/VRT/nativeclient-group/SUB/releases. Copy the files in a folder (the SUB is a dynamic library).

Input, output, Configuration: The input is a MPEG-DASH stream URL. The configuration is done automatically from the content of the stream. The output is a timed sequence of binary buffers with metadata, to be given to a decoder/renderer.

Description

The component receives a live rtmp stream from a 3d stereoscopic camera using nginx and nginx-rtmp (both bsd-2 license), crops the left and right eye to fit the presenter and transcodes it into a final stream (using FFMpeg which is GPL) which can be fetched by the player using rtmp. The whole pipeline is low latency and runs in a docker. This component also handles dash format as input and output.

License:

This component uses Nginx for the reception and publishing, and FFMpeg for the transcoding.

Description

(Virtualized) Holoconferencing cloud-based component that aims at reducing the end-user client computational resources and bandwidth usage, providing the following key features: fusion of different volumetric videos, Level of Detail (LoD) adjustment and Field of View (FoV) aware delivery.

License:

License: To Be announced

Links:

• Demo video
• NOSSDAV 2020 paper

Description:

Unity-based player capable in charge of the reception, integration and presentation of all available streams and VR content for the envisioned VR-Together scenarios. It also supports various types of interaction and synchronization features.

Licenses: 

• LGPL
• EULA
• Oculus SDK

Links: 

• Demo Video

Description:

The main aim of the RGBD Capture module is to provide a lightweight, simple to setup sensing solution that is easy to deploy (e.g. in peoples homes). Thus the capture module is responsible for a photo-realistic capture of the user, based on a single RGBD sensor (i.e. Kinect / RealSense). The module has an adaptable interface to connect to different RGBD drivers in order to get the colour image and depth image from the hardware sensor. As of now this may include foreground+background removal and replacement of the HMD with a pre-captured representation of the human face (HMD removal). The image is further processed and finally converted to a 2D RGB + grayscale depth image. The final image is displayed on the screen for capture in the WEB browser. The camera calibration is sent to the browser for geometrically correct 3D rendering using WebGL shaders. The capture is rendered as:

• 3D self-view for self-presence and 
• view for the other participants for shared presence 

all in real-time. All modules are optimized for capturing and rendering users in real-time in a 3D virtual environment.

Licenses: 

• To be announced

Links: 

• Capture and Transmit RGBD Data for 3D rendering (paper)
• HMD removal (video)

Description:

The WebRTC VR MCU system is capable of ingesting any number of WebRTC client inputs. The MCU system synchronises and composes its inputs into a single output stream which is then published using WebRTC. As a result, clients are able to retrieve all relevant streams together instead of separately. This optimizes the network bandwidth due to more efficient routing (each client only sends its video stream to the MCU, and no longer to all other clients), as well as the decoding resources (clients have a limited amount of hardware decoders, which can now be dedicated to decoding all streams instead of just one) of said clients.

Incoming WebRTC streams are demuxed into RTP streams and are fed into a pluggable media pipeline. The media pipeline performs the compositing operation and outputs a single RTP stream which is broadcast to each client using WebRTC return streams. The media pipeline of the MCU system has been designed as a containerized service such that given enough hardware, it is horizontally scalable over multiple parallel sessions. These services are managed using Docker Swarm and the orchestrator component. 

Licenses: 

• To be announced

Links: 

• MMSys Demo Paper

Description:

The entry point for the TogetherVR Web client is offered by the web server back-end, which is based on Node.js, React and Aframe. This allows any modern WebVR enabled browser to display the VR content on a screen or an OpenVR (https://github.com/ValveSoftware/openvr) enabled VR-HMD (a.o. the Oculus Rift CV1 or the Valve Index). Furthermore the client can access the image produced by the TNO RGBD Capture module to be displayed as self-view to the user itself or to be sent via WebRTC to one or multiple other clients. In order to support such a multiuser connection, the client is connected to a second WEB server that is handling all sorts of multimedia orchestration. The rendering of users in the VR environment is done via custom WebGL shaders that alpha-blend user representations into the surroundings for a natural visual representation. Audio is also captured, transmitted and made audible as spatial audio with the help of the Google Resonance APIs.

Licenses: 

• To be announced

Links: 

• Explanation Video
• Overview Paper
• Look & Feel of 1st 360-degree prototype (video)