The Volumetric Capturing (VolCap) system is a toolset designed to orchestrate the capturing, streaming and recording of the data acquired from a multi-sensor infrastructure.

A number of processing units each manage and collect data from a single sensor using a headless application called Eye.

A set of sensors is orchestrated by a centralized UI application, VolCap, that is also the delivery point of the connected sensor streams.

VolCap allows multi-stream synchronization and data-driven and global optimized volumetric alignment (calibration), in order for the data and metadata needed for a 3D mesh reconstruction to be finally captured, encoded and streamed. It also offers an extended variety of real-time RGBD data parameterization from the user interface regarding the image resolutions, compression parameters, sensor presets and so on.

The technical offerings of the new version include:

• Efficient, scalable and low-resource multi-stream live sensor data acquisition and recording
• Integration (and mixing) of Kinect 4 Azure and Intel RealSense 2.0 D415 devices
• Combined hardware (device-specific) and software (IEEE 1588 PTP) multi-stream synchronization
• Data-driven and global optimized volumetric alignment

The volumetric capture software has been used in various activities such as Live tele-presence in Augmented VR or Mixed/Augmented Reality settings, Performance Capture, Free Viewpoint Video (FVV), Immersive Applications (i.e. events and/or gaming) and Motion Capture.

In CERTH premises, we had the chance to test the final software release and simultaneously capture in high quality human user representations with 4 Microsoft Kinect4Azure and 4 Intel RealSense D415 ToF technology of the Kinect4Azure (K4A). The 4D reconstruction shows promising results that can increase sense of immersion in any application.

The new release comes with a new, organized documentation page available at https://vcl3d.github.io/VolumetricCapture

Pilots are checkpoints to evaluate the creative and technical challenges of the project, aiming at assessing the performance of the technological developments, validating and refining the defined evaluation methodology and assessing the appropriateness of the technology and created scenarios / content to provide truly realistic and interactive social VR experiences.

CERTH successfully completed 2 full 2-user sessions with TVM v3 (mesh-based volumetric video) representation between 4 users in total. In Pilot 3, the experience attempts to increase the sense of immersion and togetherness. While having a maximum of five users with a live representation and a variety of different representation options (3D avatar, 2D video, single and multi camera point clouds, TVMs), users have also the ability to join a session as spectators or even with no representation at all. The Pilot3 VRTogether scenario resumes and concludes the same storyline, while taking part in a virtual apartment of great detail. It also includes three 3D avatars which interact with the users through voice instructions for specific actions using the Oculus Rift controllers and a simple form of dialog resulting in binary answer questions.

In CERTH premises we set up two capturing nodes with four Kinect4Azure depth sensors each. Following the Pilot protocol and all the hygiene protection rules designed by our consortium partner, CWI, we were able to conduct two 2-user sessions using the Pilot3 VR scenario of VRTogether’s platform. Both users were represented as TVMs and were able to successfully complete the scene’s interactions through the Oculus Rift’s controllers and its microphone.

Unfortunately we were not able to proceed with a bigger number of participants, as a result of the local COVID19 restrictions. Participants filled questionnaires before and after the sessions, as instructed by the used protocol, while all metrics integrated in VRTOGETHER’s platform were captured.

HUMAN4D is a new multimodal human-centric 4D dataset containing a large corpus with more than 50K samples. By capturing 2 female and 2 male professional actors performing various full-body movements and expressions, HUMAN4D provides a diverse set of motions and poses encountered as part of single- and multi-person daily, physical and social activities (jumping, dancing, etc.), along with multi-RGBD (mRGBD), volumetric and audio data.

Despite the existence of multi-view color datasets captured with the use of hardware (HW) synchronization, HUMAN4D is the first and only public resource that provides volumetric depth maps with high synchronization precision due to the use of intra- and inter-sensor HW-SYNC.

VRTogether consortium members CERTH, CWI and Artanim made all the data (http://dx.doi.org/10.21227/xjzb-4y45) and code (https://github.com/tofis/human4d_dataset)  available online, including the respective synchronization, calibration and camera parameters, along with data loaders and other processing, visualization and evaluation tools, for academic use and further research.

The involved consortium members commit to continuously maintain the dataset for the community by adding new tools, baselines and captures. Despite the continuous maintenance of the dataset, benchmarking subsets will remain constant to allow the assessment and comparison between new state-of-the-art methods on the same datasets.

HUMAN4D and its associated tools will stimulate further research in computer vision and data driven approaches, enabling research on human pose estimation, real-time volumetric video reconstruction and compression, with the use of consumer-grade RGBD cameras sensors.