What is photogrammetry

The VRTogether project is characterized by the use of state of the art technology to offer high quality content and hyper realistic material. One of these technologies is named photogrammetry, and it has been used successfully for this project to digitally recreate specific environments.

But first of all, what is photogrammetry? Photogrammetry is the technique used to obtain information about objects and the environment by using recording and measuring devices, to later digitally interpret this information. Even though this technique is commonly used in fields such as topographic mapping, architecture and geology, among others, the VRTogether artistic partners found a way to apply it to the objectives of the project. This technique is also commonly used in movies and TV shows to computer-generate content in post-production.

What did we use photogrammetry for

Among others, one of the conclusions of the previously announced on this blog first Joint Business Clinic was the preparation and elaboration of different use cases that would focus on several fields. One of these fields was the preservation and showcasing of art and heritage, offering this service to museums, theatrical companies and all culture industries and institutions.

Using this technology to recreate part of the collection of a museum was just an example of how this technique can be used with the technology developed within the VRTogether project to demonstrate that museums and art exhibition rooms have the possibility to use this type of technology, not only to eventually increase their cash flow and visitors, but to achieve, among others, the following objectives:

• To preserve the museum heritage, helping its conservation and correct visualization and enjoyment.
• To preserve the history of the city in which the museum is located.
• To bring heritage closer to a diverse audience, adapting its characteristics and needs by offering an interactive experience.
• To create a product that allows to get the most out of the facilities and cultural heritage of the museum, making it attractive and unique.
• To offer experiences to groups of people, all of them being able to experiment cultural activities at the same time, allowing a positive flow of word of mouth in and out of the museum.

Even though the final use of this technology is focused on increasing the cash flow and number of visitors to the museum, this technology also allows the generation and display of new types of content, exploring the different ways in which art and culture can be preserved and presented to the audience. The inclusion of different social groups is another factor in which VR has great potential. To resume the path of sustainable competitiveness that gets us out of the current crisis, the productive system must be able to take advantage of all the potential that culture and the arts have by imagining new scenarios, future artifacts, or reinterpreting interfaces. If the artists have something in common, designers and creatives are good at making understandable in a way close to the newest concepts. This is transcendental for society and people, embracing the innovative solutions that global challenges need. VR is committed to eliminating barriers that separate the worlds of the humanities and technologies and therefore increase the competitiveness and sustainability of our innovation system. This is in line with the bet of the European Union itself, which establishes that the culture and our values are necessary to compete on a global scale with other regions from social cohesion, and thus ensure that innovation is sustainable at the economic, social and environmental levels.

Some of the most famous museums in the world are already using some VR techniques to offer their audience a new experience. TheMo’s team created one of the most beautiful VR immersive experiences for Museum Thyssen Bornemisza. The Louvre Museum and The National Museum of Natural History, both in Paris, or The Tate Modern Museum in London are just some examples of important museum refererents that are including VR in their activities. From the VRTogether consortium we wanted to continue within this field, bringing new immersive experiences.

The chosen museum to use this technique was the Víctor Balaguer Museum & Library, located in Vilanova i la Geltrú (Barcelona, Spain). Specifically, the XIX Century Art Gallery from this museum was the one selected to be immortalized with the photogrammetry technique.

First phase of the photogrammetry process

As the Victor Balaguer Museum & Library claims, the XIX Century Art Gallery gathers the big part of the founding painting and sculpture collection in an area that recreates the original environment of the Fine Arts halls of the time with high ceilings, velvety colored walls and plenty of paintings hanging on the walls.

The audience can find different works from the collection made by the Museum’s founder, Víctor Balaguer, and also several art pieces by artists like Ramon Casas, Santiago Rusiñol, Marià Fortuny, Ramon Martí Alsina or Joaquim Vayreda, representing styles such as Romanticism, Realism and Symbolism.

How did we use photogrammetry

The photogrammetry technique was applied in different stages, since the scanning process of this technology consists of several steps.

The first phase is the acquisition of images. In late August, TheMo team travelled to the museum to do this process. The captured pictures needed to have enough overlap in order to let the photogrammetry software locate all the information of the scenario and the cameras in a digital environment. As it can be seen in the following pictures, the mix of colors in the XIX Century Art Gallery was high, so all the pictures were taken in .raw format , so information couldn’t be lost in the digital recreation of the space.

The next step consists of reviewing every single picture taken of the scenario, making sure that no space is missing, so the number of pictures that need to be checked and reviewed is large. Generally, this process is done with specialized software, such as Adobe Photoshop Lightroom. During this stage of the photogrammetry process, every picture (if needed) is retouched in terms of light, hue, saturation and every other aspect that requires an extra retouch.

Normally, this process is focused on lightening those parts of the picture that may be a little bit dark, or even the opposite process, darkening parts that are overexposed. This process has to be very detail oriented, since all the digitally recreated rooms have to look natural, and lighting is one delicate step in order to make that happen. Once this review and retouch process is done, the pictures need to be exported in .jpg format, since photogrammetry softwares don’t use the previously mentioned .raw format.

Once all the pictures have been imported to the photogrammetry software, a first alignment calculation is done. This is essential in order to obtain a scattered point cloud model. The scattered point cloud model helps the technician to evaluate if more pictures are needed, if the ones chosen are the correct ones or if everyone selected is enough to continue with the process. The following gif illustrates this paragraph that is common to every photogrammetry process.

Once this step is done, a second and more precise calculation is needed to obtain a polygon mesh model of every part of the room. This process is heavier and slower to perform, but it helps to bring naturality and hyper-realism to the final product. Since it is a step in which every piece is defined, it offers as well a solid vision of how the final model is going to look like.

Finally, and once the polygonal model is obtained, the textures of the model are calculated in order to edit the final light and color information. The following pictures show the final look of the digital recreation using this photogrammetric technique. It can be appreciated all the details of every single piece of art, obtaining the desired hyper-realistic look in order to be used for the previously mentioned use case.

The Volumetric Capturing (VolCap) system is a toolset designed to orchestrate the capturing, streaming and recording of the data acquired from a multi-sensor infrastructure.

A number of processing units each manage and collect data from a single sensor using a headless application called Eye.

A set of sensors is orchestrated by a centralized UI application, VolCap, that is also the delivery point of the connected sensor streams.

VolCap allows multi-stream synchronization and data-driven and global optimized volumetric alignment (calibration), in order for the data and metadata needed for a 3D mesh reconstruction to be finally captured, encoded and streamed. It also offers an extended variety of real-time RGBD data parameterization from the user interface regarding the image resolutions, compression parameters, sensor presets and so on.

The technical offerings of the new version include:

• Efficient, scalable and low-resource multi-stream live sensor data acquisition and recording
• Integration (and mixing) of Kinect 4 Azure and Intel RealSense 2.0 D415 devices
• Combined hardware (device-specific) and software (IEEE 1588 PTP) multi-stream synchronization
• Data-driven and global optimized volumetric alignment

The volumetric capture software has been used in various activities such as Live tele-presence in Augmented VR or Mixed/Augmented Reality settings, Performance Capture, Free Viewpoint Video (FVV), Immersive Applications (i.e. events and/or gaming) and Motion Capture.

In CERTH premises, we had the chance to test the final software release and simultaneously capture in high quality human user representations with 4 Microsoft Kinect4Azure and 4 Intel RealSense D415 ToF technology of the Kinect4Azure (K4A). The 4D reconstruction shows promising results that can increase sense of immersion in any application.

The new release comes with a new, organized documentation page available at https://vcl3d.github.io/VolumetricCapture

Pilots are checkpoints to evaluate the creative and technical challenges of the project, aiming at assessing the performance of the technological developments, validating and refining the defined evaluation methodology and assessing the appropriateness of the technology and created scenarios / content to provide truly realistic and interactive social VR experiences.

CERTH successfully completed 2 full 2-user sessions with TVM v3 (mesh-based volumetric video) representation between 4 users in total. In Pilot 3, the experience attempts to increase the sense of immersion and togetherness. While having a maximum of five users with a live representation and a variety of different representation options (3D avatar, 2D video, single and multi camera point clouds, TVMs), users have also the ability to join a session as spectators or even with no representation at all. The Pilot3 VRTogether scenario resumes and concludes the same storyline, while taking part in a virtual apartment of great detail. It also includes three 3D avatars which interact with the users through voice instructions for specific actions using the Oculus Rift controllers and a simple form of dialog resulting in binary answer questions.

In CERTH premises we set up two capturing nodes with four Kinect4Azure depth sensors each. Following the Pilot protocol and all the hygiene protection rules designed by our consortium partner, CWI, we were able to conduct two 2-user sessions using the Pilot3 VR scenario of VRTogether’s platform. Both users were represented as TVMs and were able to successfully complete the scene’s interactions through the Oculus Rift’s controllers and its microphone.

Unfortunately we were not able to proceed with a bigger number of participants, as a result of the local COVID19 restrictions. Participants filled questionnaires before and after the sessions, as instructed by the used protocol, while all metrics integrated in VRTOGETHER’s platform were captured.

General testing considerations

In every software development it is important to include testing rules in the process. These rules ensure quality and a continuous evolution over new integrated features. For a developer, testing regularly allows more confidence during new implementations – the more developers test their code, the more they will guarantee the quality of it. Of course, this will take them a little while, if they just do it manually.

There are different types of tests depending on the stages of the project.

• Unit tests, which induce a step-by-step verification of the developer’s work by themselves. The idea is to verify that each unit (which can represent a function or a class depending on the projects conventions)work as intended. These tests are more numerous and have the narrowest scope of all the tests.
• Integration tests. Most applications integrate parts outside of the basic production environment (databases, file systems, network calls to other applications) that are common to dissociate during unit tests. The interactions and links that these parts have with each other are tested during integration testing.
• Functional tests, which can be divided into three types:
  • The Human-Machine Interface (HMI) tests: the purpose of the HMI tests is to verify that the corporate visual identity has been respected throughout development in two aspects:
    • The visual presentation: menus, display parameters, window properties, icon bars, screen resolution, edge effects, etc.
    • Navigation: means of navigation, shortcuts, the result of moving around on a screen, etc.
  • Application tests, which verify security and for example compatibility between different browsers
  • Load tests, which scrutinize the project’s performance and robustness.

New implementations in a project are often the vector of instabilities or anomalies, with sometimes unforeseeable consequences. That is the reason why it makes sense to perform non-regression testing, as a type of safety net. Non-regression testing can take a lot of time and effort for teams. In addition, the non-regression test phase is at the end of the chain of all projects carried out in classic or agile mode. This phase is regularly subjected to strong constraints of schedules and budget. This is why it is interesting to automate those tests.

Once operational, tests automation has many advantages:

• Time savings, due to a reduction in the workload of the teams.
• Increase in productivity.
• Reliability and quality of the tests carried out.
• Increase of the level of completeness of the tests carried out.
• Free teams from repetitive tasks in favour of tasks with greater added value (Analysis of results, definition of test cases, planning, etc.).

Automatic testing in the scope of the VRT project

As part of Work Package 3 (WP3), which focuses on the delivery of specific software and hardware components for the project, the consortium organizes regular testing sessions. Latest development and functional improvements are analysed during meetings called Test fests. Lately, these meetings highlighted the need to automate some integration tests that verify links the different components with a main component which manage all of them: the orchestrator.

These tests are time-consuming if performed by humans and seem very simple to validate via an automatic process. Moreover a solution which can emulate the user’s actions could be used later for HMI tests.

In order to evaluate efficiently testing tools, it is important to follow a precise path:

• Know when and how it is possible to automate these tests
• Translate the initial need into a test scenario: without interpretation, without forgetting any important features
• Execute the tests with rigor and follow the correction of anomalies
• Organize the tests according to the means of the company, the size and the complexity of the software First, a functional, UI (focused on the user interface) non regression test will be created.

How to choose the best suitable tools for automatic testing in Unity?

Of course, many test systems exist. The idea is to find the most suitable solution adapted to VRTogether requirements.

Test systems will be evaluated according to the following criteria:

• Windows platform
• Accordance to the concerned testing type.
• Input fields detection on unity
• Price: This research will focus on open source projects.
• Technical Support and Assistance: Automation engineers sometimes need help on critical problems.

The research for the best system induced naturally the experimentations, and so it makes sense to first create an interface that presents the milestones of the VRTogeter’s usage scenario. It’s a way to identify clearly the testing software’s specificities without confusing on the VRT’s. For the sake of efficiency, a usage scenario reduced to its strict minimum should be created in accordance to a simplified interface made on unity.

The intention is to simulate the behavior of a user who interacts with an interface comprising of a login password section and a registration section with writing areas for the name, email, password and confirmation of the password. During a first test the users will identify themselves with a previously saved profile (it takes the form of a text file at the root of the project), and during a second test the users will create their own account in the register section.

The simplified scenario:

• Check that the program is started
• Check the existence of the login button
• Fill in the login text field with the username
• Fill in the password text field with the password
• Click on the login button
• Verify the interface change

It seemed coherent to us to initially focus our search for an automated test solution working with unity on open source solutions. If the outcome of this research had not been conclusive, our research would have turned towards profitable solutions.

This section will provide an overview of the different test solutions that have been explored. Then a more precise explanation of how AltUnity works and why it was chosen as the most adapted solution follows.

Unity Test Runner

docs.unity3d.com/2017.4/Documentation/Manual/testing-editortestsrunner

Unity Test Runner (UTR) is a tool, now directly integrated into Unity, the tests are written in C # with the nunit library. The checks are done thanks to the assert class; it allows to know if the tests are positive or not.

Of course, there is no limit to the number of possible checks.

The tests can be carried out in “edit mode” from the Unity editor which allows code checks without launching the player and in “play mode” which allows you to see the effects of your test in the unity scene. The tests in play mode add more assemblies during the construction of the project, and therefore increase the size and the build time. Unity test runner is primarily useful for unit testing and possibly for integration testing. Thus, its operation in play mode involves a sort of division of the scene to isolate the fragments of code which are tested. The unity editor can also have different performance depending on the computer used (CPU, GPU, OS …). Automated functional tests must be applied to environments as close as possible to production environments. And this is why the Unity Test Runner is not the most suitable tool.

Selenium

https://www.selenium.dev/

Selenium is “a set of web browser automation tools that uses best available techniques to remotely control browser instances and emulate a user’s interaction with the browser” (Selenium, n.d.). The notoriety of the tool and the big community around it are sufficient reasons to explore Selenium.

Selenium works on a web browser, here the experiment is done with the chrome webdriver. Writing a few lines in Java is made easier by selenium IDE. Automating a functional test on a website therefore does not present any difficulties. Unfortunately, selenium does not have a driver allowing to make the link with the Unity editor or the build of the VRT application. This constraint makes Selenium unsuitable. The Selenium solution finds its equivalent for unity, this one is called Appium.

Appium

http://appium.io/

Appium was developed by Dan Cuellar in August 2012, it is actually written in node.js. Appium works with IOS, Android and theoretically Desktop applications. Starting a game with Appium does not pose any difficulties, but identifying the elements of the scene and creating interactions presents a sticking point. These concerns can be resolved with an image recognition system using the open CV library as raised by Ru Cindrela – engineer at Altom- during her conference on Appium given in 2018. VRT’s requirements lead to find a standalone solution that can identify unity objects separately from each other. Appium is therefore ruled out.

Unium

https://github.com/gwaredd/unium

Unium gives a flexible interface that allows to remotely control and inspect a game while it is running. While the API Unium offers some very interesting possibilities to inspect game behavior, object coordinates or latency, it is not a stand-alone solution. Its compatibility with Appium circumvents one of the major problems of Appium, the object detection. The association between Unium and Appium is of great interest in the testing of mobile applications.

Unium is available for free on GitHub. This is a library for Unity that allows test verification. Even if it is Appium compatible, it’s important to clarify that this is not an automation tool. It is presented as an HTTP API which can be used by the testing tools. It comes in two parts:

– A web server directly integrated into the game

– A query system that allows you to search and manipulate the elements of the project.

Autoit

https://www.autoitscript.com

Autoit is a high-level programming language allowing to automate the user’s behavior who evolve under a Microsoft Windows OS, it gives a wide choice of function libraries. There is an editor and an Autoit IDE. If in a Windows environment, the tool has no significant limitations, the creation of functional tests for unity presents blocking points.

In the usage scenario Autoit is not able to identify the existence of the login button, but if it has received the coordinates of this one as well as its colour code, it will be able to check if the pixels of the button have the expected colour at the expected moment.

This solution which considers the coordinates of the pixels, their colour and the location of the mouse cursor makes possible to automate the behavior of a user in an environment that Autoit doesn’t know while still maintaining the execution of the scenario. To get some feedbacks, when launching the VRT project, Autoit generates in a separate document a txt file in which the verifications are notified.

Indeed, if the aspect of the project takes a different form or if it’s responsive, the Autoit script must be modified.

From this observation, the research turned to a solution capable of identifying the elements of a Unity project.

AltUnityTester

https://altom.gitlab.io/altunity/altunitytester/

AltUnity tests the ability to identify the objects present in a Unity scene and to create interactions with them. This solution is an alternative to Appium presented by Ru Cindrea at the SeleniumConf in Chicago. (SeleniumConf, 2018) AltUnity Tester is an open source test automation tool developed by the company Altom, it takes the form of a user interface that should precisely help detect objects in a game and interact with them using written tests in C #, Python or Java. These tests can be run on real devices (mobile, PC, etc.) or in the Unity editor. This unity plugin seems to meet expectations because it offers a possibility to check the functioning of the tests in the editor and even to make a build. This particularity seems to us suitable for an evolution for performance tests. In addition, this tool is compatible with Appium (It could be just as effective for phone applications in other innovation projects).

It works through WebSocket server and a client. A connection can be established via an API and an interface in the unity editor.

The third year of our project is rapidly approaching its end. Time flies when you’re having fun and all that, and we most definitely have not been sitting still. Besides overall platform improvements and other activities, we have been hard at work on our third year pilot, i.e. the projects we build on top of our platform to prove the technology. As we have reported before, the core of this pilot is all about interaction. This means interaction between users, with the virtual environment, with virtual characters and with an overarching storyline. 

In terms of content production this has been the most extensive and challenging of our pilots. We have detailed the virtual character creation before, and our amazing partners of The Modern Cultural Productions (TheMo) have created an exciting script as well as a gorgeous virtual apartment where the mystery surrounding Elena Armova’s murder gets resolved. 

Motion Capture in a Global Pandemic

The pandemic we have all been facing this year caused us to schedule, reschedule, adapt and ultimately change our approach to the motion capture we planned to do of our actors. With the actors coming from Spain, and Artanim being located in Switzerland, we faced the fact that we would either incur unknown delays or we would have to get creative. Under normal conditions actors would come to our studio and get equipped with a mocap suit and markers for body mocap. At the same time a dedicated helmet with a holder for an iPhone and Reallusion’s LIVE FACE app is used for the facial mocap. This all combines nicely into the Character Creator ecosystem we use for our characters. 

With travel restrictions and quarantine periods in place, we decided to split our mocap approach. Artanim prepared and supplied the necessary hardware and software for facial mocap and shipped it off to Spain. There the TheMo team and the original actors recorded the facial mocap, the actor audio as well as video reference footage. In the meantime at Artanim we arranged for local Swiss actors to come to our studio, where we captured their body movements as they played out the scenes in sync with the recorded reference footage.

While this did give us the raw data at a more reasonable timeframe, it also implied far more post-processing would be necessary. The various animations were mostly in sync, but required cleanup and fine-tuning. And of course not having the original actors recorded for their body movement meant that retargeting the animation, while a regular part of the process, was more involved as well. And to make our wonderful animator’s life even more challenging we decided to include more detailed finger animation and prop interaction to our characters for this third pilot. But all the effort was worth it, because the combined content in pilot 3 is a significant step up in terms of quality from the first pilot the consortium produced. 

Supporting Multiplayer Interaction

With all content created the next step is putting it all together into an experience. At an early stage of development we realised that in contrast to the first two pilots, pilot 3 would need a lot more synchronization of state and events between clients. Our storyline is non linear, users interact with the environment, props and characters, and this requires careful orchestration to ensure each participant observes the same experience as the others in a session. 

Unfortunately the networking solutions available for Unity – our engine of choice – all had various downsides. Built-in networking was in a deprecated state with no clear viable replacement. And third-party offerings often ran in the cloud, which would imply less control on our side and not being able to demonstrate or use our platform in local networks one might encounter at fairs and festivals. 

Looking for a solution it dawned upon us that our partners at Motion Spell and Viaccess-Orca were already developing their Orchestrator, which they have detailed in a series of 3 blog posts. It supported chat-like communication between users in a session. And with the addition of an event API as detailed in part 3, we realised we had all the building blocks we needed “in-house” to be able to support our experience. So we set out to develop a lightweight layer on top of the API to support a multiplayer experience in which one of the users acts as the authoritative server in charge of the session. Developers are able to freely create event message types to send data via the authoritative user to all others, and within custom components they can easily subscribe to messages of any type to handle them as they see fit. 

Using this lightweight layer we were able to support all the functionality we needed, from simple event flow triggers to the synchronization of player locations, controller positions and interactions, and more complex experience object state. This system has proven useful beyond this pilot as well, by now being used to support various other functionalities throughout our platform. 

Turning Ingredients into an Experience

Unity provides 2 core mechanisms to create a narrative experience; Timelines and AnimationControllers. As the name indicates, a Timeline allows you to organize your assets over time. In our case this would include animations, character dialog, foley sounds, event triggers and a variety of other elements with a clear place in time. However, our experience is not fully linear given that the progress depends on user actions or their inaction. This implies that character animation can undergo a variety of transitions. And this aspect is better expressed in an AnimationController, which is a directed graph of animation clips in which transitions are expressed by directional edges and imply a smooth blending between the two animation clips. But those controllers are mainly for animation content only.

So we ended up with a mix of the two concepts. Our timelines contain independent sequences in which custom clip types trigger transitions within associated AnimationControllers. And the progression through the various timelines is triggered by a main experience control component. 

In total Pilot 3 as of the last count has 18 animation controllers, 62 animation clips, 22 timelines and 31 synchronized triggers to drive a multiplayer experience with 4 characters in 3 rooms of the apartment over 4 scenes for a duration of about 10 minutes. 

In terms of interaction we have integrated a number of different types. First of all grasp and touch. Driven by controller input – for example the Touch controllers of your Oculus headset – users have the ability to grab and touch objects. Grabbable objects can be held in your hand, while touchable objects such as a button can respond to a finger press. In our experience some elements of progression depend on such input, where the experience won’t progress if users don’t trigger the right events as they were instructed to do by the characters. And to make sure users are aware of what is expected from them, the characters will make sure to remind them with instructions of increasing urgency. 

Further, at various stages throughout the experience, the characters will address the users with a yes or no question. By integrating keyword speech recognition in the experience, users can simply answer the appropriate “yes” or “no” out loud, which when recognized determines the response of the character and the animation sequence which follows. 

The former two examples are very explicit interactions, a user is instructed or asked to take an action, and when they complete it this has an effect. We have however also integrated more subtle interactions such as gaze redirection. During moments of the experience where the script indicates our characters address the users, a gaze redirection module is activated. This module overrides the pre-recorded mocap data and ensures that the characters look directly at the most relevant user. This can be a user in close proximity to a specified target, or nearest to our character. The retargeting is performed within natural articulation limits and is subtle enough to go unnoticed by the user, but it adds a natural mode of engagement and a sense of being there. 

Experiencing Pilot 3

With the Pilot 3 experience in place, from a production point of view we can say that the VRTogether platform has held up well and proved to be capable and flexible enough to support the more stringent requirements of an online multiplayer interactive VR experience. 

From here on out several partners within the consortium plan to use pilot 3 to run a variety of user studies. And in the meantime we keep our fingers crossed for the global pandemic to clear up to a point where we can proudly show off what we have produced, and have you experience the final chapter in the mystery surrounding Elena Armova’s untimely departure. 

VRTogether was present at VRDays Europe 2020, showcasing the world’s first live 3D video conference, using point clouds, over a commercial 5G network. VRTogether partners CWI and Sound presented a live demo to demonstrate the potential of the platform for emergency healthcare services. The demo showcased volumetric video conferencing (based on point clouds) between a doctor, in a medical examination room, and an (acting) injured patient outside in the street.

HUMAN4D is a new multimodal human-centric 4D dataset containing a large corpus with more than 50K samples. By capturing 2 female and 2 male professional actors performing various full-body movements and expressions, HUMAN4D provides a diverse set of motions and poses encountered as part of single- and multi-person daily, physical and social activities (jumping, dancing, etc.), along with multi-RGBD (mRGBD), volumetric and audio data.

Despite the existence of multi-view color datasets captured with the use of hardware (HW) synchronization, HUMAN4D is the first and only public resource that provides volumetric depth maps with high synchronization precision due to the use of intra- and inter-sensor HW-SYNC.

VRTogether consortium members CERTH, CWI and Artanim made all the data (http://dx.doi.org/10.21227/xjzb-4y45) and code (https://github.com/tofis/human4d_dataset)  available online, including the respective synchronization, calibration and camera parameters, along with data loaders and other processing, visualization and evaluation tools, for academic use and further research.

The involved consortium members commit to continuously maintain the dataset for the community by adding new tools, baselines and captures. Despite the continuous maintenance of the dataset, benchmarking subsets will remain constant to allow the assessment and comparison between new state-of-the-art methods on the same datasets.

HUMAN4D and its associated tools will stimulate further research in computer vision and data driven approaches, enabling research on human pose estimation, real-time volumetric video reconstruction and compression, with the use of consumer-grade RGBD cameras sensors.

VRTogether will actively participate in several activities taking place during the 3-day event:

Volumetric Video Conference over 5G: a medical emergency example – Live demo

On Wednesday, 4th November, VRTogether partners CWI and Sound will present a live demo at Amsterdam’s De Kromhouthal to demonstrate the potential of VRTogether for emergency healthcare services. The demo will showcase volumetric video conferencing (based on point clouds) over the standard 5G network of KPN.

4th November
10:55 – 11:10 h CET
De Kromhouthal / Live streaming