What is photogrammetry

The VRTogether project is characterized by the use of state of the art technology to offer high quality content and hyper realistic material. One of these technologies is named photogrammetry, and it has been used successfully for this project to digitally recreate specific environments.

But first of all, what is photogrammetry? Photogrammetry is the technique used to obtain information about objects and the environment by using recording and measuring devices, to later digitally interpret this information. Even though this technique is commonly used in fields such as topographic mapping, architecture and geology, among others, the VRTogether artistic partners found a way to apply it to the objectives of the project. This technique is also commonly used in movies and TV shows to computer-generate content in post-production.

What did we use photogrammetry for

Among others, one of the conclusions of the previously announced on this blog first Joint Business Clinic was the preparation and elaboration of different use cases that would focus on several fields. One of these fields was the preservation and showcasing of art and heritage, offering this service to museums, theatrical companies and all culture industries and institutions.

Using this technology to recreate part of the collection of a museum was just an example of how this technique can be used with the technology developed within the VRTogether project to demonstrate that museums and art exhibition rooms have the possibility to use this type of technology, not only to eventually increase their cash flow and visitors, but to achieve, among others, the following objectives:

• To preserve the museum heritage, helping its conservation and correct visualization and enjoyment.
• To preserve the history of the city in which the museum is located.
• To bring heritage closer to a diverse audience, adapting its characteristics and needs by offering an interactive experience.
• To create a product that allows to get the most out of the facilities and cultural heritage of the museum, making it attractive and unique.
• To offer experiences to groups of people, all of them being able to experiment cultural activities at the same time, allowing a positive flow of word of mouth in and out of the museum.

Even though the final use of this technology is focused on increasing the cash flow and number of visitors to the museum, this technology also allows the generation and display of new types of content, exploring the different ways in which art and culture can be preserved and presented to the audience. The inclusion of different social groups is another factor in which VR has great potential. To resume the path of sustainable competitiveness that gets us out of the current crisis, the productive system must be able to take advantage of all the potential that culture and the arts have by imagining new scenarios, future artifacts, or reinterpreting interfaces. If the artists have something in common, designers and creatives are good at making understandable in a way close to the newest concepts. This is transcendental for society and people, embracing the innovative solutions that global challenges need. VR is committed to eliminating barriers that separate the worlds of the humanities and technologies and therefore increase the competitiveness and sustainability of our innovation system. This is in line with the bet of the European Union itself, which establishes that the culture and our values are necessary to compete on a global scale with other regions from social cohesion, and thus ensure that innovation is sustainable at the economic, social and environmental levels.

Some of the most famous museums in the world are already using some VR techniques to offer their audience a new experience. TheMo’s team created one of the most beautiful VR immersive experiences for Museum Thyssen Bornemisza. The Louvre Museum and The National Museum of Natural History, both in Paris, or The Tate Modern Museum in London are just some examples of important museum refererents that are including VR in their activities. From the VRTogether consortium we wanted to continue within this field, bringing new immersive experiences.

The chosen museum to use this technique was the Víctor Balaguer Museum & Library, located in Vilanova i la Geltrú (Barcelona, Spain). Specifically, the XIX Century Art Gallery from this museum was the one selected to be immortalized with the photogrammetry technique.

First phase of the photogrammetry process

As the Victor Balaguer Museum & Library claims, the XIX Century Art Gallery gathers the big part of the founding painting and sculpture collection in an area that recreates the original environment of the Fine Arts halls of the time with high ceilings, velvety colored walls and plenty of paintings hanging on the walls.

The audience can find different works from the collection made by the Museum’s founder, Víctor Balaguer, and also several art pieces by artists like Ramon Casas, Santiago Rusiñol, Marià Fortuny, Ramon Martí Alsina or Joaquim Vayreda, representing styles such as Romanticism, Realism and Symbolism.

How did we use photogrammetry

The photogrammetry technique was applied in different stages, since the scanning process of this technology consists of several steps.

The first phase is the acquisition of images. In late August, TheMo team travelled to the museum to do this process. The captured pictures needed to have enough overlap in order to let the photogrammetry software locate all the information of the scenario and the cameras in a digital environment. As it can be seen in the following pictures, the mix of colors in the XIX Century Art Gallery was high, so all the pictures were taken in .raw format , so information couldn’t be lost in the digital recreation of the space.

The next step consists of reviewing every single picture taken of the scenario, making sure that no space is missing, so the number of pictures that need to be checked and reviewed is large. Generally, this process is done with specialized software, such as Adobe Photoshop Lightroom. During this stage of the photogrammetry process, every picture (if needed) is retouched in terms of light, hue, saturation and every other aspect that requires an extra retouch.

Normally, this process is focused on lightening those parts of the picture that may be a little bit dark, or even the opposite process, darkening parts that are overexposed. This process has to be very detail oriented, since all the digitally recreated rooms have to look natural, and lighting is one delicate step in order to make that happen. Once this review and retouch process is done, the pictures need to be exported in .jpg format, since photogrammetry softwares don’t use the previously mentioned .raw format.

Once all the pictures have been imported to the photogrammetry software, a first alignment calculation is done. This is essential in order to obtain a scattered point cloud model. The scattered point cloud model helps the technician to evaluate if more pictures are needed, if the ones chosen are the correct ones or if everyone selected is enough to continue with the process. The following gif illustrates this paragraph that is common to every photogrammetry process.

Once this step is done, a second and more precise calculation is needed to obtain a polygon mesh model of every part of the room. This process is heavier and slower to perform, but it helps to bring naturality and hyper-realism to the final product. Since it is a step in which every piece is defined, it offers as well a solid vision of how the final model is going to look like.

Finally, and once the polygonal model is obtained, the textures of the model are calculated in order to edit the final light and color information. The following pictures show the final look of the digital recreation using this photogrammetric technique. It can be appreciated all the details of every single piece of art, obtaining the desired hyper-realistic look in order to be used for the previously mentioned use case.

VRTogether was present at VRDays Europe 2020, showcasing the world’s first live 3D video conference, using point clouds, over a commercial 5G network. VRTogether partners CWI and Sound presented a live demo to demonstrate the potential of the platform for emergency healthcare services. The demo showcased volumetric video conferencing (based on point clouds) between a doctor, in a medical examination room, and an (acting) injured patient outside in the street.

VRTogether will actively participate in several activities taking place during the 3-day event:

Volumetric Video Conference over 5G: a medical emergency example – Live demo

On Wednesday, 4th November, VRTogether partners CWI and Sound will present a live demo at Amsterdam’s De Kromhouthal to demonstrate the potential of VRTogether for emergency healthcare services. The demo will showcase volumetric video conferencing (based on point clouds) over the standard 5G network of KPN.

4th November
10:55 – 11:10 h CET
De Kromhouthal / Live streaming

For the eleventh consecutive year, Stereopsia will take place in Belgium (Brussels) on 11-13 December in SQUARE & BOZAR. Also known as “World Immersion Forum”, Stereopsia brings together the world’s greatest professionals in the field of immersive technologies and content: Virtual and Mixed Reality, 3D Cinema, 3D Sound, Artificial Intelligence … This event has been uniting worldwide experts for the past 10 years.

On Thursday, 12 December, VRTogether will be present at one of the Thematic Conferences within the event on “XR for Social Interaction”. CWI will explain how the project is addressing the research challenges of Social VR in a talk by Dr. Pablo Cesar.

PROGRAM

Chairman: Louis Cacciuttolo (VRrOOm, France)

• Are trolls inherent to the fairy worlds of social XR,  Louis Cacciuttolo (VRrOOm, France)
• Social VR: a new medium for remote communication & collaboration,  Pablo Cesar (Centrum Wiskunde & Informatica (CWI), The Netherlands)
• Social VR: how to get an amazing and unique story, Ana Revilla Hernando (The Modern C.P., Spain & USA)
• XR interactions with AI-powered virtual characters, from services to social experiences, Stéphane Juffé (Bemersive, France)
• Practical tips for organizing innovative XR events,  Antony Vitillo (New Technology Walkers, Italy)
• How virtual reality can promote sustainability,  Géraldine Fauville (Stanford University, USA)

These presentations will be followed by a panel discussion and Q&As.

VRTogether will be present at the second edition of the IEEE International Conference on Artificial Intelligence and Virtual Reality, which will take place in San Diego (USA). On Tuesday, 10 December, CWI researcher Dr. Pablo Cesar will give a keynote on “Using Volumetric Video for Remote Communication and Collaboration: development and evaluation of a social VR system based on point clouds“. 

This keynote will provide an overview about the challenges to achieve such a goal, based on results from practical case studies like the VRTogether project. Dr. Pablo Cesar will discuss about different technologies, like point clouds, that can be used as the format for representing highly-realistic digital humans, and about metrics and protocols for quantifying the quality of experience. The final intention of the talk is to shed some light on social VR, as a new group of virtual reality experiences based on social photorealistic immersive content and to discuss about the challenges regarding production, technology, and user-centric processes.

The IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR 2019) is a unique event, addressing researchers and industries from all areas of AI as well as Virtual, Augmented, and Mixed Reality. It provides an international forum for the exchange between those fields, to present advances in the state of the art, identify emerging research topics, and together define the future of these research domains.