A point cloud is defined as a set of points in 3D space with specified geometry coordinates and associated attributes such as colour. Point clouds are an important emerging format for VR applications because of their simplicity and versatility. There are no restrictions on the attributes associated with every point in the cloud.

Point clouds are usually captured using multiple cameras or depth sensors and can contain millions of points in order to create a photorealistic reconstruction of an object. Compression of point cloud geometry and attributes is essential in order to efficiently store and transmit point cloud data for applications such as teleimmersive video (figure 1) and free viewpoint sports replays (figure 2).

An ad hoc group was created by MPEG in order to start the standardisation activity on point cloud compression. In April 2017 MPEG issued a call for proposals on point cloud compression [mpe17] and divided the activity into three categories for static frames, dynamic sequences and dynamically acquired/fused point clouds. Nine leading technology companies responded to the CfP and the proposals were evaluated in October 2017. In addition to objective metrics, each proposal was also evaluated through subjective testing at GBTech and CWI. The winning proposals were selected as test models for the next step of the standardization activity.

For the compression of dynamic sequences it was found that compression performance can be significantly improved by leveraging existing video codecs after performing a 3D to 2D conversion using a suitable mapping scheme. This also allows the use of hardware acceleration for video codecs such as HEVC that is supported by many current generation GPUs. In this manner, synergies with existing hardware and software infrastructure can allow rapid deployment of new immersive experiences [ctb].

The next step was to identify and investigate methods to optimize the test models by performing core experiments. The core experiments for dynamic sequences explored the use of different mapping schemes from 3D to 2D, hybrid codecs that use 3D geometry compression (using octrees) as well as video codecs and the use of motion field coding.

The AhG cross checked each test model and reviewed the results of the core experiments in the recent MPEG 121 meeting in Gwangju. Other developments from this meeting include the addition of new high quality point cloud datasets for further testing and the collection of input documents from MPEG members to further improve each of the test models.

VR Together is actively participating in this group through CWI and Motion Spell. We intend to contribute new compression technologies [Sub17], delivery mechanisms and rendering techniques to the ongoing MPEG standardisation activity.

References

[ctb]Mpeg news: a report from the 120th meeting, macau, china. https://multimediacommunication.blogspot.nl/2017/12/mpeg-news-report-from-120th-meeting.html

[ifr] Intel freed technology. https://www.intel.com/content/www/us/en/sports/technology/intel-freed-360-replay-technology.html

[MBC16] Rufael Mekuria, Kees Blom, and Pablo Cesar. Design, implementation and evaluation of a point cloud codec for tele-immersive video. IEEE Transactions on Circuits and Systems for Video Technology, January 2016.

[mpe17] Call for proposals for point cloud compression iso/iec jtc1/sc29 wg11 n16732, geneva ch. January 2017.

[Sub17] Shishir Subramanyam. Interframe compression for 3D dynamic point clouds. Master’s thesis, Delft University of Technology, The Netherlands, November 2017.

Text and pictures: Shishir Subramanyam – CWI