Spatio-temporal reprojection for virtual and augmented reality applications

Finn, Sinclair

Permalink

https://hdl.handle.net/2142/109178 Copy

Description

Title

Spatio-temporal reprojection for virtual and augmented reality applications

Author(s)

Finn, Sinclair

Contributor(s)

Adve, Sarita

Issue Date

2020-12

Keyword(s)

• reprojection
• virtual reality
• augmented reality
• reconstruction
• latency
• spacewarp

Abstract

Late-stage reprojection is a vital tool for reducing power and compute requirements and improving perceptual latency in extended reality (XR) systems. While rotational reprojection is a well-studied topic for which several implementations have been published, the more advanced type of reprojection, spatial reprojection, is largely uncharacterized and unpublished. Depth-aware spatio-temporal reprojection can reproject a depth volume to account for the full six-degrees-of- freedom of user motion. By reprojecting application frames with the full range of user motion with a sufficiently high level of accuracy, dropped frames and display latency can potentially be almost completely resolved. In addition, with the advent of remote-rendering and edge-compute-based architectures, the reprojection of rendered application frames can mitigate latencies introduced by the transmission of frames across great distances. This thesis aggregates and reviews prior related work in the domain of 3D image-based reprojection and warping algorithms, examining the applicability of previous reprojection algorithms to XR applications. We evaluate three different methods for solving the warp equation, weighing their characteristics, advantages, and disadvantages across several representative XR scenarios. Finally, the thesis presents two novel implementations of full depth-aware spatio-temporal reprojection, as well as deep performance characterizations on multiple representative platforms. In particular, the algorithms’ resilience to occlusion artifacts, high-frequency depth features, and large perspective displacements are examined and characterized. This paper also presents a detailed analysis of the structural similarity index measure (SSIM) of reprojected frames to characterize the quality of the algorithms’ output and the contribution to the overall user experience.

Type of Resource

text

Language

en

Permalink

http://hdl.handle.net/2142/109178

Owning Collections