Facebook releases DeepFocus VR tool

Facebook's virtual reality research arm, Reality Labs, has announced the public release of DeepFocus, an artificially intelligent system for rendering realistic focus effects, under a Creative Commons licence - but one which, sadly, precludes commercial exploitation.

Presented during the ACM SIGGRAPH Asia 2018 conference earlier this month by researchers Lei Xiao, Anton Kaplanyan, Alexander Fix, Matthew Chapman, and Douglas Lanman, DeepFocus is designed to make virtual reality considerably more realistic by rendering retinal blur in real-time on headsets with integrated eye-tracking capabilities. The result, the company claims, is a more convincing VR experience which, as the area not currently under focus is blurred, can run at a higher frame rate than current efforts.

'When someone wearing a DeepFocus-enabled headset looks at a nearby object, it immediately appears crisp and in focus, while background objects appear out of focus, just as they would in real life,' the researchers explain of their creation. 'This defocused blur (also known as retinal blur) is important for achieving realism and depth perception in VR. DeepFocus is the first system able to produce this effect in real time for VR applications.

'Some traditional approaches, such as using an accumulation buffer, can achieve physically accurate defocused blur. But they can’t produce the effect in real time for sophisticated, rich content, because the processing demands are too high for even state-of-the-art chips. Instead, we solved this problem using deep learning. We developed a novel end-to-end convolutional neural network that produces the image with accurate retinal blur as soon as the eye looks at different parts of a scene. The network includes new volume-preserving interleaving layers to reduce the spatial dimensions of the input while fully preserving image details. The convolutional layers of the network then operate on the same, reduced spatial resolution, with significantly reduced runtime.'

To encourage adoption of the technique, Facebook has released both the DeepFocus code and its network models and data sets under a Creative Commons licence, though it comes with a major restriction preventing it from being truly open source: While anyone is free to download, use, and modify the DeepFocus source code, they are not permitted to use the software or any derivatives commercially. The chance of seeing the technology adopted outside Facebook's own Oculus ecosystem, as a result, seems slim.

Interested parties can find the source code and data sets on the Facebook Research GitHub repository.