Facebook Develops Neural Supersampling Technology

SIGGRAPH 2020 conference is about to be held, and Facebook researchers published an article today, introducing the "neural supersampling" (neural supersampling) technology. This is an algorithm technology based on artificial intelligence machine learning to improve image clarity, which is not much different from NVIDIA's DLSS "deep learning supersampling" technology. However, this neural supersampling does not require special hardware or software to achieve the results and basically comparable to the effect of DLSS.

On the other hand, Nvidia recently released Deep Learning Supersampling (DLSS), which can process low-resolution images through neural networks in real time and enlarge the samples. In this article, there is introduce of a new method that makes modern game engines easy to integrate, does not require special hardware or software, and can be applied to many existing software platforms. For neural supersampling, the additional auxiliary information provided by motion vectors plays a key role. The motion vector determines the geometric correspondence between pixels in consecutive frames. In other words, each motion vector points to a sub-pixel location, and on these pixels, a visible surface point may appear adjacent to the previous frame. These values ​​are usually estimated by computer vision methods, but such optical flow simulation algorithms are prone to errors. In contrast, the rendering engine can directly produce dense motion vectors, so as to obtain reliable and rich input information, which is provided to neural supersampling and used to render the screen content.

The method is based on the above observations, combined with additional auxiliary information, using innovative Spatio-temporal neural network design to maximize the quality of pictures and videos, while providing real-time computing performance. In the article, Facebook mentioned that neural oversampling technology may be applied to AR and VR programs to facilitate their Oculus platform. However, if the effect is good, it should be feasible for other 3D games. Of course, new technologies need to be tested by practice, and we look forward to seeing the actual performance of this neural oversampling.