Nvidia unveils RTX Broadcast Engine SDKs

Nvidia is continuing to find new ways gamers can make use of the originally workstation-centric additional hardware in its RTX graphics card family, this time using the Tensor Cores to accelerate livestreaming features including automated 'greenscreening' and augmented reality effects.

First seen in the workstation-centric Quadro family, the features of Turing-based RTX graphics cards above GTX equivalents were very clearly developed with a professional workload in mind. The RT Cores, designed to accelerate real-time ray tracing, shine in professional visualisation but have also found a home for hybrid rendering in triple-A games; the Tensor cores, designed to accelerate deep-learning workloads, have proven a tougher sell with Nvidia relying on a game-specific performance-boosting antialiasing technique to sell their benefits to gamers.

Now, though, it may have found a better way to make use of the Tensor cores outside the workstation market: Accelerating new features in livestreaming applications, including any-background 'greenscreening' and face-tracking augmented reality effects.

'Leading into TwitchCon — the world’s top gathering of livestreamers — we’re announcing the RTX Broadcast Engine, a new set of RTX-accelerated software development kits that use the AI capabilities of GeForce RTX GPUs to transform livestreams,' explains Nvidia's Gerardo Delgado. 'Powered by dedicated AI processors called Tensor Cores on RTX GPUs, the new SDKs enable virtual greenscreens, style filters and augmented reality effects — the kind of techniques used by major broadcast networks — all using AI and without the need for special equipment.'

The 'virtual greenscreens' discussed by Delgado don't rely on traditional chroma-keying and plain-coloured backgrounds, instead using an algorithm running on the Tensor Cores to detect and isolate the human part of a live video feed. A tweaked version of the algorithm is presented as RTX AR, with the claimed ability to 'detect faces, track facial features such as eyes and mouth, and even model the surface of a face, enabling real-time augmented reality effects using a standard web camera.' Finally, the RTX Style Filters use a style transference technique previously demonstrated by Nvidia to alter a live webcam video into the style of any other image - including screenshots from games, allowing streamers to mimic the visual style of whatever game they're streaming.

The company has also confirmed it is working with Hugh Bailey to integrate RTX Greenscreen into his popular OBS streaming software. 'Nvidia has been at the top of my list when it comes to streaming and recording equipment. I’m continually impressed with what they’re doing,' claims Bailey of the partnership. 'And their technology is impressive with RTX features like RTX Greenscreen.'

More information on the new features can be found on the Nvidia blog.