Utilizing PTZ Cameras in Virtual Productions

I’ve always had a fascination with virtual sets on news programs, film FX, and live events. When I came across the Indian sport Kabbadi a couple of years ago, I was delighted not only to learn about this pro sport (PKL), but also at the technology utilized in the broadcasts. Particularly delightful was the AR graphic insertion of the “living” 3D team mascots placed onto the court when introducing the match. Since I often write about PTZ cameras, this sent me down a rabbit hole of how these productions utilize PTZ cameras in VR/AR productions.

The Rise of AI Tracking Technology

When video cameras featuring integrated robotic heads and PTZ cameras were born, new opportunities emerged for small venues to control cameras remotely with small crews, while allowing large venues to add even more coverage without the need for operators at the camera’s location. As creative productions found the addition of robotic cameras a great advancement, they were still limited within VR/AR and effects departments, as external camera trackers were still required to integrate cameras into video graphics.

As PTZ cameras evolved, the addition of enhanced, integrated AI tracking algorithms has been game-changing especially within the VR/AR production space, particularly in broadcasting, esports, live events, and corporate environments. With the combination of custom AI tracking firmware and integration with tracking protocols such as FreeD, capturing precise data for VR/AR applications with PTZ is being supported with more brands.

First, Definitions

Why PTZ Cameras?

Using PTZ cameras in virtual productions is becoming more popular as they are cost effective, prevalent in the market, multiple connectivity options, portable, easily expandable, and highly upgradable, compatible, and customizable. Constant updates are being made in the field of AI tracking, and PTZ cameras with built-in remote-control capabilities are the ideal candidate for integrating with tracking applications.

PTZ head camera tracking for broadcast and cinema cameras can also be used for your existing PTZ / digital cameras that may not have built-in tracking support. With tools such as RedSpy, you pair your camera with a tracker and a robotic PTZ head and pedestal so it can seamlessly track your subjects.

Types of Virtual Productions

• AR in broadcast news, gaming competitions

• Virtual sets with green screen and cycloramas

• Video Walls and Volume Walls

• Film and TV special effects

• Live sports

• Concerts and events

Setting up Virtual Sets

One of the most common uses of PTZ cameras with virtual production is using virtual sets. You’ve probably seen news broadcasts, such as a TV anchor desk, weather wall, or dynamic coverage of an election or sporting events with the subjects sitting or standing in front of complex graphic screens with designs or even AR graphic pop-ups on set with live stats.

To achieve an effective setup, you can utilize cameras such as PTZOptics’ Move 4K, Sony BRC series, Vizrt PTZ3 Plus, or Panasonic AW-series PTZ cameras that support the FreeD communication protocol. Connect the cameras and a controller to your broadcast network and route the video to your software to add graphics within the environment.

What is FreeD?

FreeD is an open-source data communication protocol that enables the exchange of tracking data between VR/AR software applications and the camera over IP, NDI|HX, or serial communications protocols such as RS-232 and RS-422, staples of PTZ camera control. This enables cameras to send video along with data to 3D and virtual production applications such as Unreal Engine, disguise, and Mo-Sys, so the graphics software can ensure your camera movements are integrated into the virtual environment.

What types of hardware and software support tracking and virtual sets? These are just some examples that can integrate FreeD, tracking data, and/or PTZ cameras:

Case Studies

Check out some of the case studies below that demonstrate the ease of setup and the possibilities for using PTZ cameras in virtual productions.

A case study discusses LED video wall virtual set production with Canon PTZ cameras for an award show

The show utilizes a mixed multi-camera (camcorder and PTZ) setup, LED volume wall, XC controller, the Unreal Engine for virtual lighting, and Disguise media servers to manage the graphics.

Virtual Reality set using the Reality Engine software from Zero Density, also using FreeD

Panasonic AW-UE150 PTZ cameras send their position information to the software, which then tracks every move and subject using FreeD over serial or IP.

PTZOptics discusses using Move 4K and FreeD in multiple Use Cases: AR, Virtual Sets, Film/TV Production, Events, and Sports

Virtual sets, AR graphics, overlays, special effects, dynamic event backgrounds, and live sports analytics are just some possibilities in PTZ camera productions.

The Future Beyond FreeD

The future will only bring more advancements in AI tracking and VR/AR/MR/XR production, and that also means advancements beyond FreeD into proprietary VR tracking protocols. Mo-Sys and Brainstorm already offer virtual production studio software and support FreeD, but they have now created their own tracking protocols built on the foundations of FreeD. Many PTZ cameras already have gesture control functions and built-in tracking, and those features will only get more powerful. Virtual applications will become so intertwined withing PTZ camera technology, it’ll soon just be up to your imagination to make the magic happen.

Please feel free to share a virtual production experience using PTZ cameras in the comments below.