Since the dawn of the green screen and chroma keyer the term virtual set has meant different things to different people.
In some market segments it has been referred to as only real-time 3D tracking, in others a straight chroma key onto a picture. In recent years the term has acquired the connotation that some form of camera movement is available, thanks in part to the widely marketed feature in NewTek's Tricaster series. In fairness it can be argued that anytime someone is chroma keyed in front of something other than a simple gradient or looping texture that they are being keyed into a virtual 'set'. The term virtual studio is often used as well but this would tend to limit further the content displayed to just studios.
Virtual sets are used so commonly today that most viewers aren't even aware of it. Such is the sophistication and quality of the graphics, keying, and lighting that everything from the news, to popular shows such as 'The Voice', to webcasts and product videos are being shot on green screen.
Almost every software application and hardware device that handles video has a chroma keyer in it, such is the popularity of appearing to be somewhere you are not. For good reason; a green wall can be a lot cheaper than building an elaborate studio. In this article I will detail how different systems handle virtual sets, their features and their liabilities.
Broadly speaking virtual set production can be broken into two categories. When a potential customer calls and asks how to do a virtual set or how much it will cost the first question I ask them is invariably: Are you doing this Live or in Post?
Live virtual set production requires the use of hardware specifically designed for chroma keying, or video production software running on a computer.
Hardware Based Keyers
Some examples of hardware based keyers are Ultimatte, traditional production switchers like Ross, Grass Valley, Datavideo, and Sony. These generally produce what I call a straight key, one input is keyed over another creating generally a static composite. The exception to this rule is in high end virtual productions where a real-time virtual set is created as the background source to match the camera using sophisticated tracking hardware like Vizrt, Orad, and RT-Software.
Hardware keyers are generally either good and expensive, or not so good and inexpensive. Lacking any built in method of camera movement they require external equipment like realtime 3D virtual set systems to generate movement and these systems tend to come with a six figure pricetag.
Software based systems mainly run on PCs, though there are a few Mac options available. PC based keyers run a wide gamut and are mostly part of video production software. The main PC based video production systems are Tricaster, vMix, Livestream, and Wirecast. On the Mac side of things Wirecast runs the field. There are other software based switchers, but they aren't noted for doing virtual sets.
The Tricaster and vMix use a technique called UV maps to create reflections and to position the talent and screen sources within a set. This gives the viewer a heightened sense of believability and is a popular selling with those products because of the wow factor. They also add the ability to do virtual camera moves imitating the very expensive 3d virtual set systems by doing what is called a 'trackless camera move' where the physical camera doesn't move, but virtually it appears as if the camera is doing a zoom or jib shot.
Livestream and Wirecast focus on a layered approach to virtual sets, building the sets up from different elements dictated in location by the user.
The Tricaster and vMix turnkey systems run from $6000 up to $35000. vMix, Livestream, and Wirecast software runs from $350 to $1000 not including computer or IO Hardware.
Working in post offers up a whole slew of virtual set options at lower pricepoints than live production with the only caveat being that it isn't live. Post production virtual sets can again be broadly broken down into two categories: Editors and Compositors.
Almost everyone in video has at some point worked with an editing program, they are as ubiquitous as they are varied. Less than a decade ago editing systems still sold for over $100k and today there are some that are free and many under $100. Most professional editors would not consider using them for actual production limiting the non-hobby editors to Avid, Premiere, Final Cut, Vegas, and recently Resolve. There are numerous niche high end editors and a number of low end editing programs vying for popularity and an entire article could be written on them, but from a virtual set production standpoint these 5 cover the field.
These virtual sets are usually images, sometimes video files. The images are usually .png files, which are a widely accepted file format that supports embedded alpha channels which is how we convey the ability to put people behind desks and video into monitors.
Alpha channels are a 4th channel in the image, the first 3 being red, green, and blue. This 4th channel is a grayscale which determines what is opaque and what is transparent; white being visible and black being transparent and various shades of gray in between semitransparent.
In our sets there are 2-3 different versions of each angle, a 24 bit solid image, and 1 or 2 32bit matte versions which have an alpha channel for either putting the chromakeyed talent behind a desk or video into a screen.
These matte files, usually found in the mattes folder, have an A or B in their name. We used to use FG and BG but some sets have two background mattes for different things like screens and windows.
The Virtual set will be built on your timeline and may look something like this below in your nonlinear editor. Some editors are top down and some are bottom up. In this example the NLE is SpeedEdit and it is a bottom up view (meaning the camera sees the bottom layer first and so on). Most NLEs are top down, the topmost track is what is seen first. On the bottom we have the B matte, in this case it is the table. Next is the chromakeyed talent, then the A matte which is most of the set except there is an alpha channel hole where the screen is. Last is video that is going into the monitor.
We can think of this composite being a sandwich of the above items. If you looked at it from the side it might look something like this:
But when we line everything up right we get a finished composite that looks something like this:
For many years After Effects has ruled in terms of units in the field followed distantly by Fusion, Shake, Nuke, and Motion. Recently Blackmagic Design acquired Eyeon and Fusion with it and released the product for both Mac and PC and dropped the price to free. While not exactly fair to customers who have put in over $10k over the years into Fusion it does give everyone in video the opportunity to explore a VFX quality node based compositor capable of creating amazing virtual sets and much more. That said, After Effects is still the most common compositor and amazingly full featured.
After Effects makes animating everything together pretty easy using the pick whip for selecting parent layers.
Cropping source footage is a breeze with Masks:
All these platforms give users a wide range of features and price to choose from. For multi-layered virtual sets, we used our own virtual sets from virtualsetworks.com mainly because of their support for multiple layers.
For more tutorials on how to use virtual sets with your particular system go to http://www.virtualsetworks.com/docs/index.htm