Too good not to share here, and reproduced in full from his original Linkedin post, Rob Chandler of Morden Wolf and the Starting Pixel WhatsApp group fame offers newbies to virtual production and in-camera visual effects (ICVFX) some valuable advice on what to do before you get underway.
1 – Grasp The Basics
You may have already commissioned a traditional video production before. VP is similar but not the same, so it’s crucial to understand the basics.
Firstly, you should understand why VP is a good choice in the first place. Are you trying to cut the costs that filming in some real-world locations would entail? Are you trying to reduce carbon emissions? Are you trying to keep post-production time short?
Filming in VP can give you the flexibility to meet these goals, but it doesn’t always make it the right choice.
Secondly, shooting using VP shifts the focus from “fixing in post-production” to “fixing in pre-production. Like old-fashioned celluloid films, what goes in the camera stays in the camera.
Understanding this can drastically reduce post-production costs and time. Getting this wrong, however, will cause you a vortex of pain.
Thirdly, while shooting anywhere (real or imagined) is possible, different locations have different time and resource demands.
If you’re looking for vast hyper realistic backgrounds and sweeping camera movements, you must allow for more time and budget. On the other hand, more abstract scenes, such as a colourful field of candy canes or mid to close-up shots, are super simple.
Fourthly, remember that VP is just another way to shoot. It gives you options that reflect your creative, time, stylistic and budgetary needs – but traditional filming methods are often still relevant. At times, location and set builds can be as viable (if not more so) as LED walls, green screens, and full 3D environments with live animations.
It’s also helpful to understand the different types of VP before getting started. It used to be that LED walls were the mainstay of any VP shoot, but more recently, a renaissance of green screens and small LED volumes have become just as viable for more flexible and agile shoots.
Likewise, different bits of kit are great for different things. LED walls are good for immersion, while green screens stand out for post-production flexibility.
Finally, similar to traditional content creation methods, there is no universally applicable solution in VP. If there were, there would be no need for physical set builds.
VP and traditional methods, including location shooting and set construction, have unique benefits and drawbacks that should be carefully weighed and considered in the context of each specific project.
Key takeaway
Ultimately, you don’t need to be an expert, but understanding the basics is key and deciding on VP as an option should not be taken lightly – getting it wrong can be a nightmare.
From here, you can get an idea of your creative potential.
2 – Budget & Resources
Like any shoot, budget shapes what you can do. You can dream of Avatar-level graphics, but your wallet might remind you that you’re not James Cameron.
For example, hyper-realistic graphics look amazing but can be expensive and add valuable production time. Green screens are more affordable and flexible, but reflections, dynamic lighting changes and movement are harder to achieve.
Balancing your creative ambitions with cash is key. That said, virtual production is usually cheaper on balance, with fewer environmental implications, for an increased creative outcome.
For example, shooting in multiple locations, especially overseas, is notoriously expensive and will impact any sustainability goals. In these cases, large LED screens could achieve the same effect while keeping costs down and drastically reducing CO2 impact.
Then, there is the reduction in resources. Things like ancillary security costs, catering, power supply, and transport are drastically reduced when shooting at a studio where the environment is controlled.
Again, it may be that certain scenes and shots can’t be shot in a VP stage, but that’s the magic of having another option. A production gets to choose the best method, thus reducing the resources required and targeting the right resources for the hero shots.
Key takeaway
Think of virtual production as giving you more options, not as a replacement for traditional methods. Now, your budget can be focused on the shots that matter.
3 – The Right Tech Mix
Different projects have different needs – and different tech demands.
For example, virtual production green screen shoots allow for greater post-production flexibility and control than LED walls while still displaying pre-vis on set. This is much better for complex sequences or where CGI isn’t final.
Augmented reality blends digital elements with live footage, which is helpful when you need interactive digital elements, and props aren’t always available.
Simple changes to the background allow for unlimited versions. If programmed beforehand, a simple button push can change an entire virtual location or subtle changes to lighting or virtual props, leading the way for geographically specific versions of any content.
Live Motion Capture (MoCap) captures the actors’ movements live and turns them into a digital twin of a human, monster or animal. Creating this live allows the director, cast and crew to work with an onset previs version of the CGI, removing the guesswork from a shot or relying on a green tennis ball for eye-line.
Location scouting with VR lets key crew familiarise themselves with a location without leaving their office. And, with the recent use of technologies like Gaussian splatting, someone can be placed in a near-realistic render of a real location.
There’s plenty more, but this gives an idea of the scale of tech available – you just need to ensure it’s the right tech for you.
Key takeaway
No one is an expert, so don’t feel vulnerable with your level of knowledge. Join a forum and hunt out experienced groups and a few individuals about your project.
Most knowledgeable people are really keen for people to know more and embrace VP. Just be open to new approaches, push hard to understand experience levels and get excited about creating magic.
4 – Balancing Real & Virtual Elements
Using props can add the right level of realism to bring a VP environment to life.
In its simplest form, an actor shot in a mid or closer can make the entire shot. But, adding in foreground props that are lit in the same way brings a shot to life, especially if the actor interacts with it, and there are digital twins of a prop in the real-time CGI.
Chairs, tables, and physical props add authenticity, while virtual backgrounds provide creative possibilities.
However, ensuring that the real objects’ lighting, shadows, and perspective match those of the virtual set is critical.
Think and plan carefully about what you need in advance and save the hassle later.
Key takeaway
If you have an art department (or if you are the art department), realism will emerge from blending the real and the unreal (excuse the (geeky) pun). It’s the little things that matter.
5 – Lighting
While we wouldn’t want to diminish the importance of everything other than lighting, lighting is the key to achieving a seamless VP shoot. This is especially true when trying to achieve ‘final pixel’ (meaning all the action is caught in the camera without post-production).
Getting this wrong will ultimately make the shots look 2.5D (depth being achieved by 2D objects being placed at different distances from the camera)
If the lighting in VP were a person, it would be Goldilocks who needs everything just right.
Aligning virtual lighting with real-world conditions ensures that a scene’s physical and digital elements appear as though they exist in the same space.
Virtual production can use complex software and lighting rigs to mimic natural and artificial light. But VP-experienced DoPs have as many tricks up their sleeves as they would for a traditional shoot; they just don’t overlap that much.
So, if a scene is set outdoors at golden hour, the virtual lighting must replicate the sunlight’s direction, intensity, colour temperature, reflections and how it casts shadows. Also, the golden hour can be maintained all day and changed to nighttime within minutes, if not instantly.
Lighting problems can break the illusion. It’s a challenge, but it’s essential for maintaining the continuity and realism of the scene.
Lighting a VP shoot isn’t just the actual lights; it also includes the virtual scene lighting. For instance, let’s say the DoP struggles to get the lighting to match the CGI. Rather than burning time, it could be (and often is) solved by changing the lighting of the CGI.
Like real lighting, you can change colour, temperature, intensity and position. While this is never encouraged to do live on set, it can solve a seemingly big challenge with the right communication.
At the minimum, ensure your camera team has someone who knows how to interact with the onset tech artist – changing direction, temperature, colour, size, and intensity of the lighting isn’t hard, but knowing what the implications are key to a speedy change.
Key takeaway
There is a saying “There are 3 rules to achieving realism with Virtual Production”
- Lighting
- Lighting
- Lighting.
Let the lighting teams, both real and virtual, have as much opportunity to succeed. At the very least, ensure your lighting team spends time with the on-set tech artists to understand what’s possible to change on-set and how long it will take. Agree on terminology (from both sides) and build rapport.
Once this is built, the magic will really happen.
6 – Pre-production
Unless you enjoy surprises and not the fun kind, the planning stage is vital. You should already know what you want, but take the time to map it out.
Areas like storyboards, concept art, pre-shoots, and scripts should all be made VP-friendly to outline the project’s visual and narrative direction.
Ensure your teams have VP shooting knowledge; if not, get some. It doesn’t take long, but you don’t want them to learn it on the shoot.
One thing that VP isn’t, is just sourcing and booking a VP LED stage, turning up with a virtual set on a thumb drive, and expecting it all to work. This will leave crews with PTSD (I wish I was joking).
VP should be integrated into everything if it will be a sizable part of the production. Even teams like wardrobe, hair and make-up can be impacted by the tech – jewellery, sequins and frizzy hair, just to mention a few, can be affected by the tech.
As a producer, it’s worth considering using a VP producer or VP supervisor if you’ve not shot VP before. It’ll give you breathing room and allow you to learn on the job while focusing on the other key aspects of the shoot.
Key takeaway
Select your VP team or studio as soon as you can. Get them involved; they will help, not hinder. Get your crew VP enlightened. Most studios will be willing to spend time with a crew before the shoot as it makes their lives easier, and they want the shoot to go as well as you do.
Insist on VP lock (which is like a script or picture lock) at least a week before the shoot.
7 – The team
Choosing the right people can be a challenge, but the success of a virtual production project often hinges on the team.
VP supervisors not only help oversee technical delivery and ensure the technological aspects align with the creative vision, but they can save the day with their Swiss army knife of knowledge and contacts.
A tech artist or art department manages the digital content, bringing the virtual world to life with attention to detail and artistry. Ensure they have experience in designing for VP. There are a lot of Unreal Engine artists who are amazing, but the approach is very different.
So, to avoid over-complicated and unused environments, they need to have experience, be overseen by a VP Supervisor, or get some training. The training isn’t long or challenging, but it’ll ensure focus and effort are put into what the camera will see and not out of shot.
This team must have VP shoot experience. Understanding shots and the shooting process and communicating with the first AD, art and photography team honestly, openly and in the correct language will create the magic of the good kind.
An AD who can ask the digital art team the right questions is key on set. Knowing that a system restart might take 20 minutes and not the 5 minutes quoted by the nearest tech who wants to be helpful, is key.
Meanwhile, a project manager organises the project to keep it on track and within budget.
It’s good to have adaptable team members who can communicate across disciplines, as virtual production often involves a blend of different skill sets and perspectives.
Key takeaway
Communication and experience are essential. Even the smallest amount of knowledge before the shoot will help enormously and make the process almost joyful.
8 – Digital assets
Creating and managing digital assets in virtual production is a time-intensive but crucial process.
It covers a range of jobs, from 3D modelling and scanning real objects for use in virtual spaces to keeping a secure and organised asset library.
Be highly focused on the experience of the team. Many highly skilled Unreal Engine artists will be able to create the most amazing environments. However, if a virtual location calls for a bathroom and the shot will have depth of field, then there’s little to be gained in building an entire house to minuscule detail.
The number one priority to ensure a timely turnover is to have VP lock (no exceptions) at least a week before the shoot. Without experience, UE artist will assume their art will work out of the box, but this seldom happens. The final virtual set has to be fully tested.
Optimisation, plugins, colour, frame rates, and resolution must be thoroughly tested, ideally with the DoP, so lighting can be discussed in a non-pressured environment. That way, if there are issues, they can be solved way before turnover.
Properly managed, these digital assets are crucial to creating a believable and engaging virtual environment.
Key takeaway
There are 3 rules to ensuring a timely turnover:
- Testing
- Testing
- Testing
Depth of field is your friend. There’s no point in spending time and budget on highly detailed digital sets if it will be out of focus often.
Critically, ensure the VAD (Virtual Art Dept) team delivers to the VP lock date a week before the shoot, and ideally, check in the files throughout the development so the studio’s tech artists can give feedback on any foreseeable optimisation or visual anomalies.
No exceptions.
These simple checks will likely save your production and make it run super smoothly.
9 – Leveraging Unreal Engine for storyboarding
Storyboarding with Unreal Engine (UE) is like having a crystal ball that works. It allows for real-time feedback and a clearer understanding of the final product’s appearance.
A small LED studio or a green screen with VP capabilities allows for proxy UE scenes, shots, and camera movements to be tested before the shooting day. This means saving time and money while taking the guesswork out.
Directors, DoPs and Heads of Design can even remotely use VR headsets and experience the virtual location in real-time – deciding on camera angles, shots, lighting and set dressing.
This can significantly streamline the production process, saving time and resources. Also, it lets you experiment with what you want to do and opens up creative possibilities.
Being flexible in this way can help you be more creative, make better decisions, and achieve better results.
Key Takeaway
VP isn’t just limited to shoot day. It offers new approaches that were impossible up until recently.
Talk through the production with an experienced VP Supervisor. They can set expectations, support the innovations and work hard to deliver on them.
10 – Flexibility
Flexibility on set is crucial, especially when working with complex tech. Much of the time, these technologies enable real-time content and environmental adjustments based on camera feedback.
While not encouraged for many reasons, quick changes to lighting, background imagery, or other visual elements can be made to accommodate the dynamics of a scene or an actor’s performance.
The ability to make fast adjustments ensures that the final product aligns with your creative vision and is one of the crowning benefits of VP.
Key takeaway
Understand the limitations, time implications and risks involved with moving, changing and updating VP locations live. While moving something like a bookshelf a few metres should take only seconds, knock-on effects may only surface when the change has been made, and may impact your 1st AD’s blood pressure.
Have an open and honest conversation to understand the implications before it is agreed to proceed with a change.
At Mondatum we have countless years of experience solving complex in-camera and virtual production problems. For advice, guidance and support, please contact John Rowe (john@mondatum.com) or Colin Birch (colin@mondatum.com).
Main image ℅ AV Magazine
MoCap image ℅ Cartoon Brew
Object-based lighting image ℅ Broadcast
VP team image ℅ Frame IO Insider
Unreal Engine storyboarding image ℅ The Unreal Engine Marketplace