Technical Directing for a Film World in Transition

"A lot has happened in that one year," says Simon Spielmann as he steps off a crane truck at Albrecht Ade Studio. The senior engineer of Animationsinstitut’s research and development department is installing the motion-capture system on the new LED wall. Compared to last year, he said, the wall is larger at 10 by 4 meters and curved by 3 degrees per element. "The curvature makes it even easier to light the set with the screens," he explains. In a very short time, he says, the technology has evolved tremendously.

More and More Studios Rely on LED Walls

LED sets are popping up around the world, he adds. "Every reputable facility has set up these studios. In Germany, for example, there are already some in Cologne, Berlin, Mannheim and Munich." The software to operate these facilities has also taken a giant leap. "A year ago, it was all very prototypical, but now many things are more user-friendly."

On film sets, the large LED screens fulfill the role of a virtual stage scenery. The technology is increasingly replacing blue and green screens, which were previously used when film scenes played out against computer-generated landscapes or photographic backgrounds. Unlike with blue and green screens, the design of the visual environment of the scenes is not done after the fact, but in real time on set. This makes it easier for directors and actors to work with the virtual images.

Film Industry Wants to Drastically Reduce CO2 Emissions

The worldwide boom of the LED wall technology is partly due to the corona pandemic. When productions around the world were restricted by hygienic measures, the technology was increasingly used as an alternative to shoot at real locations to avoid crew travel. For example, it was used on the Disney series Mandalorian or the Netflix production 1899.

But even without the restrictions caused by the pandemic, the technology is on the rise. That is because the film industry is in a period of upheaval. While climate change is showing its effects increasingly, the industry is questioning its contribution to global warming as well. In March 2021, the Sustainable Production Alliance, in which the world's most important film companies are organized, put the average CO2 emissions of a film production at an exorbitant 3,370 tons. Per shooting day, that is about 33 tons. In addition to the direct consumption of fossil fuels, the transport of goods and people is the most significant factor.

More and more industry players now want to get a grip on this problem. The streaming service Netflix, for example, announced an ambitious plan in 2021 to make the group and its products climate-neutral by the end of 2022. With a carbon footprint of 1.13 million metric tons that Netflix caused in 2020 alone, that is a daunting task. One of the first steps, according to Netflix, is to minimize crew travel and rely more on local teams. Filming methods using LED wall technology are likely to play a role here, as they are not bound to one location.

Film as an Exciting Application Area for Computer Scientists

Filmakademie’s Animationsinstitut, with its Research and Development Department and the postgraduate course Technical Directing, has been researching the new technology for some time. It is considered one of the leading research institutions in Europe in the field of virtual production. Its students complete the so-called Set Extension Workshop during their studies. In this workshop, they learn, together with students from the specializations of Cinematography and Production Design, how to combine animation and VFX technology with film set recordings. In the course of the workshop module, the students develop and realize a short film as a team.

In 2020, an LED wall was set up in Ludwigsburg for the first time as part of this project. Furthermore, the research department undertook numerous tests on it. And on top of this, students were able to shoot their projects with the LED wall. The task of Technical Directing (TD) students within the teams is to ensure the technical realization of the films. They have a computer science background and develop tools and software applications for the so-called pipeline processes that are necessary for the creation of films with animation and VFX components. In their studies, the use of LED wall technology is taking on an increasingly important role.

LED Walls Enable New XR Experiences and Interactive Applications

In 2021, for example, the team of the film Cybercity Lovestory is using the studio setup. The project is a pilot episode that follows a novel transmedia and immersive storytelling approach: Cybercity Lovestory is both a sci-fi film and an interactive VR game.

Pascal Schober studies Technical Directing and is responsible for technical support in the Cybercity Lovestory team. "It's a great luxury to be able to shoot and experiment with the LED wall technology here at Animationsinstitut," says Pascal. He believes the new technology has a lot of potential for interactive applications like the one the Cybercity team has in mind. Here, actors interact with a game environment on the wall.

Pascal notes that the technology is already establishing itself in the advertising industry. That is because properly preparing images for LED shoots could save time and money. "The environmental aspect also plays a role," says the technical director, "especially for large productions, very long and energy-intensive render times per frame are often common. LED walls open up an alternative for that."

There Is Still a Lot to be Improved in Terms of Technology

Pascal's current main project is his diploma film Mirage of a Sea. It is a live-action film with visual effects that he is shooting with students from Filmakademie. After he undertook tests with the wall for it last year, however, he has decided not to continue using the LED screens. For some applications, he thinks the technology is not mature enough yet: "Especially with live-action filming, you're often still better off doing it in a real environment," the TD student explains. "Until a smooth photorealistic workflow is possible, we still have to wait for newer generations of more powerful graphics cards and screens."

Scenes in front of an LED wall are often shot with camera settings that blur the background, he adds. If the camera focuses on the wall, the light grid of the LED screens can be seen in the recorded image. This is known as the moiré effect. "To compensate for it, it is still necessary to combine it with conventional technology, as well as a more often than not complex postproduction."

Simon also emphasizes that there is still much that can be improved. Students and researchers at Animationsinstitut are currently looking for solutions for several technological construction sites. For example, they are experimenting with specific light panels to enhance the color quality. A combination with conventional green screen technology is also currently being tested. In this case, the LED wall shows a green area between the images, which is used as a green screen. Solutions are also being sought for simpler postproduction processes.

Switching Workflows from Offline to Real Time.

"In general, we are interested in investigating the new workflow with the technology," says the media computer scientist. Simon is certain that the introduction of new workflows from offline to real time with LED walls will come eventually. After all, he says, using computer-generated content live on set and not having to rely on a lengthy postproduction offers many advantages. To optimize the work processes, the so-called pipeline, Animationsinstitut cooperated in research projects with VFX production companies, such as the London-based DNEG. Currently, Animationsinstitut is planning further research on extended reality applications (XR) on LED walls.

In addition, Animationsinstitut is currently researching ways to integrate artificial intelligence into virtual productions as part of the KI (German for "AI") Lab “Animation & VFX”. In the future, it should be possible to have characters animated independently by an AI through machine learning. This will enable faster and more resource-efficient production while keeping artistic control. The KI Lab "Animation & VFX" is an initiative of the Ministry of Economic Affairs, Labour and Tourism of Baden-Württemberg.

VFX companies, as well as the staff at Animationinstitut, know that screen-shooting technology can create a new business field. Because they can be installed digitally in any LED screen studio, lighting setups and VFX set design presets for LED walls may be offered as a service. Building databases of 3D art for such studios is likely to become a viable sector for the future.

Pascal Schober will receive his diploma in Technical Directing in the summer of 2022. He is not quite sure yet where he will go after his studies. At the moment, he is too busy with his projects to think about it. Numerous job opportunities arise from the diploma in the postgraduate course at Animationsinstitut. However, he is sure that LED wall technology will play a role in his future work. Thanks to his studies at Animationsinstitut and the expertise in the field of virtual production that it gives him, he will be one of the experts who can lead the film world into the future with LED screens in any case.