Set Extension for Next-Level VFX

State-of-the-art technology for animation and VFX in live-action films

"The film is all about atmosphere and the emotional mood it creates," explains Théophile Farant. The animation student is the director of the short film project. The story could be read as a metaphor for the empowering connection between humankind and nature, but it can be interpreted in different ways, he says. "To develop the storyline, we formed three teams in the workshop, each of which came up with a storyline for the VFX film," the director continues. In the end, there was a vote to decide which story would be realized. "The script I worked on at first was not elected, but this made the challenge of directing even more exciting for me." Théo is actually studying at MoPa L'ecole internationale du cinéma d'animation in Arles, France. He is completing an exchange semester at Animationsinstitut in the winter term of 2021/22. His role as director of AWAKENING was also determined by the participants of the SEW in a vote.

The SEW is part of the curriculum of Filmakademie Baden-Württemberg. The participants of the SEW come from the study specializations Cinematography, Production Design as well as the programs animation and VFX of Animationsinstitut. The course is about learning the state-of-the-art techniques that can be used to combine animation and visual effects with live-action footage. In 2021, students were supervised and trained by animator and illustrator David Maas, technical artist Tonio Freitag, set designer Thomas Stammer and cinematographer Thomas Merker.

Animationsinstitut is a pioneer in LED Wall Set Extension

Since 2020, shooting with an LED wall has been part of the SEW. The wall has been set up during the first half of the respective winter semester in one of the studios on the campus of Filmakademie. Visual effects in live-action films are increasingly being created with the help of such high-resolution LED screen walls. The new technology offers the possibility of shooting footage on film sets in real time with computer-generated imagery (CGI). The content on the screens serves as a virtual set that can be merged with the conventional set design in the camera’s image. One of the first major productions to make impressive use of the technology was the Disney series MANDOLORIAN. Nowadays, there are studios around the world that have set up LED screen sets, in Germany for example in Cologne, Berlin, Mannheim and Munich.

In recent years, Animationsinstitut of Filmakademie Baden-Württemberg contributed to the development of the cinematic use of LED screens with its research work. Its Research & Development department is regarded as one of the leading research institutions in the field of virtual production worldwide. As a result, Animationsinstitut also plays a leading role in the training of animation and VFX experts who can work and realize films with LED screens.

"It's incredible that students at Animationsinstitut can try out these new tools and shoot with them."

"The fact that we have the chance to shoot with the LED wall is a great opportunity," says Théo. "When we were shooting, it felt like we were really working with the newest thing possible in the industry. It's incredible that students at Animationsinstitut can try out these new tools and shoot with them." However, he says it was also particularly challenging. For him, working with the screen was something completely novel: "In a way, the technology is still innocent. As is so often the case with a new technology, it comes down to figuring out how to use it well," the director emphasizes. Artistically, he says, new technology can advance films on the one hand, but on the other hand, it can be quite challenging when technical limitations restrict the content. "I've only done 3D animation before, and there, you can basically do anything you imagine," Théo explains. With AWAKENING, he had to think a little differently because of the technology. Nevertheless, the technically versatile team managed to implement all plans in the end.

It's a matter of finding the right workflow with the new technology

Théo emphasizes, however, that, now, the general task is to find a good workflow with the LED screens. The shortcomings of the technology must be worked around in a certain way or suitable solutions have to be found. As of yet, he says, it was still difficult to shoot long shots from a greater distance with the LED wall on set. "You had to be careful not to accidentally film beyond the screen on set," Théo notes. In addition, there was a common problem of not being able to focus on the LED wall. This is due to the so-called moirée effect, which makes the light grid of the LED screens visible in the recorded image.

The SEW team consisting of Benjamin Gätzschmann, Philipp Dörrer, Eileen Kammer, Jan Klingner, David Sick and Justus Henne was looking for an improved way of working. They succeeded by shooting in a particular way: They filmed a 90-second sequence shot in one go with complicated camera movements, where the acting took place on a rotating stage. In the process, they found a way to film with the camera beyond the edges of the LED wall. This was achieved thanks to a motion capture system and with trackers that allowed the digital images to be synchronized with the real camera. This technique fundamentally simplifies visual set extension with LED walls, as it allows wide shots to be taken even with small LED walls.

The VFX supervisors and 3D artists of the SEW team circumvented further problems by creating detailed pre-visualizations of the set and a digital model of the studio. This allowed them to precisely plan the shooting. With that, the team avoided accidentally shooting scenes with problematic camera angles. "Our shoot at the studio was also limited to only two days, so we had limited time to experiment. Therefore, this planning helped us immensely," Théo explains. In addition, for many frames of the film, the team also worked with full CG images, with traditional VFX methods, to create film images entirely digitally on a computer. These were finally combined with the set shots, and the result was visually stunning.

From storytelling to set design – LED walls have huge potential

In general, the director of AWAKENING is enthusiastic about LED screen shooting technology. "Especially for acting, the technology has enormous advantages. In front of a green screen, the actress would not have been able to play the scene that authentically," he enthuses. If the technical workflows can be adjusted and the imponderables can be solved – something that is being researched intensively at Animationsinstitut and to which the experiences of the SEW team further contribute – the LED wall technology holds great opportunities in his opinion; also in terms of storytelling. "We are just experiencing the convergence of cinema and video games," says Théo, "It's very exciting to be able to be so close to this new development here at Animationsinstitut."