SARA - Stylized Animations for Reasearch on Autism
Processing faces or categorizing emotional facial expressions are some of the impairments faced by people with Autism Spectrum Disorders (ASD). This conveys an inability to relate to others in a socially meaningful way. Previous studies have indicated that this might be due to an atypical local-oriented strategy while processing faces, or to the amount of details conveyed by the human face, or to the focus of attention on the other's face (i.e. in which part of the face an individual with ASD fixes his eyes when interacting with another person).
The DFG-funded project SARA aims to investigate in a novel and artistic way the causes for deficits in social communication and emotion recognition in children and adolescents with Autism Spectrum Disorders (ASD) and Attention-Deficit/Hyperactivity Disorder (ADHD). It brings together the areas: clinical psychology (University Medical Center Freiburg), real-time non-photorealistic rendering (University of Konstanz) and character animation (Animationsinstitut, Filmakademie Baden-Württemberg), providing the framework to carry out novel and interdisciplinary research.
One of the novelties and main focus of the project is the real-time generation, parameterization and stylization of emotional facial expressions of virtual characters, the latter achieved using Non-Photorealistic Rendering (NPR) algorithms. Using different versions of the DECT (Dynamic Emotion Categorization Test), an interaction with a virtual characters is established, allowing us to test the ability of children and young teenagers with these neurodevelopmental disorders to read emotional states in the face of the characters.
Partners of the project
Download the SARA Frapper Deploy:
Styles
We investigate how stylization of the faces of virtual characters can influence the categorization of emotions by children and adolescents with ASD and ADHD. To that end, our partners at the University of Konstanz developed non-photorealistic rendering (NPR) algorithms that are not only parameterizable, but also abstract the geometry of our characters in real-time. The implementation was done using Frapper.
Image Abstraction
This style is inspired by paintings and painting-like images, where absence of fine-grained texture details and increased sharpness of edges are two relevant visual characteristics.
Coherent Line Drawing
Line drawing effectively conveys shapes and outlines to the viewer with simple primitives: lines. We use the algorithm of Kang et al. [2007] which creates coherent and artistic-looking lines.
Up: Hank (medium & high abstraction). Down: Nikita (medium & high abstraction)
Pencil Drawing
This is one of the most fundamental techniques in visual arts to abstract human perception of natural scenes. By converting input scenes into just lines and shading, a great number of details are removed, while keeping the object boundaries and plasticity of the rendered objects by preserving their shading.
Up: Hank (medium & high abstraction). Down: Nikita (medium & high abstraction)
Watercolors
Watercolor painting is an artistic style that creates the effect of water dissolved colors on paper, or similar surfaces. We decided to use this style not only for the abstraction it provides, but also because it is used by many individuals with ASD to express themselves through painting, or during therapies that introduce artistic elements [Tataroglu 2013].Our algorithm is based on the work of Luft et al. [2006] whose approach simplifies the visual complexity and imitates the natural effects of watercolor.
Up: Hank (medium & high abstraction). Down: Nikita (medium & high abstraction)
Loose & Sketchy (not tested in SARA)
Another algorithm initially implemented as a collaboration between Filmakademie Baden-Württemberg and University of Konstanz, and recently improved by the latter was Loose & Sketchy.
To improve the visual quality, instead of drawing thin lines from the seed points along the vector field, real strokes are generated. For this, the geometry shader stage in GL3+ is used to create a quad-strip with texture coordinates and apply a real stroke texture. This is done is 2 steps: once from the seed position in direction along the vector field and once from the seed point against the direction of the vector field. This is still work-in-progress.
Down: Nikita (medium & high abstraction)
Characters
Hank, Nikita, Sara and Gunnar
In addition to the already existing characters (Nikita - a young female, and Hank - an elderly male), we created counterparts that would give more validity (Gunnar- a young male character, and Sara - an elderly female character). All characters' rigs have been created using the Facial Animation Toolset (FAT).
In order to harmonize the visual appearance of the four characters, an artist was in charged of creating light conditions and adapting the existing shaders for all the characters. All this was carried out in our development software Frapper, which was enhanced to support the new shading and lighting functionalities.
Thanks to our supporting team: Kai Götz (creator of Sara), Solveigh Jäger (creator of Gunnar), Steven Stahlberg (creator of the original Nikita), Leszek Plichta (visual harmonization of the characters), Markus Rapp (real-time hair generation), Simon Spielmann (lead software developer Frapper) and Nils Zweiling (external software developer Frapper).
The characters are distributed as part of our Frapper software deploy under the Creative Commons Attribution Non Commercial Share Alike 3.0 Unported License.
New characters appearance (from left to right): Nikita, Gunnar, Sara, Hank.
Dect
The DECT (Dynamic Emotion Categorization Test) was in its first version an interactive computer-based tool created to assess the feasibility of using real-time animations by comparing virtual characters to video clips of human actors. The implementation of the test using our software development framework Frapper, brought together the teams at the University Medical Center Freiburg and Filmakademie Baden-Württemberg for the first time in 2009.
In SARA, the DECT has gone under some modifications, resulting in three versions (currently), which are used in the different experiments carried at the University Medical Center Freiburg. The first version, R-DECT was implemented and executed in the context of a pilot study to assess “Rapid Social Cognition" of children and adolescents with Autism Spectrum Disorders (ASD).
The second version called NPR-DECT comprises one of the novelties of our project: the use of NPR algorithms to abstract and manipulate visuo-spatial information in the faces of our virtual characters, reducing the information load in the characters’ facial expressions.
The third version called i-DECT is the one with the highest interactivity, stimulating a visual interaction between the participant and the virtual character. This test will study the differences in eye contact and mutual gaze between neurotypical subjects and subjects with ASD. The experiments with the i-DECT are currently being carried out in Freiburg.
