Kristen Stewart, the controversial Hollywood actress, directed a short film, called Come Swim, after which she co-authored a scientific paper on artificial intelligence, describing the style transfer process she employed in the movie.
The paper, called "Bringing Impressionism to Life with Neural Style Transfer in Come Swim," was published as a case study on how to use the machine learning process in films.
Neural Style Transfer, The Face Of The Cinema
Fans of both Stewart and AI were equally surprised to find the news, especially since the actress is such a controversial Hollywood figure. AI researchers hurried to make jokes on Twitter, celebrating the event.
"I used to tell Twilight jokes in my Intro to AI class. I take it all back," noted Mark Riedl, associate professor at Georgia Tech School of Interactive Computing, through a tweet.
The short creation was the directorial debut of the Twilight actress, and it was created through a machine learning technique called "style transfer." The process consists of the aesthetics of an image or video applied to another image or video, so as to create an impressionistic style.
"Neural Style Transfer is a striking, recently-developed technique that uses neural networks to artistically redraw an image in the style of a source style image. This paper explores the use of this technique in a production setting, applying Neural Style Transfer to redraw key scenes in Come Swim in the style of the impressionistic painting that inspired the film," noted the research abstract.
The paper, written in collaboration with the engineer of special effects, Bhautik J. Joshi, and the producer David Shapiro, was published in the renowned online repository for non-peer reviewed work, called arXiv.
The three-page paper doesn't describe the first attempt to incorporate style transfer in cinema. Aside from the recently published paper about employing the same technique in the movie Interstellar, other low-profile artistic videos used it in the past.
But the paper was not written as to quantify the complexity or the volume of work required to employ this technique, but as a guide for other creators who would be interested in using it as part of their movies.
Creating Art With Artificial Intelligence
Machine learning is a method of data analysis consisting of the automation of analytical model building. As the model is exposed to new data, it will adapt independently and it will incorporate the new information in future analyses. Machine learning is not a new scientific process. However, the complexity of the mathematical algorithms that it incorporates since the development of computing technologies makes it increasingly sophisticated.
The team involved the existing neural networks in transferring the style of the painting into a test frame, then adding blocks of color and texture. The result was a painting-like appearance, and the technique is not among the most complicated machine learning procedures.
"Come Swim is a poetic, impressionistic portrait of a heartbroken man underwater. The film itself is grounded in a painting (figure 2 by coauthor Kristen Stewart) of man rousing from sleep. We take a novel artistic step by applying Neural Style Transfer to redraw key scenes in the movie in the style of the painting, realizing them almost literally painting that underpins the film," the paper noted.
