Action Painting (Masculine Expressionism) is a series of adrenaline-filled painterly animations composed using shapes and sounds sampled from Hollywood action film sequences. The project is presented as a multichannel video installation, projected onto large-scale painting canvases.
Action Painting was created using iconic sequences from over 40 highly-popular action flicks, such as The Terminator, Rocky, Bourne Identity, Fight Club, The Matrix, Armageddon, etc. Each animation was based on a different action film trope: the car chase, the fist fight, the shootout, and the explosion.
As action occurs in the original film, the photographic shapes within the frame (the car, the fist, the explosion) are highlighted, copied and transformed into abstract painterly forms that splatter onto a canvas. As time passes, and more action occurs in the films, the shapes accumulate and the canvas is completed. In this case, the painting canvas is entirely virtual.
Similarly, the creative "gesture" is not a physical expression like the complex and highly articulated paint flicking and dripping of Jackson Pollock; rather the gesture is the sublime and spectacular movement found within the frames of an action film. By combining the gestures with the original films' foley sounds, they are given a powerful and synchronized ummph as they splatter onto the canvas.
Action Painting consists of four distinct single-channel works that explore the visual language of a different action film trope. Pitter Patter Splatter uses smoke and fire from pistols, handguns and automatic weapons. Monochromatic Bursts of Color is composed of the bursts, flames and smoke of explosions. Revving Motors, Spinning Wheels features spatially arranged car chases, humming motors, crashes, and screeching turns. And Unrelenting Physical Aggression contains an onslaught of viscious punches, kicks, and smacks.
Action Painting was created using custom software that was written in C++ — using a wonderful open source toolkit for artists and designers called OpenFrameworks. OpenFrameworks includes software libraries that can analyse frames of a movie to look for specific colors, specific shapes, tracking movement, etc. — and draw vector graphic shapes. These tools were incredibly useful for making this project happen.
I started Action Painting during the fall of 2008, while working as a researcher at NYU's Interactive Telecommunication Program, and completed it almost 4 years later in 2011. The project also received some support from the Banff New Media Institute's Interactive Screens Program in 2009.
This post documents some of the technical processes I used to create the four animations and a series of large-scale prints. For those of you who are interested in the production tools, I've also included some software that demonstrates the different processes used to make the artwork.
Painting Cinema
Cinema is intended to be a frozen form, a "hot medium" as media theorist Marshall McLuhan liked to point out, or a "read-only" medium as Lawrence Lessig distinguishes it from the "read/write" experience of the Internet. Cinema is meant to be consumed and viewed, and most definitely not to be interacted with. In painting cinema, I'm looking for ways to explore cinema as a malleable medium (just like paint, which can be brushed, splattered, dripped, etc.), open for personal expression and creativity.
The practice of painting is an act of interpretation, of understanding a form before finding a way to express it using one's own capabilities. Making this project was an opportunity to dissect cinematic composition, and gave me a chance to dig into the language of Hollywood filmmaking.
The act of the gesture, the painter's creative expression, is a major aspect of this project. As opposed to being a painting in the traditional sense, Action Painting reflects the creative expression in our digital age: simulated/virtual as opposed to physical, an act of selecting (or curating) as apposed to an act of making, and frequently immitating or pastiching an existing style.
You might be wondering ... does this re-use of film constitute breaking the law? Historically speaking, yes and no. Some artists have been arrested for using pre-existing media that they didn't have the rights to use. In some notable cases, Andy Warhol, Shepard Fairey, Luke Dubois and others have run into trouble for their work. But it's not necessarily illegal; US copyright law leaves room for projects that use other media for what's called fair-use — for purposes of commentary, criticism, reporting, education, etc.
Fair use is a grey area that really depends on how the media is being used. In this case, I'm abstracting my footage to the point of un-recognizablility, which also means that viewing Action Painting doesn't compete econimically with the experience of viewing the films that I'm sampling. Those are two considereations (of four) for whether a use of media without permission is legal or not.
On Violence & Beauty
Film theorists and cultural critics frequently discuss the "aestheticization of violence" (see Margaret Bruder's essay of the same name): how violence in cinema and video games has been idealized and rendered in poetically beautiful sequences.
Compare an action sequence in a contemporary film with one from 20+ years ago and you'll see how contemporary films are edited (using short, quickly-edited sequences) to help increase the intensity of the violence. The most recent Transformers movies (if you've seen them) are a perfect example: the viewer is actually overwhelmed by the amount of motion in the fight sequences, making it challenging to understand what's actually happening on screen.
This is another motivation for Action Painting, which essentially takes these most violent cinematic gestures and turns them into shiny, beautiful paintings. It also influenced how I edited the films that were sampled for the animation. In making this project, I preferred to not explicitly make a blanket statement along the lines of "violence is a detriment to culture and society", but rather decided to make a project that focuses on the construction and experience of that violence. I also wanted to see what happens when we encounter that high-intensity violence with such great repetition. Does it satisfy the viewer's hunger, make them nauseous, or perhaps bring them to laughter?
Selecting the Films
The 40+ films that were used to create Action Painting were culled from top 10 lists online. I typically googled "top 10 movie punches" or "top 10 car chases" and came across dozens of websites and blog posts and YouTube channels listing these cinematic spectacles. Using top 10 list allowed me to step back from the curation of the clips and allow the project to widely reflect popular cultural tastes.
I watched close to 100 films, noting the times that significant action occurred and cataloged them in a database — before settling on which films would be used for each composition. Contemporary modes of distributing films online made it considerably easier to produce this project.
Using "top 10" lists was also a nod to Jason Salavon's artwork Top Grossing Film of All Time, 1 x 1, which introduced me to the practice of making digital art using film as a material.
"Behind The Scenes"
Action Painting was produced using custom software tools. The purpose of these digital tools was to select shapes from the films and transform them into splatters of color. This required analyzing each frame of a film sequence to find movement, colors, shapes, etc.
In the animation world, this is commonly called rotoscoping and it involves going frame by frame through a film sequence, tracing specific shapes by hand and then using those drawings in a new animation. Software tools allowed me to automate this process, though not fully.
I actually stumbled upon this process of going frame by frame through a film as a kid equipped with a VCR and a VHS copy of Robocop, my favorite childhood movie. I would hit pause and step through the scene the scene when Clarence truck drives right through Emil after he had been covered in a radioactive sludge that was slowly breaking down his body. I was totally fascinated by way in which Emil's head flies over the front end of the car on impact. And part of me was convinced that it was real.
Anyway, to anaylze and dissect each action film trope, I wrote a different piece of software.
Explosions
Selecting shapes from explosions involves searching for pixels of specific color: red, orange, white, yellow, and sometimes even blue (for the explosions at the end of Armageddon). In the world of computer vision, this is typically referred to as "color tracking." Color tracking involves searching within an image for areas (actually, neighboring groups of pixels) of a specific color. The process returns the boundaries of the shapes that it finds.
The above video demonstrates the color tracking process using slowed down footage from the film Armageddon. Here, the software searches each and every frame of the explosion sequence, looking for areas that are fiery orange. The shapes it finds are highlighted in red.
Car Chases
In order to selecting shapes from car chases, I used "motion tracking" techniques to compare consecutive frames of video for changed color values. The idea is that, when an object moves within the frame of a film, the colors of the pixels within that frame change. They might be the same colors, but they show up in different areas of the frame. For example, the bumper of a car that is no longer at a specific space in the road reveals the road — which is typically a different color.
In the demo video above, frames from the film Ronin are compared for movement — the areas that have changed significantly between frames are highlighted in red. You'll see how this process typically reveals the outline of the car.
You'll also see that this process reveals all of the other moving shapes within the frame, which is a problematic result when the camera moves. These shapes are called "false positives" because they are correct (according to the search rules) but also unwanted shapes. The tool I programmed allowed me to manually remove the false positives and only track the movement for the cars.
Gunfire
Selecting shapes from gunfire scenes was a much more manual process. I had to go manually frame by frame through each sequence looking for the bright flare emenating from the barrel of the gun after it was fired. By the way, in the industry, these are called "muzzle flashes". Everytime I clicked on that flare, the software would record the color of the pixel I clicked on and looked for shapes within the frame that were of that particular color (yes, this is pretty similar to the selection tool in Photoshop). The software would then save the oulines and colors of those matching shapes.
Altough this process of manually rotoscoping the films was a lot more painstaking and time-consuming than the automatic processes for motion or color tracking, it allowed me to be very specific and precise about the shapes I selected. Another reason for taking this approach was that there's often too much camera movement (which means more false positives) within gunfire sequences to do effective motion tracking.
Fist Fights
Selecting shapes from fist fight scenes was also done using rotoscoping: I manually went frame by frame through each of these sequences, clicking on the fists and feet of the actors fighting in the film. In order to emphase the impact of the punches, and also emphasise the splatter, I higlighted the face or body of the actor being struck by the punch.
This very laborious process of shape collection took a few months of clicking — and it was fascinating to see how the we, as viewers, could see punches when there was such serious discontinuities in the movement of attacking characters in between frames. No doubt, the use of sound in film is very useful in making these moments convincing.
Splattering Colors
Once the software locates the shapes within the frame of a film, it determines each shape's color, and then save its contours (outlines) and color to a custom data file. The software determines the color by going through all of the pixels that make up that shape and picking a few of its containing pixels at random, and then averaging those colors to get a single color, which is then used for the splatter.
The shapes' contours are analysed and then smoothed to create forms that were much closer to the wet contours of paint, so that the compositions looked more natural and less computer generated.
The shapes' scale, position and rotation were also randomlized slightly to give the compositions a looser feel. Otherwise, all of the shapes would have appeared rigidly connected to one another.
In the above demo, using a scene from Robocop, you can see how the splattered shapes get their colors from the original film — and the spatial and temporal relationships between the splatter shapes on the canvas and the action taking plance on screen.
Computer Vision
Within the computer science field, these processes of locating and tracking forms is referred to as computer vision. It's a practice that's been used for decades to locate specific shapes or track movement typically from a video camera, it's used regularly in military and industrial applications and has widely been embraced by digital artists looking to add interaction into their works (from Myron Kruger to Kinect hacks). Intel provides a free, open-source library for computer vision called OpenCV, which comes along with OpenFrameworks.
Sound Design
Sound design plays a critical role in the cognitive experience of films. In action films especially, it works to create a sense of realism, using sounds that help tie together the (sometimes simulated using CG) images on screen with the action portrayed.
For the creation of Action Painting, I decided to use the original sound (soundtrack & foley) from the films. Hearing the original sound brings materiality and meaning to the abstract forms from the animations, creating something more coherent and recognizable — a splatter of color gets becomes associated with the punch or gunfire that actually triggered it.
In his book Audio Vision French scholar and musician Michel Chion, refers to this moment as the "point of synchronization", which is when a films' sound and the image come together to describe the same event or phenomenon. This cinematic 'trick' was critical for the experience of the work, giving each of the gestures an extra burst of energy and intensity when they splattere on screen.
Composing the Paintings
Once all of the sequences had been transformed into painterly animated gestures, I wrote an additional piece of software to orchestrate them like voices in a musical composition. This was actually a timeline-based software not unlike what you'd see in After Effects or Final Cut Pro. But the reason I chose not to use these tools was so that I could write my own rules to select the clips and generate unique paintings. I carefully tagged each of the sequences based on their visual content, sonic experience, intensity of violence, and the qualities of their movement. For the car chases, for example, I would tag the type of engine sounds, the movement of the cars on screen, the speed of the movement, whether there was a crash or it was a police chase. After that, I could use a database to help me select the clips and create a composition using only tags.
A ruleset for selecting the clips for a composition of car chases might look something like this:
{ driving away, speeding up, revving motor, police siren, screeching turn, speeding across the screen, closeup of driver, car flying }
The software would select the clips from the database and then render the painted shapes and sounds to a high definition (720p) QuickTime file, which is what you'll see on Vimeo or in the installation format.
Action Painting Prints
The shapes collected from the sequences were also used to generate a series of large-scale high resolution prints. Since they are still images, they allow for a better experience of the subtleties of the form. Viewers can immerse themselves in the image in ways that aren't possible on screen (because of the limited detail of computer displays and projectors).
Unlike the painterly animations, these prints were composed by more-randomly splattering the shapes onto the canvas, abstracting the content from the original films and more closely emulating Pollock's own improvisational process.
So far, I've produced four different prints, (again) one for each action film trope. I still intend on making one final: "Best of Action Painting" print, utilizing shapes from the different tropes. Limited edition versions of these images printed on archival fine art paper are available for sale.
Code Samples
I've included three applications demonstrating how to perform basic color and motion tracking in OpenFrameworks. In order to run these demos, you'll need to download the OpenFrameworks project for your operating system (Note: these examples use the most recent version at the time of publishing: 0.71).
The fourth application show how a single blob's (that's the term used in computer vision to describe a shape) contours can be transformed into a smoother, more painterly shape using Bezier curves. This technique was based on this tutorial written by Maxim Shemanarev for the Antigrain graphics rendering engine.
The applications are available for viewing, disecting and expanding upon on GitHub.
— Jeremy Rotsztain. September 2012.
