For their final project in associate professor Sergio Alvarez鈥檚 鈥淎rtificial Intelligence鈥� course in the computer science department, Andrew Francl 鈥�16 and Jesse Mu 鈥�17 chose to adapt the work of computational neuroscientists at Germany's University of T眉bingen, who in September 2015 reported on an algorithm they鈥檇 developed that extracts the brush-stroke style of a painting and applies it to another image. Francl, a double major in computer science and biology with a minor in mathematics, and Mu, a computer science major with a minor in mathematics, reconstructed and extended the T眉bingen algorithm to produce, among other 鈥渁rtwork,鈥� this scene鈥攁 photograph of Gasson Hall at sunset鈥攕uggesting the distinctive textures of Vincent van Gogh鈥檚 The Starry Night (1889).
Unlike computer apps that are programmed to impose specific painterly effects鈥攁n impressionist haze, say鈥攐nto another image, the algorithm that made this image can autonomously extract the style of any artist. The process, known as deep learning, involves algorithms, modeled on human neural systems, that enable a computer to learn from experience, modifying its behavior in response to mathematical rules as it takes on more data. The network Francl and Mu worked with, called a convolutional neural network, was modeled on the visual cortex, the part of the brain primarily involved with processing visual information.
Using a programming language called Python and an algorithm template titled Caffe, Francl and Mu translated mathematical formulations from the T眉bingen paper into some 650 lines of code, sitting side by side in the Fulton Hall computer science lab. The work required hours of 鈥渢rial-and-error and debugging,鈥� said Francl. 鈥淲e had to make sure our computational intuitions aligned with the mathematical equations,鈥� added Mu. The image above is the 512th step by the computer in improving its faithfulness to van Gogh鈥檚 brush-strokes (the quality plateaued thereafter), and represents some six hours of computing time.
The students tested the algorithm on a range of brush styles, from the proto-impressionism of J. M. W. Turner to the pointillism of Georges Seurat, transposing those styles onto color photographs of, for instance, the Back Bay skyline. In doing so, they encountered shortcomings in their program: Notably, it failed to make sense of art that lacked a distinctive surface texture鈥攆or example, Salvador Dal铆鈥檚 sleek聽The Persistence of Memory听(1931).
To capture van Gogh鈥檚 use of color, Francl and Mu wrote an extension to the T眉bingen algorithm resulting in the sky shown here, but its application proved limited. The pair also attempted to extract an artist鈥檚 overall style, using 45 cubist works by Picasso. Their conclusion: For that artist at least, 鈥渢he average becomes meaningless.鈥�
Francl will study cognitive sciences at MIT in the fall. Mu is a 2016 Goldwater Scholar.