In 2018, a distorted portrait created by an artificial intelligence (AI) algorithm was sold at Christie’s for $432,500 to an unknown phone bidder. This event had a massive impact on both the general public and art communities.
Inspired by this groundbreaking purchase, a group of researchers from the University of Colorado Boulder conducted a study asking people to tell the difference between art made by AI and by humans. The study concluded that people were not particularly good at telling the two apart.
Is it possible to “teach” a machine how to paint? Data scientists and AI engineers are working on new ways to find out. Although computer vision has been around for much longer than many people think, the introduction of machine learning (ML) has significantly shifted the paradigm.
The underlying idea of computer vision is to detect and recognize what the machine sees. Something so easy for humans turned out to be a tough challenge for machines. Here’s where convolutional neural networks come to play. With sufficient training data, they learn to identify and retrieve necessary features without human “assistance.”
What started as face recognition tools to unlock your phone or Snapchat’s funny filters have transformed into sophisticated AI algorithms that can identify artistic styles, restore paintings, and even create unique art pieces.
What Is a Convolutional Neural Network and How Does It Work?
An artificial neural network (ANN) is an algorithm that receives data, processes it in several stages, and then outputs the already modified data. A single stage of such processing is called a neural network layer. A convolutional neural network (CNN) is a type of ANN with additional layers – convolutional and pooling. It makes CNNs a significant advancement in deep learning, as these networks can have hundreds of such layers, and each of them aims to detect different imaging features.
The ML algorithms combine all possible features in all possible variations. But the ANN doesn’t stop at the first stage; it then combines features to get new intermediate variables. On the next layers, it creates combinations of combinations of combinations, and so on. And that’s the underlying principle of how a CNN works.
- First, the algorithm selects very specific and low-level features in the original image. These can be groups of pixels that are close to some color spot.
- By highlighting new elementary features in the resulting pictures, the algorithm combines and gradually complicates them.
- The original image is transformed into countless combinations of feature maps, where some pixels are activated.
- Then, the original image “shrinks” to a single point – a signal transmitted by a neuron. Such a signal combines with other signals and activates a chain of neurons in a CNN.
- At the end of the network, another single neuron “comes up” with the output depending on the information transmitted by other neurons in the chain.
This model has shown impressive results in natural language processing and computer vision, among other fields. Moreover, the largest image classification image database ImageNet was created with CNNs. Provided the model’s outstanding performance in image recognition, it eventually became a dominant ML tool in painting restoration.
Identifying Paintings With Neural Networks
Machine learning algorithms have been a major tool of authorship attribution and textual data processing for a while now. But image recognition instruments inevitably brought scientists to the creation of painting identification patterns. Such approaches have a great potential for real-life applications, including:
- Detecting fakes;
- Captioning artwork;
- Transforming art pieces into photo-realistic images;
- Analyzing representativity;
- Visual question answering.
Some open-source algorithms are available already, and some are only being proposed based on empirical scientific research. CNNs showed outstanding results when trained on large datasets of distorted pictures – scaled, lens-distorted, rotated, etc. Such distortions imitate real-life circumstances like the appearance of paintings on TV, in photos, or in movies.
In general terms, CNNs prove to be effective in artist identification and show the possibility of replacing human experts in performing this task in the future. Moreover, with sufficient training datasets (for example, 300 paintings by 57 well-known artists), CNNs successfully learned to identify a painting style.
Restoring Art Pieces With Neural Networks
Traditional painting restoration techniques include various methods, including multispectral imaging, as well as physical and chemical procedures. Today, scientists find more efficient ways for art conservation and restoration using neural networks. There are several ways machine learning algorithms have already been successfully applied in this field.
Convolutional neural networks prove to be effective in cleaning the varnish layer that deteriorates the look of old paintings when they turn yellow over time. If the traditional approach implies physically cleaning the varnish layer, in virtual cleaning, the CNN trains on artificially yellowed images of various kinds (rural and urban areas, people, and color pallets) and the images’ original versions. The trained network is applied to paintings labeled with similar content. In their research, the scientists used this method to restore the Mona Lisa and The Virgin and Child With Saint Anne. The algorithm outperformed the physical cleaning method and showed quantitative and qualitative improvements in the color quality.
CNNs are used in controllable image inpainting techniques that deal with flaking, degradation, and cracking. The framework is based on deep learning prediction and matching the nearest neighbor pixel. In this approach, the CNN provides crude estimations of what the painting would look like by filling in missing areas. Here, by matching the nearest neighbor pixel, the CNN provides high-quality outputs and maps a mid-frequency estimation in a controllable way. As a result, CNNs generate controllable high-frequency photo-realistic inpainting results and efficiently predict information in large missing areas.
Network Inversion As a Visualization Method
Some research attempts to understand how CNNs work as a whole in contrast to most visualization methods that focus on a single feature or a neuron at a time. Network inversion searches for input data that produces a specific image in the output. By identifying the network’s “preferences” for producing the output, the researchers proposed a visualization method effective for image restoration. The evaluation experiments showed that the network inversion efficiently restored the images damaged by haze or water. This network architecture demonstrates great potential for handling multiple tasks like inpainting image denoising and colorization.
Creating Paintings With Neural Networks
Creating a picture can be imagined as a mathematically difficult task. Here, the picture is represented as a set of numbers that specify the colors of corresponding pixels. The neural network attempts to produce a set of numbers similar to the pictures in which the neural network was trained.
Although creating art pieces may seem a part of a sci-fi narrative, it has become available for pretty much everyone these days. Tom Simonite from Wired demonstrates it vividly by creating digital paintings using open-source artificial intelligence tools.
In fact, neural art has become a term recognized by art dealers, collectors, and artists themselves. The Neural Algorithm of Artistic Style designed by German, Swiss, and Belgian scientists allows users to apply particular art styles to uploaded photographs. Moreover, digital artists create communities where they share their experiences with each other and everyone interested.
Using machine learning tools like image recognition or segmentation enables specialists to see the object from different angles. Convolutional neural networks solve visual problems: they turn visual data into conceptual units. And while some means of implementation like face recognition are already well-known to every smartphone user, there are plenty of other ways to use neural networks in the art field.
Deep learning is visual art that advances industry-related research exponentially. Notably, convolutional neural networks apply to a wide range of tasks – from “simple” object detection and recognition to artwork authorship attribution to actual content creation. AI algorithms may soon replace human art experts and even become the hottest trend in the high-end art market.