Rembrandts and Robots: Using Neural Networks to Explore Authorship in Painting

We use convolutional neural networks (CNNs) to analyze authorship questions surrounding works of representational art. Trained on the works of an artist under study and visually comparable works of other artists, our system can identify forgeries and provide attributions. Our system can also assign classification probabilities within a painting, revealing mixed authorship and identifying regions painted by different hands. We describe and illustrate performance in connection with Rembrandt portraits and van Gogh landscapes. With one interesting exception, our system’s attributions match those of authoritative experts for works whose authorship has been the subject of longstanding controversy. In less than a decade, neural networks have achieved widespread deployment in applications ranging from self-driving cars to real-time language processing to medical diagnostics. The rate and volume level at which new potential uses are announced obscures how few human workers have actually been displaced. For applications whose value is high enough to justify the cost of research and development, the risks of taking human expertise out of the loop may be too great. For applications involving creativity and subjective judgment, the uncertainties may be irreducible. Analysis of artwork represents one such creative domain. Neural networks have been used to categorize art images by movement and style (Lecoutre et al., 2017; Balakrishnan et al., 2017; Tan et al., 2016; Saleh et al., 2015; Bar et al., 2014), to classify paintings by artist rather than style (van Noord et al., 2015), and to extract and analyze artists’ brushstrokes (Li et al., 2012). These techniques have achieved varying degrees of success but few efforts have addressed the more difficult question of attribution and forgery detection. The reason for this stems from some fundamental limitations of neural networks. Organized in a highly interconnected brain-like fashion, they can analyze and recognize patterns in a wide range of complex input  an image, spoken words, stock prices, or weather data, for example. But a neural network must be trained for a given task. Convolutional neural networks (CNNs) have