art.faces - AI sculpts 3D faces from famous paintings

Exploration on face detection and 3D reconstruction using the "Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression" code by Aaron S. JacksonAdrian Bulat, Vasileios Argyriou and Georgios Tzimiropoulos.

We selected 8 Famous Paintings and let the Convolutional Neural Network (CNN) perform a direct regression of a volumetric representation of the 3D facial geometry from a single 2D image.

01 Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg

Mona Lisa

Leonardo da Vinci
c. 1503–06, perhaps continuing until c. 1517
Oil on poplar panel
Subject: Lisa Gherardini
77 cm × 53 cm
Musée du Louvre, Paris

03 Meisje_met_de_parel.jpg

Girl with a Pearl Earring

Meisje met de parel
Johannes Vermeer
c. 1665
Oil on canvas
44.5 cm × 39 cm
Mauritshuis, The Hague, Netherlands

05 Sandro_Botticelli_-_La_nascita_di_Venere_-_Google_Art_Project_-_edited.jpg

The Birth of Venus

Nascita di Venere
Sandro Botticelli,
The Birth of Venus (c. 1484-86).
Tempera on canvas.
172.5 cm × 278.9 cm.
Uffizi, Florence

Las Meninas

Diego Velázquez
Oil on canvas
318 cm × 276 cm
Museo del Prado, Madrid

02 Vincent_van_Gogh_-_Self-Portrait_-_Google_Art_Project.jpg


Vincent van Gogh
September 1889
Oil on canvas
65 × 54 cm
Musée d'Orsay, Paris.
This may have been Van Gogh's last self-portrait.

04 Cavalier_soldier_Hals-1624x.jpg

Laughing Cavalier

Frans Hals
oil on canvas
83 cm × 67.3 cm
Wallace Collection, London

06 Grant_Wood_-_American_Gothic_-_Google_Art_Project.jpg

American Gothic

Grant Wood
Oil on beaverboard
78 cm × 65.3 cm
Art Institute of Chicago

08 The_Nightwatch_by_Rembrandt.jpg

The Night Watch

Rembrandt van Rijn
Oil on canvas
363 cm × 437 cm
Rijksmuseum, Amsterdam

The technology behind: 

Code available on GitHub
Test your images with this demo,

This tech is fantastic and amazing, but same as with DeepFakes, it should be handled with care… Once we can generate accurate 3D representations of anyone with minimal input data, there’s the possibility to use that 3D output as digital identity theft.

As well questioned in this VentureBeat article : 

So what happens when the technology further improves (which it will) and becomes accessible to marketers and brands (which it always does)? Imagine a casting call where a dozen actors are digitally and convincingly superimposed on a stand-in model prior to engaging the actors in real life.

We can imagine a scenario where videos are created using someone’s face (and body?) to make them do whatever they want, without consent.

On the other hand, AI is getting so smart, that in some cases, it doesn’t need a real input to go wild and imagine possible realities.

In this paper by NVIDIA, a generative adversarial network is able to picture imaginary celebrities.

Particularly in the 3D space, we see a huge potential for this kind of technology.