Sketch a Face to Crack the Case
DeepFaceDrawing is a deep learning framework that generates synthetic face images from rough sketches.
DeepFaceDrawing (DFD) is a new method developed by Chen et al. that can turn a Pollock into a da Vinci. Supplied with a sketch of a human face, DFD will generate a realistic synthetic face image.
A very rough drawing is sufficient to get started; the DFD method can use limited information to guide the generation of a plausible face image. Adding additional detail further refines the result.
Training data for DFD was retrieved from the CelebAMask-HQ dataset, consisting of 17,000 images. To approximate hand-drawn sketches for pairing with the images, the authors applied the combination of a Photoshop filter and a sketch simplification algorithm.
DFD consists of three main stages, CE (Component Embedding), FM (Feature Mapping), and IS (Image Synthesis). The CE module learns to describe five features (“left eye," “right eye," “mouth," “nose," and “remainder”) in the sketches using an autoencoder artificial neural network. FM and IS work together to map feature vectors to realistic images, and to generate a final synthetic face image.
With relatively modest hardware (Intel i7-7700 CPU, 16GB RAM and an NVIDIA GTX 1080Ti GPU) the authors were able to achieve real-time feedback from sketches.
Previous solutions to this sketch-to-image problem do exist (e.g. Pix2pixHD, Lines2FacePhoto), but tend to overfit the input sketches, necessitating a high level of talent in creating the sketch. Feeding these tools with insufficiently detailed sketches is more likely to supply you with fuel for horrific nightmares than realistic results.
In addition to just being lots of fun to play with, DFD has a clear potential for use in criminal investigation applications. Imagine a crime victim being given the power to sketch and refine realistic representations of the image in their mind in real-time.