Abstract

Content-aware image completion or in-painting is a fundamental tool for the correction of defects or removal of objects in images. We propose a non-parametric in-painting algorithm that enforces both structural and aesthetic (style) consistency within the resulting image. Our contributions are two-fold: 1) we explicitly disentangle image structure and style during patch search and selection to ensure a visually consistent look and feel within the target image. 2) we perform adaptive stylization of patches to conform the aesthetics of selected patches to the target image, so harmonizing the integration of selected patches into the final composition. We show that explicit consideration of visual style during in-painting delivers excellent qualitative and quantitative results across the varied image styles and content, over the Places2 scene photographic dataset and a challenging new in-painting dataset of artwork derived from BAM!

Dataset

A subset of 1000 digital artworks were sampled from BAM! (Behance Media) dataset! across 8 artistic styles( watercolor; vectorart; 3D; graphite; pen; oil; comic; photo). Each image was re-sized to have longest side of 600px and a random mask defining the ’hole’ for in-painting of side 150-250 pixels positioned at random.

Source Images: Mattes:

Sample results

missing
Comparison with existing works, AIC million image[8], PatchMatch[1], Img Melding[3], and Context Encoder[20]. Existing works produce reasonable results but with excessive blurring or missing key details absent in the image due to inability to identify patches that are coherent in structure and style. Our approach identifies patches from similar structure and style images to hallucinate detail that is not present in the image, and also then restyle the patches further to ensure a visually coherent result.

Addtional results

Addtional results can be found in the supplementary material

Paper

Disentangling Structure and Aesthetics for Style-aware Image Completion
Andrew Gilbert, John Collomosse , Hailin Jin and Brian Price
CVPR 2018







Citation

    @inproceedings{Gilbert:CVPR:2018,
        AUTHOR = "Gilbert, Andrew and Collomosse, John and Jin, Hailin and Price, Brian",
        TITLE = "Disentangling Structure and Aesthetics for Style-aware Image Completion",
    	BOOKTITLE = "2018 Conference on Computer Vision and Pattern Recognition (CVPR'18)",
        YEAR = "2018",
    }

Acknowledgments

This work was supported by a charitable donation from Adobe Systems, and also in part by the InnovateUK TotalCapture project grant agreement 102685 and the Visual Media project (EU H2020 grant 687800).