Introduction

At Flixstock in addition to garment to on-model imagery, we need to ensure that the garment appears in its original form without any sort of imperfections, like dust, wrinkles etc. that might have gotten introduced during logistics. There is a common problem of dust clinging to articles such as bags and shoes, though very small, it might reduce user experience when seen on high resolution images we deal with. Imagine that even after spending a fortune staging a product that companies are selling; some persistent dust particles always show up in the high-resolution images that are used at e-commerce sites?

Overview

Recent advances in image segmentation from computer vision-based algorithms to deep learning-based approaches have enabled us to segment out parts in visual input at pixel level accuracy. Mostly image segmentation models employ CNN based encoder to create a feature map that has high semantic information and use it to classify each pixel to which category of segment it belongs. To process segments as small as dust, we employ image segmentation models to identify pixels where dust is present and then we can inpaint them ensuring texture of article remains intact.

Approach

We generate a binary mask to identify pixels where dust is present using a segmentation network. Major challenges of this task are data preparation and ensuring that texture over garment remains intact. Data preparation for semantic segmentation has always been a challenge for humans, since it requires pixel level accuracy. This problem makes it very hard for us to identify small segments such as dust when dealing with high resolution images. To overcome this challenge, we have employed Generative Adversarial Networks for data preparation. GANs have been known for their realistic image modelling. We were able to model the distribution of dust using GANs over an arbitrary mask. This is also known as image translation. By optimizing our data preparation process, we were able to generate lots of data which helped our model to avoid overfitting as well as recognize differences between texture and dust. 

Picture 1
Figure I

Examples     

Dusty bag                                                            Dust-removed bag

        Dusty shoe                                          Dust-removed shoe

  • These denote the dust particles on the articles, which needs to be removed.
  • These denote cleaned area after detection and removal of dust particles.

References

[1] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde Farley,Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets.

In Advances in neural information processing systems, pages 2672-2680, 2014.
[2] Figure I is taken from article: An overview of semantic image segmentation, by Jeremy Jordan.