Original Image
Sample Transferred Image
Partially Transferred with Gaussian Smoothing
Introduction
In recent years, image style blending or style transfer has gone into public life from theoretical papers in academic. In particular, Russian-based Prisma Labs launched the mobile-phone application Prisma, which transfer the deep learning based style transfer technique into user-friendly filter, gaining millions of users. However, by far, most of the style transfer technique focusing on transferring the entire image. So I thought it might be interesting to dive in and achieve this localized style transfer.
Approach
The naive approach to achive this localized or partial style transfer can be demonstrated through this flow chart below
​Step 1: Using pre-trained model to obtain style image
In this step, I simply used pre-trained image models for fast style transfer. In particular, I modified and tweaked around the loss function in CNN to achieve somewhat more desirable results. I also modified the program to train my own models so it generates style model that fits my need.
​Step 2: Using pre-trained model to obtain style image
​Step 2: Using pre-trained model to obtain style image
In this step, I found a model using markov random field recurrent neural network trained on VGG to conduct semantic segmentation, which generates labeled object results
​Step 3: mask boundary selection, smoothing and image overlay
I wrote a python program to complete this task. The input should be original image, labled image generated by semantic segmentation programs, and desired style images. The program first transfer the colored label mask into binary mask image and then extract the object pixels label and background pixels label to achieve separation. A note is if the mask image has multiple objects, it will be further processed by a finding connected component algorithm to extract different object masks, which lay foundations for future multiple objects style transfer. Another note is that dilation and erosion technique is used to smooth out the binary mask.
​
Then a boundary is extracted by using skimage package functions finding contours. I have to further give the boundary some padding to give it additional layer for further smoothing works.
​
Then we simply overlay the original pixels from objects or backgrounds on style images to achieve this naive partial style transfer
Program will further process image by using either gaussian smoothing or markov random field blending to smooth out the boundary pixels.
​
Finally, I constructed a pipeline scripts to achieve one step style transfer, semantic segmentation, mask processing and localize transfer
​
Markov Random Field and Gaussian Smoothing comparison
From below comparison, MRFB apparently achieve smoother boundary result
notice the horse feet area
​Fully Transferred Image
Markov Random Field Blending
​Fully Transferred Image
Markov Random Field Blending
Benchmark naive approach image
Gaussian Smoothing
Benchmark naive approach image
Gaussian Smoothing
More Explorations
I also tried to play around the videos. I used prorgram to directly decompose a short video into sequence images and then style transfer it(FFMPEG). The original thoughts was to combine this video decomposition and reassembling to videos on partially transferred images, but I am running out of times. You can view a sample video transfer and more algorithm details in my ppt.
​