Redistributing and Re-Stylizing Features for Training a Fast Photorealistic Stylizer

Style transfer studies can be categorized into two types - artistic and photorealistic. The high-speed transfer has been well-studied for artistic styles but remains challenging for photorealistic styles. To guarantee semantic accuracy and style faithfulness, prior photorealistic style transfer techniques often rely on intensive feature matching, hierarchical stylization, and complex auxiliary smoothing. Such high design complexity severely limits the space of transfer speed improvement. In this paper, we propose to accelerate the transfer through a single-level stylization without complex auxiliary smoothing. We design a two-stage 'stylization and re-stylization' training pipeline to enhance style faithfulness. The stylization/re-stylization stage consists of two core steps: feature aggregation and redistribution. A new type of layers, Feature Aggregation (FA) layers, is proposed to gradually aggregate multi-scale style features into content features at each spatial location. A Spatially coherent Content-style Preserving (SCP) loss at feature map level is then used to preserve semantic accuracy. The SCP loss provides effective guidance on redistributing the aggregated features between locations to enforce spatial coherence of style-sensitive content semantic. Experimental results show that compared to previous competitive methods, our method reduces at least 72% run time while achieving better image synthesis quality based on both subjective and objective evaluation metrics. Ablation studies validate the major contribution of our proposed SCP loss and re-stylization to the quality of our synthesized images.