This work addresses the problem of semantic foggy scene understanding (SFSU). Although extensive research has been performed on image dehazing and on semantic scene understanding with clear-weather images, little attention has been paid to SFSU. Due to the difficulty of collecting and annotating foggy images, we choose to generate synthetic fog on real images that depict clear-weather outdoor scenes, and then leverage these partially synthetic data for SFSU by employing state-of-the-art convolutional neural networks (CNN). In particular, a complete pipeline to add synthetic fog to real, clear-weather images using incomplete depth information is developed. We apply our fog synthesis on the Cityscapes dataset and generate Foggy Cityscapes with 20550 images. SFSU is tackled in two ways: 1) with typical supervised learning, and 2) with a novel type of semi-supervised learning, which combines 1) with an unsupervised supervision transfer from clear-weather images to their synthetic foggy counterparts. In addition, we carefully study the usefulness of image dehazing for SFSU. For evaluation, we present Foggy Driving, a dataset with 101 real-world images depicting foggy driving scenes, which come with ground truth annotations for semantic segmentation and object detection. Extensive experiments show that 1) supervised learning with our synthetic data significantly improves the performance of state-of-the-art CNN for SFSU on Foggy Driving; 2) our semi-supervised learning strategy further improves performance; and 3) image dehazing marginally advances SFSU with our learning strategy. The datasets, models and code are made publicly available.
翻译:这项工作解决了语义模糊的场景理解问题。 尽管已经对图像脱色和用清晰的天文图像对语义场景理解进行了广泛的研究,但我们很少注意SFSU。 由于难以收集和注解雾状图像,我们选择在真实图像上产生合成雾,描绘清晰的天气户外场景,然后将这些部分合成数据用于SFSU, 方法是使用最先进的电动神经神经网络(CNN ) 。 特别是, 开发了一个完整的管道, 利用不完全的深度信息将合成雾添加到真实的、清晰的天气图像中。 我们在市景数据集中应用我们的雾合成合成合成合成, 并用20550个图像生成Foggy城市景色。 SFSU以两种方式处理: 1) 典型的监控学习, 2) 与一种新型的半监控外观学习相结合, 1) 与一种不受监控的监督从清晰的天气代码转换到合成的雾性对等。 此外,我们仔细研究图像脱色图像对SFSU的效用。 为了评估,我们展示了底色图像的深度图像,我们展示了底基的图像,我们展示了一种真实的模型, 演示的演化模型, 展示了一种真实的演化图像, 和演化的演化的演化的演化的演化的图像, 我们展示了一种可以显著的演化的演化的演化的演进的演制, 。