报告主题: Neural Architecture Search and Beyond

报告简介:

神经网络结构搜索(NAS)是一种自动化设计人工神经网络的技术。由于NAS能设计出与手工设计神经网络结构相当甚至优于手工设计结构的网络,而成为近两年深度学习社区的研究热点。来自Google的科学家Barret Zoph,ICCV2019上做了《Neural Architecture Search and Beyond》的报告,讲述了Google在NAS方面的最新研究进展。

嘉宾介绍:

Barret Zoph目前是谷歌大脑团队的高级研究科学家。之前,在信息科学研究所与Kevin Knight教授和Daniel Marcu教授一起研究与神经网络机器翻译相关的课题。

下载链接: https://neuralarchitects.org/slides/zoph-slides.pdf

成为VIP会员查看完整内容
0
20

相关内容

Barret Zoph目前是谷歌大脑团队的高级研究科学家。之前,在信息科学研究所与Kevin Knight教授和Daniel Marcu教授一起研究与神经网络机器翻译相关的课题。

题目: A Survey of the Recent Architectures of Deep Convolutional Neural Networks

摘要:

深度卷积神经网络(CNNs)是一种特殊类型的神经网络,在计算机视觉和图像处理等领域的多项竞赛中均有出色的表现。CNN有趣的应用领域包括图像分类与分割、目标检测、视频处理、自然语言处理、语音识别等。深度卷积神经网络强大的学习能力很大程度上是由于它使用了多个特征提取阶段,可以从数据中自动学习表示。大量数据的可用性和硬件技术的改进加速了CNNs的研究,最近出现了非常有趣的深度卷积神经网络架构。事实上,人们已经探索了几个有趣的想法来促进CNNs的发展,比如使用不同的激活和丢失函数、参数优化、正则化和架构创新。然而,深度卷积神经网络的代表性能力的主要提升是通过架构上的创新实现的。特别是利用空间和信道信息、建筑的深度和宽度以及多路径信息处理的思想得到了广泛的关注。同样,使用一组层作为结构单元的想法也越来越流行。因此,本次调查的重点是最近报道的深度CNN架构的内在分类,因此,将CNN架构的最新创新分为七个不同的类别。这七个类别分别基于空间开发、深度、多路径、宽度、特征图开发、通道提升和注意力。对CNN的组成部分、当前CNN面临的挑战和应用进行了初步的了解。

成为VIP会员查看完整内容
0
37

Current state-of-the-art convolutional architectures for object detection are manually designed. Here we aim to learn a better architecture of feature pyramid network for object detection. We adopt Neural Architecture Search and discover a new feature pyramid architecture in a novel scalable search space covering all cross-scale connections. The discovered architecture, named NAS-FPN, consists of a combination of top-down and bottom-up connections to fuse features across scales. NAS-FPN, combined with various backbone models in the RetinaNet framework, achieves better accuracy and latency tradeoff compared to state-of-the-art object detection models. NAS-FPN improves mobile detection accuracy by 2 AP compared to state-of-the-art SSDLite with MobileNetV2 model in [32] and achieves 48.3 AP which surpasses Mask R-CNN [10] detection accuracy with less computation time.

0
6
下载
预览

Deep Convolutional Neural Networks (CNNs) are a special type of Neural Networks, which have shown state-of-the-art results on various competitive benchmarks. The powerful learning ability of deep CNN is largely achieved with the use of multiple non-linear feature extraction stages that can automatically learn hierarchical representation from the data. Availability of a large amount of data and improvements in the hardware processing units have accelerated the research in CNNs and recently very interesting deep CNN architectures are reported. The recent race in deep CNN architectures for achieving high performance on the challenging benchmarks has shown that the innovative architectural ideas, as well as parameter optimization, can improve the CNN performance on various vision-related tasks. In this regard, different ideas in the CNN design have been explored such as use of different activation and loss functions, parameter optimization, regularization, and restructuring of processing units. However, the major improvement in representational capacity is achieved by the restructuring of the processing units. Especially, the idea of using a block as a structural unit instead of a layer is gaining substantial appreciation. This survey thus focuses on the intrinsic taxonomy present in the recently reported CNN architectures and consequently, classifies the recent innovations in CNN architectures into seven different categories. These seven categories are based on spatial exploitation, depth, multi-path, width, feature map exploitation, channel boosting and attention. Additionally, it covers the elementary understanding of the CNN components and sheds light on the current challenges and applications of CNNs.

0
36
下载
预览

Deep Learning has enabled remarkable progress over the last years on a variety of tasks, such as image recognition, speech recognition, and machine translation. One crucial aspect for this progress are novel neural architectures. Currently employed architectures have mostly been developed manually by human experts, which is a time-consuming and error-prone process. Because of this, there is growing interest in automated neural architecture search methods. We provide an overview of existing work in this field of research and categorize them according to three dimensions: search space, search strategy, and performance estimation strategy.

0
10
下载
预览

Designing convolutional neural networks (CNN) models for mobile devices is challenging because mobile models need to be small and fast, yet still accurate. Although significant effort has been dedicated to design and improve mobile models on all three dimensions, it is challenging to manually balance these trade-offs when there are so many architectural possibilities to consider. In this paper, we propose an automated neural architecture search approach for designing resource-constrained mobile CNN models. We propose to explicitly incorporate latency information into the main objective so that the search can identify a model that achieves a good trade-off between accuracy and latency. Unlike in previous work, where mobile latency is considered via another, often inaccurate proxy (e.g., FLOPS), in our experiments, we directly measure real-world inference latency by executing the model on a particular platform, e.g., Pixel phones. To further strike the right balance between flexibility and search space size, we propose a novel factorized hierarchical search space that permits layer diversity throughout the network. Experimental results show that our approach consistently outperforms state-of-the-art mobile CNN models across multiple vision tasks. On the ImageNet classification task, our model achieves 74.0% top-1 accuracy with 76ms latency on a Pixel phone, which is 1.5x faster than MobileNetV2 (Sandler et al. 2018) and 2.4x faster than NASNet (Zoph et al. 2018) with the same top-1 accuracy. On the COCO object detection task, our model family achieves both higher mAP quality and lower latency than MobileNets.

0
4
下载
预览

This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques.

0
3
下载
预览
小贴士
相关VIP内容
专知会员服务
34+阅读 · 2020年3月19日
相关论文
Golnaz Ghiasi,Tsung-Yi Lin,Ruoming Pang,Quoc V. Le
6+阅读 · 2019年4月16日
A Survey of the Recent Architectures of Deep Convolutional Neural Networks
Asifullah Khan,Anabia Sohail,Umme Zahoora,Aqsa Saeed Qureshi
36+阅读 · 2019年1月17日
IRLAS: Inverse Reinforcement Learning for Architecture Search
Minghao Guo,Zhao Zhong,Wei Wu,Dahua Lin,Junjie Yan
4+阅读 · 2018年12月14日
Neural Architecture Optimization
Renqian Luo,Fei Tian,Tao Qin,Enhong Chen,Tie-Yan Liu
7+阅读 · 2018年9月5日
Thomas Elsken,Jan Hendrik Metzen,Frank Hutter
10+阅读 · 2018年9月5日
Mingxing Tan,Bo Chen,Ruoming Pang,Vijay Vasudevan,Quoc V. Le
4+阅读 · 2018年7月31日
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
Ningning Ma,Xiangyu Zhang,Hai-Tao Zheng,Jian Sun
3+阅读 · 2018年7月30日
Hierarchical Graph Representation Learning with Differentiable Pooling
Rex Ying,Jiaxuan You,Christopher Morris,Xiang Ren,William L. Hamilton,Jure Leskovec
7+阅读 · 2018年6月26日
Hanxiao Liu,Karen Simonyan,Yiming Yang
3+阅读 · 2018年6月24日
Parth Shah,Vishvajit Bakarola,Supriya Pati
20+阅读 · 2018年1月17日
Top