Computational optical imaging (COI) systems leverage optical coding elements (CE) in their setups to encode a high-dimensional scene in a single or multiple snapshots and decode it by using computational algorithms. The performance of COI systems highly depends on the design of its main components: the CE pattern and the computational method used to perform a given task. Conventional approaches rely on random patterns or analytical designs to set the distribution of the CE. However, the available data and algorithm capabilities of deep neural networks (DNNs) have opened a new horizon in CE data-driven designs that jointly consider the optical encoder and computational decoder. Specifically, by modeling the COI measurements through a fully differentiable image formation model that considers the physics-based propagation of light and its interaction with the CEs, the parameters that define the CE and the computational decoder can be optimized in an end-to-end (E2E) manner. Moreover, by optimizing just CEs in the same framework, inference tasks can be performed from pure optics. This work surveys the recent advances on CE data-driven design and provides guidelines on how to parametrize different optical elements to include them in the E2E framework. Since the E2E framework can handle different inference applications by changing the loss function and the DNN, we present low-level tasks such as spectral imaging reconstruction or high-level tasks such as pose estimation with privacy preserving enhanced by using optimal task-based optical architectures. Finally, we illustrate classification and 3D object recognition applications performed at the speed of the light using all-optics DNN.
翻译:光学成像(COI)系统在设置中利用光学编码要素(CE)来将高维场景编码成一个单一或多个快照,并通过计算算法解码解码。COI系统的性能在很大程度上取决于其主要组成部分的设计:CE模式和用于执行特定任务的计算方法。常规方法依靠随机模式或分析设计来确定CE的分布。然而,深神经网络(DNN)的现有数据和算法能力在CE数据驱动设计中打开了新的视野,共同考虑光学编码器和计算解码器。具体来说,通过一个完全不同的图像形成模型来模拟COI的测量,该模型考虑到基于物理的光的传播及其与CEE的相互作用,确定CE和计算解码器的参数可以以端对端(E2E2E)的方式优化。此外,通过利用纯光学和计算,从纯光学应用来完成推断任务。