We develop a compositional approach for automatic and symbolic differentiation based on categorical constructions in functional analysis where derivatives are linear functions on abstract vectors rather than being limited to scalars, vectors, matrices or tensors represented as multi-dimensional arrays. We show that both symbolic and automatic differentiation can be performed using a differential calculus for generating linear functions representing Fr\'echet derivatives based on rules for primitive, constant, linear and bilinear functions as well as their sequential and parallel composition. Linear functions are represented in a combinatory domain-specific language. Finally, we provide a calculus for symbolically computing the adjoint of a derivative without using matrices, which are too inefficient to use on high-dimensional spaces. The resulting symbolic representation of a derivative retains the data-parallel operations from the input program. The combination of combinatory differentiation and computing formal adjoints turns out to be behaviorally equivalent to reverse-mode automatic differentiation. In particular, it provides opportunities for optimizations where matrices are too inefficient to represent linear functions.
翻译:我们根据功能分析的绝对结构,制定了一种自动和象征性区别的构成方法,即衍生物是抽象矢量上的线性功能,而不是局限于作为多维阵列代表的标量、矢量、矩阵或压强。我们表明,可以使用一种不同的微分法来进行象征和自动区分,以产生代表Fr\'echet衍生物的线性功能,这种功能基于原始、常数、线性、双线性功能的规则,以及其相继和平行的构成。线性功能以组合式域特定语言表示。最后,我们提供了一种微分法,用于象征性地计算衍生物的连接,而不使用矩阵,因为矩阵效率太低,无法在高维空间使用。由此产生的衍生物象征性表示保留了输入程序的数据单体操作。组合式区分和计算正式连接在行为上等同于反向模式自动区分。特别是,它为在矩阵效率太低、无法代表线性功能的情况下进行优化提供了机会。