Stream Processing With Dependency-Guided Synchronization

Real-time data processing applications with low latency requirements have led to the increasing popularity of stream processing systems. While such systems offer convenient APIs that can be used to achieve data parallelism automatically, they offer limited support for computations which require synchronization between parallel nodes. In this paper we propose \emph{dependency-guided synchronization (DGS)}, an alternative programming model and stream processing API for stateful streaming computations with complex synchronization requirements. In a nutshell, using our API the input is viewed as partially ordered, and the program consists of a set of parallelization constructs which are applied to decompose the partial order and process events independently. Our API maps to an execution model called \emph{synchronization plans} which supports synchronization between parallel nodes. Our evaluation shows that APIs offered by two widely used systems (Flink and Timely Dataflow) cannot suitably expose parallelism in some representative applications. In contrast, DGS enables implementations which scale automatically, the resulting synchronization plans offer throughput improvements when implemented in existing systems, and the programming overhead is small compared to writing sequential code.