Data-driven solutions for the investment industry require event-based backend systems to process high-volume financial data feeds with low latency, high throughput, and guaranteed delivery modes. At vwd we process an average of 18 billion incoming event notifications from 500+ data sources for 30 million symbols per day and peak rates of 1+ million notifications per second using custom-built platforms that keep audit logs of every event. We currently assess modern open source event-processing platforms such as Kafka, NATS, Redis, Flink or Storm for the use in our ticker plant to reduce the maintenance effort for cross-cutting concerns and leverage hybrid deployment models. For comparability and repeatability we benchmark candidates with a standardized workload we derived from our real data feeds. We have enhanced an existing light-weight open source benchmarking tool in its processing, logging, and reporting capabilities to cope with our workloads. The resulting tool wrench can simulate workloads or replay snapshots in volume and dynamics like those we process in our ticker plant. We provide the tool as open source. As part of ongoing work we contribute details on (a) our workload and requirements for benchmarking candidate platforms for financial feed processing; (b) the current state of the tool wrench.
翻译:投资业的数据驱动解决方案需要基于事件的后端系统,以低延迟、高吞吐量和有保证的交付模式处理大量金融数据。 在进行处理时,我们平均处理来自500+数据源的180亿次事件通知,每天3 000万个符号,高峰率为每秒100万个通知,使用定制平台对每件事件进行审计日志。我们目前评估的是现代开放源事件处理平台,如Kafka、NATS、Redis、Flink或Storm等,供我们的圆心厂使用,以减少对交叉问题和混合部署模式的维护努力。为了可比性和重复性,我们用我们从实际数据输入中得出的标准化工作量来衡量候选人。我们加强了现有的轻量开放源基准工具的处理、记录和报告能力,以应对我们的工作量。由此产生的工具扳手可以模拟工作量或像我们在滚心厂中处理的那样在数量和动态上重新播放快照。我们提供了工具作为开放源。作为目前工作的一部分,我们提供以下方面的细节:(a)我们为财务供货工具处理基准候选平台的工作量和要求的现状。 (b)