Alaska: A Flexible Benchmark for Data Integration Tasks

Data integration is a long-standing interest of the data management community and has many disparate applications, including business, government and Web search. We have recently witnessed impressive results in isolated data integration tasks, such as Entity Resolution, thanks to the increasing availability of benchmarks for training and testing purposes. Unfortunately, such benchmarks often come with their own task definition and it can be difficult to leverage them for more complex pipelines. As a result, evaluating automated pipelines for the entire data integration process is still an elusive goal. In this work, we present the Alaska benchmark, the first real-world dataset to support seamlessly multiple tasks and tasks variants of the data integration pipeline. It consists of a wide and heterogeneous selection of product specifications from different electronics e-commerce websites and providing hundreds of different product properties. Our benchmark comes with profiling meta-data, pre-defined use cases with different characteristics, and an extensive manually curated ground truth. We demonstrate the flexibility of our benchmark by focusing on two crucial data integration tasks, Schema Matching and Entity Resolution, and some of their popular variants. Our benchmark allows us to compare on the same stage a variety of methods that previously were difficult to picture together, and we hope that it can foster the design of more holistic data integration solutions.