COLLABORATORE DI RICERCA presso: Dipartimento di Ingegneria "Enzo Ferrari"OSPITE CON ACCESSO AL SERVIZIO VPN presso: Dipartimento di Ingegneria "Enzo Ferrari"
- Sopj: A scalable online provenance join for data integration
[Relazione in Atti di Convegno]
Zhu, Song; S., Email Author; Fiameni, Giuseppe; G., Email Author; Simonini, Giovanni; G., Email Author; Bergamaschi, S.
Data integration is a technique used to combine different sources of data together to provide an unified view among them. MOMIS is an open-source data integration framework developed by the DBGroup1. The goal of our work is to make MOMIS be able to scale-out as the input data sources increase without introducing noticeable performance penalty. In particular, we present a full outer join method capable to efficiently integrate multiple sources at the same time by using data streams and provenance information. To evaluate the scalability of this innovative approach, we developed a join engine employing a distributed data processing framework. Our solution is able to process input data sources in the form of continuous stream, execute the join operation on-the-fly and produce outputs as soon as they are generated. In this way, the join can return partial results before the input streams have been completely received or processed optimizing the entire execution.