Effective data versioning for collaborative data analytics

Huang, Silu

Permalink

https://hdl.handle.net/2142/105669 Copy

Description

Title

Effective data versioning for collaborative data analytics

Author(s)

Huang, Silu

Issue Date

2019-07-11

Director of Research (if dissertation) or Advisor (if thesis)

Parameswaran, Aditya

Doctoral Committee Chair(s)

Parameswaran, Aditya

Committee Member(s)

• Han, Jiawei
• Sinha, Saurabh
• Elmore, Aaron

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

Data Versioning, Collaborative Data Analytics, Partitioning, Data Representation

Abstract

With the massive proliferation of datasets in a variety of sectors, data science teams in these sectors spend vast amounts of time collaboratively constructing, curating, and analyzing these datasets. Versions of datasets are routinely generated during this data science process, via various data processing operations like data transformation and cleaning, feature engineering and normalization, among others. However, no existing systems enable us to effectively store, track, and query these versioned datasets, leading to massive redundancy in versioned data storage and making true collaboration and sharing impossible. In this thesis, we develop solutions for versioned data management for collaborative data analytics. In the first part of this thesis, we extend a relational database to support versioning of structured data. Specifically, we build a system, OrpheusDB, on top of a relational database with a carefully designed data representation and an intelligent partitioning algorithm for fast version control operations. OrpheusDB inherits much of the same benefits of relational databases, while also compactly storing, keeping track of, and recreating versions on demand. However, OrpheusDB implicitly makes a few assumptions, namely that: (a) the SQL assumption: a SQL-like language is the best fit for querying data and versioning information; (b) the structural assumption: the data is in a relational format with a regular structure; (c) the from-scratch assumption: users adopt OrpheusDB from the very beginning of their project and register each data version along with full metadata in the system. In the second part of this thesis, we remove each of these assumptions, one at a time. First, we remove the SQL assumption and propose a generalized query language for querying data along with versioning and provenance information. Second, we remove the structural assumption and develop solutions for compact storage and fast retrieval of arbitrary data representations. Finally, we remove the “from-scratch” assumption, by developing techniques to infer lineage relationships among versions residing in an existing data repository.

Graduation Semester

2019-08

Type of Resource

text

Permalink

http://hdl.handle.net/2142/105669

Copyright and License Information

Copyright 2019 Silu Huang

Owning Collections