Rapidly growing data volumes have led many developers to abandon traditional relational databases in favor of distributed key/values stores and map/reduce programs. While these alternatives often provide trivial scalability, they lack many of the benefits of high-level declarative languages such as optimization and data-independence. Instead, we propose extending the the relational model with scale independence, a new type of data independence, that ensures consistent performance for all queries in an application, independent of the data size. Our implementation, PIQL, provides a scale independent relational system on top of existing distributed key/values stores by changing the objective function for optimization and automatically selecting and maintaining required indexes and materialized views. The PIQL system also integrates with the Scala compiler to provide language integrated schema specification and a LINQ-like query language.
SCADS is a research prototype key/value store written in Scala. Built using BDB-JE, its design is focused on modularity and easy deployment for running experiments. The system has served as the storage system for the director (FAST'11), PIQL execution engine, RAD Lab Stack, and the multi-datacenter concurrency control project.
The RAD lab's moon-shot vision statement was to enable a single person to design, analyze, deploy and operate the next multi-million user website in only a single weekend. I led the effort to integrate the various projects of the lab, including SCADS, PIQL, Mesos, the director, Spark, and deploylib into a single unified demo stack. At the at the end of project celebration on February 24th 2011 we demonstrated three web applications written by undergrads, including one completed the previous weekend. Using the stack we scaled them to 300+ EC2 instances over the course of an afternoon.
Deploylib is a scala DSL for deploying experiments and other software on clusters of machines, including Amazon's EC2. It was used to run the experiments for the PIQL and director papers as well as the RAD Lab Final Demo. It provides developers with the following constructs: