Today, Pivotal open sourced HAWQ and MADlib, contributing them to the Apache Software Foundation (ASF) and the global Apache Hadoop community. Both projects are now in official incubating status with the ASF. You can find the HAWQ project page here, and MADLib here.
What this means for Hadoop, its users, and the larger community is that there is now a full featured, SQL standards compliant, battle tested and proven interactive SQL engine purpose built for demanding, analytical workloads and business transformation available in open source.
Data is, increasingly, one of the most valuable assets a company can possess. As technologists, those companies we admire and consider to be a gold standard in working with data (Uber, Netflix, etc) have built data infrastructure for themselves that integrates directly with their applications, their users and their business. With the contribution of HAWQ and MADlib, we hope that we’ve brought some of the most important missing building blocks into the open so that all can benefit.
It also means that these same businesses can gain an enormous head start on solving real business problems by leveraging the work of hundreds of commercial, academic and open source developers that formed the basis for the MADlib framework. With an open source library of over 30 principle in-database, scalable algorithms, industries of all types can advance their machine learning agendas.
Both HAWQ and MADlib represent more than a decade of R&D that has taken place at Pivotal and, before that, Greenplum. HAWQ incorporates the SQL processor and relational query engine of the Greenplum Database. What started as Greenplum on Hadoop has evolved significantly to a system recast in terms of Hadoop.
Similarly, MADlib grew out of a collaboration between researchers at UC Berkeley, University of Wisconsin, University of Florida and engineers and computer scientists at Pivotal (formerly EMC/Greenplum). Designed for in-database analytics, MADlib relies on the massively parallel processing power of Greenplum Database and HAWQ.
Developed with and proven in some of the most demanding environments, across a wide variety of industries, we feel that the combination of these two technologies enriches the Hadoop ecosystem and can help accelerate adoption and success of the platform.
So why move them from Pivotals engineering to the open governance of the Apache Software Foundation? The rationale lies in the fact that over the last decade, the database industry has seen radical transformation. To give but a few examples:
These industry level shifts have lead to the inevitability of Hadoop as the fundamental substrate of new generation data warehousing. As the technology is finding solid ground in more and more enterprises, the demand is accelerating, and the need for these tools is bigger than Pivotal, or Pivotals customers.
We feel that by contributing HAWQ and MADlib to the ASF, making them bigger than Pivotal, and continuing to integrate them deeply into the Hadoop ecosystem is a first big step toward building not only a Hadoop Native SQL engine, but ultimately an entire Hadoop Native, data center-class, high performance analytic database infrastructure.
We still have work to do to realize such a vision. Today is but the first big step. We hope youll join us on this journey.
Editors Note: Apache, Apache Hadoop, and Hadoop are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.