Joining Forces for an Arrow-Native Future

Joint Post from Wes McKinney and Josh PattersonAllow us to reintroduce ourselves

Too often people say "let���s do something together" in passing, anddon���t. There's the occasional inter-project collaboration, but rarely willpeople take that next step. There are countless reasons why this happens, andaligning goals is challenging to say the least. But after spending the lastseveral years working separately on related problems in the data ecosystem, werealized our best hope to make lasting progress was to build a stronger,unified foundation. We needed to do something radically different.

A Brief History

Wes helped start the Apache Arrow project in2015, and since then has continued to build a developer community to achieveArrow���s dual goals. The first goal is to be an efficient, language-independentopen standard for columnar data interchange. The second goal is to be aportable, high-performance computing foundation for doing analytics on thatcolumnar data. To pursue these goals, Wes formed UrsaLabs in 2018 and UrsaComputing in 2020.

In parallel, Josh and colleagues at NVIDIAforesaw the potential of GPUs to accelerate analytics workloads. In 2017, theycreated the GPU Open AnalyticsInitiativeand later RAPIDS, which has demonstrated the potential of acceleratedhigh-performance columnar analytics. Josh and the cuDF developers collaboratedextensively with BlazingSQL to bringGPU-accelerated Arrow analytics not only to the Python community, but to modernSQL workloads as well.

Over the last 5 years, Arrow has been rapidly adopted as the gold standard fortabular data interchange across the data warehousing and data scienceecosystems, bringing massive performance and efficiency improvements to manyuse cases. Arrow is also taking Flight (punintended)as a replacement for slow database access protocols like ODBC and JDBC. Theseorganizations worked across numerous projects, but individually, each onlyaddressed some of the community's needs.

United Foundation

The next stage of growth is to see Arrow adopted not only as the standard forfast data movement but also as the native format for cost-effective analyticalcomputing. We envision a ubiquitous, hardware-optimized foundation thatsimplifies and accelerates data analytics workloads across programminglanguages.

Today, we are launching a new company, Voltron Data,that reflects this unified vision. The Ursa Computing and BlazingSQL teamstogether with pioneers of RAPIDS and other open-source projects have joinedforces to form Voltron Data. Additionally, Ursa Labs is now VoltronLabs, and it will continue to work for thebenefit of the open-source ecosystem around Apache Arrow. Josh and Wes are nowVoltron Data���s CEO and CTO, respectively. You���ll see us doing even more work inthe Arrow community than we have in the past, and we look forward to increasingArrow���s footprint in the world. Together we are unifying our collectiveexpertise in performance, portability, and programmability to build morebridges across the data ecosystem to improve the tools you know and love.

We look forward to sharing more about Voltron Data in the coming months. In themeantime, we have many open roles and arelooking for talented software engineers around the globe to further ourmission. Join us!