In-Memory Analytics with Apache Arrow: Perform fast and efficient data analytics on both flat and hierarchical structured data

by Matthew Topol, Wes McKinney (Foreword)

Process tabular data and build high-performance query engines on modern CPUs and GPUs using Apache Arrow, a standardized language-independent memory format, for optimal performance

Key FeaturesLearn about Apache Arrow's data types and interoperability with pandas and ParquetWork with Apache Arrow Flight RPC, Compute, and Dataset APIs to produce and consume tabular dataReviewed, contributed, and supported by Dremio, the co-creator of Apache ArrowBook DescriptionApache Arrow is designed to accelerate analytics and allow the exchange of data across big data systems easily.

In-Memory Analytics with Apache Arrow begins with a quick overview of the Apache Arrow format, before moving on to helping you to understand Arrow's versatility and benefits as you walk through a variety of real-world use cases. You'll cover key tasks such as enhancing data science workflows with Arrow, using Arrow and Apache Parquet with Apache Spark and Jupyter for better performance and hassle-free data translation, as well as working with Perspective, an open source interactive graphical and tabular analysis tool for browsers. As you advance, you'll explore the different data interchange and storage formats and become well-versed with the relationships between Arrow, Parquet, Feather, Protobuf, Flatbuffers, JSON, and CSV. In addition to understanding the basic structure of the Arrow Flight and Flight SQL protocols, you'll learn about Dremio's usage of Apache Arrow to enhance SQL analytics and discover how Arrow can be used in web-based browser apps. Finally, you'll get to grips with the upcoming features of Arrow to help you stay ahead of the curve.

By the end of this book, you will have all the building blocks to create useful, efficient, and powerful analytical services and utilities with Apache Arrow.

What you will learnUse Apache Arrow libraries to access data files both locally and in the cloudUnderstand the zero-copy elements of the Apache Arrow formatImprove read performance by memory-mapping files with Apache ArrowProduce or consume Apache Arrow data efficiently using a C APIUse the Apache Arrow Compute APIs to perform complex operationsCreate Arrow Flight servers and clients for transferring data quicklyBuild the Arrow libraries locally and contribute back to the communityWho this book is forThis book is for developers, data analysts, and data scientists looking to explore the capabilities of Apache Arrow from the ground up. This book will also be useful for any engineers who are working on building utilities for data analytics and query engines, or otherwise working with tabular data, regardless of the programming language. Some familiarity with basic concepts of data analysis will help you to get the most out of this book but isn't required. Code examples are provided in the C++, Go, and Python programming languages.

Table of ContentsGetting Started with Apache ArrowWorking with Key Arrow SpecificationsData Science with Apache ArrowFormat and Memory HandlingCrossing the Language Barrier with the Arrow C Data APILeveraging the Arrow Compute APIsUsing the Arrow Datasets APIExploring Apache Arrow Flight RPCPowered By Apache ArrowHow to Leave Your Mark on ArrowFuture Development and Plans

392 pages, Kindle Edition

Published June 24, 2022

Reviews

[Rik Bauwens · Rating 3 out of 5]

Good compilation of how Arrow works internally, how to use it, and how tools are using it. I also liked the future development part. The main benefit is that it contains some information about Arrow that I haven't found elsewhere on the internet. The author clearly knows a lot about Arrow specifically, but also about the data space in general.

What could have been better: many parts of the book are either too light on detail, or not very clearly explained. It reads like it was written in a rush.

Overall, I would recommend picking it up if you're interested in where modern data tools are going.

[Szymon Kulec · Rating 5 out of 5]

5 out of 5, an amazing read.

This is how technical books should be written. I started reading it to understand Arrow and this books makes you sold on it from the first pages. First, it's not about Arrow itself but also covers WHY things are designed this way (batching, CPU & SIMD, GPU ready). Then it provides a lot of comparisons showing that Arrow can bring a lot in regards to performance. This is the crux I believe as if you think about Arrowing your data processing, you are probably in performance area. The last but not least, it shows the connection between single items, creating schema for Arrow and bulking the data in. If you think that the book omits IPC and benefits of shared memory, you're wrong as this is covered as well. The last but not least, Flight with its fully distributed system is there for you.

This book provided a lot of insights that were missing in the initial read of the original doco. It showed WHY, showed the way to make it work with Arrow and your data. Even if you skip the exercises (I did), you'll learn a lot.