Database Internals: A deep-dive into how distributed data systems work

by Alex Petrov

When it comes to choosing, using, and maintaining a database, understanding its internals is essential. But with so many distributed databases and tools available today, it’s often difficult to understand what each one offers and how they differ. With this practical guide, Alex Petrov guides developers through the concepts behind modern database and storage engine internals.

Throughout the book, you’ll explore relevant material gleaned from numerous books, papers, blog posts, and the source code of several open source databases. These resources are listed at the end of parts one and two. You’ll discover that the most significant distinctions among many modern databases reside in subsystems that determine how storage is organized and how data is distributed.

This book examines:

Storage engines: Explore storage classification and taxonomy, and dive into B-Tree-based and immutable log structured storage engines, with differences and use-cases for each
Distributed systems: Learn step-by-step how nodes and processes connect and build complex communication patterns, from UDP to reliable consensus protocols
Database clusters: Discover how to achieve consistent models for replicated data

Genres: Technology, Programming, Computer Science, Technical, Software, Engineering, Nonfiction

376 pages, Paperback

Published November 4, 2019

Reviews

[Sebastian Gebski · Rating 5 out of 5]

One of the best tech books I've read in the last 12 months.
It consists of 2 parts: DB internals & DB distribution internals.

The 1st part is pure gold - one can learn about B*-trees, LSM-trees, differences between locks and latches, memory VS disk optimizations, rebalancing, concurrency models for transactions and much, much more. I can't recall any single book that covers as much deep-level knowledge on these topics.

The 2nd part is less unique - there are other good resources on distributed systems. What I liked was a solid description of Paxos algorithm (including variants: multi- and fast-). A chapter about anti-entropy was very solid as well (it's a pity I didn't have any comparable resource on the topic before I've started working with Cassandra).

No point in extending this review further - it's a great book, just grab it (if you're keen on the topic) - you won't find anything better.

[Bugzmanov · Rating 5 out of 5]

I liked this one a lot.

It complements nicely "Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems" by Martin Kleppmann.
While Kleppmanns' book provides pretty solid overview of data processing landscape, this book goes deeper into implementation details, data structures and algorithms.
It's a bit more dry and more technical, but still it's a relatively easy read.

[Emre Sevinç · Rating 5 out of 5]

“Database Internals: A Deep Dive Into How Distributed Data Systems Work” by Alex Petrov, belongs to a very special category of O’Reilly books such as “Designing Data-Intensive Applications” and “Cassandra: The Definitive Guide“, in the sense that it is a serious deep dive into the most fundamental and challenging aspects of big and distributed data systems that we rely on daily basis.

Today there’s an unprecedented proliferation of distributed database technologies, combined with an ever-growing multitude of cloud computing services offering them, as well as rapid advances in physical storage systems such as NVMe SSDs that force considering different trade-offs and algorithms. A working software developer, system engineer, solution architect, or a CTO can easily be overwhelmed with so many distributed NoSQL, newSQL, time-series, graph, document, key-value, embedded databases in addition to typical, traditional, enterprise RDBMS variants. Luckily, all of these fancy distributed database technologies are built on a limited number of concepts, techniques, and algorithms which are concisely introduced and surveyed in “Database Internals” book.

Read the rest of the review...

[Adrian · Rating 3 out of 5]

Unfortunately i have read this book after Martin Kelppman's Designing Data Intensive apps and probably this is the reason i rated it poorly.

The book starts really good by describing the intetnal systems that are encompassed in any dbms ( Connection Listener layer , Query parser+ optimizer layer , execution layer and of course the storage layer).

I wanted the book to explore more on this subject , how the components are drsigned and how they deal with concurrency etc.

The book then took a deep dive into tree-based data structures ..as in really deep.
I found myself at times wondering why am i reading this.It was way too terse.

Now after this part the book became interesting once again by the time it started tackling distributed transactions,consensus ,replication,byzantine faults, paxos algorhitms.

I was already familiar with all of those which in my humble opinion were tackled slighty better in the first book i mentioned.

All in all a good book , but i wouldn't classify it as complementing Martin Klepmann's

[Mikhail Filatov · Rating 2 out of 5]

The book is a strange mix - the second part is really about distributed data systems and it's ok - while "Designing Data Intensive applications" is better in this part.
The first part contains a lot of descriptions of different implementations of B*-Trees (replace * with any other symbol(s)) - most of them unreadable.

[Bilal · Rating 3 out of 5]

The book is divided into two parts: The first part deals with storage on hard disk and solid state storage but in the context of a singular system; while the second part deals with distributed systems. In this sense it differs from most other books on distributed storage that typically do not discuss the topics in the first part of this book.

I found the book informative, but not very effective in building a solid understanding of concepts. I felt the author jumps from idea to (related) idea too frequently in the manner of short paragraphs, and in so doing doesn't see an idea through to the end in enough detail for it to be learned properly. Perhaps the first part was better presented; the second was not.

[Povilas Balzaravičius · Rating 4 out of 5]

Good and interesting content. But some chapters are scattered with missing transitioning between topics or algorithms. Some other parts are developed well. Diagrams are missing where I have expected better explanation or are present for obvious things. The writing is 3/5, but the content is 5. So it is 4/5. I'm glad I've read the book and definitely will get back to some chapters to refresh some details.

[Łukasz Słonina · Rating 3 out of 5]

I like this book for the content, if you would like to know more about databases and distributed systems plus get long list of further reads then go for this book. What I don't like is actually that this material does not read like book (e.g. DDIA), it's more like compendium of algorithms, data structures and theories. Some of the algorithms could be better presented (more diagrams).

[Marcin Golenia · Rating 5 out of 5]

The book is organized into 2 parts and let me review the book in two parts.

1. Storage Engines (5/5)
I didn't expect that we will get so much into internals. Hats off! The knowledge of the author is extensive and nicely presented by the book text and illustrations. Everything you need to know - hardware (HDD, SSD) and the relation between them and data storage algorithms, transactions, slotted pages, b-trees variants, LSM. Great stuff.

2. Distributed Systems (4/5)
There is a gap between complexity level in introduction of topics and the actual "meat" that is described. It starts nice and easy then boom! "Percolator transaction execution", "Calvin" and "Spanner". Hard stuff and this part may keep you reading the page few times so you can understand it. I failed to do so in few places but I am fine with this - The author provides many references so you can learn the intermediate knowledge from there.

I know I will forget some of the advanced things that Alex tried to explain to me, but I will know where should I look for it ;) All in all the book will help you to build nice end2end understanding of how the database really work in both places - your computer and a big cluster.

[Ahmad hosseini · Rating 3 out of 5]

I part 1, book explains internal database structure in details and examines its parts like storage engine very well.
“The storage engine (or database engine) is a software component of a database management system responsible for storing, retrieving, and managing data in memory and on disk, designed to capture a persistent, long-term memory of each node.”
Part 2 explains distributed systems characteristics in general and examine some specific topics related to distributed databases.
Book also introduces good sources for further reading

[Bartosz Sypytkowski · Rating 5 out of 5]

This book comes along nicely together with "Designing Data-Intensive Applications" by Martin Klepmann: they both focus on core, fundamental concepts of persistent, distributed systems, providing wide variety of known algorithms and protocols for common problems in that area, including rationale behind each one, which helps to build intuition about their trade offs. It's also full of references for anyone, who wants to continue more in-depth exploration for a given topic.

[Leonid · Rating 5 out of 5]

I've found some similarities between this one and my favourite "wild board book" ("Designing Data-Intensive Applications" by Martin Kleppmann), though in the first part, dedicated to DB internals, it goes much more in-depth, and there are tons of awesome stuff there. Second part, dedicated to Distributed systems, is a bit less unique, but still very good, and worth reading.
So, wholeheartedly recommend to anyone working with DBs, or just distributed systems in general.

[Elijah Oyekunle · Rating 5 out of 5]

Love it!

[Eric · Rating 3 out of 5]

This book really feels like two incomplete books in one. The first half of the book focuses on database internals, file formats, caching strategies etc. The second half of the book switches gears and dives into the components (algorithms, and strategies) used by distributed systems.

The problem is that there is nothing to tie the first and second parts of the book together. You could be reading entirely different books. The second issue is that even within each part, you are presented with a lot of great information, but there is no guidance (imo) on how you may want to logically put the pieces together to build a complete system.

True to the book's title, it does a great job of exploring the internals of database systems. If you are looking for how a specific component is built (ie you want to learn more about RAFT consensus), this is a great resource. If you are looking for a book that ties these concepts together however, I would suggest looking elsewhere.

[Vishwanath · Rating 2 out of 5]

Informative but would have preferred more examples with practical scenarios. No code and this all mostly conceptual. Some good references to papers for subsequent reading. The first part of the book deals primarily with storage and covers an in-depth discussion of b-trees and types. The second half is focused on distributed systems and has useful sections on consensus protocols. Concepts like "2-phase commits" are explained well with figures. However, the lack of practical examples/code and overall dry subject matter made this a laborious read. Good book to reference theoretical concepts.

[Lauro Caetano · Rating 5 out of 5]

Excellent book! It goes a bit in the direction of what Design Data-intensive applications goes when it talks about distributed systems, dist transactions and so on.
But this book goes some steps further: explaining how the db represents data internally, and also explaining distributed systems algorithms.

Excellent read!

[Ivan · Rating 4 out of 5]

Детальное, но без больших подробностей (это искупается большим числом ссылок и рекомендаций для дальнейшего изучения) описание структур и алгоритмов для современных систем.

[Verisimilitude · Rating 2 out of 5]

Great, content-wise. Although appeared to have lost the flow/transitions in describing concepts.
Had to continuously take notes and cross-refer them so as to prevent myself from loosing the flow

[Ricardo Hernández · Rating 5 out of 5]

A book that I really appreciated to expand and have a 360 degree refresh on Database essentials. The progression of the book is built in an organic way, parting from basic concepts at low level implementation to modern distributed challenges. This helps you build comprehension naturally, in constructs with some other technical books that jump from topic to topic without any respect to cognitive challenges.

Totally recommended to get a solid understanding on databases to help solve contemporary problems.

[Manav Rathod · Rating 4 out of 5]

Solid coverage of concepts related to database technologies. I felt some concepts weren't super intriguing and I didn't think connected well. The distributed systems portion was not as well-done. I did learn some core ideas, but I don't feel as well-read on them compared to the database concepts. I also think a real-world section of what services are commonly used in the real world to handle these problems as well as their trade-offs would be a nice addition.

[Idir Yacine · Rating 4 out of 5]

I believe this book is meant specifically for database admins and engineers (low level stuff) , As such if you are trying to get a high level understanding (practical code examples ,real life implementation examples) to get the job done this is not the book for it . All said the book is still pretty decent and worth the read .

[Lưu Khánh · Rating 3 out of 5]

I had a mixed feeling about this book. The first part deals with the concepts behind DBMS, storage engines, B-tree, disk storage,... which is very informative. The second part is really about distributed data system. It's not like that I was reading a book, but like I was reading theories or slides from lectures at university. Again, the book is very informative but not easy to read.

[Mikhail Turnovskiy · Rating 4 out of 5]

This book seams unbalanced for me.
Too complex for technical overview, too shallow for deep dive.
But it is a good starting point to learn about different areas of database and distributed system design.

[Lu Pan · Rating 4 out of 5]

The first part of the book is better. The part two which focuses on distributed system is less than a deep dive and I agree with other reviewer that Data Intensive is a better book on distributed system. But this is still a great book on single host database!

[Aboullaite Mohammed · Rating 5 out of 5]

I liked very much reading this book! It complements nicely "Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems" Book by Martin Kleppmann. I highly recommend!

[Edmund · Rating 3 out of 5]

I finally managed to complete this book on my third reading. While the book has a lot of valuable information in it, I personally think it tries to do too much in too little space. This makes the writing style quite terse and difficult to get through.

[Matt Eland · Rating 5 out of 5]

This book nearly made me cry. Such depth and detail, but it's a good look into the problems a database must overcome and the various algorithms that enable reliable, high performance, distributed database systems.

[Ken · Rating 5 out of 5]

Probably the densest technical book I have read which featured very little (if any) code. That being said, my admiration for databases (and understanding) that we take for granted to simply 'work' has increased.

[Arun Mahendrakar · Rating 4 out of 5]

* goes quite in-depth; I took what I wanted
* types of database systems - HTAP was interesting to read about
* compares row- and column-store databases
* B-Tree, LSM, Skiplist, Hybrid Gossip protocol are some things that are discussed in detail

[SolidM · Rating 4 out of 5]

Le livre est partagé en 2 parties : algorithmiques et distributed systems.
Un must-read pour approfondir ses connaissances sur les DB.