Generative Deep Learning: Teaching Machines To Paint, Write, Compose, and Play

by David Foster

Generative AI is the hottest topic in tech. This practical book teaches machine learning engineers and data scientists how to use TensorFlow and Keras to create impressive generative deep learning models from scratch, including variational autoencoders (VAEs), generative adversarial networks (GANs), Transformers, normalizing flows, energy-based models, and denoising diffusion models. The book starts with the basics of deep learning and progresses to cutting-edge architectures. Through tips and tricks, you'll understand how to make your models learn more efficiently and become more creative. This book also explores the future of generative AI and how individuals and companies can proactively begin to leverage this remarkable new technology to create competitive advantage.

Genres: Artificial Intelligence, Computer Science, Technology, Programming, Textbooks, Science, Nonfiction

707 pages, Kindle Edition

Published June 28, 2022

Reviews

[☘Misericordia☘ ⚡ϟ⚡⛈⚡☁ ❇️❤❣ · Rating 5 out of 5]

Allows the reader to inspect both philosophical and tech considerations of deep learning aimed to generate new-ish content.

[Nerd Sniper · Rating 3 out of 5]

Usually I don't review the books due to my incompetence, but seeing none of the reviews yet, I am obliged to provide one.

Thanks to a tweet, I saw this book and it's kind of both author and O'Reilly to share it. It was quite disappointing to see the Goodfellow's own book (it's not his fault - GAN was still in it's first year then) having a brief mention of the topic. No doubt, a compelling need for a (detailed) book on the topic was pressing on - especially with every new advancement in the field.

This book is not short of flaws: Typos, pretty dull examples (still trying to get over the terminologies used in the GAN one) and then all the sudden throwing you in the hot waters of complex mathematics, history of GAN beginning with Goodfellow's talk at NIPS and ignoring the interesting story of it's discovery, mentioning LSTM in advance of RNNs, using existing Keras-GAN repository for code, etc.

But despite all these cons, it would be premature to write this book off. This is one of the first books on the topic, gives many insights where blogs fail, even a dummy like me can begin generating data due to detailed explanation, and shows author is up-to-date (as we see in nearly all the book but especially ch. 8 and 9).

I think this book can grow to be an excellent one (provided if author really wants to - he was bit laid-back when he cross-referenced to other blogs for detailed explanations while reader expects them here) if next edition would encompass exercises, further expansion of the concepts and maybe some co-author as well. I definitely enjoyed reading it - learnt lot of things which I didn't know before. Looking forward to up the ante by shifting to practical side now (playing with it's notebooks).

[Amir · Rating 4 out of 5]

The author has explained the theoretical aspects of Generative Deep Learning, Amazingly. He used fictions for clarifying complex topic. thats great!
the programming aspects, the book has several issues. The codes are correct, but there is no comment. Author didnt mentioned some aspects of program, because he supposed that we know what he thought, but reader is not in his mind. In other words, if you haven't implemented generative deep learning programs prior reading book, it would be hard for you to use the programs.

[Ravi Teja · Rating 3 out of 5]

3.5/5

It's a great book, awesome explanations but falls short because of some rushed up chapters. The chapters related to computer vision are very good. I'm a Computer Vision Engineer and I loved reading it. Some metaphors used were really good particularly the ones about the variational autoencoders, but some are cheesy and weren't really helpful, I didn't mind these.

The parts related to NLP and attention mechanisms were not good enough. With all due respect to the author, I felt that I could have explained some things better. The intuition part was not covered in these areas as much as when dealing with images. I'd have loved lesser focus on code and more emphasis on the explanation.

And some parts I was not good enough to understand, the ones where he talks of music generation, despite good familiarity with the underlying technologies. So these parts are definitely not for non-practioners. I'd have to do some more reading to understand this music part.

I felt the final few chapters were very rushed. Particularly when NLP started. I'd have loved seeing a fatter book, which is something I don't say often.

After all is said and done, this is still a wonderful book and very entertaining. A must read.

[Ben Eckart · Rating 3 out of 5]

I like this book, though I'm not sure who the target audience is.

The author covers a lot of interesting concepts using fun metaphorical stories-- e.g. explaining LSTMs with a prison guard trying to coordinate groups of prisoners in cells to write him short stories, or explaining VAEs with art collectors tryng to decide where on their gallery wall to place their paintings. These stories are memorable for me, but I wonder if it's because I pretty much already understood the math/theory behind them. I have a feeling that less technical readers would get less from these fun expositions, which is a problem because the author's intent seems to be to target exactly these non-technical readers, as evidenced by the fact that there's so little math in this book.

Regardless, I think the author's included Keras code is really elegant, and it shows how accessible deep learning software can be, even when re-implementing cutting edge research ideas that are only a few years old. For example, David Ha and Jurgen Schmidhuber's amazing 2018 NeurIPS paper "World Models" (https://worldmodels.github.io/) is given an entire chapter and covered in great detail. If you are interested in AI-based art, music, or language generation, this book might be a good start.

[Antonis Maronikolakis · Rating 4 out of 5]

A really useful read, great for whetting your appetite for a wide range of generative topics. From image generation to text and music, this book covers all the main areas of generation, while also touching on some less known areas in the field. The concepts are explained, more or less, clearly and the accompanying parables are very useful in getting the idea across (although some of them are a bit forced, like the apples/oranges one). The book does not go into a lot of details, which is not a bad thing since it allows the reader to more easily pick and choose the topics they want to invest more time in.

Overall, a nice read. Recommend it to everyone who wants an overview of the field, although at times you need to skim through some sections.

[Freeread · Rating 3 out of 5]

ก่อนอ่าน
เป้าหมายการอ่าน : ใช้ AI ให้เกิดประโยชน์ ในสิ่งที่ยังไม่เคยสั่งให้มันทำ และสั่งในสิ่งที่เคยทำให้ดีขึ้น
คำถามที่อยากรู้ : prompt ที่สั่งต้องซับซ้อนหรอถึงจะทำได้ดี simpleๆ ได้ไหม
คาดหวังผลลัพธ์ : ชั้นใช่ AI ได้ดีขึ้น ประหยัดเวลาขึ้น สั่งตรงจุดขึ้น วัดยังไง?
บทที่น่าสนใจ : ส่วนที่ 3 ประยุกต์ใช้

หลังอ่าน
ข้อคิดสำคัญ: อย่าอ่านอะไรที่ไม่มีพื้นฐานพอ มันต้องใช้ความพยายามหาข้อมูลเพิ่มเยอะ ทั้งคำศัพท์ ทั้งสมการคณิตศาสตร์ ทั้งโค้ด
ความเห็น: ยากกว่ามือใหม่จะใช้ได้ ง่ายและกลวงกว่ามือเก่าจะเอาไปใช้ มีตัวอย่างยัดมาเต็ม แบบไม่มีลึก แต่ก็ไม่เห็นภาพ
จะเรียกหนังสือเล่มนี้ว่า: คู่มือแนะนำคราวๆ สำหรับผู้มีความรู้

**** ใช้ chat gpt ช่วย
🧠 พื้นฐานที่ต้องรู้

เชื่อมกับความรู้เดิม: keyword หลายคำเอาไปใช้ได้
1. Deep Learning คืออะไร?
เหมือนเราสร้าง "หอคอยสมอง" มีหลายชั้น
ข้อมูล (รูปภาพ, เสียง, คำพูด) จะไหลขึ้นไปทีละชั้น
แต่ละชั้นจะค่อย ๆ เข้าใจมากขึ้น เช่น

ชั้นล่าง: รู้ว่านี่คือเส้น

ชั้นกลาง: รู้ว่านี่คือหู ตา ปาก

ชั้นบนสุด: รู้ว่านี่คือ "แมว" หรือ "หมา"

ยิ่งชั้นเยอะ = ยิ่งเข้าใจลึก = Deep Learning!

2. Generative คืออะไร?
คือการที่ "สมองหอคอย" สร้างของใหม่ได้เอง!
เช่น

วาดภาพแมวขึ้นมาใหม่

แต่งเพลงที่ไม่เคยมีใครฟังมาก่อน

แต่งนิทานใหม่เกี่ยวกับหมีพูห์

ไม่ใช่แค่ดู/จำ แต่ "คิดและสร้าง" ของเองได้!

🧮 เครื่องมือวิเศษที่ใช้
3. Gradient (เกรเดียนต์)
ทางลาดเอียง → บอกว่าเราควรปีนขึ้นหรือไหลลง เพื่อให้หอคอยฉลาดขึ้น

4. Jacobian (จาโคเบียน)
แผนที่ทางลาด → บอกว่าส่วนไหนเอียงยังไง เวลาข้อมูลเปลี่ยน

5. Exponential (เอ็กซ์โปเนนเชียล)
โตเร็วแบบจรวด → ตัวเลขพุ่งขึ้นแบบ 2, 4, 8, 16, 32, 64...

🎨 วิธีทำให้สมองสร้างของ
6. Embedding
แปลงของทุกอย่าง (รูป เสียง คำพูด) เป็นเม็ดลูกปัด (ตัวเลข) ที่เครื่องเข้าใจได้

7. Convolution (คอนโวลูชัน)
แว่นขยาย → ใช้ส่องหารูปทรงสำคัญในภาพ เช่น เส้น, วงกลม

8. Attention (แอตเทนชัน)
ไฟฉาย → ส่องเฉพาะจุดที่สำคัญ ไม่เสียเวลามองทั้งภาพ

9. Residual
ทางลัด → ถ้ามีอะไรยากไป กระโดดข้ามได้ แต่ยังจำได้ว่าเคยผ่านมา

10. Upsampling
เป่ายางลม → ขยายรูปหรือเสียงให้ใหญ่ขึ้น โดยเติมของใหม่เข้าไป

🏗 โมเดลยอดมนุษย์ที่สร้างของใหม่
11. VAE (Variational Autoencoder)
เครื่องบีบข้อมูลลงกระเป๋า แล้วปล่อยกลับออกมาใหม่ได้

12. GAN (Generative Adversarial Network)
การแข่งกันระหว่าง "นักปลอม" กับ "นักจับผิด" เพื่อสร้างของให้เหมือนจริงที่สุด

13. WGAN
GAN เวอร์ชันใจเย็นขึ้น → แข่งกันแบบค่อย ๆ ให้ดีขึ้น

14. Diffusion
เป่าเมฆ → เอาภาพที่มีแต่ฝุ่นกลับมากลายเป็นภาพชัด ๆ

15. EBM (Energy-Based Model)
ดูพลังงาน → ของจริงต้องใช้พลังงานน้อย ของปลอมพลังงานเยอะ

🛡 เทคนิคช่วยให้เรียนดีขึ้น
16. Regularization
สอนให้คิดแบบ "พอดี" อย่าเลียนแบบทุกอย่าง → ไม่งั้นเด็กจะจำผิด หรือคิดซับซ้อนเกินจริง

17. Batch Normalization
จัดระเบียบเพื่อนในห้องให้ตัวสูงพอ ๆ กัน → จะได้เรียนรู้เร็วขึ้น

🧳 ความลับในกระเป๋า
18. Latent Space
กระเป๋าเก็บความลับ → ซ่อนข้อมูลย่อ ๆ เอาไว้ สร้างของใหม่จากตรงนี้ได้

🧩 จัดหมวดหมู่เข้าใจโลก
19. Taxonomy
จัดเรียงของต่าง ๆ เป็นกลุ่ม ๆ เช่น หมา แมว ปลา คน

🔥 พิเศษ
20. Flamingo
สมองยืดหยุ่น → อ่านภาพได้, อ่านคำได้, คิดได้พร้อมกัน (ทั้งตา+หู)

ตัวอย่าง
อยากทำ AI วาดรูป → ใช้ Diffusion + Attention + Embedding

อยากทำ AI แต่งเพลง → ใช้ Generative + Deep Learning + Latent Space

อยากทำ AI สร้างเกม → ใช้ GAN หรือ WGAN แข่งกันสร้างไอเดียใหม่

อยากทำ AI เขียนนิยาย → ใช้ Flamingo (อ่านหลายแบบได้) + Transformer (attention-based)
****

[Jakob · Rating 5 out of 5]

"Generatives Deep Learning" von David Foster ist ein faszinierendes Buch, das tief in die Welt des generativen Deep Learning eintaucht. Es bietet eine fundierte Untersuchung der Techniken, Modelle und Anwendungen, die in diesem neuen Bereich der künstlichen Intelligenz entwickelt wurden. Programme, welche dies nutzen sind unter anderem Open Ai's Chat GPT oder Google Bards.

Besonders gut gefallen mir die technischen Details und Erklärungen, die das Buch bietet. Foster gelingt es, komplexe Konzepte verständlich zu machen, indem er klare Beispiele und präzise Erklärungen liefert. Die Beispiele sind besonders hilfreich, da sie den Lesern ermöglichen, Deep Learning besser zu verstehen, sei es in klassischen Anwendungen wie der Zahlenerkennung (Bilderkennung durch Inputs) oder in anspruchsvolleren Projekten wie der Musikgenerierung.
Zusätzlich sind die zahlreichen Analogien serh anschaulich, die Foster verwendet, um die komplexen Konzepte zugänglicher zu machen. Diese Analogien helfen dem Leser, abstrakte Ideen besser zu erfassen und erleichtern das Verständnis der zugrunde liegenden Prinzipien.

Allerdings ist es wichtig zu betonen, dass aufgrund der Komplexität des Themas "Generatives Deep Learning" möglicherweise nicht das ideale Buch für absolute Anfänger ist. Personen, die sich gerade erst mit Deep Learning vertraut machen, könnten möglicherweise von einem Buch wie "Deep Learning - Grundlagen und Implementierung" profitieren, das einen einfacheren Einstieg in die Materie bietet.

Insgesamt ist "Generatives Deep Learning" von David Foster ein äußerst informatives und gut geschriebenes Buch, das einen tiefen Einblick in die Welt des generativen Deep Learning bietet. Es ist eine wertvolle Ressource für alle, die ihr Wissen auf diesem Gebiet erweitern möchten.

[Brian Powell · Rating 4 out of 5]

This is an accessible and practically useful guide to generative AI, with an emphasis on pre-transformer models including variational autoencoders, GANs, and recurrent neural networks. This is a heavily application-driven text, with code and datasets available for you to play along (though requiring for sure some adjustments to get the assuredly out-of-date repository to install properly). I felt the example projects were fun: facial image generation with VAEs, artistic style transfer with GANs, and music composition using RNNs with attention, among several others. Transformers are covered quickly towards the end, and though GPT was only released up to version 2.5 when this book was written, the author can tell big things are happening with these powerful language models.

But, as with any text that covers difficult, complicated matters at a high-level, in a sort of 'just start getting your hands dirty' kind of way, key concepts aren't always given the attention necessary for maximum groking. Sometimes statements of considerable depth are made without elaboration or support, and this is annoying. But, here you can just dig a little deeper into the original literature if you've got an itch that Foster doesn't scratch (he does consistently reference key prior work).

If you're new to generative AI and want to get up to speed with this world-changing technology, this is a sound starting point.

[Moran Danieli-Cohen · Rating 5 out of 5]

This book is an essential read for anyone looking to deepen their understanding of AI, especially in the context of generative models. This book excels in its comprehensive coverage of critical topics like variational autoencoders, Generative Adversarial Networks (GANs), and advanced architectures including the Transformer and sophisticated GAN models.

The book's exploration of world models is particularly intriguing, offering insights into how AI can simulate and predict complex environments. It skillfully bridges the gap between theoretical concepts and practical application, making complex topics accessible through clear explanations and practical code examples. This approach not only enhances understanding but also encourages practical experimentation.

Overall, the book is a valuable resource for both AI enthusiasts and seasoned professionals, offering a deep dive into the exciting and rapidly evolving field of generative deep learning.

[Nguyen Le · Rating 3 out of 5]

I would definitely rate this book higher once the code base is documented. Overall, the concept explained is done very well. However, the level of details, and shall I say effort, are not the same in the latter part of the book. Especially for the generating music chapter. I understand that the book goes over quickly on a few part, otherwise it would be a few hundred of pages more ( I wouldn't mind that at all, considering how well the author explains the concept when he tries to do so). But some parts are just very difficult to understand without the proper documentation of the code, and any explanation in the book. I hope the author will start commenting his code on git hub soon enough, allowing for more readers to have a clearer understanding of what is going on under the hood

[to'c · Rating 5 out of 5]

I wanted a nice theoretical overview of the current state of neural net based AI. Boy howdy, did this book deliver! A very deep dive into the generative aspects of modern AI. I believe I got far more than I expected.

That said, this is not meant to be a theoretical introduction. Mr. Foster encourages experimentation and gives you plenty of tools and pointers to pursue that on your own. I didn't have the time nor inclination to do so on this read but will certainly build some nice nets on my next read. And there will be a next read. This book is chock full of facts and ideas.

The concluding chapter (aptly titled "Conclusion") presents encouraging words about the future of AI tools and how they will enhance skills we humans already possess. Take heed, young'uns, this is your future.

[Ankit Jain · Rating 5 out of 5]

A comprehensive guide to Gen AI that I needed!

Very glad I chanced upon this book. This guide to state-of-the-art Gen AI research is both comprehensive and deep. It helps you grasp the underlying architecture, building blocks and mathematical intuition of wide variety of gen AI models (ranging from text to images to music to multi modality models). The book expects some background in stats and probability theory. It can be a heavy read at times but that's when you know this has the depth to actually get what's going on in these models and not just learn to call a few APIs in a phyton program. Although, it does include a set of coding exercises too. Strongly recommended.

[Ozan · Rating 4 out of 5]

This is a fine book, however it’s hard to say who the target audience is. On one hand, the author explains concepts using not-so-helpful examples (like a prisoner guard trying to write prose with the help of the prisoners) assuming that you’re so far off from the topic and that you will not understand LSTMs without such a story, on the other hand he tries to explain and reproduce state-of-the research in AI using quite some mathematical detail. This makes the whole reading experience dull for both hobbyists and professionals in different ways.

[Kautuk Kundan · Rating 5 out of 5]

The book is well written. There is a perfect balance of theory and hands-on task. The code provided along with the book is also top quality, well documented and readable.

My favourite aspect of the book are the short stories which opens every chapter. These stories immediately tells about the content of the chapter while staying brief and generates an interest to learn more about it.
I wish more technical authors adopt this style of writing.

5/5 recommended if you wish to learn about generative modelling.

[Lucas Moda · Rating 5 out of 5]

Impressionante como em menos de um ano da publicação já vimos ainda mais avanços (GPT-4, por exemplo). Esse não é um livro que vai te ensinar a treinar um LLM ou modelo multimodal num passo a passo, mas não deixa de ser valioso por conta disso. Apesar da (longa) parte introdutória, é altamente recomendado que se tenha uma boa base em Deep Learning, ML, Python e afins para absorver melhor o conteúdo. De maneira geral, livro bem escrito, organizado e que oferece uma boa base dos conceitos fundamentais para ficar por dentro do boom de GenAI. Vale a pena.

[Himanshu Sharma · Rating 3 out of 5]

The book provides a high level summary of important models/ papers published in past few years in the field of Generative ML. Good for beginners or for revising concepts.

Pro - Cherry picks best papers and explains intuitively
Cons - Good for first few chapters but for complex papers seems all over the place.

[Kacxxia · Rating 3 out of 5]

This book isn't really for beginners like me who ask a lot of "why" questions. The author jumps right into deep learning, briefly covering basic math concepts. But honestly, the simplified math explanations and the classification task exercises just left me even more confused.

[Pablo Ferreiro · Rating 3 out of 5]

It’s a good review of generative deep learning. But it has a problem with it public. Is not for basic users, but it lacks the depth of more advanced texts. It also has some gaps in things that should have been explained.

[Susan · Rating 5 out of 5]

Very readable, with some cute analogies and generally clear explanations. Great overview of a lot of info. The concluding material also helps bring the book up to date and into context with today’s latest developments.

[Stein Karlsen · Rating 5 out of 5]

Great book! I particularly enjoyed the parts on autoencoders and have used the ideas from the book professionally. Will probably read it again

[Filip Cvetko]

Great!

[Dayton Outar · Rating 3 out of 5]

A good summary of GANs but too many allegories. I don't really appreciate too many detours into stories that distracts from understanding the behaviour of systems.

[Özgür · Rating 4 out of 5]

A good intro to Generative AI.
Looking forward to 2nd edition in the summer.

[Lupin V · Rating 4 out of 5]

very technical book for me. I scanned over a lot of parts to understand the high level first. I may go back or use this book as a reference to learn other details.

[Tenzin Wangdhen · Rating 4 out of 5]

Currently one of the few books that dives deep into the technical details of generative technologies like GPT and StableDiffusion. The examples are in Keras, I’d have preferred PyTorch. At the beginning of the book he has a cutsy metaphor for the technologies but he gets lazy about that towards the end.

[Rick Sam · Rating 4 out of 5]

Great Work, helped me to understand a bit of Generative Models.

My Contention:

I didn't want fancy stories, those were distracting to me.

So, What did I desire?



"How do I mathematically express this?"

"What does it mean, if I express mathematically this way?"

"Where does one apply this in real-world?"

"What happens if we didn't have this way of doing things?"

"At times, I see pictures of mathematical expressions in my mind"

"If I tweak the concrete mathematical representation, what would happen?"



Recommended to Computer Scientists, Practitioners in Machine Learning.

Deus Vult,
Gottfried

[Gautam · Rating 4 out of 5]

The book covers deep learning algorithms and did background on different arts.