What do you think?
Rate this book
The MIT Press Essential Knowledge
Deep Learning
MP3 CD Format Deep learning is an artificial intelligence technology that enables computer vision, speech recognition in mobile phones, machine translation, AI games, driverless cars, and other applications. When we use consumer products from Google, Microsoft, Facebook, Apple, or Baidu, we are often interacting with a deep learning system. In this volume in the MIT Press Essential Knowledge series, computer scientist John Kelleher offers an accessible and concise but comprehensive introduction to the fundamental technology at the heart of the artificial intelligence revolution.
Kelleher explains some of the basic concepts in deep learning, presents a history of advances in the field, and discusses the current state of the art. He describes the most important deep learning architectures, including autoencoders, recurrent neural networks, and long short-term networks, as well as such recent developments as Generative Adversarial Networks and capsule networks. He also provides a comprehensive introduction to the two fundamental algorithms in deep gradient descent and backpropagation. Finally, Kelleher considers the future of deep learning—major trends, possible developments, and significant challenges.
Kelleher explains some of the basic concepts in deep learning, presents a history of advances in the field, and discusses the current state of the art. He describes the most important deep learning architectures, including autoencoders, recurrent neural networks, and long short-term networks, as well as such recent developments as Generative Adversarial Networks and capsule networks. He also provides a comprehensive introduction to the two fundamental algorithms in deep gradient descent and backpropagation. Finally, Kelleher considers the future of deep learning—major trends, possible developments, and significant challenges.
6 pages, Audio CD
First published January 1, 2019
About the author
John D. Kelleher
5 books27 followersJohn D. Kelleher is a Professor of Computer Science and the Academic Leader of the Information, Communication, and Entertainment Research Institute at the Dublin Institute of Technology. He is the coauthor of Fundamentals of Machine Learning for Predictive Data Analytics (MIT Press).
Ratings & Reviews
Friends & Following
Create a free account to discover what your friends think of this book!
Community Reviews
Displaying 1 - 30 of 77 reviews
November 5, 2019
This is an entry in a series from the MIT Press that selects a small part of a topic (in this case, a subset of artificial intelligence) and gives it an 'essential knowledge' introduction. The problem is, there seems to be no consistency over the target audience of the series.
I previously reviewed Virtual Reality in the same series and it kept things relatively simple and approachable to the general reader, even if it did overdo the hype. This book by John Kelleher starts gently, but by about half way through it has become a full-blown simplified textbook with far too much in-depth technical content. That's exactly what you don't want in a popular science title.
What we get is plenty of detail of what deep learning-based systems are and how they work at the technical level, but there is practically nothing on how they fit with applications (unless you count playing games), which are described but not really explained, nor is there anything much on the problems that arise when deep learning is used for real world applications. There is a passing reference, admittedly to the difficulties of understanding how a deep learning AI system came to a decision and how this clashes with the EU's GDPR requirement for transparency and explanation, but if feels more like this is done to criticise the naivety of the legislation than the danger of using such systems.
Similarly, I saw nothing about the dangers of deep learning systems using big data picking up on correlations that don't involve any causal link, nor does the book discuss the long tail problems that arise with inputs that are relatively uncommon and so are unlikely to turn up in the training data. Similarly we read nothing about the dangers of adversarial attacks, which can fool the systems into misinterpreting inputs with tiny changes, or the difficulties such systems have with real, messy environments as opposed to the rigid rules of a game.
Overall, the book is both pitched wrong and doesn't cover the aspects that really matter to the public. It may well do fine as an introductory text for a computer science student, but that doesn't fit with the blurb on the back, which implies it is for public consumption.
I previously reviewed Virtual Reality in the same series and it kept things relatively simple and approachable to the general reader, even if it did overdo the hype. This book by John Kelleher starts gently, but by about half way through it has become a full-blown simplified textbook with far too much in-depth technical content. That's exactly what you don't want in a popular science title.
What we get is plenty of detail of what deep learning-based systems are and how they work at the technical level, but there is practically nothing on how they fit with applications (unless you count playing games), which are described but not really explained, nor is there anything much on the problems that arise when deep learning is used for real world applications. There is a passing reference, admittedly to the difficulties of understanding how a deep learning AI system came to a decision and how this clashes with the EU's GDPR requirement for transparency and explanation, but if feels more like this is done to criticise the naivety of the legislation than the danger of using such systems.
Similarly, I saw nothing about the dangers of deep learning systems using big data picking up on correlations that don't involve any causal link, nor does the book discuss the long tail problems that arise with inputs that are relatively uncommon and so are unlikely to turn up in the training data. Similarly we read nothing about the dangers of adversarial attacks, which can fool the systems into misinterpreting inputs with tiny changes, or the difficulties such systems have with real, messy environments as opposed to the rigid rules of a game.
Overall, the book is both pitched wrong and doesn't cover the aspects that really matter to the public. It may well do fine as an introductory text for a computer science student, but that doesn't fit with the blurb on the back, which implies it is for public consumption.
October 25, 2019
This book satisfied most of the initial expectations I had about it's content: It provided a comprehensive overview of the origins of the field of machine learning and AI, covered all the technical bases of modern neural networks such as RNNs (for language processing), CNNs (for image processing), GANs (for game theory), and concluded with a short but insightful comment on the "future of AI".
The mathematical deep dives into foundational concepts such as neurons, weighted sums, activation functions, classic neural network models of just a few layers (that could learn boolean functions such as AND and XOR), pre-training neural-nets with auto-encoders (to filter out extraneous features in data sets), modern neural-net models like LSTM and the average CNN, the gradient-decent algorithm (for training "deep convolutional neural-networks" with dozens of layers), and finally the nuances and differential-calculus behind the back-propagation algorithm (the solution to the "credit assignment" problem that afflicts gradient-decent in deep neural-nets), we're all greatly appreciated and mostly-well explained.
The reason I rated it 4/5 stars is because I felt that this book could have spent more time on the technical details. I was reading the book for the in hopes of gaining broader social, technical, and mathematical contexts with respect to the subject of "deep-learning" in general-- and for the most part, I think I got that. However, I did feel that the explanation of back-propagation deserved a bit more attention. As someone who spends a fair amount of time more than the average person in familiarizing myself with mathematics relevant to engineering, I still felt that I had to re-read certain sentences and paragraphs, and stare at the equations and visual depictions more times than I expected, for the reason that the information wasn't presented as clearly as it could have been. I felt the author elided a certain depth of explanation in order to assure the book had few enough pages to potentially appeal to more people than might actually finish it.
In conclusion, I enjoyed this book enough to readily recommend this book to anyone who wants a thorough introduction to the topic of "deep learning". Those with at least a light familiarity to differential calculus and a tangential understanding of Machine Learning would greatly benefit from reading; perhaps the average data-scientist, the software engineer working for an ML company, or the mathematician/computer scientist looking to expand the breadth of their mathematical world.
The mathematical deep dives into foundational concepts such as neurons, weighted sums, activation functions, classic neural network models of just a few layers (that could learn boolean functions such as AND and XOR), pre-training neural-nets with auto-encoders (to filter out extraneous features in data sets), modern neural-net models like LSTM and the average CNN, the gradient-decent algorithm (for training "deep convolutional neural-networks" with dozens of layers), and finally the nuances and differential-calculus behind the back-propagation algorithm (the solution to the "credit assignment" problem that afflicts gradient-decent in deep neural-nets), we're all greatly appreciated and mostly-well explained.
The reason I rated it 4/5 stars is because I felt that this book could have spent more time on the technical details. I was reading the book for the in hopes of gaining broader social, technical, and mathematical contexts with respect to the subject of "deep-learning" in general-- and for the most part, I think I got that. However, I did feel that the explanation of back-propagation deserved a bit more attention. As someone who spends a fair amount of time more than the average person in familiarizing myself with mathematics relevant to engineering, I still felt that I had to re-read certain sentences and paragraphs, and stare at the equations and visual depictions more times than I expected, for the reason that the information wasn't presented as clearly as it could have been. I felt the author elided a certain depth of explanation in order to assure the book had few enough pages to potentially appeal to more people than might actually finish it.
In conclusion, I enjoyed this book enough to readily recommend this book to anyone who wants a thorough introduction to the topic of "deep learning". Those with at least a light familiarity to differential calculus and a tangential understanding of Machine Learning would greatly benefit from reading; perhaps the average data-scientist, the software engineer working for an ML company, or the mathematician/computer scientist looking to expand the breadth of their mathematical world.
November 27, 2023
This book is actually pretty technical. With a good amount of mathematics. And as such may not be SUPER useful to a general audience.
So why read it?
Well…
I include myself in the average reader - general audience category. I’m a therapist with some training in cognitive neuroscience and learning theory. But beyond that. I don’t have advanced math training. And I don’t write code. So I had to read a lot of this book VERY loosely, meaning focused on the big concepts rather than a more granular understanding.
Given all that.
I found this book to be very interesting and eye opening.
Deep learning is very biomimetic, meaning that the design approaches to machine learning are very analogous to biological/animal learning. From topographical stimulus/response reward conditioning. To cognitive/associative learning. All the way down to neuronal level processes.
There is a LARGE amount of convergence between learning theory, cognitive neuroscience, and artificial intelligence (AI) and machine learning (ML).
NURAL NETWORKS:
In AI/ML a neural network refers to a computational technique inspired by brain/nervous system structure and function.
In biological brains, neurons that “fire together wire together”, meaning that neuronal pathways form and become stronger with use. In essence, this is how animals (including people) learn.
A neural network consists of interconnected artificial neurons, which process information in a way that is mimetic of biological neurons. With “weighted” connections between artificial neurons that can self adjust in priority/salience based on their programming goals and the data they are processing.
DEEP LEARNING:
Neural networks are organized into layers (also mimetic of brain structure/function). With (a) input layers receiving and sending data to (b) one (or more) hidden (deep) data processing layers, which in turn send processed information to (c) an output layer to produce the final results.
AI/ML uses and deep learning to recognize patterns, make decisions, and solve complex problems. for instance, neural networks ability to learn from big data sets makes it particularly useful/effective for pattern recognition tasked such as: image and speech recognition, and language translation. Additionally deep learning can be trained on YOU. And as such manipulate you into doing ALL KINDS of stuff including clinking, buying and even voting.
Deep learning is beginning to surpass human performance in ALL KINDS of ways. And as such, is generating a LOT of fear/excitement about the future of stuff like education, work, democracy and capitalism (just to name a few).
I became interested in evolutionary biology, neuroscience, and psychology in the early 2000’s. Some earlier mediation and psychedelic experiences got me REALLY curious about mind and consciousness.
I was also in a LOT of emotional pain at the time, occurring at the intersection of addiction and trauma. I was becoming increasingly dissatisfied and disillusioned with the 1960s era psychology and spirituality. And I was desperately seeking a more parsimonious, naturalistic and reliable explanation for human suffering and liberation.
Research is “me-search” as we say 😀.
More recently, my interest in AI/ML and deep learning has been piqued by the STUNNING advances in AI applications like CHAT-GPT and DALL-E. The awe and wonder I felt the first time I played with those technologies felt like real live MAGIC. That coupled with instant TERROR at the prospect of becoming TOTALLY irrelevant.
I offer ALL of that as a long preface to a short book. Despite being untrained in math and computer science. There is DEFINITELY something of value in being exposed to a more technical explanation of this tech.
As with neuroscience. I wouldn’t be qualified to do brain surgery after becoming more than casually acquainted with the field. But it can and does inform my world view, and profoundly impacts my work as a therapist.
Similarly I won’t be able to program AI/ML after reading this. But I will be (at least a little) more able to understand and follow the emergence of these INSANELY important technologies.
It’s hard to say how (precisely) AI/ML will impact our near to distant future. But, barring some horrible disasters, it will. And I have to assume know at least a little bit about how it works will confer at least a little bit of advantage.
Lastly, there is something oddly soothing about this book.
I think just increasing my understanding just a little bit more helps me relax a bit. And the exiting prospect of witnessing the emergence of this WILD new thing is also very exciting.
At minimum.
It’s good nerdy fun.
5/5 stars ⭐️
So why read it?
Well…
I include myself in the average reader - general audience category. I’m a therapist with some training in cognitive neuroscience and learning theory. But beyond that. I don’t have advanced math training. And I don’t write code. So I had to read a lot of this book VERY loosely, meaning focused on the big concepts rather than a more granular understanding.
Given all that.
I found this book to be very interesting and eye opening.
Deep learning is very biomimetic, meaning that the design approaches to machine learning are very analogous to biological/animal learning. From topographical stimulus/response reward conditioning. To cognitive/associative learning. All the way down to neuronal level processes.
There is a LARGE amount of convergence between learning theory, cognitive neuroscience, and artificial intelligence (AI) and machine learning (ML).
NURAL NETWORKS:
In AI/ML a neural network refers to a computational technique inspired by brain/nervous system structure and function.
In biological brains, neurons that “fire together wire together”, meaning that neuronal pathways form and become stronger with use. In essence, this is how animals (including people) learn.
A neural network consists of interconnected artificial neurons, which process information in a way that is mimetic of biological neurons. With “weighted” connections between artificial neurons that can self adjust in priority/salience based on their programming goals and the data they are processing.
DEEP LEARNING:
Neural networks are organized into layers (also mimetic of brain structure/function). With (a) input layers receiving and sending data to (b) one (or more) hidden (deep) data processing layers, which in turn send processed information to (c) an output layer to produce the final results.
AI/ML uses and deep learning to recognize patterns, make decisions, and solve complex problems. for instance, neural networks ability to learn from big data sets makes it particularly useful/effective for pattern recognition tasked such as: image and speech recognition, and language translation. Additionally deep learning can be trained on YOU. And as such manipulate you into doing ALL KINDS of stuff including clinking, buying and even voting.
Deep learning is beginning to surpass human performance in ALL KINDS of ways. And as such, is generating a LOT of fear/excitement about the future of stuff like education, work, democracy and capitalism (just to name a few).
I became interested in evolutionary biology, neuroscience, and psychology in the early 2000’s. Some earlier mediation and psychedelic experiences got me REALLY curious about mind and consciousness.
I was also in a LOT of emotional pain at the time, occurring at the intersection of addiction and trauma. I was becoming increasingly dissatisfied and disillusioned with the 1960s era psychology and spirituality. And I was desperately seeking a more parsimonious, naturalistic and reliable explanation for human suffering and liberation.
Research is “me-search” as we say 😀.
More recently, my interest in AI/ML and deep learning has been piqued by the STUNNING advances in AI applications like CHAT-GPT and DALL-E. The awe and wonder I felt the first time I played with those technologies felt like real live MAGIC. That coupled with instant TERROR at the prospect of becoming TOTALLY irrelevant.
I offer ALL of that as a long preface to a short book. Despite being untrained in math and computer science. There is DEFINITELY something of value in being exposed to a more technical explanation of this tech.
As with neuroscience. I wouldn’t be qualified to do brain surgery after becoming more than casually acquainted with the field. But it can and does inform my world view, and profoundly impacts my work as a therapist.
Similarly I won’t be able to program AI/ML after reading this. But I will be (at least a little) more able to understand and follow the emergence of these INSANELY important technologies.
It’s hard to say how (precisely) AI/ML will impact our near to distant future. But, barring some horrible disasters, it will. And I have to assume know at least a little bit about how it works will confer at least a little bit of advantage.
Lastly, there is something oddly soothing about this book.
I think just increasing my understanding just a little bit more helps me relax a bit. And the exiting prospect of witnessing the emergence of this WILD new thing is also very exciting.
At minimum.
It’s good nerdy fun.
5/5 stars ⭐️
October 24, 2020
This book never really delivers on the promise of being "an accessible introduction" to deep learning, despite the fact that the two first chapters are written on a very clear language that strikes a good balance between precision and understandability for the layperson. After that, however, the book does not find its voice: some parts seem to reach out to a general audience, but, only a few paragraphs later, the author goes back to speaking to an audience of computer scientists.
As a result, much of the core of the book is inaccessible to the general reader while failing to provide enough detail for a technically-minded introduction. Which is sad, as the book occasionally delivers insights that are useful for either audience. So, it seems that the author would have been better served by picking one public or the other, instead of aiming at both and satisfying neither.
Besides this general complaint, there is one specific thing that annoyed me A LOT: figure positioning. At times, a figure (such as a network diagram) is mentioned in the text, but it only appears several pages later, after the author has already spent a few paragraphs describing what is going on. This does not help with understanding what is going on.
As a result, much of the core of the book is inaccessible to the general reader while failing to provide enough detail for a technically-minded introduction. Which is sad, as the book occasionally delivers insights that are useful for either audience. So, it seems that the author would have been better served by picking one public or the other, instead of aiming at both and satisfying neither.
Besides this general complaint, there is one specific thing that annoyed me A LOT: figure positioning. At times, a figure (such as a network diagram) is mentioned in the text, but it only appears several pages later, after the author has already spent a few paragraphs describing what is going on. This does not help with understanding what is going on.
February 15, 2020
An excellent introduction to the theoretical basis for one of the most important technologies of the 21st century. Thankfully, Kelleher doesn't completely eschew mathematics; even so at times more rigorous mathematical explanations may have aided understanding, at least for those mathematically inclined.
February 25, 2025
Es una excelente introducción al Deep Learning y al campo de la inteligencia artificial moderna en general. Me gusta mucho la característica de esta serie de ser lo suficientemente introductoria para no intimidarte, pero profundizar lo suficiente en pocas páginas para que sientas que de verdad aprendiste algo.
Este libro en específico abarca la base matemática de las redes neuronales profundas y luego te habla de nuevos desarrollos que menciona como importantes y que de verdad se convirtieron en cosas que están cambiando el mundo, lo que da testimonio del conocimiento del autor.
Lo recomiendo mucho a todos las personas que quieran introducirse en el desarrollo de inteligencia artificial o quieran entender las bases de su funcionamiento para no dejarse engañar por los miles de datos falsos que salen día a día.
Escribí una reseña más completa aquí: Reseña de Deep Learning.
Este libro en específico abarca la base matemática de las redes neuronales profundas y luego te habla de nuevos desarrollos que menciona como importantes y que de verdad se convirtieron en cosas que están cambiando el mundo, lo que da testimonio del conocimiento del autor.
Lo recomiendo mucho a todos las personas que quieran introducirse en el desarrollo de inteligencia artificial o quieran entender las bases de su funcionamiento para no dejarse engañar por los miles de datos falsos que salen día a día.
Escribí una reseña más completa aquí: Reseña de Deep Learning.
Currently Reading
March 11, 2021This book gives beautiful insights on how exactly a Neural Network functions.
This helped me build a stronger intuition of the basics of Deep Learning.
It is quite a heavy read. It even feels wordy at times. However, it travels beautifully through the various fundamental concepts of deep learning. It's as if I'm reading a story, and arriving at the various concepts, through some very clear explanations, and most importantly, building a strong intuition while doing so.
This helped me build a stronger intuition of the basics of Deep Learning.
It is quite a heavy read. It even feels wordy at times. However, it travels beautifully through the various fundamental concepts of deep learning. It's as if I'm reading a story, and arriving at the various concepts, through some very clear explanations, and most importantly, building a strong intuition while doing so.
January 29, 2023
A solid introduction to the field of DL/ML. The author did a commendable job charting out a very complex subject in 250 pages. The effort to make this position accessible to a broader audience is evident, yet parts of this book may still feel beyond reach for less mathematically inclined readers. Then again, I'm not sure if it's possible to write a general overview of this subject without illustrating it with mathematical equations. What is however possible is thorough proof reading to ensure there are not typos. For a book on such a precise topic, its cavalier approach to spelling is perplexing. The typos (and unintuitive placement of illustrations) aside, this is a worthwhile overview of the subject.
August 9, 2024
interesting book that compares deep learning to the human neurological system, and details the history of deep learning. So what does deep learning mean? It's basically part of artificial intelligence, and if you drew a Venn diagram of AI then deep learning would be one of the circles. I did not realize the underlying theories of AI began as early as the 1940s. This is a good book if you have some computer background and want a better understanding of AI.
September 25, 2020
A case of false advertising, but a highly informative read nonetheless.
February 20, 2022
Good book with examples by MIT Press Essential Knowledge Series for anyone who wants to learn the fundamentals of Deep Learning. It helped me to refresh a part of the machine learning course I took.
December 22, 2022
...an average book that talks about the history of deep learning, where it is going and the functions/techniques used related to it. It was very difficult to read despite being a computer scientist myself and having some background in machine learning. This is one of the books with good content but which I would like to be explained like I'm five and it didn't do a good job of doing that.
May 29, 2022
How did Newton find out or learned the laws of motion or gravitation? First of all he had the assumption that hidden "functions" are at play. This is a big assumption because function is a specific type of relation. However, from human experience, it turns out thinking about nature in terms of function therefore in terms of relations for which one input cannot map to two or more different outputs is a fruitful path. Newton probably had a guess about the set of candidate functions that could explain the motion or gravitation. More challenging perhaps was to first find features of motion like position/coordinates, speed, acceleration, mass/inertia, force, momentum. He probably used thought experiments to efficiently search for “the best” function and compared each promising function’s performance against how well it explains motion or gravitation.
(Supervised) machine learning also tries to find “the best” function (aka model) that explains the relationship between a set of inputs and outputs. What’s the use? Like Newton's laws of motion, once the function has been learned or found, it can be used in unknown scenarios. Say predict if it will rain tomorrow. Or predict if the user will click this ad. Or say predict if the user will choose a binge watch from this short list of 20 shows. Or say predict this is an image of a dog. Or translate this Chinese sentence into english.
Machine learning, like Newton, considers a set of candidate functions that an algorithm searches through, comparing each according to some “goodness” criteria. So, in learning there are five ingredients:
1. data: example pairs of (input, output)
2. an assumption (aka inductive bias) about what kind of functions may be candidates to explain the relationship between input and output in (1)
3. a representation of the candidate functions
4. a searching algorithm
5. a goodness criteria that will be used by (4) to decide how well candidate function explains the input->output relationship
Say we have a blackbox that has a hidden function. We guess the hidden function is one of: addition, subtraction, multiplication, and division.
We have three examples or observations:
black_box(1, 3) = 3, given 1 and 3 as inputs, the box produces 3 as output.
black_box(2, 2) = 4
black_box(3, 1) = 3
One way to find out would be to try all four possible functions and see which one matches with the most outputs, that would be our “goodness” criterion. So the goodness of the four possible functions across those three examples are:
addition has goodness = 1, therefore it explains only one example
subtraction has goodness = 0
multiplication has goodness = 3
division has goodness = 1
So, reasonably we can say the black box is doing multiplication. That would be the learned model/function of the black box.
There are a few challenges:
Data is noisy, so learning process should try to discard noise
The number of candidate functions can be way too higher than the data at hand. So, there may not be a “best” choice given data. Therefore multiple functions may explain the input->output map kind of equally well. To get around this, newton had used his and others' experience about how motion works to cut down on the possible functions, which is called inductive bias, because newton is showing bias towards a special set of functions. It is called inductive bias because in the end the whole process is a generalization from specific examples aka induction.
Deep learning is a subset of machine learning that uses neural networks to represent the candidate functions in (3). It is a very flexible representation. In other words, a neural network can represent a very large number of possible functions. Therefore it risks learning noise if data is scanty. In other words, if there are only a few examples, it fails to narrow down the number of candidate functions and ends up choosing one that just explains these few examples well but performs badly on new examples. Good thing is, with online businesses there is plenty of data. Thus the neural network is a good fit now-a-days with “big” data. Another good thing about neural networks or deep learning is, it can auto-learn how to discard noise from data. This is called feature engineering and with other machine learning models it is a very manual, time and labor-intensive process.
Think of data as a table. Each row is one example. Each column is describing one salient feature of those examples. There usually is a special column at the end, which is the output. Therefore if there are 4 columns, the first 3 makes up the input domain of the function being learned and the 4th column is the output of that function.
A artificial neural network is inspired by biological neuron network. Like a biological neuron, each artificial neuron is simple: a weighted sum followed by an activation function. The power comes from how an aggregate of neuron interacts in the network. From (3) we want a representation that could capture a sufficiently large class of functions from which we could choose a good one that mimics or learns the real process we are trying to model.
In deep learning, to find the best function we find the minimum of error surface which lies above the weight space. Moving from one point to another in the weight space amounts to choosing one function over another. Bowl-shaped convex error surfaces are nice because there is a single minimum so if we follow the local gradient of the error surface, we are sure to reach the minimum, even if in a rather circuitous manner. Since deep neural networks incorporate a non-linear activation function in neurons, the error surface may look more like a bumpy hill and the function-searcher may get stuck at a local minimum thinking myopically mistakes the local minimum as the global. There are ways to get around. A popular function searching algorithm is gradient descent. Given the difference between predicted output and actual output the gradient descent will tell us how much and which direction we need to move in the weight space to go downhill in the error surface hanging above. This works for a single neuron's error surface. If we know the error of the entire neural network and we want to update weights belonging to the many constituent neurons, we need to somehow assign the blame of error to different neurons and then use gradient descent to update the weights of a particular neuron. This blame assignment is done using another algorithm called back propagation. Basically both algorithms are fancy name for chain-rule of differentiation.
In a deep neural network, the neurons are arranges in layers, think of going left to right, input layer, hidden layer 1, hidden layer 2, etc., then output layer of neurons. The earlier layers of neurons usually learn to detect smaller sub-problems and how to solve those. The latter layers of neurons build upon those solutions of the sub-problems. So, deep learning is based on divide-and-conquer strategy for solving a complicated problem.
The books hints some exciting problems in the deep learning that are being worked on right now: GANs, neuromorphic computing, quantum computing, interpretability. Interpretability in particular looked attactive to me. It is the problem of explaining how a deep neural network came to make a particular decision. This is hard to explain with a deep neural network with many neurons. Almost, like trying to explain an idea in human mind: what caused the idea to come, how? There are some avenues folks are taking to solve this problem, one is like coming up with FMRI/PET like styles, find out certain inputs that trigger certain decision we are interested in. Once such inputs/examples are found now dig deeper into the example and how it affects the connection weights between neurons and try to explain why the networks is coming up with that decision.
(Supervised) machine learning also tries to find “the best” function (aka model) that explains the relationship between a set of inputs and outputs. What’s the use? Like Newton's laws of motion, once the function has been learned or found, it can be used in unknown scenarios. Say predict if it will rain tomorrow. Or predict if the user will click this ad. Or say predict if the user will choose a binge watch from this short list of 20 shows. Or say predict this is an image of a dog. Or translate this Chinese sentence into english.
Machine learning, like Newton, considers a set of candidate functions that an algorithm searches through, comparing each according to some “goodness” criteria. So, in learning there are five ingredients:
1. data: example pairs of (input, output)
2. an assumption (aka inductive bias) about what kind of functions may be candidates to explain the relationship between input and output in (1)
3. a representation of the candidate functions
4. a searching algorithm
5. a goodness criteria that will be used by (4) to decide how well candidate function explains the input->output relationship
Say we have a blackbox that has a hidden function. We guess the hidden function is one of: addition, subtraction, multiplication, and division.
We have three examples or observations:
black_box(1, 3) = 3, given 1 and 3 as inputs, the box produces 3 as output.
black_box(2, 2) = 4
black_box(3, 1) = 3
One way to find out would be to try all four possible functions and see which one matches with the most outputs, that would be our “goodness” criterion. So the goodness of the four possible functions across those three examples are:
addition has goodness = 1, therefore it explains only one example
subtraction has goodness = 0
multiplication has goodness = 3
division has goodness = 1
So, reasonably we can say the black box is doing multiplication. That would be the learned model/function of the black box.
There are a few challenges:
Data is noisy, so learning process should try to discard noise
The number of candidate functions can be way too higher than the data at hand. So, there may not be a “best” choice given data. Therefore multiple functions may explain the input->output map kind of equally well. To get around this, newton had used his and others' experience about how motion works to cut down on the possible functions, which is called inductive bias, because newton is showing bias towards a special set of functions. It is called inductive bias because in the end the whole process is a generalization from specific examples aka induction.
Deep learning is a subset of machine learning that uses neural networks to represent the candidate functions in (3). It is a very flexible representation. In other words, a neural network can represent a very large number of possible functions. Therefore it risks learning noise if data is scanty. In other words, if there are only a few examples, it fails to narrow down the number of candidate functions and ends up choosing one that just explains these few examples well but performs badly on new examples. Good thing is, with online businesses there is plenty of data. Thus the neural network is a good fit now-a-days with “big” data. Another good thing about neural networks or deep learning is, it can auto-learn how to discard noise from data. This is called feature engineering and with other machine learning models it is a very manual, time and labor-intensive process.
Think of data as a table. Each row is one example. Each column is describing one salient feature of those examples. There usually is a special column at the end, which is the output. Therefore if there are 4 columns, the first 3 makes up the input domain of the function being learned and the 4th column is the output of that function.
A artificial neural network is inspired by biological neuron network. Like a biological neuron, each artificial neuron is simple: a weighted sum followed by an activation function. The power comes from how an aggregate of neuron interacts in the network. From (3) we want a representation that could capture a sufficiently large class of functions from which we could choose a good one that mimics or learns the real process we are trying to model.
In deep learning, to find the best function we find the minimum of error surface which lies above the weight space. Moving from one point to another in the weight space amounts to choosing one function over another. Bowl-shaped convex error surfaces are nice because there is a single minimum so if we follow the local gradient of the error surface, we are sure to reach the minimum, even if in a rather circuitous manner. Since deep neural networks incorporate a non-linear activation function in neurons, the error surface may look more like a bumpy hill and the function-searcher may get stuck at a local minimum thinking myopically mistakes the local minimum as the global. There are ways to get around. A popular function searching algorithm is gradient descent. Given the difference between predicted output and actual output the gradient descent will tell us how much and which direction we need to move in the weight space to go downhill in the error surface hanging above. This works for a single neuron's error surface. If we know the error of the entire neural network and we want to update weights belonging to the many constituent neurons, we need to somehow assign the blame of error to different neurons and then use gradient descent to update the weights of a particular neuron. This blame assignment is done using another algorithm called back propagation. Basically both algorithms are fancy name for chain-rule of differentiation.
In a deep neural network, the neurons are arranges in layers, think of going left to right, input layer, hidden layer 1, hidden layer 2, etc., then output layer of neurons. The earlier layers of neurons usually learn to detect smaller sub-problems and how to solve those. The latter layers of neurons build upon those solutions of the sub-problems. So, deep learning is based on divide-and-conquer strategy for solving a complicated problem.
The books hints some exciting problems in the deep learning that are being worked on right now: GANs, neuromorphic computing, quantum computing, interpretability. Interpretability in particular looked attactive to me. It is the problem of explaining how a deep neural network came to make a particular decision. This is hard to explain with a deep neural network with many neurons. Almost, like trying to explain an idea in human mind: what caused the idea to come, how? There are some avenues folks are taking to solve this problem, one is like coming up with FMRI/PET like styles, find out certain inputs that trigger certain decision we are interested in. Once such inputs/examples are found now dig deeper into the example and how it affects the connection weights between neurons and try to explain why the networks is coming up with that decision.
January 2, 2020
It is very, very hard to write accessible popular science that attempts to explain systems that, while not particularly complex, depend on concepts that are not clear with middle-school math.
This is a very good attempt at explaining the basic components of deep learning. It doesn’t go to deep, nor gloss over too much. The author does commendable service explaining the underlying math, and while not ‘breezy’ it is immensely useful.
I found it simplifies topics that I struggled to grasp around back propagation and how neurons optimized outcomes. The chapter on back-propagation math was beyond my ability, but I could grasp the basic concepts.
That is the purpose of this work. It is not a how/to guide or a survey of technologies and platforms. It will not meaningfully explain to you how speech recognition or self-driving works. It will give you a working sense of what the pieces are and how the mechanism works.
That is what I was looking for, and it delivered.
This is a very good attempt at explaining the basic components of deep learning. It doesn’t go to deep, nor gloss over too much. The author does commendable service explaining the underlying math, and while not ‘breezy’ it is immensely useful.
I found it simplifies topics that I struggled to grasp around back propagation and how neurons optimized outcomes. The chapter on back-propagation math was beyond my ability, but I could grasp the basic concepts.
That is the purpose of this work. It is not a how/to guide or a survey of technologies and platforms. It will not meaningfully explain to you how speech recognition or self-driving works. It will give you a working sense of what the pieces are and how the mechanism works.
That is what I was looking for, and it delivered.
May 3, 2020
4-decent. A readable introduction. Useful to get an overhead/much-simplified view while learning about NNs and deep learning in depth in class.
March 24, 2022
I listened to the unabridged 7-hour audio version of this title (read by Joel Richards), containing an accessible introduction to AI technologies enabling computer vision, speech recognition, machine translation, driverless cars, and many of our other daily conveniences, as a way of familiarizing myself with concepts of deep learning, but then acquired and read the hard-copy paperback edition in order to gain a level of understanding not possible without paying due attention to formulas and diagrams/images.
Computer scientist John Kelleher offers a concise but comprehensive introduction to the fundamental techniques at the heart of the latest surge in artificial intelligence research & development. After reviewing the history of AI and its ups and downs, resulting from successes and disappointments, Kelleher explains that deep learning enables data-driven decisions by identifying and extracting patterns from large datasets. The rise of deep learning was in no small part due to the availability of such large datasets from sources such as healthcare records, climate research, space telescopes, and sensor networks.
Kelleher describes important deep-learning architectures, including autoencoders, recurrent neural networks, and long short-term networks, as well as more-recent developments such as generative adversarial networks and capsule networks. He also covers two fundamental algorithms in deep learning, namely, gradient descent and backpropagation. He ends by discussing possible developmental paths and challenges faced in further advancing deep learning methods and their applications.
Computer scientist John Kelleher offers a concise but comprehensive introduction to the fundamental techniques at the heart of the latest surge in artificial intelligence research & development. After reviewing the history of AI and its ups and downs, resulting from successes and disappointments, Kelleher explains that deep learning enables data-driven decisions by identifying and extracting patterns from large datasets. The rise of deep learning was in no small part due to the availability of such large datasets from sources such as healthcare records, climate research, space telescopes, and sensor networks.
Kelleher describes important deep-learning architectures, including autoencoders, recurrent neural networks, and long short-term networks, as well as more-recent developments such as generative adversarial networks and capsule networks. He also covers two fundamental algorithms in deep learning, namely, gradient descent and backpropagation. He ends by discussing possible developmental paths and challenges faced in further advancing deep learning methods and their applications.
July 28, 2026
Wow. I went in armed with high school bio, some nebulous concept of an artificial neuron, and my dinky perceptron coded in C I made in the spring for my systems class that I didn't understand until I read this. I found the chapter talking about the history the most fascinating, seeing the contributions of all the different people in furthering DL, and also appreciated the historical context for why funding ebbed and flowed throughout the years -- Not relevant to the mechanisms of the neurons but interesting + relevant auxiliary information that makes the content stick for me.
Also found the last chapter very interesting in putting other things like quantum and neuromorphic computing + transformers (hot topic of the year!!!) into context that I hear or read things about all the time but never understood the connection to DL. Also realizing that I definitely need to top up math beyond what my CS degree requires (no calc 3). I understood everything until chapter 6 with occasional difficulty but then really dropped the ball with backpropagation. This book was exactly what I wanted it to be but if the purpose of the MIT Essential Knowledge series is to be accessible to everyone (including people who aren't familiar with matrices, calculus) it's definitely a little heavy on the math. ALSO, I did not expect the number of typos that my copy had: at least 5. I stopped counting after 5. A little ridiculous for an MIT Press book.
Overall fun, interesting, informative, and definitely inspired me to learn more.
Also found the last chapter very interesting in putting other things like quantum and neuromorphic computing + transformers (hot topic of the year!!!) into context that I hear or read things about all the time but never understood the connection to DL. Also realizing that I definitely need to top up math beyond what my CS degree requires (no calc 3). I understood everything until chapter 6 with occasional difficulty but then really dropped the ball with backpropagation. This book was exactly what I wanted it to be but if the purpose of the MIT Essential Knowledge series is to be accessible to everyone (including people who aren't familiar with matrices, calculus) it's definitely a little heavy on the math. ALSO, I did not expect the number of typos that my copy had: at least 5. I stopped counting after 5. A little ridiculous for an MIT Press book.
Overall fun, interesting, informative, and definitely inspired me to learn more.
January 1, 2023
From this book I was expecting a profound overview about Deep Learning - with just enough detail to understand its theoretical fundamentals, its opportunities and challenges, its applications and trends. And I'm glad to say, the book delivered exactly that.
On 295 pages, John D. Kelleher gives a very good wrap-up of the topic, including the conceptual foundations derived from biological neurons, the main modeling building blocks, the scientific history of the topic Deep Learning, the two most prominent types of Deep Neural Networks - Convolutional Neural Networks and Recurrent Neural Networks -, training methodology including steepest descent and backpropagation and lastly an overview of current trends and an outlook towards potential future applications.
The book is quite compact, well-written and structured so that it is definitely applicable to a wide audience. It enables newcomers to quickly pick up the most important aspects, while at the same time challenging tech graduates with some mathematical proofs, therefore offering something to everyone who is either new to the topic or wants to complete their basic knowledge. Of course, as it says, this is a title from MIT Essential Knowledge Series, so do not expect finest detail, latest news or complete implementation advice. It is what it is: Essential Knowledge about the current state of Deep Learning - and as such has my absolute recommendation.
On 295 pages, John D. Kelleher gives a very good wrap-up of the topic, including the conceptual foundations derived from biological neurons, the main modeling building blocks, the scientific history of the topic Deep Learning, the two most prominent types of Deep Neural Networks - Convolutional Neural Networks and Recurrent Neural Networks -, training methodology including steepest descent and backpropagation and lastly an overview of current trends and an outlook towards potential future applications.
The book is quite compact, well-written and structured so that it is definitely applicable to a wide audience. It enables newcomers to quickly pick up the most important aspects, while at the same time challenging tech graduates with some mathematical proofs, therefore offering something to everyone who is either new to the topic or wants to complete their basic knowledge. Of course, as it says, this is a title from MIT Essential Knowledge Series, so do not expect finest detail, latest news or complete implementation advice. It is what it is: Essential Knowledge about the current state of Deep Learning - and as such has my absolute recommendation.
July 25, 2023
Three-and-a-half-stars, rounded down. I found this book a useful read personally, but the level of technical detail was ... surprising. As noted in other reviews, the author went surprisingly narrow and surprisingly deep (and for the reviewer who wanted more math, did you understand the remit for the series this book belonged to?). Given this line from the series introduction:
I was surprised to find a book that relied on math so heavily for its descriptions. (Basically, introductory college math including calculus, some light matrix algebra, and some concepts from statistics.)
But that aside, the author does do a good job of hammering home the important concepts of the field (note that I read this immediately after Machine Learning, and I have an (ancient) CS degree, so I was somewhat primed for this stuff). I do wish he had spent some more time on actual applications, social issues and general context, but these are (deliberately) short books.
Synthesizing specialized subject matter for nonspecialists and engaging critical topics through fundamentals, each of these compact volumes offers readers a point of access to complex ideas.
I was surprised to find a book that relied on math so heavily for its descriptions. (Basically, introductory college math including calculus, some light matrix algebra, and some concepts from statistics.)
But that aside, the author does do a good job of hammering home the important concepts of the field (note that I read this immediately after Machine Learning, and I have an (ancient) CS degree, so I was somewhat primed for this stuff). I do wish he had spent some more time on actual applications, social issues and general context, but these are (deliberately) short books.
June 23, 2023
Advertised as a book that could explain the working mechanisms of deep learning AIs, but is ultimately too technical for someone like me with zero background in the field. The book starts out easy to understand, only to soon devolve into a summarized textbook for those already more knowledgeable about AI and deep learning. I forced myself to read the book to the end, in hopes it would start explaining things in easier terms again, but ultimately feel like I wasted my time, especially since I took my sweet time reading the pages, really trying to soak in the information.
I was hoping to be educated on the real world applications of deep learning AIs and the theoretical problems that may arise and/or are already making themselves present, but alas. Much technical stuff about the mathematical equations that go to determining the desired results of deep learning AIs, a few lines about the history of AI that don't go any deeper than mere mentions, and not much anything else. A disappointing read all in all.
I was hoping to be educated on the real world applications of deep learning AIs and the theoretical problems that may arise and/or are already making themselves present, but alas. Much technical stuff about the mathematical equations that go to determining the desired results of deep learning AIs, a few lines about the history of AI that don't go any deeper than mere mentions, and not much anything else. A disappointing read all in all.
May 11, 2024
For anyone with a basic background in applied math, economics, natural sciences, or statistics who know what fitting a model js generally all about, parts of this book will be very elementary. However, it quickly proves its merit, running through the basics of problems and methods that the average non-AI/ML person won’t be familiar with. This book was sufficient for me to define deep learning and understand conceptually how the models are fit. A little surprised that the book dealt so little with theoreticals problems in overfitting, given how much this is emphasized in other data mining and statistical learning texts such as Hastie’s Elements of Statistical Learning. It also didn’t deal very much with practical training techniques to avoid overfitting like cross validation. Finally, it does not sufficiently cover the inherent limits of deep learning, and problems of inference caused by the models not knowing the difference between causal association and correlation. I recommend reading online the critiques of Gary Marcus for this perspective.
July 2, 2020
This is definitely very gentle, and very informative introduction to the deep learning. To me it seems that this book even will be clear to people without prior experience with deep learning. Kelleher dissects neural networks to their basic units and explains their basics along with mathematics (mostly high-level math, but for deeper understanding the reader can look for more technical books). Then, he puts all the elements together and after this there is no magic left under the hood of the deep learning algorithms.
I think that this book will help everyone to get at least general understanding, what is deep learning, because it is not only modern buzzword, but the essential part of the modern informational systems.
I think that this book will help everyone to get at least general understanding, what is deep learning, because it is not only modern buzzword, but the essential part of the modern informational systems.
August 24, 2022
Kelleher does a damned fine job here and it really should be paired with Alpaydin's Machine Learning, like a steak with um... another steak really. Very rich stuff.
Deep learning is kinda sorta the next part of machine learning but top down wise Kelleher assumes - correctly with the subject and all - that the technology is being pushed in a direction is based on some core ideas but the most important thing is how machine learning is focused on parameters, but Kelleher break that down well first by calling them "weighted" in terms of their relevance to processing.
He makes some incredibly important points about how deep learning is supposed to work vs. how it does work and also vs how it's used.
It's a good read but you'll need to drink water.
Deep learning is kinda sorta the next part of machine learning but top down wise Kelleher assumes - correctly with the subject and all - that the technology is being pushed in a direction is based on some core ideas but the most important thing is how machine learning is focused on parameters, but Kelleher break that down well first by calling them "weighted" in terms of their relevance to processing.
He makes some incredibly important points about how deep learning is supposed to work vs. how it does work and also vs how it's used.
It's a good read but you'll need to drink water.
April 23, 2024
I was pleasantly surprised by this book. I was looking for an introductory book on deep learning, and all the deep, technical, books were very expensive so I decided to first try this one. I was worried it might be a fluffy, "popular" book with little content, but I was surprised when it actually included deep and interesting intuitions on how deep neural networks actually work, and a lot of interesting details (e.g., for the first time I understood how, historically, the strange "activation functions" were chosen).
The book is not perfect - it could have been longer covering more things, and the back-propagation section was unreadable for me even though I already knew the material (it has all the right math, but written mostly as text :-)), but I would still recommend it.
The book is not perfect - it could have been longer covering more things, and the back-propagation section was unreadable for me even though I already knew the material (it has all the right math, but written mostly as text :-)), but I would still recommend it.
February 1, 2021
The book is a good introduction to the basics of neural networks and the different algorithms used in the field, as a student , the author does a great job explaining the perceptron, hidden layers, and different activation functions used, way much better than what I "learned" at school, after reading it I didn't have any problem following more complex books and videos about the topic, Excelent technical introduction and a brief history of deep learning.
Highly recomment it if you want to start learning about this topic.
Highly recomment it if you want to start learning about this topic.
November 15, 2021
A Good Introduction to Artificial Intelligence by John D. Kelleher
Not sure if anyone can read Russell's work on A.I straight in a setting.
Outline:
1 Introduction to Deep Learning
2 Conceptual Foundations
3 Neural Networks: The Building Blocks of Deep Learning
4 A Brief History of Deep Learning
5 Convolutional and Recurrent Neural Networks
6 Learning Functions
7 The Future of Deep Learning
In other words, Deep Learning can also be defined as representation learning.
Not sure if anyone can read Russell's work on A.I straight in a setting.
Outline:
1 Introduction to Deep Learning
2 Conceptual Foundations
3 Neural Networks: The Building Blocks of Deep Learning
4 A Brief History of Deep Learning
5 Convolutional and Recurrent Neural Networks
6 Learning Functions
7 The Future of Deep Learning
In other words, Deep Learning can also be defined as representation learning.
October 17, 2022
A decent intro to machine learning (ML) with a focus on deep learning (neural networks). This one is definitely more technical and math oriented, with little focus on applications and ML's impact on the world which would have been a more accessible and interesting perspective for the lay-audeince it was advertised for. Additionally, there were quite a few spelling/grammatical errors and I found the author's use of metaphors to be excessive and rather distracting at times. I imagine there are much better books that serve the same function.
Read
February 22, 2024Good review of concepts, nice refresher
Personal notes:
Should revisit how to use autoencoders in keras, matrix decomposition for word embeddings, best weight initialization methods, t-SNE, markov chains, using LSTMs as a hidden layer in a reccurent NN, BERT transformer models. Also should investigate how to use weak supervision using snorkel and apply it to deep learning for effective model training, as well as most effective data labeling techniques for least computational costs. GraphQL and other graph algorithms too?
Personal notes:
Should revisit how to use autoencoders in keras, matrix decomposition for word embeddings, best weight initialization methods, t-SNE, markov chains, using LSTMs as a hidden layer in a reccurent NN, BERT transformer models. Also should investigate how to use weak supervision using snorkel and apply it to deep learning for effective model training, as well as most effective data labeling techniques for least computational costs. GraphQL and other graph algorithms too?
November 30, 2024
It was a great read, especially if you want to review Deep Learning concepts. It has good explanations and illustrations, but in my opinion, it's far from complete. If you are beginning with Deep Learning and Neural Networks, I suggest starting with Understanding Deep Learning because of better ML and math explanations and better illustrations. And then use this book as a spaced repetition tool to reinforce the DL concepts. The last chapter about the future of DL is great but also far from complete.
December 1, 2024
It was a great read, especially if you want to review Deep Learning concepts. It has good explanations and illustrations, but in my opinion, it's far from complete. If you are beginning with Deep Learning and Neural Networks, I suggest starting with Understanding Deep Learning because of better ML and math explanations and better illustrations. And then use this book as a spaced repetition tool to reinforce the DL concepts. The last chapter about the future of DL is great but also far from complete.
Displaying 1 - 30 of 77 reviews