Information Theory, Inference, and Learning Algorithms

by David J.C. MacKay

Information theory and inference, often taught separately, are here united in one entertaining textbook. These topics lie at the heart of many exciting areas of contemporary science and engineering - communication, signal processing, data mining, machine learning, pattern recognition, computational neuroscience, bioinformatics, and cryptography. This textbook introduces theory in tandem with applications. Information theory is taught alongside practical communication systems, such as arithmetic coding for data compression and sparse-graph codes for error-correction. A toolbox of inference techniques, including message-passing algorithms, Monte Carlo methods, and variational approximations, are developed alongside applications of these tools to clustering, convolutional codes, independent component analysis, and neural networks. The final part of the book describes the state of the art in error-correcting codes, including low-density parity-check codes, turbo codes, and digital fountain codes -- the twenty-first century standards for satellite communications, disk drives, and data broadcast. Richly illustrated, filled with worked examples and over 400 exercises, some with detailed solutions, David MacKay's groundbreaking book is ideal for self-learning and for undergraduate or graduate courses. Interludes on crosswords, evolution, and sex provide entertainment along the way. In sum, this is a textbook on information, communication, and coding for a new generation of students, and an unparalleled entry point into these subjects for professionals in areas as diverse as computational biology, financial engineering, and machine learning.

Genres: Computer Science, Mathematics, Science, Textbooks, Artificial Intelligence, Nonfiction, Technical

642 pages, Paperback

First published June 15, 2002

About the author

David MacKay was a Professor in the Department of Physics at the University of Cambridge. He studied Natural Sciences at Cambridge and then obtained his PhD in Computation and Neural Systems at the California Institute of Technology. He returned to Cambridge as a Royal Society research fellow at Darwin College. He was internationally known for his research in machine learning, information theory, and communication systems, including the invention of Dasher, a software interface that enables efficient communication in any language with any muscle. He has taught Physics in Cambridge since 1995. Since 2005, he devoted much of his time to public teaching about energy. He was a member of the World Economic Forum Global agenda Council on Climate Change.

Reviews

[Maurizio Codogno]

Teoria dell'informazione ma non solo

Questo libro è piuttosto strano per come mischia argomenti di solito ben separati. Io l'ho preso per la parte di teoria dell'informazione (vista in parte con un approccio bayesiano), ma si trovano temi come l'inferenza bayesiana per l'appunto, temi fisici come il modello di Ising, reti neurali... il tutto con una visione per l'appunto direi quasi olistica, il che permette di capire meglio come funzionano le cose (oltre che capire perché il mantra "bisogna che le parole in codice siano tutte massimamente separate tra loro" non è necessariamente la scelta migliore da fare: tanto i teoremi di Shannon su un canale rumoroso non possono per ipotesi assicurare il 100% di corretta decodifica, e allora tanto vale lasciare qualche possibile errore, se si può correggere meglio il resto.)
Altra cosa bella di questo libro è che è possibile scaricarlo (ma non è permesso stamparlo: non che io stamperei 640 pagine di testo) dal sito dell'autore, www.inference.org.uk/mackay/itila/boo... .