Python High Performance Programming

by Gabriele Lanaro

Boost the performance of your Python programs using advanced techniques

Overview

Identify the bottlenecks in your applications and solve them using the best profiling techniques
Write efficient numerical code in NumPy and Cython
Adapt your programs to run on multiple processors with parallel programming
In Detail

Python is a programming language with a vibrant community known for its simplicity, code readability, and expressiveness. The massive selection of third party libraries make it suitable for a wide range of applications. This also allows programmers to express concepts in fewer lines of code than would be possible in similar languages. The availability of high quality numerically-focused tools has made Python an excellent choice for high performance computing. The speed of applications comes down to how well the code is written. Poorly written code means poorly performing applications, which means unsatisfied customers.

This book is an example-oriented guide to the techniques used to dramatically improve the performance of your Python programs. It will teach optimization techniques by using pure python tricks, high performance libraries, and the python-C integration. The book will also include a section on how to write and run parallel code.

This book will teach you how to take any program and make it run much faster. You will learn state-of the art techniques by applying them to practical examples. This book will also guide you through different profiling tools which will help you identify performance issues in your program. You will learn how to speed up your numerical code using NumPy and Cython. The book will also introduce you to parallel programming so you can take advantage of modern multi-core processors.

This is the perfect guide to help you achieve the best possible performance in your Python applications.

What you will learn from this book

Assess the performance of your programs using benchmarks
Spot the bottlenecks in your code using the Python profiling tools
Speed up your code by replacing Python loops with NumPy
Boost NumPy performance using the numexpr compiler
Use Cython to reach performance on par with the C language
Write code for multiple processors
Profile, optimize, and rewrite an application from start to finish
Approach

An exciting, easy-to-follow guide illustrating the techniques to boost the performance of Python code, and their applications with plenty of hands-on examples.

Who this book is written for

If you are a programmer who likes the power and simplicity of Python and would like to use this language for performance-critical applications, this book is ideal for you. All that is required is a basic knowledge of the Python programming language. The book will cover basic and advanced topics so will be great for you whether you are a new or a seasoned Python developer.

Genres: Programming, Reference, Nonfiction

108 pages, Paperback

First published January 1, 2013

About the author

Gabriele Lanaro is a PhD student in Chemistry at the University of British Columbia, in the field of Molecular Simulation. He writes high performance Python code to analyze chemical systems in large-scale simulations. He is the creator of Chemlab—a high performance visualization software in Python—and emacs-for-python—a collection of emacs extensions that facilitate working with Python code in the emacs text editor. This book builds on his experience in writing scientific Python code for his research and personal projects.

Reviews

[Dgg32 · Rating 5 out of 5]

A nice primer on speeding up python programs. Very compact four chapters, non-nonse illustration, including profiling, numpy, cython and multiprocessing.

[Pyhead · Rating 5 out of 5]

The author has done an excellent job condensing his practical knowledge in this difficult and sometimes very technical field. The choice of content and format of presentation has resulted in a book where each chapter adds to a set of techniques that, without further study, can be immediately applied to get sometimes impressive performance increases. Although the book has a number-crunching theme much of the information applies equally to other domains, including the emphasis on benchmarking, profiling and checking that solutions scale well. It has also managed to avoid becoming a shopping-list affair of technologies/techniques and this helps keep the book focused and short enough to be a very practical introduction to the field.

I nearly skipped the chapter on NumPy but it turned out to be a very good and concise refresher. I would recommend it as a quick introduction for anyone not already familiar with NumPy. Even if you do already know NumPy basics, this whole chapter is fully justified towards the end with the introduction of the "numexpr" package and again in the third chapter when NumPy is combined with Cython's "buffer interface" and "typed memoryview". My past use of NumPy typically ended when I thought the code was "fast enough" but I'm now eager to revisit old code and breath new life into it :-)

With the rise of Numba it could be tempting to dismiss Cython but Numba is a relatively recent development, possibly after this book's creation and almost definitely after the bulk of the work it is based on (large sim's & "Chemlab"). Even if Numba is now a credible drop-in replacement, I suspect Cython might still provide more fine control and remains the current choice for many developers. It would be good to see a supplementary article by the author on this topic and other options.

The final chapter deals with parallel programming and, like other chapters, is sufficiently self contained to be a stand-alone introduction while still offering up techniques that are immediately useful. It manages to adequately cover the three main parallel options (including IPython's facilities) in less than 20 pages. This amazing feat is a testament to the authors' focus and the amount of thought put into the planning of this whole book.

Niggles: not many. The book is generally well written but maybe a little rushed in the first few pages of the first chapter where it sets up the example used for benchmarking/profiling and throughout the rest of the book (hint: just jump in, import simul.py and start benchmarking & profiling on your own machine). It would be nice for comparison to know the spec's of the system the author used for timings. There is no overall summary saying something like "This is what we started with, this is what we ended up with" and showing gains at each stage/technique. Likewise, to wrap the whole book up, it would have been good to include a short final chapter to set real world expectations and provide some pointers for further exploration.

Despite these niggles, the book is still an absolute bargain and that's even without factoring in the time it would take to pull all this information together yourself, from a multitude of sources, and judge what will get you the biggest gains for the least effort.

Well done Gabriele Lanaro (first book?) and Packt Publishing!

[Kami Bee · Rating 5 out of 5]

Probably the first Packt book I've read in a while that I thought was really worth my time. I was looking for a brief overview of how Cython worked, and the relevant chapter certainly delivered. I also particularly liked the coverage of parallel processing in the final chapter. It is definitely just a primer on all of the topics, but everything is explained well, there are few mistakes, and it is overall a good introduction to high performance computing techniques you can practically implement with your own equipment.

[Jon · Rating 2 out of 5]

Just about useless. Nothing here isn't covered better in the cProfile and cython docs. The example benchmark/testing suite setup throws everything into a single function; I was hoping for demonstration of some better style. The one plus side is an accessible, fairly complete discussion of numpy indexing and its weirdness.

[TN · Rating 3 out of 5]

It is a short book on high performance programming in Python. I found it a bit difficult to follow all the concepts.