More on this book
Kindle Notes & Highlights
Read between
October 6, 2017 - September 22, 2018
Protected instance attributes should be in _leading_underscore format. • Private instance attributes should be in __double_leading_underscore format.
Instance methods in classes should use self as the name of the first parameter (which refers to the object). • Class methods should use cls as the name of the first parameter (which refers to the class).
• Use inline negation (if a is not b) instead of negation of positive expressions (if not a is b).
Imports should be in sections in the following order: standard library modules, third-party modules, your own modules. Each subsection should have imports in alphabetical order.
To convert Unicode characters to binary data, you must use the encode method. To convert binary data to Unicode characters, you must use the decode method.
Slicing deals properly with start and end indexes that are beyond the boundaries of the list. That makes it easy for your code to establish a maximum length to consider for an input sequence.
If you leave out both the start and the end indexes when slicing, you’ll end up with a copy of the original list.
If you assign a slice with no start or end indexes, you’ll replace its entire contents with a copy of what’s referenced (instead of allocating a new list).
Python has special syntax for the stride of a slice in the form somelist[start:end:stride]. This lets you take every nth item when slicing a sequence.
The problem is that the stride syntax often causes unexpected behavior that can introduce bugs. For example, a common Python trick for reversing a byte string is to slice the string with a stride of -1. x = b'mongoose' y = x[::-1] print(y) >>> b'esoognom'
indexes. If you must use stride with start or end indexes, consider using one assignment to stride and another to slice. Click here to view code image b = a[::2] # ['a', 'c', 'e', 'g'] c = b[1:-1] # ['c', 'e'] Slicing and then striding will create an extra shallow copy of the data. The first operation should try to reduce the size of the resulting slice by as much as possible.
list. You can do this by providing the expression for your computation and the input sequence to loop over. Click here to view code image a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] squares = [x**2 for x in a] print(squares) >>> [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
Unless you’re applying a single-argument function, list comprehensions are clearer than the map built-in function for simple cases. map requires creating a lambda function for the computation, which is visually noisy. Click here to view code image squares = map(lambda x: x ** 2, a) Unlike map, list comprehensions let you easily filter items from the input list, removing corresponding outputs from the result.
Here, I do this with a list comprehension by including two for expressions. These expressions run in the order provided from left to right. Click here to view code image matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] flat = [x for row in matrix for x in row] print(flat) >>> [1, 2, 3, 4, 5, 6, 7, 8, 9]
my_lists = [ [[1, 2, 3], [4, 5, 6]], # ... ] flat = [x for sublist1 in my_lists for sublist2 in sublist1 for x in sublist2] At this point, the multiline comprehension isn’t much shorter than the alternative. Here, I produce the same result using normal loop statements. The indentation of this version makes the looping clearer than the list comprehension. flat = [] for sublist1 in my_lists: for sublist2 in sublist1: flat.extend(sublist2)
Multiple conditions at the same loop level are an implicit and expression.
The problem with list comprehensions (see Item 7: “Use List Comprehensions Instead of map and filter”) is that they may create a whole new list containing one item for each value in the input sequence. This is fine for small inputs, but for large inputs this could consume significant amounts of memory and cause your program to crash. For example, say you want to read a file and return the number of characters on each line. Doing this with a list comprehension would require holding the length of every line of the file in memory. If the file is absolutely enormous or perhaps a never-ending
...more
Generator expressions don’t materialize the whole output sequence when they’re run. Instead, generator expressions evaluate to an iterator that yields one item at a time from the expression. A generator expression is created by putting list-comprehension-like syntax between () characters. Here, I use a generator expression that is equivalent to the code above. However, the generator expression immediately evaluates to an iterator and doesn’t make any forward progress. Click here to view code image it = (len(x) for x in open('/tmp/my_file.txt'))
Chaining generators like this executes very quickly in Python. When you’re looking for a way to compose functionality that’s operating on a large stream of input, generator expressions are the best tool for the job.
Python provides the enumerate built-in function for addressing this situation. enumerate wraps any iterator with a lazy generator. This generator yields pairs of the loop index and the next value from the iterator. The resulting code is much clearer.
enumerate provides concise syntax for looping over an iterator and getting the index of each item from the iterator as
Prefer enumerate instead of looping over a range and indexing into a sequence.
To make this code clearer, Python provides the zip built-in function. In Python 3, zip wraps two or more iterators with a lazy generator. The zip generator yields tuples containing the next value from each iterator. The resulting code is much cleaner than indexing into multiple lists.
This is just how zip works. It keeps yielding tuples until a wrapped iterator is exhausted. This approach works fine when you know that the iterators are of the same length, which is often the case for derived lists created by list comprehensions. In many other cases, the truncating behavior of zip is surprising and bad. If you aren’t confident that the lengths of the lists you want to zip are equal, consider using the zip_longest function from the itertools built-in module instead (also called izip_longest in Python 2).
Python has special syntax that allows else blocks to immediately follow for and while loop interior blocks. The else block after a loop only runs if the loop body did not encounter a break statement. Avoid using else blocks after loops because their behavior isn’t intuitive and can be confusing.
Use try/except/else to make it clear which exceptions will be handled by your code and which exceptions will propagate up. When the try block doesn’t raise an exception, the else block will run. The else block helps you minimize the amount of code in the try block and improves
Functions that return None to indicate special meaning are error prone because None and other values (e.g., zero, the empty string) all evaluate to False in conditional expressions. Raise exceptions to indicate special situations instead of returning None. Expect the calling code to handle exceptions properly when they’re documented.
Functions are first-class objects in Python, meaning you can refer to them directly, assign them to variables, pass them as arguments to other functions, compare them in expressions and if statements, etc. This is how the sort method can accept a closure function as the key argument.
Avoid making dictionaries with values that are other dictionaries or long tuples. Use namedtuple for lightweight, immutable data containers before you need the flexibility of a full class. Move your bookkeeping code to use multiple helper classes when your internal state dictionaries get complicated.
Only consider using private attributes to avoid naming conflicts with subclasses that are out of your control.
you perform type checking and validation on values passed to your class. Here, I define a class that ensures all resistance values are above zero ohms:
Concurrency is when a computer does many different things seemingly at the same time. For example, on a computer with one CPU core, the operating system will rapidly change which program is running on the single processor. This interleaves execution of the programs, providing the illusion that the programs are running simultaneously. Parallelism is actually doing many different things at the same time. Computers with multiple CPU cores can execute multiple programs simultaneously. Each CPU core runs the instructions of a separate program, allowing each program to make forward progress during
...more
Python makes it easy to write concurrent programs. Python can also be used to do parallel work through system calls, subprocesses, and C-extensions. But it can be very difficult to make concurrent Python code truly run in parallel. It’s important to understand how to best utilize Python in these subtly different situations.
The standard implementation of Python is called CPython. CPython runs a Python program in two steps. First, it parses and compiles the source text into bytecode. Then, it runs the bytecode using a stack-based interpreter. The bytecode interpreter has state that must be maintained and coherent while the Python program executes. Python enforces coherence with a mechanism called the global interpreter lock (GIL). Essentially, the GIL is a mutual-exclusion lock (mutex) that prevents CPython from being affected by preemptive multithreading, where one thread takes control of a program by
...more
There are ways to get CPython to utilize multiple cores, but it doesn’t work with the standard Thread class (see Item 41: “Consider concurrent.futures for True Parallelism”) and it can require substantial effort. Knowing these limitations you may wonder, why does Python support threads at all? There are two good reasons. First, multiple threads make it easy for your program to seem like it’s doing multiple things at the same time. Managing the juggling act of simultaneous tasks is difficult to implement yourself (see Item 40: “Consider Coroutines to Run Many Functions Concurrently” for an
...more
This highlight has been truncated due to consecutive passage length restrictions.
The problem is that while the slow_systemcall function is running, my program can’t make any other progress. My program’s main thread of execution is blocked on the select system call. This situation is awful in practice. You need to be able to compute your helicopter’s next move while you’re sending it a signal, otherwise it’ll crash. When you find yourself needing to do blocking I/O and computation simultaneously, it’s time to consider moving your system calls to threads.
The parallel time is 5× less than the serial time. This shows that the system calls will all run in parallel from multiple Python threads even though they’re limited by the GIL. The GIL prevents my Python code from running in parallel, but it has no negative effect on system calls. This works because Python threads release the GIL just before they make system calls and reacquire the GIL as soon as the system calls are done.
There are many other ways to deal with blocking I/O besides threads, such as the asyncio built-in module, and these alternatives have important benefits. But these options also require extra work in refactoring your code to fit a different model of execution (see Item 40: “Consider Coroutines to Run Many Functions Concurrently”). Using threads is the simplest way to do blocking I/O in parallel with minimal changes to your program.
Things to Remember Python threads can’t run bytecode in parallel on multiple CPU cores because of the global interpreter lock (GIL). Python threads are still useful despite the GIL because they provide an easy way to do multiple things at seemingly the same time. Use Python threads to make multiple system call...
This highlight has been truncated due to consecutive passage length restrictions.
Decorators have the ability to run additional code before and after any calls to the functions they wrap. This allows them to access and modify input arguments and return values. This functionality can be useful for enforcing semantics, debugging, registering functions, and more.
The solution is to use the wraps helper function from the functools built-in module. This is a decorator that helps you write decorators. Applying it to the wrapper function will copy all of the important metadata about the inner function to the outer function.
Decorators are Python syntax for allowing one function to modify another function at runtime. Using decorators can cause strange behaviors in tools that do introspection, such as debuggers. Use the wraps decorator from the functools built-in module when you define your own decorators to avoid any issues.
For example, you can add documentation by providing a docstring immediately after the def statement of a function.
A standard way of defining documentation makes it easy to build tools that convert the text into more appealing formats (like HTML). This has led to excellent documentation-generation tools for the Python community, such as Sphinx (http://sphinx-doc.org). It’s also enabled community-funded sites like Read the Docs (https://readthedocs.org) that provide free hosting of beautiful-looking documentation for open source Python projects.
Once you’ve written docstrings for your modules, it’s important to keep the documentation up to date. The doctest built-in module makes it easy to exercise usage examples embedded in docstrings to ensure that your source code and its documentation don’t diverge over time.
Python can limit the surface area exposed to API consumers by using the __all__ special attribute of a module or package. The value of __all__ is a list of every name to export from the module as part of its public API. When consuming code does from foo import *, only the attributes in foo.__all__ will be imported from foo. If __all__ isn’t present in foo, then only public attributes, those without a leading underscore, are imported (see Item 27: “Prefer Public Attributes Over Private Ones”).
Simple packages are defined by adding an __init__.py file to a directory that contains other source files. These files become the child modules of the directory’s package. Package directories may also contain other packages.
In some cases, using ValueError makes sense, but for APIs it’s much more powerful to define your own hierarchy of exceptions. You can do this by providing a root Exception in your module. Then, have all other exceptions raised by that module inherit from the root exception.
Having a root exception in a module makes it easy for consumers of your API to catch all of the exceptions that you raise on purpose. For example, here a consumer of your API makes a function call with a try/except statement that catches your root exception:
This try/except prevents your API’s exceptions from propagating too far upward and breaking the calling program. It insulates the calling code from your API. This insulation has three helpful effects.