Fluent Python: Clear, Concise, and Effective Programming

by Luciano Ramalho

The is operator is faster than ==, because it cannot be overloaded, so Python does not have to find and invoke special methods to evaluate it, and computing is as simple as comparing two integer IDs.

Tuples, like most Python collections—lists, dicts, sets, etc.—are containers: they hold references to objects.

the immutability of tuples really refers to the physical contents of the tuple data structure (i.e., the references it holds), and does not extend to the referenced objects.

What can never change in a tuple is the identity of the items it contains.

However, using the constructor or [:] produces a shallow copy (i.e., the outermost container is duplicated, but the copy is filled with references to the same items held by the original container). This saves memory and causes no problems if all the items are immutable. But if there are mutable items, this may lead to unpleasant surprises.

The only mode of parameter passing in Python is call by sharing. That is the same mode used in most object-oriented languages, including JavaScript, Ruby, and Java (this applies to Java reference types; primitive types use call by value). Call by sharing means that each formal parameter of the function gets a copy of each reference in the arguments. In other words, the parameters inside the function become aliases of the actual arguments.

The problem is that each default value is evaluated when the function is defined—i.e., usually when the module is loaded—and the default values become attributes of the function object. So if a default value is a mutable object, and you change it, the change will affect every future call of the function.

Unless a method is explicitly intended to mutate an object received as an argument, you should think twice before aliasing the argument object by simply assigning it to an instance variable in your class. If in doubt, make a copy. Your clients will be happier. Of course, making a copy is not free: there is a cost in CPU and memory. However, an API that causes subtle bugs is usually a bigger problem than one that is a little slower or uses more resources.

Objects are never explicitly destroyed; however, when they become unreachable they may be garbage-collected.

The second surprising fact is that del deletes references, not objects. Python’s garbage collector may discard an object from memory as an indirect result of del, if the deleted variable was the last reference to the object.

In CPython, the primary algorithm for garbage collection is reference counting. Essentially, each object keeps count of how many references point to it. As soon as that refcount reaches zero, the object is immediately destroyed: CPython calls the __del__ method on the object (if defined) and then frees the memory allocated to the object.

del does not delete objects, but objects may be deleted as a consequence of being unreachable after del is used.

Weak references to an object do not increase its reference count. Therefore, a weak reference does not prevent the target object from being garbage collected. Weak references are useful in caching applications because you don’t want the cached objects to be kept alive just because they are referenced by the cache.

The sharing of string literals is an optimization technique called interning. CPython uses a similar technique with small integers to avoid unnecessary duplication of numbers that appear frequently in programs like 0, 1, –1, etc. Note that CPython does not intern all strings or integers, and the criteria it uses to do so is an undocumented implementation detail.

Never depend on str or int interning! Always use == instead of is to compare strings or integers for equality. Interning is an optimization for internal use of the Python interpreter.

Every Python object has an identity, a type, and a value. Only the value of an object may change over time.

Simple assignment does not create copies.

Augmented assignment with += or *= creates new objects if the lefthand variable is bound to an immutable object, but may modify a mutable object in place.

Assigning a new value to an existing variable does not change the object previously bound to it. This is called a rebinding: the variable is now bound to a different object. If that variable was the last reference to ...

Function parameters are passed as aliases, which means the function may change any mutable object received as an argument. There is no way to prevent this, except making local copies or using immutable objects (e.g., passing a tuple instead of a list).

Using mutable objects as default values for function parameters is dangerous because if the parameters are changed in place, then the default is changed, affecting every future call that relies on the default.

A popular way of explaining how parameter passing works in Python is the phrase: “Parameters are passed by value, but the values are references.”

In Python, the function gets a copy of the arguments, but the arguments are always references. So the value of the referenced objects may be changed, if they are mutable, but their identity cannot. Also, because the function gets a copy of the reference in an argument, rebinding it in the function body has no effect outside of the function.

Functions in Python are first-class objects. Programming language researchers define a “first-class object” as a program entity that can be: Created at runtime Assigned to a variable or element in a data structure Passed as an argument to a function Returned as the result of a function Integers, strings, and dictionaries are other examples of first-class objects in Python—nothing fancy here.

Calling map(function, iterable) returns an iterable where each item is the result of calling the first argument (a function) to successive elements of the second argument (an iterable),

A function that takes a function as an argument or returns a function as the result is a higher-order function.

The map, filter, and reduce higher-order functions are still around, but better alternatives are available for most of their use cases,

In Python 3, map and filter return generators—a form of iterator—so their direct substitute is now a generator expression (in Python 2, these functions returned lists, therefore their closest alternative was a listcomp).

The common idea of sum and reduce is to apply some operation to successive items in a series, accumulating previous results, thus reducing a series of values to a single value.

The lambda keyword creates an anonymous function within a Python expression.

The best use of anonymous functions is in the context of an argument list for a higher-order function.

The lambda syntax is just syntactic sugar: a lambda expression creates a function object just like the def statement. That is just one of several kinds of callable objects in Python.

Given the variety of existing callable types in Python, the safest way to determine whether an object is callable is to use the callable() built-in:

A class implementing __call__ is an easy way to create function-like objects that have some internal state that must be kept across invocations,

Another good use case for __call__ is implementing decorators. Decorators must be callable, and it is sometimes convenient to “remember” something between calls of the decorator (e.g., for memoization—caching the results of expensive computations for later use) or to split a complex implementation into separate methods.

The functional approach to creating functions with internal state...

Closely related are the use of * and ** to unpack iterables and mappings into separate arguments when we call a function.

Keyword-only arguments are a feature of Python 3. In Example 7-9, the class_ parameter can only be given as a keyword argument—it will never capture unnamed positional arguments. To specify keyword-only arguments when defining a function, name them after the argument prefixed with *. If you don’t want to support variable positional arguments but still want keyword-only arguments, put a * by itself in the signature,

Note that keyword-only arguments do not need to have a default value: they can be mandatory, like b in the preceding example.

To define a function requiring positional-only parameters, use / in the parameter list.

All arguments to the left of the / are positional-only. After the /, you may specify other arguments, which work as usual.

The / in the parameter list is a syntax error in Python 3.7 or earlier.

partial takes a callable as first argument, followed by an arbitrary number of positional and keyword arguments to bind.

Programming language “paradigms” are a moribund and tedious legacy of a bygone age. Modern language designers pay them no respect, so why do our courses slavishly adhere to them?

What else to make of a language like Python, Ruby, or Perl? Their designers have no patience for the niceties of these Linnaean hierarchies; they borrow features as they wish, creating mélanges that utterly defy characterization.

Python is not, by design, a functional language—whatever that means. Python just borrows a few good ideas from functional languages.

Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention.

The goal is to help developer tools find bugs in Python codebases via static analysis, i.e., without actually running the code through tests.

You may very well write an excellent piece of Python code, with good test coverage and passing tests, but still be unable to add type hints that satisfy a type checker. That’s OK; just leave out the problematic type hints and ship it!

Seeking 100% coverage of type hints is likely to stimulate type hinting without proper thought, only to satisfy the metric. It will also prevent teams from making the most of the power and flexibility of Python. Code without type hints should naturally be accepted when annotations would make an API less user-friendly, or unduly complicate its implementation.