Fluent Python: Clear, Concise, and Effective Programming

by Luciano Ramalho

This is the typical behavior of a decorator: it replaces the decorated function with a new function that accepts the same arguments and (usually) returns whatever the decorated function was supposed to return, while also doing some extra processing.

functools.wraps decorator to copy the relevant attributes from func to clocked.

The functools.cache decorator implements memoization:5 an optimization technique that works by saving the results of previous invocations of an expensive function, avoiding repeat computations on previously used arguments.

All the arguments taken by the decorated function must be hashable, because the underlying lru_cache uses a dict to store the results, and the keys are made from the positional and keyword arguments used in the calls.

functools.cache can consume all available memory if there is a very large number of cache entries. I consider it more suitable for use in short-lived command-line scripts. In long-running processes, I recommend using functools.lru_cache with a suitable maxsize parameter, as explained in the next section.

The functools.cache decorator is actually a simple wrapper around the older functools.lru_cache function, which is more flexible and compatible with Python 3.8 and earlier versions.

The main advantage of @lru_cache is that its memory usage is bounded by the maxsize parameter, which has a rather conservative default value of 128—which means the cache will hold at most 128 entries at any time.

The acronym LRU stands for Least Recently Used, meaning that older entries that have not been read for a while are di...

The functools.singledispatch decorator allows different modules to contribute to the overall solution, and lets you easily provide specialized functions even for types that belong to third-party packages that you can’t edit. If you decorate a plain function with @singledispatch, it becomes the entry point for a generic function: a group of functions to perform the same operation in different ways, depending on the type of the first argument. This is what is meant by the term single dispatch. If more arguments were used to select the specific functions, we’d have multiple dispatch.

bool is a subtype-of numbers.Integral, but the singledispatch logic seeks the implementation with the most specific matching type, regardless of the order they appear in the code.

If you don’t want to, or cannot, add type hints to the decorated function, you can pass a type to the @«base».register decorator. This syntax works in Python 3.4 or later.

The @«base».register decorator returns the undecorated function, so it’s possible to stack them to register two or more types on the same implementation.

A notable quality of the singledispatch mechanism is that you can register specialized functions anywhere in the system, in any module. If you later add a module with a new user-defined type, you can easily provide a new custom function to handle that type. And you can write custom functions for classes that you did not write and can’t change.

@singledispatch is not designed to bring Java-style method overloading to Python. A single class with many overloaded variations of a method is better than a single function with a lengthy stretch of if/elif/elif/elif blocks. But both solutions are flawed because they concentrate too much responsibility in a single code unit—the class or the function. The advantage of @singledispatch is supporting modular extension: each module can register a specialized function for each type it supports. In a realistic use case, you would not have all the implementations of generic functions in the same module

When parsing a decorator in source code, Python takes the decorated function and passes it as the first argument to the decorator function. So how do you make a decorator accept other arguments? The answer is: make a decorator factory that takes those arguments and returns a decorator, which is then applied to the function to be decorated.

Conceptually, the new register function is not a decorator but a decorator factory. When called, it returns the actual decorator that will be applied to the target function.

To accept parameters, the new register decorator must be called as a function

Lennart Regebro—a technical reviewer for the first edition—argues that decorators are best coded as classes implementing __call__, and not as functions like the examples in this chapter.

The question is: how to evaluate the free variables? The first and simplest answer is “dynamic scope.” This means that free variables are evaluated by looking into the environment where the function is invoked.

Functions should be opaque, with their implementation hidden from users. But with dynamic scope, if a function uses free variables, the programmer has to know its internals to set up an environment where it works correctly.

It’s an eloquent testimony to the dangers of dynamic scope that even the very first example of higher-order Lisp functions was broken because of it. It may be that McCarthy was not fully aware of the implications of dynamic scope in 1960. Dynamic scope remained in Lisp implementations for a surprisingly long time—until Sussman and Steele developed Scheme in 1975. Lexical scope does not complicate the definition of eval very much, but it may make compilers harder to write.

Today, lexical scope is the norm: free variables are evaluated considering the environment where the function is defined. Lexical scope complicates the implementation of languages with first-class functions, because it requires the support of closures. On the other hand, lexical scope makes source code easier to read. Most languages invented since Algol have lexical scope. One notable exception is JavaScript, where the special variable this is confusing because it can be lexically or dynamically scoped, depending on how the code is written.

Python function decorators fit the general description of decorator given by Gamma et al. in Design Patterns: “Attach additional responsibilities to an object dynamically. Decorators provide a flexible alternative to subclassing for extending functionality.”

At the implementation level, Python decorators do not resemble the classic decorator design pattern, but an analogy can be made.

The decorator conforms to the interface of the component it decorates so that its presence is transparent to the component’s clients. The decorator forwards requests to the component and may perform additional actions (such as drawing a border) before or after forwarding. Transparency lets you nest decorators recursively, thereby allowing an unlimited number of added responsibilities.”

Note that I am not suggesting that function decorators should be used to implement the decorator pattern in Python programs. Although this can be done in specific situations, in general the decorator pattern is best implemented with classes to represent the decorator and the components it will wrap.

Python does not have a program global scope, only module global scopes.

Conformity to patterns is not a measure of goodness.

In software engineering, a design pattern is a general recipe for solving a common design problem.

The choice of programming language is important because it influences one’s point of view. Our patterns assume Smalltalk/C++-level language features, and that choice determines what can and cannot be implemented easily. If we assumed procedural languages, we might have included design patterns called “Inheritance,” “Encapsulation,” and “Polymorphism.” Similarly, some of our patterns are supported directly by the less common object-oriented languages. CLOS has multi-methods, for example, which lessen the need for a pattern such as Visitor.

Define a family of algorithms, encapsulate each one, and make them interchangeable. Strategy lets the algorithm vary independently from clients that use it.

Once you get used to the idea that functions are first-class objects, it naturally follows that building data structures holding functions often makes sense.

globals() Return a dictionary representing the current global symbol table. This is always the dictionary of the current module (inside a function or method, this is the module where it is defined, not the module from which it is called).

The function inspect.getmembers returns the attributes of an object—in this case, the promotions module—optionally filtered by a predicate (a boolean function). We use inspect.isfunction to get only the functions from the module.

The goal of Command is to decouple an object that invokes an operation (the invoker) from the provider object that implements it (the receiver).

The idea is to put a Command object between the two, implementing an interface with a single method, execute, which calls some method in the receiver to perform the desired operation. That way the invoker does not need to know the interface of the receiver, and different receivers can be adapted through different Command subclasses. The invoker is configured with a concrete command and calls its execute method to operate it.

Quoting from Design Patterns, “Commands are an object-oriented replacement for callbacks.”

sometimes you may encounter a design pattern or an API that requires that components implement an interface with a single method, and that method has a generic-sounding name such as “execute,” “run,” or “do_it.” Such patterns or APIs often can be implemented with less boilerplate code in Python using functions as first-class objects.

Matching design patterns to language features is not an exact science.

For a library or framework to be Pythonic is to make it as easy and natural as possible for a Python programmer to pick up how to perform a task.

repr() Return a string representing the object as the developer wants to see it. It’s what you get when the Python console or a debugger shows an object.

str() Return a string representing the object as the user wants to see it. It’s what you get when you print() an object.

classmethod changes the way the method is called, so it receives the class itself as the first argument, instead of an instance. Its most common use is for alternative constructors, like frombytes

the staticmethod decorator changes a method so that it receives no special first argument. In essence, a static method is just like a plain function that happens to live in a class body, instead of being defined at the module level.

The f-strings, the format() built-in function, and the str.format() method delegate the actual formatting to each type by calling their .__format__(format_spec) method.

To make Vector2d work with positional patterns, we need to add a class attribute named __match_args__ , listing the instance attributes in the order they will be used for positional pattern matching:

In Python, there is no way to create private variables like there is with the private modifier in Java. What we have in Python is a simple mechanism to prevent accidental overwriting of a “private” attribute in a subclass.

if you name an instance attribute in the form __mood (two leading underscores and zero or at most one trailing underscore), Python stores the name in the instance __dict__ prefixed with a leading underscore and the class name, so in the Dog class, __mood becomes _Dog__mood, and in Beagle it’s _Beagle__mood. This language feature goes by the lovely name of name mangling.

Never, ever use two leading underscores. This is annoyingly private. If name clashes are a concern, use explicit name mangling instead (e.g., _MyThing_blahblah). This is essentially the same thing as double-underscore, only it’s transparent where double underscore obscures.

The single underscore prefix has no special meaning to the Python interpreter when used in attribute names, but it’s a very strong convention among Python programmers that you should not access such attributes from outside the class.8 It’s easy to respect the privacy of an object that marks its attributes with a single _, just as it’s easy respect the convention that variables in ALL_CAPS should be treated as constants.