Jason Brownlee's Blog, page 24
February 23, 2023
Forking Processes is 20x Faster Than Spawning in Python
Forking a process is faster than spawning a process in Python. This is generally known, but how much faster is forking and when should we consider adopting the fork start method for child processes over spawning? In this tutorial, you will discover the speed differences between fork and spawn start methods. Let’s get started. Fork […]
February 21, 2023
ThreadPool Producer-Consumer Pattern in Python
You can create a producer thread pool and a consumer thread pool connected by a shared queue. This allows many producer tasks to run concurrently as well as many consumer tasks to run concurrently, allowing the producer-consumer pattern to scale with the amount of work or capabilities of the system. In this tutorial, you will […]
February 19, 2023
Multiprocessing Deadlock in Python
You can identify multiprocessing deadlocks by seeing examples and developing an intuition for their common causes. In most cases, deadlocks can be avoided by using best practices in concurrency programming, such as lock order, using time outs on waits, and using context managers when acquiring locks. In this tutorial, you will discover how to identify […]
February 16, 2023
Multiprocessing Pool with Different Function Types
You can execute functions, methods, and static methods as tasks in the multiprocessing pool. Some other types of functions, such as lambda functions and nested inner functions cannot be executed in the multiprocessing pool because they cannot be pickled. In this tutorial, you will discover how to execute many different function types in the multiprocessing […]
February 14, 2023
Process-Safe Counter in Python
You can develop a process-safe counter class using a multiprocessing.Value and a mutex lock. In this tutorial, you will discover how to develop a process-safe counter in Python. Let’s get started. Need a Process-Safe Counter A counter is an object that maintains a private variable that changes via methods, e.g. incremented, and accessed. We often […]
February 12, 2023
How to Order Parallel Tasks in Python
You can order parallel by having the tasks coordinate themselves or by splitting tasks into parallel and sequential subtasks. In this tutorial, you will discover how to order tasks that are executed in parallel in Python. This tutorial was inspired by questions and discussions with Casey M. Thank you deeply! If you have a question […]
February 9, 2023
Why Python Developers Hate Asyncio
Asyncio is strongly disliked, perhaps hated by many Python developers. This can be seen in the comments on social media when asyncio in Python is discussed. I believe this mostly stems from Python developers misunderstanding the promise of asyncio (e.g. not speed or ease of use) and not taking asyncio seriously as an alternative programming […]
February 7, 2023
Why Do Python Developers Hate The GIL?
Python does not support parallelism via threads because of the Global Interpreter Lock or GIL. As such, many Python developers HATE the GIL. This refrain is then repeated and repeated by newer developers, leaving behind the nuance of the limitations of the GIL and leading to statements like “Python does not support concurrency“, which is […]
February 5, 2023
How to Use 100% of All CPU Cores in Python
You can use all CPU cores in your system at nearly 100% utilization by using process-based concurrency. This is suited for tasks that are CPU-bound, that is, run as fast as your CPU cores can execute. In this tutorial, you will discover how to update Python programs to use all CPU cores on your system. […]
February 2, 2023
What are the Two Asyncio APIs
The asyncio module in Python provides a low-level and a high-level API. The low-level API is for library and framework developers, whereas the high-level API builds on top of the low-level API and is intended for application developers. Nevertheless, the presence and wide use of both APIs make things confusing to those just getting started […]
