Jason Brownlee's Blog, page 16

Share NumPy Array Using Memory-Mapped File

You can share a NumPy array between processes by using a memory-mapped file. Each process is able to access the memory-mapped file directly. This means that we don’t have the overhead of sharing the array directly between processes, or the handle to the file. In this tutorial, you will discover how to share a NumPy […]

ThreadPoolExecutor Concurrent List Comprehension

You can execute a list comprehension concurrently with threads by using the ThreadPoolExecutor with either the submit() or map() methods. In this tutorial, you will discover how to execute a list comprehension concurrently using the ThreadPoolExecutor. Let’s get started. Need a Concurrent List Comprehension A list comprehension is a Python syntax for creating a list […]

ThreadPoolExecutor Concurrent For Loop

You can convert a for-loop to be concurrent using the ThreadPoolExecutor class. In this tutorial, you will discover how to convert a for-loop to be concurrently using the ThreadPoolExecutor. Let’s get started. Need a ConcurrentFor-Loop You have a for-loop and you want to execute each iteration concurrently. This is a common situation. The loop involves […]

How to Use ThreadPoolExecutor Timeouts

You can set a timeout when waiting for the ThreadPoolExecutor. This involves setting a “timeout” argument when processing task results via the ThreadPoolExecutor.map() method, Future.result() method, and Future.exception() method, as well as when waiting for tasks to complete via the concurrent.futures.wait() function and concurrent.futures.as_completed() function. In this tutorial, you will discover how to use a […]

concurrent.futures.Future and asyncio.Future Not Compatible

You can mix concurrent.futures.Future and asyncio.Future objects in your Python program because they are not compatible. This means that instances of the asyncio.Future class cannot be used in concurrent.futures module functions like concurrent.futures.wait() and concurrent.futures.as_completed() as it will cause an exception to be raised. Similarly, instances of the concurrent.futures.Future class cannot be used in asyncio […]

concurrent.futures.Future vs asyncio.Future

The Python standard library provides two Future classes. The first is in the concurrent.futures module and the second is in the asyncio module: This raises the question: What is the difference between the concurrent.futures.Future and asyncio.Future classes? Are they compatible with each other? For example, can we use asyncio.Future instances in the concurrent.futures.wait() and concurrent.futures.as_completed() […]

ThreadPoolExecutor Fill NumPy Array

You can fill a NumPy array in parallel using Python threads. NumPy will release the global interpreter lock (GIL) when calling a fill function, allowing Python worker threads in the ThreadPoolExecutor to run in parallel and populate different sub-arrays of a large shared array. This can offer up to a 3x speed-up when filling NumPy […]

ThreadPoolExecutor Pipeline For Multi-Step Tasks

You can execute multi-step concurrent tasks using a pipeline of thread pools in Python. In this tutorial, you will discover how to execute multi-step tasks using a ThreadPoolExecutors pipeline. Let’s get started. Need a Pipeline To Execute Multi-Step Tasks The ThreadPoolExecutor provides a pool of reusable worker threads using the executor design pattern. Tasks executed […]

Run One-Off File I/O Tasks in the Background

You can run one-off file IO tasks in the background using a new thread or new child process. Running a file IO task in the background allows the main thread of the program to continue on with other tasks. In this tutorial, you will discover how to run one-off file IO tasks in the background. […]

ThreadPoolExecutor Thread That Runs Done Callback Functions

The worker thread that executes the task is typically the same thread that executes the done callback function for the task. In this tutorial, you will discover which thread runs the done callback functions in the ThreadPoolExecutor. Let’s get started. What Thread Runs Done Callback Functions The ThreadPoolExecutor provides a pool of reusable worker threads […]