Jason Brownlee's Blog, page 23

Stop Main Process and Leave Child Processes Running in Python

You can stop the main process and allow child processes to continue running. This can be achieved by allowing the main process to stop executing custom code and automatically join the child processes, or to have the main process exit itself and release all resources. In this tutorial, you will discover how to stop the […]

Inheriting is 34x Faster Than Sending Data Between Processes in Python

Transmitting data between processes using a queue or a pipe requires that the data be pickled. This is much slower than a child process inheriting the data from a parent process either as a global variable or being passed the data as an argument when the process is created. In this tutorial, you will discover […]

Return Value From Child Process Without Changing The Target Function

You can execute a target function that returns a value in a child process without changing the target function by using a wrapper function to handle the transmission of the return value back to the caller automatically. An alternate approach is to define a callback function and have the wrapper function call the callback function […]

Fix a Broken Mutex Lock When Terminating Child Processes

You can fix and recover a mutex lock used in a terminated child process by first releasing it before using it again, and handling the case where the lock had not been acquired. In this tutorial, you will discover how to recover a mutex lock used in a terminated child process. Let’s get started. Broken […]

Instance Variables Are Not Shared Between Processes in Python

You can share changes to Python object instance variables among processes by changing instance variables to be shared ctypes. In this tutorial, you will discover how to share changes to Python object instance variables between processes. Let’s get started. Object Instance Variables Not Shared With Processes Python objects can be shared among processes. For example, […]

Execute Task in Child Process With a Timeout

You can run a task in a child process with a timeout by waiting for the task to complete for a fixed number of seconds by calling the join() method, then terminate the task if the child process is still running via the terminate() method. In this tutorial, you will discover how to execute a […]

How to Execute a Task with a Delay in a Child Process in Python

You can execute a task in a child process with a delay using a wrapper function that sleeps first before executing the target function. A more elaborate approach can be developed that extends the Process class allowing an arbitrary target function to be executed after a delay. The delayed task can also be canceled by […]

How to Share a Queue with a Multiprocessing Pool

You can share a queue with workers in the pool using the fork start method and inheritance of global variables or by sharing proxy objects for a queue hosted using a manager. In this tutorial, you will discover how to share a multiprocessing queue with tasks executed by child process workers in the multiprocessing pool […]

Threads are 4x Faster at Sharing Data Than Processes in Python

Processes are slower at transmitting data than threads. The rationale is that all data transmitted between processes requires the use of inter-process communication, whereas threads can directly access shared memory. We can design and run a controlled experiment to explicitly measure how much slower data transmission is between processes than between threads. In this tutorial, […]

Processes Are About 40x Slower Than Threads in Python

Processes are slow to start, threads are faster. In fact, threads are about 40x faster to create than processes in Python. The difference in time taken to create threads and processes depends on the specifics of the system and the start method used to create the threads. The difference in time can be calculated using […]