Python multiprocessing shared list

by

Python Multiprocessing Shared List

Multiprocessing is a powerful module in Python that allows you to create and manage multiple processes, which can run concurrently and independently. When using multiprocessing, each individual process has its own memory space.

However, sometimes it becomes necessary to share data between multiple processes. In such cases, the multiprocessing.Manager() class can be used to create a shared list that can be accessed and modified by different processes.

Example:

from multiprocessing import Process, Manager

def square_list(my_list, result_list):
    for num in my_list:
        result_list.append(num ** 2)

if __name__ == "__main__":
    shared_list_manager = Manager()
    shared_list = shared_list_manager.list()

    input_list = [1, 2, 3, 4, 5]
    processes = []

    for i in range(3):
        start_index = i * len(input_list) // 3
        end_index = (i + 1) * len(input_list) // 3

        p = Process(target=square_list, args=(input_list[start_index:end_index], shared_list))
        processes.append(p)
        p.start()
    
    for p in processes:
        p.join()

    print(f"Shared List: {shared_list}")

In the above example, we are creating a shared list using the multiprocessing.Manager().list() method. This shared list is assigned to the variable shared_list.

We then define a function square_list() that takes two arguments: my_list and result_list. Here, my_list represents a portion of the original list that will be squared, and result_list represents the shared list where the squared values will be appended.

Within the square_list() function, we iterate over each number in my_list and calculate its square. The resulting squared values are then appended to the result_list, which is the shared list.

In the main block, we create a list of processes and divide the input list into three equal parts. Each process will operate on one of these parts to square the numbers and append them to the shared list.

Finally, we start and join all the processes, ensuring that they complete before printing the contents of the shared list.

Leave a comment