part of Course 131 Data Munging Tips and Tricks

If you spend a lot of time waiting around for your code to run, and you've tried all the easier things, you can sometimes get a big speedup by parallelizing it — breaking it into chunks and running each chunk separately. Laptops typically typically have at least two cores, and many have four. Desktop boxes can have as many as 32. But Python is single-threaded. By default it will only run on one core at a time. The multiprocessing package helps us to use as many cores as we want.

Here's a minimal example that you can copy and paste to get started with.

from multiprocessing import Pool
import os
import numpy as np

def f(n):
    return np.var(np.random.sample((n, n)))

result_objs = []
n = 1000
with Pool(processes=os.cpu_count() - 1) as pool:
    for _ in range(n):
        result = pool.apply_async(f, (n,))
        result_objs.append(result)
    
    results = [result.get() for result in result_objs]
    print(len(results), np.mean(results), np.var(results))

• f(n) is a function with a burdensome calculation.

• Pool is a collection of worker processes.

• When initializing a Pool, the processes keyword argument chooses how many workers to create. It doesn't make sense to create more workers than you have processors. os.cpu_count() tells you exactly how many that is. If you leave one processor free you can still run Firefox and listen to Spotify while your code runs.

• Pool.apply_async() assigns a job to your worker pool, but doesn't wait around for the result. Instead it returns a placeholder. We can use the placeholder to get the actual result by calling result_placeholder.get().

• Pool.apply_async() takes a function as a first argument and a tuple of arguments for that function as a second argument. Because we want each worker to run f(n), we pass apply_async(f, (n,)).

• If you were to immediately call get() on the result placeholder from apply_async(), it would hold up the for loop while it waited for the result. In parallelization terminology, it blocks all the other workers from getting new jobs. In this case, you would still have several workers, but they would each take turns doing one job while the others stood around and waited. Better to collect all the result placeholders and gather up the results when the works have done their jobs.

The multiprocessing package is incredibly powerful. This is only the tiniest part of its capabilities. But hopefully it's a part that you'll find particularly useful.