Concurrently Run CLI Processes with Python

Problem

I needed to run multiple processes via CLI simultaneously as part of a test set up I had to perform. Each of these commands follows a template with a few parameters that differs from each other. This required me to copy and paste multiple different strings again and again. I think you can agree that it is very tedious. The following snippet allows me to concurrently run these CLIs using Python’s subprocess module.

Solution

Define a new function to run the command.

import subprocess

def run(command):
    subprocess.Popen(command, cwd="Your/Working/Directory")

Use a loop to iterate over a list of my commands or parameters to run them all and store them in a list. The second while loop periodically checks the status of each process using poll and removes them from the list. The polling interval can be customized by modifying the waiting time.

import time

processes = []

for i in j:
    processes.append(run(i))

while processes:
    for process in processes:
        status = process.poll()

        if status is not None:
            processes.remove(process)
        else:
            time.sleep(15)
            continue