I need to write some code in python that launches 8 processes in parallel and that, for each process, is able to report their return value and their duration in microseconds.

Unfortunately I don't know how to do it.

I imagined launching the processes with:

p = subprocess.Popen([f"{my_process}"], shell=True, universal_newlines = True, stderr=subprocess.STDOUT, stdout=DEVNULL)

and waiting for their termination with:

p.wait()

measuring their start and stop time with:

datetime.now()

but what I don't know how to do is capture the durations of each process in parallel, also taking the return value of each.

Do you think this task is too difficult for python? I guess not, but I can't get out of it!

Can someone help me please?

Thanks

I need to write some code in python that launches 8 processes in parallel and that, for each process, is able to report their return value and their duration in microseconds.

Unfortunately I don't know how to do it.

I imagined launching the processes with:

p = subprocess.Popen([f"{my_process}"], shell=True, universal_newlines = True, stderr=subprocess.STDOUT, stdout=DEVNULL)

and waiting for their termination with:

p.wait()

measuring their start and stop time with:

datetime.now()

but what I don't know how to do is capture the durations of each process in parallel, also taking the return value of each.

Do you think this task is too difficult for python? I guess not, but I can't get out of it!

Can someone help me please?

Thanks

• python
• subprocess
• return-value
• duration

2 Answers 2

actually you are on the right track, what I can advise you is that to start processes in parallel you use concurrent.futures, which is also thread-safe so it is better, and then to calculate the time I used the time library and I calculated the delta-time which gives us the time taken to start the process, i leave the code below:

import concurrent.futures
import subprocess
import time

def open_chrome(n):
    start_time = time.time()
    print(f"Opening Chrome instance {n}...")
    process = subprocess.run(('start chrome.exe'), shell=True, universal_newlines=True, stderr=subprocess.STDOUT) #i start chrome as process but you can try with whatever process you want
    end_time = time.time()
    elapsed_time = end_time - start_time #delta-time
    return f"Chrome instance {n} opened in {elapsed_time:.2f} seconds.", elapsed_time

if __name__ == "__main__":
    with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
        futures = {executor.submit(open_chrome, i): i for i in range(8)}
        
        for future in concurrent.futures.as_completed(futures):
            result, time_taken = future.result()
            print(result)

Let me know if everything works for you or if you have any other questions.

Using the scric's code base, which I thank very much, I developed the following code that suits my needs well. I post it here for common use.

First, to simulate the 8 processes, I wrote a small code that, before exiting, waits for a number of seconds passed from the command line and that returns 0 or 1 depending on whether the number of seconds waited is even or odd. The code, saved in test_process.c, is the following:

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>


int main (int argc, char** argv)
{
    int i, duration, quiet=0;

    if (   (argc < 2)
        || ( (duration = atoi(argv[1]))  <= 0 ))
        return -1;

    for (i=2; i<argc && !quiet; i++) 
        quiet = (0 == strcmp(argv[i], "quiet"));

    if (! quiet) for (i=2; i<argc; i++) printf("par_%d='%s' ", i, argv[i]);

    if (! quiet) printf("\nStart sleep for %d sec!\n", duration);
    sleep(duration);
    if (! quiet) printf("END sleep for %d sec!\n", duration);

    return duration & 1;
}

and it is compiled with:

cc test_process.c -o test_process

Secondly, I took the scric's code and put it in a python script called parallel.py

#!/usr/bin/python3

import concurrent.futures
import subprocess
from datetime import datetime 
from datetime import timedelta

class process :

    def __init__ (self, cmd) :
        self.invocation     = cmd
        self.duration       = None
        self.return_value   = None

    def __str__(self) :
        return f"invocation = '{self.invocation}', \tduration = {self.duration} msec, \treturn_value = {self.return_value}\n"

    def __repr__(self) :
        return f"<process: invocation = '{self.invocation}', \tduration = {self.duration} msec, \treturn_value = {self.return_value}>\n"

pars = [process("1 quiet tanks 4 your support!"),
        process("2 quiet 0xdead 0xbeef"  ),
        process("3 quiet three params here" ),
        process("4 quiet 2 parameters "   ),
        process("5 quiet many parameters here: one two three" ),
        process("6 quiet --1-- --6--"     ), 
        process("7 quiet ----- -----"    ),
        process("8 quiet ===== =====")]


def run_process(string_command, index):
    start_time = datetime.now()
    process    = subprocess.run((f'{string_command}'), shell=True, universal_newlines=True, stderr=subprocess.STDOUT)
    end_time   = datetime.now()
    delta_time = end_time - start_time 
    return process.returncode, delta_time, index

def main():
    with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
        futures = {executor.submit(run_process, f"./test_process {pars[i].invocation}", f"{i}"): i for i in range(8)}
        
        for future in concurrent.futures.as_completed(futures):
            result, time_taken, index       = future.result()
            pars[int(index)].duration       = time_taken / timedelta(milliseconds=1)
            pars[int(index)].return_value   = result
            print(f"{index}: result={result}, time_taken={time_taken}")

main()
print(pars)

Running parallel.py from the command line you get the following output:

./parallel.py 
0: result=1, time_taken=0:00:01.006900
1: result=0, time_taken=0:00:02.003389
2: result=1, time_taken=0:00:03.013232
3: result=0, time_taken=0:00:04.003463
4: result=1, time_taken=0:00:05.002579
5: result=0, time_taken=0:00:06.002372
6: result=1, time_taken=0:00:07.021016
7: result=0, time_taken=0:00:08.003653
[<process: invocation = '1 quiet tanks 4 your support!',    duration = 1006.9 msec,     return_value = 1>
, <process: invocation = '2 quiet 0xdead 0xbeef',   duration = 2003.389 msec,   return_value = 0>
, <process: invocation = '3 quiet three params here',   duration = 3013.232 msec,   return_value = 1>
, <process: invocation = '4 quiet 2 parameters ',   duration = 4003.463 msec,   return_value = 0>
, <process:  futures = {executor.submit(run_process, f"./test_process {pars[i].invocation}", f"{i}"): i for i in range(8)}
 = '5 quiet many parameters here: one two three',   duration = 5002.579 msec,   return_value = 1>
, <process: invocation = '6 quiet --1-- --6--',     duration = 6002.372 msec,   return_value = 0>
, <process: invocation = '7 quiet ----- -----',     duration = 7021.016 msec,   return_value = 1>
, <process: invocation = '8 quiet ===== =====',     duration = 8003.653 msec,   return_value = 0>
]

In this way all the information I need is saved in the <pars> object.

If you need to call different processes, just put the name of the process in self.invocation and change the line

 futures = {executor.submit(run_process, f"./test_process {pars[i].invocation}", f"{i}"): i for i in range(8)}

in

 futures = {executor.submit(run_process, f"{pars[i].invocation}", f"{i}"): i for i in range(8)}

obviously changing the definition of pars in this way

pars = [process("./test_process 1 quiet tanks 4 your support!"),
        process("./test_process 2 quiet 0xdead 0xbeef"  ),
        process("./test_process 3 quiet three params here" ),
        process("./test_process 4 quiet 2 parameters "   ),
        process("./test_process 5 quiet many parameters here: one two three" ),
        process("./test_process 6 quiet --1-- --6--"     ), 
        process("./test_process 7 quiet ----- -----"    ),
        process("./test_process 8 quiet ===== =====")]

Thanks to everyone!