I have a fastapi aplication and I want to launch a another process with a python script in an non-blocking way:

process = subprocess.Popen(
                ["python","my_script.py"],
                stdout=subprocess.DEVNULL,
                stderr=subprocess.DEVNULL,
                stdin=subprocess.DEVNULL,
                start_new_session=True  # Detaches the process
)

Now, the problem is if that process fails it still lingers on as a zombie process. Only if I restart the fastapi aplication the process really dies. If I try to use os.fork in my_script.py to make sure I detach from the parent then a new process with new pid will be created and the original one will remain zombie.

The question is, from a system engineering point of view what is the most reliable way (and production suitable) to start/trigger python scripts from a fastapi aplication that can die properly if there is a problem ? I can not use celery because I'm trying to train machine learning models and this can take a lot of time and resources and I'm afraid celery workers are not suitable for that but please correct me if I'm wrong.