I have written a Python GUI for a media player (a shared library I created from ffplay in ffmpeg). It uses replay gain to adjust volume so that most music sounds about equally loud. I was not able to find any shared library that calculates replay gain so I run ffmpeg as a subprocess to return replay gain via a pipe through its stderr output.

I run an ffmpeg subprocess for each music file in a separate thread. I use threads because running the subprocess commands from the main GUI causes short gaps or hiccups in the playback of files (the player also runs in a thread). What I have discovered is that if more than one of the threads for ffmpeg runs concurrently, the Python GUI will not shutdown cleanly.

I should explain what I mean by not shutting down cleanly. I use Geany for my IDE; it runs the program in a terminal window. When everything terminates as it should, messages appear in the window "(program exited with code: 0) Press return to continue." By pressing Enter, the window closes immediately.

What I observe instead is that when more than one of the threads runs concurrently, the same messages appear but pressing Enter will not close the window. I must either close it by clicking on the X in the upper left corner or pressing ctrl+C.

I use a timer to create and start threads. On my system, most threads complete in about 2 sec. By setting the timer interval to 3 sec or more, everything shuts down properly. If I shorten the interval to about 2 sec or less, the problem I describe above occurs. I currently have the timer set to 100 msec. For a variety of reasons, I do not what to use an interval longer than that.

I have pared the GUI down to a minimal example. This example does not have the player. It is just a widget that starts a sequence of threads that each run ffmpeg in a subprocess.

The example code is shown below. I am running Python 3.12, PyQt 5.15 on Linux Mint 22.1. Just in case it matters, I am using ffmpeg 7.1.

    import sys, subprocess, queue, threading
    from PyQt5 import QtWidgets, QtGui, QtCore

    class AudioPlayerDialog(QtWidgets.QDialog):
        def __init__(self, mediaList):
            super().__init__()
            vbox = QtWidgets.QVBoxLayout()
            vbox.addWidget(QtWidgets.QLabel('WIDGET'))
            self.setLayout(vbox)
            self.calcGainTimer = None
            self.gdCalcThreads = None
            self.finished.connect(self.closeEvent)

            self.mediaList = mediaList
            self.mediaListLen = len(mediaList)
            self.normGainList = [-1. for x in mediaList]
            self.buildNormGainList()

        def closeEvent(self, event):
            print('**** close event')
            if self.calcGainTimer is not None:
                print('**** stopping calcGainTimer')
                self.calcGainTimer.stop()
            if self.gdCalcThreads is not None:
                print('**** checking gain calc threads')
                for idx, t in self.gdCalcThreads.copy().items():
                    if t.is_alive():
                        print('**** waiting for thread {} to finish'.format(idx))
                        t.join()
            print('**** closing ...')

        def buildNormGainList(self):
            self.calcGainQueue = queue.LifoQueue(maxsize=self.mediaListLen + 10)
            for i in range(self.mediaListLen-1, -1, -1):
                self.calcGainQueue.put_nowait(i)
            # setup gain/dur calculation thread dictionary
            self.gdCalcThreads = dict()
            # For each expiration of the following timer, a thread will be
            # started to calculate the replay gain for a particular file
            # and a check will determine if all files have been processed.
            print('@@@@ starting updt timer')
            self.calcGainTimer = QtCore.QTimer()
            self.calcGainTimer.timeout.connect(self.stopGainDurCalcWhenDone)
            self.calcGainTimer.setInterval(100) # interval in msec.
            self.calcGainTimer.start()

        def stopGainDurCalcWhenDone(self):
            # this is the target of the updt timer
            try:
                # start a thread if the queue is not empty
                idx = self.calcGainQueue.get_nowait()
                if idx not in self.gdCalcThreads:
                    print('@@@@ creating thread for', idx)
                    thread = threading.Thread(None, self.requestGain, args=(idx,))
                    self.gdCalcThreads[idx] = thread
                    thread.start()
            except queue.Empty:
                pass
            except Exception as err:
                print('**** ERROR: get request from calcGainQueue failed.\n\t'\
                                                            '{}'.format(err))
            # check if any thread finished but updates not yet finished
            for idx,t in self.gdCalcThreads.items():
                if not t.is_alive() and self.normGainList[idx]  < 0.:
                    # Thread finished; update not yet completed. Invert the
                    # sign of the gain, which indicates update completed.
                    self.normGainList[idx] = -self.normGainList[idx]
                    print('@@@@ update finished for', idx)

            # check if all files have been processed; stop updt timer if so
            done = True     # assume all calc done
            for i in range(self.mediaListLen):
                if self.normGainList[i] < 0.:
                    done = False
                    break
            if done:
                print('@@@@ all calc are done -----------------------')
                # all files have been processed
                print('@@@@ stopping updt timer')
                self.calcGainTimer.stop()
                self.calcGainTimer = None
                self.gdCalcThreads = None

        TEST = False  # set True to pass dummy filename of ffmpeg

        def requestGain(self, idx):
            print('@@@@ start subprocess for', idx)
            if self.TEST:
                fileName = 'XXXX'
            else:
                fileName = self.mediaList[idx]
            cmd = ['ffmpeg', '-i', '{}'.format(fileName), '-hide_banner',
                   '-af', 'replaygain', '-f', 'null', '-']
            proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            try:
                _,err = procmunicate(timeout=30)
                text = err.decode('utf8')
            except subprocess.TimeoutExpired:
                print('**** Wait for report from subprocess expired')
                proc.kill()
                text = ''
            except Exception as err:
               print('**** Unable to get report from subprocess: {}'.format(err))
               proc.kill()
               text = ''
            # actual program parses gain from text
            gain = 1.
            self.normGainList[idx] = -gain  # save negative gain


    if __name__ == '__main__':

        app = QtWidgets.QApplication(sys.argv)

        # put full-qualified filenames of media in list
        mediaList = [
            ]

        apd = AudioPlayerDialog(mediaList)
        apd.exec()

        print('\nFINI')
        sys.exit()

I tried running the subprocess without a PIPE. That did not make a difference.

A variable named TEST appears immediately above the requestGain() routine. By setting it True, a dummy filename is passed to ffmpeg. This causes the program to terminate quickly. Nothing useful is done but it prevents two threads from running concurrently and demonstrates the program will shutdown cleanly.

As you may see, the problem occurs even though all threads have completed.

It may be important to point out again that the the player is running in a threading.Thread instance. The player thread plus a single thread for the ffmpeg subprocess does not cause a problem. Two ffmpeg subprocess threads alone (without the player) cause the problem.

I must be doing something wrong but I cannot see what it is. Perhaps you can see it?

EDIT: I did more testing. I opened two terminal windows and ran top in one window to monitor processes and ran the Python GUI program in the other window.

top reported a Python process for the program and an ffmpeg process for each of the threads started to run those processes.

The Python program printed on the console a message as each thread ended and top showed the ffmpeg processes launched by those threads also ended. Eventually, all ffmpeg processes ended and only the Python process and its bash window remained. I terminated the Python program and its processes also ended and disappeared.

The bash window where I ran the Python program returned to the prompt but it did not respond to any keyboard input except for a ctrl+C which caused a new prompt to appear. Clicking on the X in the upper-right corner closed the window and that bash process disappeared in top.

Does anyone have any suggestions on how I can diagnose what is happening? How can I discover what is not terminating correctly?

EDIT2: I revised the try/except block by removing the wait, adding a timeout value for receiving communication and checking specifically for the TimeExpired exception. These changes did not solve the problem.

Something is preventing the process that ran the Python program from terminating. Any suggestions for discovering more about this will be appreciated.

EDIT3: I revised the code to use the ThreadPoolExecutor in concurrent.futures. I can provide the example if anyone is interested.

This change did not help; same experience. The program appears to shutdown OK but something is left hanging as I describe above.

EDIT 4: I replaced the subprocess call with a time.sleep(5) call. Multiple threads ran concurrently BUT the program shutdown cleanly. There is something about the subprocess call that is not terminating completely. As I mentioned above, these processes disappear from what top shows but there is something not visible to top that lingers. I am going to edit the title of this post to mention subprocess.

EDIT 5: I changed the title of the post to refer specifically to ffmpeg. See my answer below.

I have written a Python GUI for a media player (a shared library I created from ffplay in ffmpeg). It uses replay gain to adjust volume so that most music sounds about equally loud. I was not able to find any shared library that calculates replay gain so I run ffmpeg as a subprocess to return replay gain via a pipe through its stderr output.

I run an ffmpeg subprocess for each music file in a separate thread. I use threads because running the subprocess commands from the main GUI causes short gaps or hiccups in the playback of files (the player also runs in a thread). What I have discovered is that if more than one of the threads for ffmpeg runs concurrently, the Python GUI will not shutdown cleanly.

I should explain what I mean by not shutting down cleanly. I use Geany for my IDE; it runs the program in a terminal window. When everything terminates as it should, messages appear in the window "(program exited with code: 0) Press return to continue." By pressing Enter, the window closes immediately.

What I observe instead is that when more than one of the threads runs concurrently, the same messages appear but pressing Enter will not close the window. I must either close it by clicking on the X in the upper left corner or pressing ctrl+C.

I use a timer to create and start threads. On my system, most threads complete in about 2 sec. By setting the timer interval to 3 sec or more, everything shuts down properly. If I shorten the interval to about 2 sec or less, the problem I describe above occurs. I currently have the timer set to 100 msec. For a variety of reasons, I do not what to use an interval longer than that.

I have pared the GUI down to a minimal example. This example does not have the player. It is just a widget that starts a sequence of threads that each run ffmpeg in a subprocess.

The example code is shown below. I am running Python 3.12, PyQt 5.15 on Linux Mint 22.1. Just in case it matters, I am using ffmpeg 7.1.

    import sys, subprocess, queue, threading
    from PyQt5 import QtWidgets, QtGui, QtCore

    class AudioPlayerDialog(QtWidgets.QDialog):
        def __init__(self, mediaList):
            super().__init__()
            vbox = QtWidgets.QVBoxLayout()
            vbox.addWidget(QtWidgets.QLabel('WIDGET'))
            self.setLayout(vbox)
            self.calcGainTimer = None
            self.gdCalcThreads = None
            self.finished.connect(self.closeEvent)

            self.mediaList = mediaList
            self.mediaListLen = len(mediaList)
            self.normGainList = [-1. for x in mediaList]
            self.buildNormGainList()

        def closeEvent(self, event):
            print('**** close event')
            if self.calcGainTimer is not None:
                print('**** stopping calcGainTimer')
                self.calcGainTimer.stop()
            if self.gdCalcThreads is not None:
                print('**** checking gain calc threads')
                for idx, t in self.gdCalcThreads.copy().items():
                    if t.is_alive():
                        print('**** waiting for thread {} to finish'.format(idx))
                        t.join()
            print('**** closing ...')

        def buildNormGainList(self):
            self.calcGainQueue = queue.LifoQueue(maxsize=self.mediaListLen + 10)
            for i in range(self.mediaListLen-1, -1, -1):
                self.calcGainQueue.put_nowait(i)
            # setup gain/dur calculation thread dictionary
            self.gdCalcThreads = dict()
            # For each expiration of the following timer, a thread will be
            # started to calculate the replay gain for a particular file
            # and a check will determine if all files have been processed.
            print('@@@@ starting updt timer')
            self.calcGainTimer = QtCore.QTimer()
            self.calcGainTimer.timeout.connect(self.stopGainDurCalcWhenDone)
            self.calcGainTimer.setInterval(100) # interval in msec.
            self.calcGainTimer.start()

        def stopGainDurCalcWhenDone(self):
            # this is the target of the updt timer
            try:
                # start a thread if the queue is not empty
                idx = self.calcGainQueue.get_nowait()
                if idx not in self.gdCalcThreads:
                    print('@@@@ creating thread for', idx)
                    thread = threading.Thread(None, self.requestGain, args=(idx,))
                    self.gdCalcThreads[idx] = thread
                    thread.start()
            except queue.Empty:
                pass
            except Exception as err:
                print('**** ERROR: get request from calcGainQueue failed.\n\t'\
                                                            '{}'.format(err))
            # check if any thread finished but updates not yet finished
            for idx,t in self.gdCalcThreads.items():
                if not t.is_alive() and self.normGainList[idx]  < 0.:
                    # Thread finished; update not yet completed. Invert the
                    # sign of the gain, which indicates update completed.
                    self.normGainList[idx] = -self.normGainList[idx]
                    print('@@@@ update finished for', idx)

            # check if all files have been processed; stop updt timer if so
            done = True     # assume all calc done
            for i in range(self.mediaListLen):
                if self.normGainList[i] < 0.:
                    done = False
                    break
            if done:
                print('@@@@ all calc are done -----------------------')
                # all files have been processed
                print('@@@@ stopping updt timer')
                self.calcGainTimer.stop()
                self.calcGainTimer = None
                self.gdCalcThreads = None

        TEST = False  # set True to pass dummy filename of ffmpeg

        def requestGain(self, idx):
            print('@@@@ start subprocess for', idx)
            if self.TEST:
                fileName = 'XXXX'
            else:
                fileName = self.mediaList[idx]
            cmd = ['ffmpeg', '-i', '{}'.format(fileName), '-hide_banner',
                   '-af', 'replaygain', '-f', 'null', '-']
            proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            try:
                _,err = procmunicate(timeout=30)
                text = err.decode('utf8')
            except subprocess.TimeoutExpired:
                print('**** Wait for report from subprocess expired')
                proc.kill()
                text = ''
            except Exception as err:
               print('**** Unable to get report from subprocess: {}'.format(err))
               proc.kill()
               text = ''
            # actual program parses gain from text
            gain = 1.
            self.normGainList[idx] = -gain  # save negative gain


    if __name__ == '__main__':

        app = QtWidgets.QApplication(sys.argv)

        # put full-qualified filenames of media in list
        mediaList = [
            ]

        apd = AudioPlayerDialog(mediaList)
        apd.exec()

        print('\nFINI')
        sys.exit()

I tried running the subprocess without a PIPE. That did not make a difference.

A variable named TEST appears immediately above the requestGain() routine. By setting it True, a dummy filename is passed to ffmpeg. This causes the program to terminate quickly. Nothing useful is done but it prevents two threads from running concurrently and demonstrates the program will shutdown cleanly.

As you may see, the problem occurs even though all threads have completed.

It may be important to point out again that the the player is running in a threading.Thread instance. The player thread plus a single thread for the ffmpeg subprocess does not cause a problem. Two ffmpeg subprocess threads alone (without the player) cause the problem.

I must be doing something wrong but I cannot see what it is. Perhaps you can see it?

EDIT: I did more testing. I opened two terminal windows and ran top in one window to monitor processes and ran the Python GUI program in the other window.

top reported a Python process for the program and an ffmpeg process for each of the threads started to run those processes.

The Python program printed on the console a message as each thread ended and top showed the ffmpeg processes launched by those threads also ended. Eventually, all ffmpeg processes ended and only the Python process and its bash window remained. I terminated the Python program and its processes also ended and disappeared.

The bash window where I ran the Python program returned to the prompt but it did not respond to any keyboard input except for a ctrl+C which caused a new prompt to appear. Clicking on the X in the upper-right corner closed the window and that bash process disappeared in top.

Does anyone have any suggestions on how I can diagnose what is happening? How can I discover what is not terminating correctly?

EDIT2: I revised the try/except block by removing the wait, adding a timeout value for receiving communication and checking specifically for the TimeExpired exception. These changes did not solve the problem.

Something is preventing the process that ran the Python program from terminating. Any suggestions for discovering more about this will be appreciated.

EDIT3: I revised the code to use the ThreadPoolExecutor in concurrent.futures. I can provide the example if anyone is interested.

This change did not help; same experience. The program appears to shutdown OK but something is left hanging as I describe above.

EDIT 4: I replaced the subprocess call with a time.sleep(5) call. Multiple threads ran concurrently BUT the program shutdown cleanly. There is something about the subprocess call that is not terminating completely. As I mentioned above, these processes disappear from what top shows but there is something not visible to top that lingers. I am going to edit the title of this post to mention subprocess.

EDIT 5: I changed the title of the post to refer specifically to ffmpeg. See my answer below.

• python
• multithreading
• subprocess
• shutdown

• There's a bunch of issues here. For one, you don't want to confuse threads and processes. Threads don't run concurrently, processes do. Your main issue is likely with the .wait then communicate and the .kill in the except section - you can end up in a deadlock there. – Grismar Commented Mar 18 at 22:49
• I do understand the difference between processes and threads. I merely use the word "concurrently" as it is often used in the literature For decades (at least since IBM's MFT OS) there has been a distinction made between concurrently and simultaneously. As for the kill, it doesn't run of course unless there is an exception. I put it there because the Python docs advise it -- if the wait times out, kill the process. – dave Commented Mar 19 at 2:25
• in Python: use threading for IO-bound tasks, processing for CPU-bound tasks. See concurrent from the standard library for a high level interface – cards Commented Mar 19 at 10:21
• @cards Thank you but I am already well aware of what you said. I do not see how what you mentioned helps address the problem I described. Do you have any thoughts about how to solve the problem I am having or how to diagnose the cause so I can develop a solution? – dave Commented Mar 19 at 16:56
• It closed for me when I add a keyPressEvent which calls self.close() – Kenny Ostrom Commented Mar 19 at 22:08

1 Answer 1

I have learned more about the problem that I will share this just in case someone else runs into a similar situation.

First, the problem is very subtle. if you run the program in a terminal window, it appears to terminate successfully. The messages from all print statements appear in the terminal window, including the 'FINI' message at the end of the program. The command line prompt appears next. All looks good until you try to do anything else in that window. It ignores all key strokes except for a ctrl+C and that causes only another command line prompt to appear. Nothing can be entered; no command can be executed.

The problem occurs when two or more of the threads that launch submprocess.Popen instances run concurrently. If they do not overlap in time, no problem occurs.

I include another sample program below that reproduces the problem on my systems, running Linux Mint 21.3 and 22.1 with Python 3.10 and 3.12. It is much simpler than the code I included in the original post. If you set TEST to True, each invocation of ffmpeg returns very quickly and avoids running concurrently with the next thread / subprocess invocation. With TEST set to True, the problem does not occur, which shows that concurrency is required for the problem to arise.

The latest thing I have learned is that this problem occurs if more than instance of ffmpeg runs concurrently but it does not occur for other programs like Python running a script, ffplay plalying a media file, or grep searching for files.

My investigation continues.

EDIT: Someone helping me discovered that when the program ran with threads and their processes running concurrently, the command line window appeared to become unresponsive because key presses were no longer echoed to the screen. By running stty sane, echoing could be restored and the window behaved normally.

I have since tried launching the program from a context menu in my file manager and by irexec in the lirc package using an IR remote. The program can be launched repeatedly with no apparent problem. There probably is a bug in ffmpeg and/or Python subprocess management but not important enough to devote any resources to find it and fix it.

#!/usr/bin/env python3
#

import sys, subprocess, threading, time

TEST = False

def doSubProcess():
    if TEST:
        fileName = 'XXXX'
    else:
        fileName = '<<fully-qualified file name of an audio file>>'

    cmd = ['ffmpeg', '-i', '{}'.format(fileName), '-hide_banner',
           '-af', 'replaygain', '-f', 'null', '-']
    #cmd = ['ffplay', '"'<<fully-qualified file name of an audio file>>'"', '-autoexit']
    #cmd = ['grep', '-rw', '/usr', '-e', '"XYZ"']

    print('starting subprocess')
    proc = subprocess.Popen(cmd, stderr=subprocess.PIPE)

    try:
        _,err = procmunicate(timeout=60)
    except Exception as err:
        print('**** ERROR: for {} from subprocess: {}'.format(fileName, err))
        proc.kill()
    print('ending subprocess for', fileName)


if __name__ == '__main__':

    threadDict = dict()

    for i in range(5):
        print('starting thread for', i)
        thread = threading.Thread(None, doSubProcess)
        threadDict[i] = thread
        thread.start()
        time.sleep(.05)

    threadDict = dict()

    for i,t in threadDict.items():
        if t.is_alive():
            print('waiting on thread', i)
            t.join()

    print('\nFINI')
    sys.exit()