I am trying to reduce the memory usage of an image processing (Python + OpenCV) app. When running the memory_profiler on a test script, I get a maximum memory usage of 211.4 MiB, however the python thread is using almost 3 GiB.

$ mprof run mem_Dewarper.py; mprof plot

Filename: mem_Dewarper.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     3     20.2 MiB     20.2 MiB           1   @profile
     4                                         def run_dewarper_test():
     5     20.2 MiB      0.0 MiB           1       import os, sys 
     6     20.2 MiB      0.0 MiB           1       import resource
     7     20.2 MiB      0.0 MiB           1       from os.path import join, abspath
     8                                         
     9     20.2 MiB      0.0 MiB           1       sys.path.append('./classes')
    10    101.6 MiB     81.4 MiB           1       from classes.dewarper import Dewarper
    11                                         
    12    101.6 MiB      0.0 MiB           1       PATH = "./"
    13    101.6 MiB      0.0 MiB           1       PATH_REF = abspath( join(PATH, "v07-100.png")  )
    14    101.6 MiB      0.0 MiB           1       path_img = "./aOriginal.jpg"
    15                                         
    16                                         
    17                                             ### Deskew the image
    18    150.3 MiB     48.7 MiB           1       dw = Dewarper(PATH_REF, path_img)
    19    164.0 MiB     13.7 MiB           1       dw.kpd_ref = dw.load_kpd("./kpd_v07.pkl")
    20    211.4 MiB     47.4 MiB           1       dw.dewarp()
    21                                         
    22    136.7 MiB    -74.7 MiB           1       del dw

Furthermore, when I restrict the script's memory resource to 3.5 GiB, it crashes with memory errors.

# mem_Dewarper.py
@profile
def run_dewarper_test():
    import os, sys 
    import resource
    from os.path import join, abspath

    os.environ['OPENBLAS_NUM_THREADS'] = '1'
    
    # Set maximum heap size (e.g., 500 MB)
    max_heap_size = 3500 * 1024 * 1024  # 3500 MB in bytes

    # Set memory limit
    resource.setrlimit(resource.RLIMIT_AS, (max_heap_size, max_heap_size))


    sys.path.append('./classes')
    from classes.dewarper import Dewarper

    PATH = "./"
    PATH_REF = abspath( join(PATH, "v07-100.png")  )
    path_img = "./aOriginal.jpg"


    ### Deskew the image
    dw = Dewarper(PATH_REF, path_img)
    dw.kpd_ref = dw.load_kpd("./kpd_v07.pkl")
    dw.dewarp()

    del dw

$ mprof run mem_Dewarper.py; mprof plot
mprof: Sampling memory every 0.1s
running new process
running as a Python program...
Traceback (most recent call last):
  File "/home/rh/.local/share/mambafe-pypy3/envs/test_scorer/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/rh/.local/share/mambafe-pypy3/envs/test_scorer/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/rh/.local/share/mambafe-pypy3/envs/test_scorer/lib/python3.9/site-packages/memory_profiler.py", line 1351, in <module>
    exec_with_profiler(script_filename, prof, args.backend, script_args)
  File "/home/rh/.local/share/mambafe-pypy3/envs/test_scorer/lib/python3.9/site-packages/memory_profiler.py", line 1252, in exec_with_profiler
    exec(compile(f.read(), filename, 'exec'), ns, ns)
  File "mem_Dewarper.py", line 46, in <module>
    run_dewarper_test()
  File "/home/rh/.local/share/mambafe-pypy3/envs/test_scorer/lib/python3.9/site-packages/memory_profiler.py", line 1188, in wrapper
    val = prof(func)(*args, **kwargs)
  File "/home/rh/.local/share/mambafe-pypy3/envs/test_scorer/lib/python3.9/site-packages/memory_profiler.py", line 761, in f
    return func(*args, **kwds)
  File "mem_Dewarper.py", line 29, in run_dewarper_test
    dw.dewarp()
  File "/home/rh/.local/share/mambafe-pypy3/envs/test_scorer/lib/python3.9/site-packages/memory_profiler.py", line 1188, in wrapper
    val = prof(func)(*args, **kwargs)
  File "/home/rh/.local/share/mambafe-pypy3/envs/test_scorer/lib/python3.9/site-packages/memory_profiler.py", line 761, in f
    return func(*args, **kwds)
  File "/home/rh/Dev/ts/source/classes/dewarper.py", line 441, in dewarp
    self.sift('img')
  File "/home/rh/Dev/ts/source/classes/dewarper.py", line 268, in sift
    kp, des = self.sifter.detectAndCompute(self.img, mask=None)
cv2.error: OpenCV(4.7.0) /home/conda/feedstock_root/build_artifacts/libopencv_1675729965945/work/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 192000000 bytes in function 'OutOfMemoryError'

Using last profile data.

The custom Dewarper class holds a maximum of 4 images in memory:

• 4.3 MiB
• 180 KiB
• 500 KiB
• 500 KiB

Its internal calls to OpenCV methods total 47.0 MiB. It does not contain any loops.

1. What does memory_profiler actually show?
2. How can I investigate why the python thread uses over 3 GiB when the line-by-line profile shows a max usage of 211 MiB?
3. Might this have something to do with OpenCV's C++ memory heap?