I am trying to build a Python package that interops with C++ code. I created a PyBind11 wrapper to do the interop. I created a setup.py file following the instructions here:

from pybind11.setup_helpers import Pybind11Extension

ext_modules = [
    Pybind11Extension(
        "python_example",
        sorted(glob("src/*.cpp")),  # Sort source files for reproducibility
extra_link_args=[f"-Wl,-rpath,$ORIGIN"]
    ),
]

setup(..., ext_modules=ext_modules)

I set extra_link_args=[f"-Wl,-rpath,$ORIGIN"] as I understand it will embed the origin to the search path of native libraries. (I have also tried without it with no luck). My C++ module depends on a shared library that I put right beside __init__.py. But when I try to import my C++ module I get :

ImportError: libxxx: cannot open shared object file: No such file or directory

How can I fix the error?

Adding more details and will be offering a bounty tomorrow as I am at the end of my wits:

1. Verify that dependency is placed in same directory as pybind module:

$ tree -L 1 venv/lib/python3.10/site-packages/my_package/
venv/lib/python3.10/site-packages/my_package/
├── __init__.py
├── __pycache__
├── my_data
├── libxxx.so.1
├── libyyy.so
├── libyyy.so.1.14.1
└── my_package_cpp.cpython-310-x86_64-linux-gnu.so

2. Verify there is an entry setting RUNPATH to $ORIGIN which is supposed to instruct the dynamic linker to add directory containing the module to its search path:

$ readelf -d venv/lib/python3.10/site-packages/my_package/my_package.cpython-310-x86_64-linux-gnu.so | grep -E 'RPATH|RUNPATH'
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN]

3. And yet the dynamic linker searches everywhere except for the directory containing the module itself (the $ORIGIN):

LD_DEBUG=libs python3 -c "import my_pybind_module"

4. ldd does not complain:

$ ldd venv/lib/python3.10/site-packages/my_package/my_package_cpp.cpython-310-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffea75f7000)
        libxxx.so.1 => /home/me/my_project/venv/lib/python3.10/site-packages/my_package/libxxx.so.1 (0x00007f072f352000)
        libyyy.so.1.14.1 => /home/me/my_project/venv/lib/python3.10/site-packages/my_package/libyyy.so.1.14.1 (0x00007f072e3bb000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f072e187000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f072e167000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f072df3e000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f072de55000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f072de50000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f072de4b000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f072de46000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f072f48c000)