When deploying the multimodal model ARIA (), I need to install the grouped_gemm acceleration framework to speed up the model's inference.

When running pip install grouped_gemm==0.1.6, I encountered an issue where the CUDA version in my environment did not match the PyTorch version, causing the installation of grouped_gemm to fail. However, I have repeatedly confirmed that I do not have any PyTorch version compiled with CUDA 12.4 installed.

Does this mean that grouped_gemm, during installation in the temporary build directory (/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/torch/), downloaded and used a PyTorch version compiled with CUDA 12.4 on its own, bypassing the locally installed PyTorch? Does this mean that grouped_gemm must be downloaded in a CUDA 12.4 environment?

Collecting grouped_gemm==0.1.6
Using cached grouped_gemm-0.1.6.tar.gz (978 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: grouped_gemm
Building wheel for grouped_gemm (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for grouped_gemm (pyproject.toml) did not run successfully.
│ exit code: 1
╰─\> \[67 lines of output\]
/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/torch/\_subclasses/functional_tensor.py:275: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:81.)
cpu = \_conversion_method_template(device=torch.device("cpu"))
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-310
creating build/lib.linux-x86_64-cpython-310/grouped_gemm
copying grouped_gemm/ops_test.py -\> build/lib.linux-x86_64-cpython-310/grouped_gemm
copying grouped_gemm/__init__.py -\> build/lib.linux-x86_64-cpython-310/grouped_gemm
copying grouped_gemm/ops.py -\> build/lib.linux-x86_64-cpython-310/grouped_gemm
copying grouped_gemm/backend.py -\> build/lib.linux-x86_64-cpython-310/grouped_gemm
running build_ext
Traceback (most recent call last):
File "/root/anaconda3/envs/unsloth_env3/lib/python3.10/site-packages/pip/\_vendor/pyproject_hooks/\_in_process/\_in_process.py", line 353, in \<module\>
main()
File "/root/anaconda3/envs/unsloth_env3/lib/python3.10/site-packages/pip/\_vendor/pyproject_hooks/\_in_process/\_in_process.py", line 335, in main
json_out\['return_val'\] = hook(\*\*hook_input\['kwargs'\])
File "/root/anaconda3/envs/unsloth_env3/lib/python3.10/site-packages/pip/\_vendor/pyproject_hooks/\_in_process/\_in_process.py", line 251, in build_wheel
return \_build_backend().build_wheel(wheel_directory, config_settings,
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 410, in build_wheel
return self.\_build_with_temp_dir(
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 395, in \_build_with_temp_dir
self.run_setup()
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
exec(code, locals())
File "\<string\>", line 58, in \<module\>
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 104, in setup
return distutils.core.setup(\*\*attrs)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/\_distutils/core.py", line 184, in setup
return run_commands(dist)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/\_distutils/core.py", line 200, in run_commands
dist.run_commands()
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/\_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
super().run_command(command)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/\_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-uuagnhy5/normal/lib/python3.10/site-packages/wheel/\_bdist_wheel.py", line 387, in run
self.run_command("build")
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/\_distutils/cmd.py", line 316, in run_command
self.distribution.run_command(command)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
super().run_command(command)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/\_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/\_distutils/command/build.py", line 132, in run
self.run_command(cmd_name)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/\_distutils/cmd.py", line 316, in run_command
self.distribution.run_command(command)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
super().run_command(command)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/\_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 91, in run
\_build_ext.run(self)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/\_distutils/command/build_ext.py", line 359, in run
self.build_extensions()
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 552, in build_extensions
\_check_cuda_version(compiler_name, compiler_version)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 447, in \_check_cuda_version
raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
RuntimeError:
The detected CUDA version (11.8) mismatches the version that was used to compile
PyTorch (12.4). Please make sure to use the same CUDA versions.

      [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for grouped_gemm
Failed to build grouped_gemm
ERROR: Could not build wheels for grouped_gemm, which is required to install pyproject.toml-based projects

I installed the PyTorch version corresponding to CUDA 11.8 using the following command:

pip install torch==2.1.1+cu118 torchvision==0.16.1+cu118 torchaudio==2.1.1 --index-url 

Possible Cause 1: Using PyTorch from a Different Environment

Currently, I have three environments:

• Windows (local machine)
• Linux (WSL)
• Conda environment within Linux

Check 1: Running nvcc --version in all three environments

• Windows (local machine): CUDA 12.1
• WSL (Linux): CUDA 11.8
• Conda environment (Linux): CUDA 11.8

Suspicious point: There is no CUDA 12.4 installed, so why is there a PyTorch version compiled with CUDA 12.4?

Check 2: Running common PyTorch commands (inside the Conda environment)

print(f"PyTorch version: {torch.__version__}")  # Confirms PyTorch version is 2.1.1+cu118  
print(f"CUDA version used by PyTorch: {torch.version.cuda}")  # Confirms CUDA version is 11.8  
print(f"Is CUDA available: {torch.cuda.is_available()}")  # Checks if CUDA is available  

Additionally, running nvidia-smi shows that my system's driver only supports up to CUDA 12.3.

Check 3: Checking environment paths

echo $CUDA_HOME  
echo $PATH  
echo $PYTHONPATH  
echo $LD_LIBRARY_PATH  

Check 4: Installing with a specified path

CUDA_HOME=/usr/local/cuda-11.8 pip install grouped_gemm==0.1.6  
/root/anaconda3/envs/unsloth_env3/bin/python -m pip install grouped_gemm==0.1.6  

None of the above methods worked. This leads me to believe that when downloading grouped_gemm, it forcibly downloads and uses a PyTorch version compiled with CUDA 12.4. This essentially means that grouped_gemm requires a CUDA 12.4 environment in order to be installed.