text-generation-webui加载codellama报错DLL load failed while importing flash_attn_2_cuda: 找不到指定的模块。

使用text-generation-webui加载codellama，报错：

1. Traceback (most recent call last):
3. File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\utils\import_utils.py", line 1353, in _get_module
6. return importlib.import_module("." + module_name, self.__name__)
7. File "D:\Anaconda\Anaconda\envs\codellama\lib\importlib_init_.py", line 126, in import_module
10. return _bootstrap._gcd_import(name[level:], package, level)
11. File "", line 1050, in _gcd_import
13. File "", line 1027, in _find_and_load
15. File "", line 1006, in _find_and_load_unlocked
17. File "", line 688, in _load_unlocked
19. File "", line 883, in exec_module
21. File "", line 241, in _call_with_frames_removed
23. File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\llama\modeling_llama.py", line 48, in
26. from flash_attn import flash_attn_func, flash_attn_varlen_func
27. File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\flash_attn_init_.py", line 3, in
30. from flash_attn.flash_attn_interface import (
31. File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\flash_attn\flash_attn_interface.py", line 8, in
34. import flash_attn_2_cuda as flash_attn_cuda
35. ImportError: DLL load failed while importing flash_attn_2_cuda: 找不到指定的模块。
37. The above exception was the direct cause of the following exception:
39. Traceback (most recent call last):
41. File "E:\模型\text-generation-webui\text-generation-webui\modules\ui_model_menu.py", line 209, in load_model_wrapper
44. shared.model, shared.tokenizer = load_model(shared.model_name, loader)
45. File "E:\模型\text-generation-webui\text-generation-webui\modules\models.py", line 85, in load_model
48. output = load_func_map[loader](model_name)
49. File "E:\模型\text-generation-webui\text-generation-webui\modules\models.py", line 155, in huggingface_loader
52. model = LoaderClass.from_pretrained(path_to_model, **params)
53. File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 565, in from_pretrained
56. model_class = _get_model_class(config, cls._model_mapping)
57. File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 387, in _get_model_class
60. supported_models = model_mapping[type(config)]
61. File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 740, in getitem
64. return self._load_attr_from_module(model_type, model_name)
65. File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 754, in _load_attr_from_module
68. return getattribute_from_module(self._modules[module_name], attr)
69. File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 698, in getattribute_from_module
72. if hasattr(module, attr):
73. File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\utils\import_utils.py", line 1343, in getattr
76. module = self._get_module(self._class_to_module[name])
77. File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\utils\import_utils.py", line 1355, in _get_module
80. raise RuntimeError(
81. RuntimeError: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback):
83. DLL load failed while importing flash_attn_2_cuda: 找不到指定的模块。

一开始排查是以为transformers的版本不对，先确定了transformers的版本，transformers的版本应该大于4.35.0

把transformers升级为4.35.0后仍然报错

接着排查cuda和torch的版本

最后发现是cuda版本与torch版本不匹配

1. >>> print(torch.version.cuda) # 检查CUDA版本
2. >>> 11.8

控制台运行nvcc --version ：

输出：

1. nvcc: NVIDIA (R) Cuda compiler driver
2. Copyright (c) 2005-2023 NVIDIA Corporation
3. Built on Wed_Feb__8_05:53:42_Coordinated_Universal_Time_2023
4. Cuda compilation tools, release 12.1, V12.1.66
5. Build cuda_12.1.r12.1/compiler.32415258_0

最后解决：

先卸载原本的torch：

pip uninstall torch torchvision torchaudio

然后安装12.1的：

pip install torch torchvision torchaudio -f https://download.pytorch.org/whl/cu121/torch_stable.html

最后加载成功codellama