failed with 'CUDA_ERROR_INVALID_HANDLE'

[captainsalih]

Dear All,

I want to install this latest version of deepmd. I'm use Ubuntu, and NVIDIA 5090 graphics card. So here are what I had done

1. Create new env and install tensorflow and cuda. Here is the detail

+-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 570.148.08 Driver Version: 570.148.08 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+

---

TF: 2.20.0
CUDA: 12.5.1
cuDNN: 9

conda list | egrep 'cudatoolkit|cudnn|cuda'
nvidia-cuda-cupti-cu12 12.9.79 pypi_0 pypi
nvidia-cuda-nvcc-cu12 12.9.86 pypi_0 pypi
nvidia-cuda-nvrtc-cu12 12.9.86 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.9.79 pypi_0 pypi
nvidia-cudnn-cu12 9.12.0.46 pypi_0 pypi

2. Download 0,1,2 sh file and combine
3. run the sh file. up to this everything went normal

but when I try to train here is the error

2025-08-19 19:20:22.120030: W tensorflow/compiler/mlir/tools/kernel_gen/tf_gpu_runtime_wrappers.cc:40] 'cuModuleGetFunction(&function, module, kernel_name)' failed with 'CUDA_ERROR_INVALID_HANDLE'

2025-08-19 19:20:22.120039: W tensorflow/core/framework/op_kernel.cc:1829] INTERNAL: 'cuLaunchKernel(function, gridX, gridY, gridZ, blockX, blockY, blockZ, 0, reinterpret_cast(stream), params, nullptr)' failed with 'CUDA_ERROR_INVALID_HANDLE'
2025-08-19 19:20:22.122646: W tensorflow/compiler/mlir/tools/kernel_gen/tf_gpu_runtime_wrappers.cc:40] 'cuModuleLoadData(&module, data)' failed with 'CUDA_ERROR_INVALID_PTX'

2025-08-19 19:20:22.122659: W tensorflow/compiler/mlir/tools/kernel_gen/tf_gpu_runtime_wrappers.cc:40] 'cuModuleGetFunction(&function, module, kernel_name)' failed with 'CUDA_ERROR_INVALID_HANDLE'

2025-08-19 19:20:22.122665: W tensorflow/core/framework/op_kernel.cc:1829] INTERNAL: 'cuLaunchKernel(function, gridX, gridY, gridZ, blockX, blockY, blockZ, 0, reinterpret_cast(stream), params, nullptr)' failed with 'CUDA_ERROR_INVALID_HANDLE'
2025-08-19 19:20:22.125076: W tensorflow/compiler/mlir/tools/kernel_gen/tf_gpu_runtime_wrappers.cc:40] 'cuModuleLoadData(&module, data)' failed with 'CUDA_ERROR_INVALID_PTX'

2025-08-19 19:20:22.125088: W tensorflow/compiler/mlir/tools/kernel_gen/tf_gpu_runtime_wrappers.cc:40] 'cuModuleGetFunction(&function, module, kernel_name)' failed with 'CUDA_ERROR_INVALID_HANDLE'

thank you very much for the help