Run on multiple GPUs

ITensor Julia Questions
1

Hello everybody,

I am running a tensor-network code using ITensors (Julia version) with GPUs (Nvidia A100) on a HPC cluster. I tried to parallelize it to use both GPUs available on a node, but I realized the code is actually only using one.

Is there a way to use both? I couldn’t find anything in the forum.

Thanks in advance!

2

Hi!

I tried this two years ago but never found a way to make it work efficiently. I am not sure if maybe now, as the gpu implementation works better, there may be a way to do it (please let me now if you find one). Anyway, maybe some of what I learned during the process can be useful for you. In principle, you can use the function CUDA.device!() to switch which GPU you are using.

For example. if you have two devices and run CUDA.devices() your output will be something like:

CUDA.DeviceIterator() for 2 devices:
0. Tesla V100-SXM2-32GB
1. Tesla V100-SXM2-32GB

Then CUDA.device!(0) will choose the first one for usage and CUDA.device!(1) the second one. I never found a way to use both at the same time, but in principle you could use the first one until the memory is full and then switch to the second one (for example checking the memory during the sweeps of a DMRG code and use a custom observer for switching the GPU), but I could never make that work efficiently.

I have this GitHub repository with some tests I did in the past about it: ITensor_wGPU/DMRG - Multiple GPUs.ipynb at main · Tensor-Networks-PITT-MEMS/ITensor_wGPU · GitHub no sure if that can help you.

3

Hi,

Thanks a lot for the information! I will give it a try.