Strange Precompilation Error

ITensor Julia Questions
1

Hello,

I have been developing my own package built on top of ITensors. I have never encountered the following issue. I have a project set up using DrWatson in which I am generating simulation runs. In that project I have my package dev’d, for example in the Project.toml you’d see

MyTensorPackage vXXX 'src/MyTensorPackage.jl

My package had ITensor v0.3.38. I recently increased the size of some simulations and I started getting warnings about having too many indices. Within my own package I have this resolved because I included the set_warn_order(Max_Indices) but outside the package, with the project environment, I also needed to turn it off because I was contracting and manipulating the result of the simulation within the DrWatson project (for post-processing/plotting purposes for example).

I reason then that I should add ITensors to the DrWatson project and then in my startup script for the project add the same line set_warn_order(Max_Indices). When I added ITensors to the project it automatically pulled the latest which was v0.3.42.

But now I find I cannot even run my simulations anymore. I now get the following error. I have no idea what it means or why it is occurring. I have never used disable_auto_fermion. I have no idea why it even started. Any help getting this resolved would be greatly appreciated.

I tried updating my own package’s version of ITensors to v0.3.42 but the issue persists.

Failed to precompile ITensors [9136182c-28ba-11e9-034c-db9fb085ebd5] to "/home/USERNAME/.julia/compiled/v1.9/ITensors/jl_kBFMD7".
ERROR: LoadError: UndefVarError: `disable_auto_fermion` not defined
2

I seemed to have fixed this but I’m not sure it was the right way. So I reverted back to v0.3.38 within my package and that fixed the issue. But it appears the DrWatson project Manifest.toml was saying the dep for MyTensorPackage.jl was ITensor still at v0.3.42. Trying Pkg.resolve() did not fix the issue.

So I manually edited the DrWatson Manifest.toml, copying the [deps] entry of ITensor from MyTensorPackage.jl. That appears to have fixed it.

3

It sounds like maybe you have a discrepancy in the versions of NDTensors and ITensors. We should be more strict about the compat versions to guide people towards using compatible versions, but generally you should either use the latest registered versions of both, or the latest dev’d versions of both, since they are often developed together.

4

Thanks for the reply. So I’ve updated ITensors to v0.3.42 and I’m getting the same error, can’t precompile.

In my package’s Manifest.toml I see that NDTensors is

[[deps.NDTensors]]
deps = ["Adapt", "Compat", "Dictionaries", "FLoops", "Folds", "Functors", "HDF5", "LinearAlgebra", "Random", "Requires", "SimpleTraits", "SplitApplyCombine", "StaticArrays", "Strided", "TimerOutputs", "TupleTools"]
git-tree-sha1 = "5ba23649531a5638b3f260a78187710f852deac1"
uuid = "23ae76d9-e61a-49c4-8f12-3f1a16adf9cf"
version = "0.1.51"

Is this the problem? If so, how can I update the dependency of a dependency? NDTensors is not a direct dependency of the package I am developing (only ITensors is). Should I directly update the manifest.toml? (I was under the impression generally you shouldn’t need to do that)

5

The latest version of NDTensors is v0.2.11, so your version is out of date and causing the error you are seeing. Have you tried updating all of your packages with ] up?

If that doesn’t work, you can find out more information about why some packages may not be updating with julia> ] st --outdated at the command prompt. Also julia> ] add NDTensors@0.2.11 is a way to try to force Julia to add a certain package version, which should either succeed or throw an error explaining why it can’t be updated to that version. You shouldn’t modify the Manifest.toml.

6

Thanks for the advice - I will give these things a shot. Just to be clear though, NDTensors is not a direct dependency of my package, ie its not listed when I do pkg] st. By doing julia> ] add NDTensors@0.2.11 I guess I will be making it a direct dependency of my package? Hopefully I am making sense.

7

I understand. I think you should try adding it explicitly to try to debug this problem, and then hopefully when you sort out the route cause then you can try to avoid adding it as a direct dependency. It is hard to debug/analyze issues with indirect dependencies.

Any chance you have ITensorGPU installed? Also have you tried just deleting the Manifest.toml to force Julia to make a new one?

8

I don’t - though I am planning on adding or at least experimenting with adding GPU support for my package. I had not decided yet to add it as an extension (and extend my package methods with new ones that return basically gpu(MPS(...)) instead of returning MPS(...)) or just add it as a direct dependency. I’d appreciate any thoughts/opinions you have on that approach.

But more generally to your point - is there something I should be worried about regarding adding ITensorGPU?

9

ITensorGPU is only compatible with NDTensors v0.1 since we have been making a lot of internal changes to NDTensors and are moving towards a package extension (NDTensorsCUDAExt), so ITensorGPU will soon be deprecated. I was just curious if it was holding back your environment’s ability to update to NDTensors v0.2 but it doesn’t sound like that’s an issue.

The package extension gets loaded if you load ITensors and CUDA so nothing else needs to be installed by the end user/dependent packages. It has all of the same functionality as ITensorGPU so can be used as a drop-in replacement, however there are a few performance issues at the moment. Those will be addressed by [NDTensors][NDTensorsCUDAExt] Fill tensors on GPU devices by kmp5VT · Pull Request #1194 · ITensor/ITensors.jl · GitHub, after which ITensorGPU can be fully deprecated in favor of the new package extension.

10

That is great to know - I appreciate you telling me as I was in the process of troubleshooting the setup of ITensorsGPU on our cluster. I doubt its worth it now that I know it will be deprecated.

I just tried the ITensorsGPU dmrg example here. I modified the example just slightly, changing

using ITensors
using ITensorGPU

to

using CUDA, ITensors

Its giving me the following error. Would appreciate any insights you have on this. I believe CUDA is set up correctly - I did that first prior to running the script (made sure the CUDA runtime matches the version of CUDA I have loaded (11.8)).

ERROR: LoadError: Scalar indexing is disallowed.
Invocation of getindex resulted in scalar indexing of a GPU array.
This is typically caused by calling an iterating implementation of a method.
Such implementations *do not* execute on the GPU, but very slowly on the CPU,
and therefore are only permitted from the REPL for prototyping purposes.
If you did intend to index this array, annotate the caller with @allowscalar.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] assertscalar(op::String)
    @ GPUArraysCore ~/.julia/packages/GPUArraysCore/uOYfN/src/GPUArraysCore.jl:103
  [3] getindex(xs::CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}, I::Int64)
    @ GPUArrays ~/.julia/packages/GPUArrays/EZkix/src/host/indexing.jl:9
  [4] getindex
    @ ~/.julia/packages/StridedViews/bIjcq/src/stridedview.jl:97 [inlined]
  [5] macro expansion
    @ ~/.julia/packages/Strided/l1vm3/src/mapreduce.jl:330 [inlined]
  [6] macro expansion
    @ ./simdloop.jl:77 [inlined]
  [7] macro expansion
    @ ~/.julia/packages/Strided/l1vm3/src/mapreduce.jl:329 [inlined]
  [8] _mapreduce_kernel!(f::Strided.CaptureArgs{NDTensors.var"#69#70", Tuple{Strided.Arg, Strided.Arg}}, op::Nothing, initop::Nothing, dims::Tuple{Int64, Int64, Int64}, blocks::Tuple{Int64, Int64, Int64}, arrays::Tuple{StridedViews.StridedView{Float32, 3, CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}, typeof(identity)}, StridedViews.StridedView{Float32, 3, CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}, typeof(identity)}, StridedViews.StridedView{Float32, 3, CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}, typeof(identity)}}, strides::Tuple{Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}}, offsets::Tuple{Int64, Int64, Int64})
    @ Strided ~/.julia/packages/Strided/l1vm3/src/mapreduce.jl:229
  [9] _mapreduce_block!(f::Any, op::Any, initop::Any, dims::Tuple{Int64, Int64, Int64}, strides::Tuple{Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}}, offsets::Tuple{Int64, Int64, Int64}, costs::Tuple{Int64, Int64, Int64}, arrays::Tuple{StridedViews.StridedView{Float32, 3, CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}, typeof(identity)}, StridedViews.StridedView{Float32, 3, CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}, typeof(identity)}, StridedViews.StridedView{Float32, 3, CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}, typeof(identity)}})
    @ Strided ~/.julia/packages/Strided/l1vm3/src/mapreduce.jl:152
 [10] _mapreduce_order!(f::Any, op::Any, initop::Any, dims::Tuple{Int64, Int64, Int64}, strides::Tuple{Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}}, arrays::Tuple{StridedViews.StridedView{Float32, 3, CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}, typeof(identity)}, StridedViews.StridedView{Float32, 3, CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}, typeof(identity)}, StridedViews.StridedView{Float32, 3, CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}, typeof(identity)}})
    @ Strided ~/.julia/packages/Strided/l1vm3/src/mapreduce.jl:138
 [11] _mapreduce_fuse!(f::Any, op::Any, initop::Any, dims::Tuple{Int64, Int64, Int64}, arrays::Tuple{StridedViews.StridedView{Float32, 3, CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}, typeof(identity)}, StridedViews.StridedView{Float32, 3, CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}, typeof(identity)}, StridedViews.StridedView{Float32, 3, CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}, typeof(identity)}})
    @ Strided ~/.julia/packages/Strided/l1vm3/src/mapreduce.jl:116
 [12] copyto!
    @ ~/.julia/packages/Strided/l1vm3/src/broadcast.jl:35 [inlined]
 [13] materialize!
    @ ./broadcast.jl:884 [inlined]
 [14] materialize!
    @ ./broadcast.jl:881 [inlined]
 [15] permutedims!(R::NDTensors.DenseTensor{Float32, 3, Tuple{Index{Int64}, Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, T::NDTensors.DenseTensor{Float32, 3, Tuple{Index{Int64}, Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, perm::Tuple{Int64, Int64, Int64}, f::NDTensors.var"#69#70")
    @ NDTensors ~/.julia/packages/NDTensors/pUJ6b/src/dense/densetensor.jl:246
 [16] permutedims!!
    @ ~/.julia/packages/NDTensors/pUJ6b/src/dense/densetensor.jl:188 [inlined]
 [17] permutedims!!
    @ ~/.julia/packages/NDTensors/pUJ6b/src/dense/densetensor.jl:186 [inlined]
 [18] permutedims(tensor::NDTensors.DenseTensor{Float32, 3, Tuple{Index{Int64}, Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, perm::Tuple{Int64, Int64, Int64})
    @ NDTensors ~/.julia/packages/NDTensors/pUJ6b/src/tensoralgebra/generic_tensor_operations.jl:5
 [19] permutedims
    @ ~/.julia/packages/ITensors/yZbTa/src/tensor_operations/permutations.jl:64 [inlined]
 [20] _permute(as::NDTensors.NeverAlias, T::NDTensors.DenseTensor{Float32, 3, Tuple{Index{Int64}, Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, new_inds::Tuple{Index{Int64}, Index{Int64}, Index{Int64}})
    @ ITensors ~/.julia/packages/ITensors/yZbTa/src/tensor_operations/permutations.jl:69
 [21] permute(as::NDTensors.NeverAlias, T::ITensor, new_inds::Tuple{Index{Int64}, Index{Int64}, Index{Int64}})
    @ ITensors ~/.julia/packages/ITensors/yZbTa/src/tensor_operations/permutations.jl:73
 [22] #permute#271
    @ ~/.julia/packages/ITensors/yZbTa/src/tensor_operations/permutations.jl:54 [inlined]
 [23] permute(T::ITensor, new_inds::Tuple{Index{Int64}, Index{Int64}, Index{Int64}})
    @ ITensors ~/.julia/packages/ITensors/yZbTa/src/tensor_operations/permutations.jl:38
 [24] permute(M::MPO, #unused#::Tuple{typeof(linkind), typeof(siteinds), typeof(linkind)})
    @ ITensors ~/.julia/packages/ITensors/yZbTa/src/mps/dmrg.jl:14
 [25] #dmrg#1052
    @ ~/.julia/packages/ITensors/yZbTa/src/mps/dmrg.jl:25 [inlined]
 [26] dmrg(H::MPO, psi0::MPS, sweeps::Sweeps)
    @ ITensors ~/.julia/packages/ITensors/yZbTa/src/mps/dmrg.jl:20
 [27] top-level scope
    @ ./timing.jl:273 [inlined]
 [28] top-level scope
    @ ~/projects/dmrg_gpu_test2/DMRG_GPU.jl:0
in expression starting at /home/projects/dmrg_gpu_test2/DMRG_GPU.jl:25
11

That issue will be fixed by [NDTensors][NDTensorsCUDAExt] Fill tensors on GPU devices by kmp5VT · Pull Request #1194 · ITensor/ITensors.jl · GitHub.

12

Awesome - thank you!

13

FYI - I just tried again using ITensors v0.3.46 and CUDA v5.0.0 and the example here works!

I had one other question. I can make a separate post if you like. To summarize, I had some other test scripts I was trying with quantum numbers (e.g., siteinds("Electron", N, conserve_qns = true) but it throws an error (I do not get the error if I do conserve_qns = false). Do you know if there are any plans to support quantum numbers on the GPU?

14

Many block sparse tensor operations are working on GPU, but full DMRG runs are not working yet. It shouldn’t take much effort to get that working and we plan to fix that soon, so stay tuned.

See also [ANN] Initial release of new ITensor GPU backends and ITensor on GPU and conserved Quantum numbers.

15

Just in case, for anyone else who might have this issue, I fixed it by deleting the manifest and reinstantiating. After that, everything updated to latest and precompiled just fine.