Large memory issue in time evolution using ITensors.jl with apply function

Hi!

I am just asking about another memory issue here with ITensors.jl. So I wrote a code for simulating time evolution of a spin-1/2 system with both n.n. and n.n.n couplings. I ran it on an HPC, and noticed that the memory could be a bit large and sometimes exceeded the memory limit and got killed. Here is the code:

let
    # Parameters
    N = parse(Int64,ARGS[1]);
    T = 1
    h = 1*pi*(1-0.1)/T
    M = (1-0.02)*pi/T
    J = 1/2/T 
    ω = 2*pi/T
    t_tot = parse(Int64,ARGS[2])*T
    dt = 0.01
    measure_step = 100
    Nsteps = Int(t_tot/dt);
    Nperiods = Int(Nsteps/measure_step)
    maxdim = parse(Int64,ARGS[3];)
    path = string(ARGS[4]);
    cut_off = 1E-8;

    # MPS time evolution
    ## Initial MPS preparation
    s = siteinds("S=1/2",N;conserve_qns=false)
    ψ = MPS(ComplexF64,s);
    for i=1:N
        ψ[i][1,1,1]=1;
        ψ[i][1,2,1]=0;
    end
    orthogonalize!(ψ,1);

    # Initial measurement
    Szlist = Array{Float64,1}(undef,0);
    tlist = Array{Float64,1}(undef,0);
    temp = sum(2*expect(ψ,"Sz";sites=1:2:N))/(N/2);
    append!(tlist,0)
    append!(Szlist,temp)
   

    for k = 1:Nperiods
        ψ = single_period_Trotter_1st_order(J,h,N,s,ω,M,
                                             T,
                                             dt,
                                             ψ,
                                             cut_off,
                                             k,
                                             maxdim)
        temp1 = sum(2*expect(ψ,"Sz";sites=1:2:N))/(N/2);
        append!(Szlist,temp1)
        append!(tlist,k*measure_step*dt)
        # Save data
        # print(k)
        filename = "N"*ARGS[1]*"ttotal"*ARGS[2]*"maxdim"*ARGS[3]*".h5"
        h5open(path * "/" * filename,"w") do file
            g = create_group(file, "Parameters") # create a group
            g["N"] = N
            g["T"] = T
            g["h"] = h
            g["M"] = M
            g["dt"] = dt
            g["ttotal"] = t_tot
            # g["initialstate"] = initial_state_type
            g["cutoff"] = cut_off
            g["maxbd"] = maxdim
            g["measurestep"] = measure_step

            R = create_group(file,"Results")
            write(file, "Results/Mz", Szlist)
            write(file, "Results/tlist", tlist)
            # attributes(g)["Description"] = "This group contains only a single dataset" # an attribute
        end

    end

Note that the function \texttt{single\_period\_Trotter\_1st\_order} is similar to the example given in the ITensors.jl documentation, and the key step of the apply all those Trotterized gates is:

# e^{-i H_{odd} dt}|ψ>
ψ = apply(UOdd,ψ;cutoff=cutoff,maxdim=maxdim)
# e^{-i H_{even} dt}|ψ>
ψ = apply(UEven,ψ;cotoff=cutoff,maxdim=maxdim) 

and I am trying to save the data on the fly with HDF5.

It is found that for N=24, it could already take up to more than 64 GB’s memory, and for N=56, it exceeded 80 GB (PBS killed in the middle so I wasn’t able to know the upper bound). May I ask is it normal, or is there anything I could optimize my code to save the memory? Or it has something to do with the HDF5 saving? Here, I use \texttt{maxdim}=100 for all calculations.

Thanks so much!

Tianqi Chen

Hi Tianqi, it’s quite possible that it is the HDF5 part that is causing so much memory usage. Did you try running the code with the HDF5 parts commented out or disabled to check?

Hi Miles, yes, I commented that HDF5 parts. However for N=56 it still exceeded 80 GB limit. I just pasted the error message below for reference:

=>> PBS: job killed: mem 84771500kb exceeded limit 83886080kb

[18664] signal (15): Terminated
in expression starting at /home/users/ntu/tianqich/4T-DTC/test_memoryv2.jl:11
sweep_malloced_arrays at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:1366 [inlined]
gc_sweep_other at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:1729 [inlined]
jl_gc_collect at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:3537
ijl_gc_collect at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:3707
maybe_collect at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:1078 [inlined]
jl_gc_pool_alloc_inner at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:1443 [inlined]
jl_gc_pool_alloc_noinline at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:1504 [inlined]
jl_gc_alloc
at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia_internal.h:460 [inlined]
jl_gc_alloc at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:3754
new_array at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/array.c:144
_new_array at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/array.c:198 [inlined]
ijl_new_array at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/array.c:431
Array at ./boot.jl:489 [inlined]
similar at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/NDTensors/oXCSC/src/abstractarray/similar.jl:56 [inlined]
similar at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/NDTensors/oXCSC/src/tensorstorage/similar.jl:40 [inlined]
similar at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/NDTensors/oXCSC/src/tensor/similar.jl:22 [inlined]
contraction_output at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/NDTensors/oXCSC/src/dense/tensoralgebra/contract.jl:79 [inlined]
contraction_output at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/NDTensors/oXCSC/src/tensoralgebra/generic_tensor_operations.jl:47 [inlined]
contract at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/NDTensors/oXCSC/src/tensoralgebra/generic_tensor_operations.jl:93
unknown function (ip: 0x2aab88346f47)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
contract at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/NDTensors/oXCSC/src/tensoralgebra/generic_tensor_operations.jl:76
contract at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/SimpleTraits/l1ZsK/src/SimpleTraits.jl:331 [inlined]
_contract at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor_operations/tensor_algebra.jl:3
unknown function (ip: 0x2aab8833e996)
_jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
_contract at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor_operations/tensor_algebra.jl:9
contract at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor_operations/tensor_algebra.jl:104

  • at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor_operations/tensor_algebra.jl:91 [inlined]
    #factorize_eigen#325 at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor_operations/matrix_decomposition.jl:560
    factorize_eigen at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor_operations/matrix_decomposition.jl:548 [inlined]
    #factorize#327 at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor_operations/matrix_decomposition.jl:676
    factorize at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor_operations/matrix_decomposition.jl:623
    unknown function (ip: 0x2aab8831aead)
    jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
    ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
    #
    #739 at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:1941
    AbstractMPS at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:1919
    unknown function (ip: 0x2aab8834c07d)
    _jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
    ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
    #setindex!#729 at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:1881
    setindex! at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:1809
    unknown function (ip: 0x2aab88312c02)
    _jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
    ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
    #setindex!#734 at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:1892
    setindex! at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:1889 [inlined]
    #product#746 at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:2120
    product at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:2093 [inlined]
    product at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:2093 [inlined]
    #product#749 at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:2235
    product at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:2226 [inlined]
    evoST_1st_order_one_step at /home/users/ntu/tianqich/4T-DTC/fourTEvolution.jl:146 [inlined]
    single_period_Trotter_1st_order at /home/users/ntu/tianqich/4T-DTC/fourTEvolution.jl:176
    unknown function (ip: 0x2aab669a1175)
    _jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
    ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
    top-level scope at /home/users/ntu/tianqich/4T-DTC/test_memoryv2.jl:50
    jl_toplevel_eval_flex at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/toplevel.c:903
    jl_toplevel_eval_flex at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/toplevel.c:856
    ijl_toplevel_eval_in at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/toplevel.c:971
    eval at ./boot.jl:370 [inlined]
    include_string at ./loading.jl:1864
    _jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
    ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
    _include at ./loading.jl:1924
    include at ./Base.jl:457
    jfptr_include_43521.clone_1 at /home/users/ntu/tianqich/julia-1.9.0/lib/julia/sys.so (unknown line)
    _jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
    ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
    exec_options at ./client.jl:307
    _start at ./client.jl:522
    jfptr__start_37386.clone_1 at /home/users/ntu/tianqich/julia-1.9.0/lib/julia/sys.so (unknown line)
    _jl_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
    ijl_apply_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940
    jl_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
    true_main at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/jlapi.c:573
    jl_repl_entrypoint at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/jlapi.c:717
    main at julia (unknown line)
    __libc_start_main at /lib64/libc.so.6 (unknown line)
    unknown function (ip: 0x401098)
    unknown function (ip: (nil))
    Allocations: 534481077 (Pool: 528171528; Big: 6309549); GC: 37930

May I ask what you suggest to debug this?

Let me follow up a little bit here: after I insert a line of

GC.gc()

at the end of each loop, the problem seems to be solved. I gather that this is still a workaround. I suspected it is the way that I formed all my even and odd bonds MPO which leads to so much memory: as my Hamiltonian is actually time-dependent (periodic), for each loop, I just generate a list of them, and therefore there’re so many of my MPO’s. Is that the reason why my memory went out so quickly?

Hi Tianqi,
Thanks for investigating this more on your own. If you are remaking the Hamiltonian on each loop of the innermost loop of a big calculation, then I could see how this could make “garbage” pile up more quickly than usual. Unfortunately this is a weakness of Julia right now, about how it can be too slow to collect freed memory. What you did is a good workaround for now, of calling the GC to collect at certain points. If it’s slowing down your code too much, you can put an if statement that makes the GC call happen only every other loop iteration, or every 10th iteration, say.

However, you may be in luck with a new feature in the recently released Julia 1.9. This feature is a “GC memory usage hint” flag you can pass when starting Julia:
https://julialang.org/blog/2023/04/julia-1.9-highlights/#memory_usage_hint_for_the_gc_with_--heap-size-hint
You can put a size of heap that you want the GC to respect and it will try to use closer to that amount of memory. So maybe you can try putting something like 75% of the RAM on your system, for example. I’d be curious if this flag works for you (with the GC.gc() line now commented out). If so, I will tell other ITensor users about it if they run into memory issues.

Best regards,
Miles

2 Likes

Hi Miles, thanks for the solutions and detailed answers on GC.gc(), as well as GC memory usage hint. Sorry for the slow reply, as I have been in a job transition these few months, and I forgot to reply to this thread any time sooner.

Yes, this flag really helps to reduce the RAM usage. I supplied 0.5 of the RAM on the HPC limit, and turns out that it indeed used less than that memory (but is closer to that value) as shown in the output file, with GC.gc() being commented. I think in that case, we could let other ITensors.jl users be aware of this?

That’s a good suggestion, thanks. Glad that flag has been helpful for you. I’ll add a new part to the ITensor FAQ page in the documentation about high performance computing and mention this flag there.

1 Like