# Large memory issue in time evolution using ITensors.jl with apply function

**URL:** <https://itensor.discourse.group/t/large-memory-issue-in-time-evolution-using-itensors-jl-with-apply-function/861>\
**Category:** ITensor Julia Questions\
**Tags:** julia, mps\
**Created:** [May 13, 2023, 12:03pm UTC](https://itensor.discourse.group/t/large-memory-issue-in-time-evolution-using-itensors-jl-with-apply-function/861 "2023-05-13T12:03:54Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![talentkeychen](https://yyz2.discourse-cdn.com/free1/user_avatar/itensor.discourse.group/talentkeychen/32/272_2.png) [@talentkeychen](https://itensor.discourse.group/u/talentkeychen)\
**Post date:** [May 13, 2023, 12:03pm UTC](https://itensor.discourse.group/t/large-memory-issue-in-time-evolution-using-itensors-jl-with-apply-function/861/1 "2023-05-13T12:03:54Z")

</div>

Hi!

I am just asking about another memory issue here with ITensors.jl. So I wrote a code for simulating time evolution of a spin-1/2 system with both n.n. and n.n.n couplings. I ran it on an HPC, and noticed that the memory could be a bit large and sometimes exceeded the memory limit and got killed. Here is the code:

```julia
let
    # Parameters
    N = parse(Int64,ARGS[1]);
    T = 1
    h = 1*pi*(1-0.1)/T
    M = (1-0.02)*pi/T
    J = 1/2/T 
    ω = 2*pi/T
    t_tot = parse(Int64,ARGS[2])*T
    dt = 0.01
    measure_step = 100
    Nsteps = Int(t_tot/dt);
    Nperiods = Int(Nsteps/measure_step)
    maxdim = parse(Int64,ARGS[3];)
    path = string(ARGS[4]);
    cut_off = 1E-8;

    # MPS time evolution
    ## Initial MPS preparation
    s = siteinds("S=1/2",N;conserve_qns=false)
    ψ = MPS(ComplexF64,s);
    for i=1:N
        ψ[i][1,1,1]=1;
        ψ[i][1,2,1]=0;
    end
    orthogonalize!(ψ,1);

    # Initial measurement
    Szlist = Array{Float64,1}(undef,0);
    tlist = Array{Float64,1}(undef,0);
    temp = sum(2*expect(ψ,"Sz";sites=1:2:N))/(N/2);
    append!(tlist,0)
    append!(Szlist,temp)
   

    for k = 1:Nperiods
        ψ = single_period_Trotter_1st_order(J,h,N,s,ω,M,
                                             T,
                                             dt,
                                             ψ,
                                             cut_off,
                                             k,
                                             maxdim)
        temp1 = sum(2*expect(ψ,"Sz";sites=1:2:N))/(N/2);
        append!(Szlist,temp1)
        append!(tlist,k*measure_step*dt)
        # Save data
        # print(k)
        filename = "N"*ARGS[1]*"ttotal"*ARGS[2]*"maxdim"*ARGS[3]*".h5"
        h5open(path * "/" * filename,"w") do file
            g = create_group(file, "Parameters") # create a group
            g["N"] = N
            g["T"] = T
            g["h"] = h
            g["M"] = M
            g["dt"] = dt
            g["ttotal"] = t_tot
            # g["initialstate"] = initial_state_type
            g["cutoff"] = cut_off
            g["maxbd"] = maxdim
            g["measurestep"] = measure_step

            R = create_group(file,"Results")
            write(file, "Results/Mz", Szlist)
            write(file, "Results/tlist", tlist)
            # attributes(g)["Description"] = "This group contains only a single dataset" # an attribute
        end

    end

```

Note that the function \texttt{single\_period\_Trotter\_1st\_order} is similar to the example given in the ITensors.jl documentation, and the key step of the apply all those Trotterized gates is:

```julia
# e^{-i H_{odd} dt}|ψ>
ψ = apply(UOdd,ψ;cutoff=cutoff,maxdim=maxdim)
# e^{-i H_{even} dt}|ψ>
ψ = apply(UEven,ψ;cotoff=cutoff,maxdim=maxdim) 

```

and I am trying to save the data on the fly with HDF5.

It is found that for N=24, it could already take up to more than 64 GB’s memory, and for N=56, it exceeded 80 GB (PBS killed in the middle so I wasn’t able to know the upper bound). May I ask is it normal, or is there anything I could optimize my code to save the memory? Or it has something to do with the HDF5 saving? Here, I use \texttt{maxdim}=100 for all calculations.

Thanks so much!

Tianqi Chen

---

<div class="post-metadata">

**Author:** ![miles](https://yyz2.discourse-cdn.com/free1/user_avatar/itensor.discourse.group/miles/32/6_2.png) [@miles](https://itensor.discourse.group/u/miles)\
**Post date:** [May 13, 2023, 1:20pm UTC](https://itensor.discourse.group/t/large-memory-issue-in-time-evolution-using-itensors-jl-with-apply-function/861/2 "2023-05-13T13:20:19Z")

</div>

Hi Tianqi, it’s quite possible that it is the HDF5 part that is causing so much memory usage. Did you try running the code with the HDF5 parts commented out or disabled to check?

---

<div class="post-metadata">

**Author:** ![talentkeychen](https://yyz2.discourse-cdn.com/free1/user_avatar/itensor.discourse.group/talentkeychen/32/272_2.png) [@talentkeychen](https://itensor.discourse.group/u/talentkeychen)\
**Post date:** [May 13, 2023, 2:27pm UTC](https://itensor.discourse.group/t/large-memory-issue-in-time-evolution-using-itensors-jl-with-apply-function/861/3 "2023-05-13T14:27:26Z")

</div>

Hi Miles, yes, I commented that HDF5 parts. However for N=56 it still exceeded 80 GB limit. I just pasted the error message below for reference:

=\>\> PBS: job killed: mem 84771500kb exceeded limit 83886080kb

[18664] signal (15): Terminated  
in expression starting at /home/users/ntu/tianqich/4T-DTC/test\_memoryv2.jl:11  
sweep\_malloced\_arrays at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:1366 [inlined]  
gc\_sweep\_other at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:1729 [inlined]  
_jl\_gc\_collect at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:3537  
ijl\_gc\_collect at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:3707  
maybe\_collect at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:1078 [inlined]  
jl\_gc\_pool\_alloc\_inner at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:1443 [inlined]  
jl\_gc\_pool\_alloc\_noinline at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:1504 [inlined]  
jl\_gc\_alloc_ at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia\_internal.h:460 [inlined]  
jl\_gc\_alloc at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gc.c:3754  
_new\_array_ at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/array.c:144  
\_new\_array at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/array.c:198 [inlined]  
ijl\_new\_array at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/array.c:431  
Array at ./boot.jl:489 [inlined]  
similar at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/NDTensors/oXCSC/src/abstractarray/similar.jl:56 [inlined]  
similar at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/NDTensors/oXCSC/src/tensorstorage/similar.jl:40 [inlined]  
similar at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/NDTensors/oXCSC/src/tensor/similar.jl:22 [inlined]  
contraction\_output at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/NDTensors/oXCSC/src/dense/tensoralgebra/contract.jl:79 [inlined]  
contraction\_output at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/NDTensors/oXCSC/src/tensoralgebra/generic\_tensor\_operations.jl:47 [inlined]  
contract at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/NDTensors/oXCSC/src/tensoralgebra/generic\_tensor\_operations.jl:93  
unknown function (ip: 0x2aab88346f47)  
\_jl\_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]  
ijl\_apply\_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940  
contract at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/NDTensors/oXCSC/src/tensoralgebra/generic\_tensor\_operations.jl:76  
contract at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/SimpleTraits/l1ZsK/src/SimpleTraits.jl:331 [inlined]  
\_contract at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor\_operations/tensor\_algebra.jl:3  
unknown function (ip: 0x2aab8833e996)  
\_jl\_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]  
ijl\_apply\_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940  
\_contract at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor\_operations/tensor\_algebra.jl:9  
contract at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor\_operations/tensor\_algebra.jl:104

- at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor\_operations/tensor\_algebra.jl:91 [inlined]  
#factorize\_eigen#325 at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor\_operations/matrix\_decomposition.jl:560  
factorize\_eigen at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor\_operations/matrix\_decomposition.jl:548 [inlined]  
#factorize#327 at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor\_operations/matrix\_decomposition.jl:676  
factorize at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/tensor\_operations/matrix\_decomposition.jl:623  
unknown function (ip: 0x2aab8831aead)  
_jl\_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]  
ijl\_apply\_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940  
#_#739 at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:1941  
AbstractMPS at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:1919  
unknown function (ip: 0x2aab8834c07d)  
\_jl\_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]  
ijl\_apply\_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940  
#setindex!#729 at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:1881  
setindex! at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:1809  
unknown function (ip: 0x2aab88312c02)  
\_jl\_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]  
ijl\_apply\_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940  
#setindex!#734 at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:1892  
setindex! at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:1889 [inlined]  
#product#746 at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:2120  
product at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:2093 [inlined]  
product at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:2093 [inlined]  
#product#749 at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:2235  
product at /home/users/ntu/tianqich/juliapkg/nsccenv/packages/ITensors/HjjU3/src/mps/abstractmps.jl:2226 [inlined]  
evoST\_1st\_order\_one\_step at /home/users/ntu/tianqich/4T-DTC/fourTEvolution.jl:146 [inlined]  
single\_period\_Trotter\_1st\_order at /home/users/ntu/tianqich/4T-DTC/fourTEvolution.jl:176  
unknown function (ip: 0x2aab669a1175)  
\_jl\_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]  
ijl\_apply\_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940  
top-level scope at /home/users/ntu/tianqich/4T-DTC/test\_memoryv2.jl:50  
jl\_toplevel\_eval\_flex at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/toplevel.c:903  
jl\_toplevel\_eval\_flex at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/toplevel.c:856  
ijl\_toplevel\_eval\_in at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/toplevel.c:971  
eval at ./boot.jl:370 [inlined]  
include\_string at ./loading.jl:1864  
\_jl\_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]  
ijl\_apply\_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940  
\_include at ./loading.jl:1924  
include at ./Base.jl:457  
jfptr\_include\_43521.clone\_1 at /home/users/ntu/tianqich/julia-1.9.0/lib/julia/sys.so (unknown line)  
\_jl\_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]  
ijl\_apply\_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940  
exec\_options at ./client.jl:307  
\_start at ./client.jl:522  
jfptr\_\_start\_37386.clone\_1 at /home/users/ntu/tianqich/julia-1.9.0/lib/julia/sys.so (unknown line)  
\_jl\_invoke at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]  
ijl\_apply\_generic at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/gf.c:2940  
jl\_apply at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]  
true\_main at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/jlapi.c:573  
jl\_repl\_entrypoint at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/src/jlapi.c:717  
main at julia (unknown line)  
\_\_libc\_start\_main at /lib64/libc.so.6 (unknown line)  
unknown function (ip: 0x401098)  
unknown function (ip: (nil))  
Allocations: 534481077 (Pool: 528171528; Big: 6309549); GC: 37930

May I ask what you suggest to debug this?

---

<div class="post-metadata">

**Author:** ![talentkeychen](https://yyz2.discourse-cdn.com/free1/user_avatar/itensor.discourse.group/talentkeychen/32/272_2.png) [@talentkeychen](https://itensor.discourse.group/u/talentkeychen)\
**Post date:** [May 14, 2023, 6:23am UTC](https://itensor.discourse.group/t/large-memory-issue-in-time-evolution-using-itensors-jl-with-apply-function/861/4 "2023-05-14T06:23:19Z")

</div>

Let me follow up a little bit here: after I insert a line of

```julia
GC.gc()

```

at the end of each loop, the problem seems to be solved. I gather that this is still a workaround. I suspected it is the way that I formed all my even and odd bonds MPO which leads to so much memory: as my Hamiltonian is actually time-dependent (periodic), for each loop, I just generate a list of them, and therefore there’re so many of my MPO’s. Is that the reason why my memory went out so quickly?

---

<div class="post-metadata">

**Author:** ![miles](https://yyz2.discourse-cdn.com/free1/user_avatar/itensor.discourse.group/miles/32/6_2.png) [@miles](https://itensor.discourse.group/u/miles)\
**Post date:** [May 14, 2023, 6:21pm UTC](https://itensor.discourse.group/t/large-memory-issue-in-time-evolution-using-itensors-jl-with-apply-function/861/5 "2023-05-14T18:21:57Z")

</div>

Hi Tianqi,  
Thanks for investigating this more on your own. If you are remaking the Hamiltonian on each loop of the innermost loop of a big calculation, then I could see how this could make “garbage” pile up more quickly than usual. Unfortunately this is a weakness of Julia right now, about how it can be too slow to collect freed memory. What you did is a good workaround for now, of calling the GC to collect at certain points. If it’s slowing down your code too much, you can put an if statement that makes the GC call happen only every other loop iteration, or every 10th iteration, say.

However, you may be in luck with a new feature in the recently released Julia 1.9. This feature is a “GC memory usage hint” flag you can pass when starting Julia:  
[https://julialang.org/blog/2023/04/julia-1.9-highlights/#memory\_usage\_hint\_for\_the\_gc\_with\_--heap-size-hint](https://julialang.org/blog/2023/04/julia-1.9-highlights/#memory_usage_hint_for_the_gc_with_--heap-size-hint)  
You can put a size of heap that you want the GC to respect and it will try to use closer to that amount of memory. So maybe you can try putting something like 75% of the RAM on your system, for example. I’d be curious if this flag works for you (with the GC.gc() line now commented out). If so, I will tell other ITensor users about it if they run into memory issues.

Best regards,  
Miles

---

<div class="post-metadata">

**Author:** ![talentkeychen](https://yyz2.discourse-cdn.com/free1/user_avatar/itensor.discourse.group/talentkeychen/32/272_2.png) [@talentkeychen](https://itensor.discourse.group/u/talentkeychen)\
**Post date:** [August 19, 2023, 4:36pm UTC](https://itensor.discourse.group/t/large-memory-issue-in-time-evolution-using-itensors-jl-with-apply-function/861/6 "2023-08-19T16:36:08Z")

</div>

Hi Miles, thanks for the solutions and detailed answers on GC.gc(), as well as GC memory usage hint. Sorry for the slow reply, as I have been in a job transition these few months, and I forgot to reply to this thread any time sooner.

Yes, this flag really helps to reduce the RAM usage. I supplied 0.5 of the RAM on the HPC limit, and turns out that it indeed used less than that memory (but is closer to that value) as shown in the output file, with GC.gc() being commented. I think in that case, we could let other ITensors.jl users be aware of this?

---

<div class="post-metadata">

**Author:** ![miles](https://yyz2.discourse-cdn.com/free1/user_avatar/itensor.discourse.group/miles/32/6_2.png) [@miles](https://itensor.discourse.group/u/miles)\
**Post date:** [August 20, 2023, 12:29am UTC](https://itensor.discourse.group/t/large-memory-issue-in-time-evolution-using-itensors-jl-with-apply-function/861/7 "2023-08-20T00:29:36Z")

</div>

That’s a good suggestion, thanks. Glad that flag has been helpful for you. I’ll add a new part to the ITensor FAQ page in the documentation about high performance computing and mention this flag there.