# Memory usage in DMRG with Julia 1.x

**URL:** <https://itensor.discourse.group/t/memory-usage-in-dmrg-with-julia-1-x/1092>\
**Category:** ITensor Julia Questions\
**Created:** [August 16, 2023, 7:57pm UTC](https://itensor.discourse.group/t/memory-usage-in-dmrg-with-julia-1-x/1092 "2023-08-16T19:57:14Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ryanlevy](https://yyz2.discourse-cdn.com/free1/user_avatar/itensor.discourse.group/ryanlevy/32/150_2.png) [@ryanlevy](https://itensor.discourse.group/u/ryanlevy)\
**Post date:** [August 16, 2023, 7:57pm UTC](https://itensor.discourse.group/t/memory-usage-in-dmrg-with-julia-1-x/1092/1 "2023-08-16T19:57:14Z")

</div>

I came across some problems with memory usage in Julia DMRG and wanted to share my findings (similar to discussions in [Large amount of memory used, when dmrg runs on cluster](https://itensor.discourse.group/t/large-amount-of-memory-used-when-dmrg-runs-on-cluster/1045/1) and [memory usage in dmrg (julia)](https://itensor.discourse.group/t/memory-usage-in-dmrg-julia/562/1) )

- _tl;dr_ add `GC.gc()` to an observer or use a small `--heap-size-hint`, or use a version other than 1.9

### Experiment

I created a test case of DMRG with no quantum numbers on a 16 site non-interacting chain of electrons (H=\sum\_{i\sigma} c^\dagger\_{i\sigma} c\_{j\sigma}). I slowly increase the bond dimension, running over 15 sweeps

> **Sweeps**
>
> ```julia
> 1 cutoff=1.0E-16, maxdim=64, mindim=1, noise=1.0E-03
> 2 cutoff=1.0E-16, maxdim=64, mindim=1, noise=1.0E-04
> 3 cutoff=1.0E-16, maxdim=64, mindim=1, noise=1.0E-08
> 4 cutoff=1.0E-16, maxdim=64, mindim=1, noise=1.0E-08
> 5 cutoff=1.0E-16, maxdim=128, mindim=1, noise=1.0E-08
> 6 cutoff=1.0E-16, maxdim=128, mindim=1, noise=1.0E-08
> 7 cutoff=1.0E-16, maxdim=128, mindim=1, noise=1.0E-08
> 8 cutoff=1.0E-16, maxdim=128, mindim=1, noise=1.0E-08
> 9 cutoff=1.0E-16, maxdim=256, mindim=1, noise=1.0E-08
> 10 cutoff=1.0E-16, maxdim=256, mindim=1, noise=1.0E-08
> 11 cutoff=1.0E-16, maxdim=256, mindim=1, noise=1.0E-08
> 12 cutoff=1.0E-16, maxdim=256, mindim=1, noise=1.0E-08
> 13 cutoff=1.0E-16, maxdim=256, mindim=1, noise=1.0E-08
> 14 cutoff=1.0E-16, maxdim=256, mindim=1, noise=1.0E-08
> 15 cutoff=1.0E-16, maxdim=256, mindim=1, noise=1.0E-08
> 
> ```

I record the **peak** memory usage (maxRSS)

> **Code**
>
> ```julia
> # see https://github.com/JuliaLang/julia/blob/master/test/netload/memtest.jl
> struct RUsage
> ru_utime_sec::Clong # user CPU time used
> ru_utime_usec::Clong # user CPU time used
> ru_stime_sec::Clong # system CPU time used
> ru_stime_usec::Clong # system CPU time used
> ru_maxrss::Clong # maximum resident set size
> ru_ixrss::Clong # integral shared memory sizeG
> ru_idrss::Clong # integral unshared data size
> ru_isrss::Clong # integral unshared stack size
> ru_minflt::Clong # page reclaims (soft page faults)
> ru_majflt::Clong # page faults (hard page faults)
> ru_nswap::Clong # swaps
> ru_inblock::Clong # block input operations
> ru_oublock::Clong # block output operations
> ru_msgsnd::Clong # IPC messages sent
> ru_msgrcv::Clong # IPC messages received
> ru_nsignals::Clong # signals received
> ru_nvcsw::Clong # voluntary context switches
> ru_nivcsw::Clong # involuntary context switches
> end
> 
> function get_vmsize()
> ru = Vector{RUsage}(undef, 1)
> ccall(:getrusage, Cint, (Cint, Ptr{Cvoid}), 0, ru)
> return ru[1].ru_maxrss
> end
> 
> ```

This machine has 188GB so plenty of room, and there’s no slurm involved.

### Results

I ran the test above on several Julia versions (1.8,1.9,1.10-beta).  
Essentially if you use the latest release version of Julia (1.9.2), the peak memory usage (maxRSS) continues to grow compared to previous and newer versions

 ![image](https://global.discourse-cdn.com/free1/uploads/itensor/original/1X/bbe138a9d5ab8238312c6631105e8590d6f64e5f.png)  
1.8.5 uses the least amount of memory, and 1.10-beta uses more (a trade off for faster compiling I believe) while 1.9.2 will slowly explode. In my production code with 1.9.2 the total RAM usage ends up at \> 1TB after enough time, so don’t let the small size here fool you.

### Possible Solutions

 ![image](https://global.discourse-cdn.com/free1/uploads/itensor/original/1X/4a2b79c50a5a6309dc38d0203e1954c8c03f4ab0.png)

First, and probably the easiest, you can add in an [observer](https://itensor.github.io/ITensors.jl/stable/Observer.html) that runs garbage collection

```julia
mutable struct GCObserver <: AbstractObserver
end

function ITensors.measure!(o::GCObserver; kwargs...)
  bond = kwargs[:bond]
  half_sweep = kwargs[:half_sweep]
  (bond==1 && half_sweep==2) && GC.gc()
end

```

I only tried the above, but let me know if anyone suggests a different frequency than 1/sweep.

That seems to work well and reduced the memory significantly but I have not benchmarked the total time differences for larger problems.  
The other solution is the often recommended [`--heap-size-hint` flag](https://julialang.org/blog/2023/04/julia-1.9-highlights/#memory_usage_hint_for_the_gc_with_--heap-size-hint). I find that you really want this number smaller than your total memory so that the garbage collection is run (purple line vs red line), if the environment is very ram sensitive.

---

<div class="post-metadata">

**Author:** ![mtfishman](https://yyz2.discourse-cdn.com/free1/user_avatar/itensor.discourse.group/mtfishman/32/11_2.png) [@mtfishman](https://itensor.discourse.group/u/mtfishman)\
**Post date:** [August 17, 2023, 7:59pm UTC](https://itensor.discourse.group/t/memory-usage-in-dmrg-with-julia-1-x/1092/2 "2023-08-17T19:59:16Z")

</div>

Thanks for the in-depth report Ryan! Really helpful.

Did you find a Julia Github issue or discourse post related to this? If not, we should ask about this on Julia discourse and see if we should raise an issue about it on Julia’s Github repository if there isn’t one already.

---

<div class="post-metadata">

**Author:** ![mtfishman](https://yyz2.discourse-cdn.com/free1/user_avatar/itensor.discourse.group/mtfishman/32/11_2.png) [@mtfishman](https://itensor.discourse.group/u/mtfishman)\
**Post date:** [August 18, 2023, 1:35pm UTC](https://itensor.discourse.group/t/memory-usage-in-dmrg-with-julia-1-x/1092/3 "2023-08-18T13:35:23Z")

</div>

May be related to:

> <https://github.com/JuliaLang/julia/issues/49761>
>
> \`\`\`julia
> steps = 799500:100:800000
> 
> function main(nx, ny, nz, steps, Axi, Axj…, Ayi, Ayj)
> 
> hx = 1.0/(nx - 1); hy = 1/(ny - 1);
> 
> for n in steps
> 		
>        
> w=rand(nx,ny,nz)
> u=rand(nx,ny,nz)
> v=rand(nx,ny,nz)
>         
> Tsb = 110.4 / T\_inf
> 
> Amu =@. 1.0 / Re \* (1.0 + Tsb) \* sqrt(T .^ 3)/ (Tsb + Tw)
> 
> ux, uy, uz = get\_grad(u, hx, hy, Axi, Axj, Ayi, Ayj, sz)
> vx, vy, vz = get\_grad(v, hx, hy, Axi, Axj, Ayi, Ayj, sz)
> wx, wy, wz = get\_grad(w, hx, hy, Axi, Axj, Ayi, Ayj, sz)
> 
> Theta = ux + vy + wz
> 
> Sigma11 = @. (ux\*2 - 2/3\*Theta) \* Amu
> Sigma22 = @. (vy\*2 - 2/3\*Theta) \* Amu
> Sigma33 = @. (wz\*2 - 2/3\*Theta) \* Amu
> Sigma12 = @. (uy + vx) \* Amu
> Sigma23 = @. (vz + wy) \* Amu
> Sigma13 = @. (uz + wx) \* Amu
> 
> println("Write OCFD$(string(n,pad=7)).jld2")
> 				
> jldopen("$(filepath1)test$(string(n,pad=7))\_sigma\_theta.jld2","w") do f
> f\["σ11"\] = Sigma11
> f\["σ22"\] = Sigma22
> f\["σ33"\] = Sigma33
> f\["σ12"\] = Sigma12
> f\["σ13"\] = Sigma13
> f\["σ23"\] = Sigma23
> f\["θ"\] = Theta
> end
> end
> 
> end
> 
> 
> @time main( nx, ny, nz, steps, Axi, Axj, Ayi, Ayj)
> \`\`\`
> 
> The memory allocated by Julia will keep increase in for loop. How to solve it, only by \`gc()\` manually? This fig shows the memory allocated when run this Julia code.
> !\[536fec9f26a3c3c4\](https://github.com/JuliaLang/julia/assets/112403698/a79cfbb9-60bd-43ca-b4ec-9b584eb15a33)

> <https://github.com/JuliaLang/julia/issues/49545>
>
> I managed to isolate a memory leak that caused runaway memory usage for long run…ning processes due to some logging I was doing. 
> 
> MWE:
> 
> \`\`\`julia
> function explode(lck, buffer, path, message; kwargs...)
> lock(buffer) do
> println(buffer, message)
> open(path, append = true) do f
> write(f, take!(buffer))
> end
> end
> end
> 
> let
> lck = ReentrantLock()
> buffer = IOBuffer()
> path = "test.log"
> while true
> Threads.@spawn begin
> explode(lck, buffer, path, "blabla")
> end
> end
> end
> \`\`\`
> 
> 
> versioninfo():
> \`\`\`
> Julia Version 1.9.0-rc2
> Commit 72aec423c2 (2023-04-01 10:41 UTC)
> Platform Info:
> OS: Linux (x86\_64-pc-linux-gnu)
> CPU: 8 × Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
> WORD\_SIZE: 64
> LIBM: libopenlibm
> LLVM: libLLVM-14.0.6 (ORCJIT, skylake)
> Threads: 8 on 8 virtual cores
> Environment:
> LD\_LIBRARY\_PATH = /opt/intel/compilers\_and\_libraries\_2020.4.304/linux/compiler/lib/intel64\_lin:/opt/intel/compilers\_and\_libraries\_2020.4.304/linux/mkl/lib/intel64\_lin:/opt/intel/compilers\_and\_libraries\_2020.4.304/linux/compiler/lib/intel64\_lin:/opt/intel/compilers\_and\_libraries\_2020.4.304/linux/mkl/lib/intel64\_lin
> JULIA\_NUM\_THREADS = 8
> \`\`\`
> 
> Compiled from source. Also tested on downloaded binaries of julia \`1.8.2\` , \`1.6.7\`

which should be fixed by:

> <https://github.com/JuliaLang/julia/pull/50144>
>
> This PR implements GC heuristics based on the amount of pages allocated instead …of live objects like was done before.
> The heuristic for new heap target is based on https://dl.acm.org/doi/10.1145/3563323 (in summary it argues that the heap target should have square root behaviour).
> From my testing this fixes https://github.com/JuliaLang/julia/issues/49545 and https://github.com/JuliaLang/julia/issues/49761

which will be included in Julia 1.10 (which just had a new beta release [Julia v1.10.0-beta2 is now available - Announcements - Julia Programming Language](https://discourse.julialang.org/t/julia-v1-10-0-beta2-is-now-available/102923)):  
[https://github.com/JuliaLang/julia/blob/v1.10.0-beta2/NEWS.md#compilerruntime-improvements](https://github.com/JuliaLang/julia/blob/v1.10.0-beta2/NEWS.md#compilerruntime-improvements)

It looks like it may not be backported to Julia 1.9 so users should upgrade to Julia 1.10, use Julia 1.6, 1.7, or 1.8, or if using Julia 1.9, manually call the garbage collector as Ryan summarized in his first post.

---

<div class="post-metadata">

**Author:** ![ryanlevy](https://yyz2.discourse-cdn.com/free1/user_avatar/itensor.discourse.group/ryanlevy/32/150_2.png) [@ryanlevy](https://itensor.discourse.group/u/ryanlevy)\
**Post date:** [August 18, 2023, 3:44pm UTC](https://itensor.discourse.group/t/memory-usage-in-dmrg-with-julia-1-x/1092/4 "2023-08-18T15:44:57Z")

</div>

Thanks for following up on this! Looks like 1.10-beta2 is definitely more sensible with the default GC

 ![image](https://global.discourse-cdn.com/free1/uploads/itensor/original/1X/2e03dbe810b4a1f9f21dd9b78d8808fd246ce428.png)

---

<div class="post-metadata">

**Author:** ![ryanlevy](https://yyz2.discourse-cdn.com/free1/user_avatar/itensor.discourse.group/ryanlevy/32/150_2.png) [@ryanlevy](https://itensor.discourse.group/u/ryanlevy)\
**Post date:** [November 13, 2023, 12:39am UTC](https://itensor.discourse.group/t/memory-usage-in-dmrg-with-julia-1-x/1092/5 "2023-11-13T00:39:32Z")

</div>

Since 1.10 has a release candidate out and and a few versions of passed, I re-ran the above benchmark

 ![image](https://global.discourse-cdn.com/free1/uploads/itensor/original/1X/5a09c76c46899540d0b09b609d350194359ea8c8.png)

Looks like 1.10 is going to be great! (Nothing new in `NEWS.md` about this)

And I managed to plot some timing data, this is only 6 sweeps at bond dimension 256  
 ![image](https://global.discourse-cdn.com/free1/uploads/itensor/original/1X/c25ecfd835f4b8c5f301bdb18864f410ebb79ba1.png)

So there is a small (~25%) performance hit for all of this, but this is probably below the “BLAS limit”

---

<div class="post-metadata">

**Author:** ![mtfishman](https://yyz2.discourse-cdn.com/free1/user_avatar/itensor.discourse.group/mtfishman/32/11_2.png) [@mtfishman](https://itensor.discourse.group/u/mtfishman)\
**Post date:** [November 13, 2023, 3:37pm UTC](https://itensor.discourse.group/t/memory-usage-in-dmrg-with-julia-1-x/1092/6 "2023-11-13T15:37:59Z")

</div>

That’s great, thanks for tracking that @ryanlevy! Seems like they are making a lot of improvements to the GC.

Curious if the new multithreaded garbage collector also helps DMRG: [Julia 1.10-rc1 gives a huge speed-up over Julia 1.9 - General Usage - Julia Programming Language](https://discourse.julialang.org/t/julia-1-10-rc1-gives-a-huge-speed-up-over-julia-1-9/106072), [Multi-Threading · The Julia Language](https://docs.julialang.org/en/v1.10.0-rc1/manual/multi-threading/#Multiple-GC-Threads).

---

<div class="post-metadata">

**Author:** ![mtfishman](https://yyz2.discourse-cdn.com/free1/user_avatar/itensor.discourse.group/mtfishman/32/11_2.png) [@mtfishman](https://itensor.discourse.group/u/mtfishman)\
**Post date:** [November 13, 2023, 3:40pm UTC](https://itensor.discourse.group/t/memory-usage-in-dmrg-with-julia-1-x/1092/7 "2023-11-13T15:40:24Z")

</div>

Also will be interesting to be able to test out alternative GC backends once that becomes available, i.e. [Introduce an interface for external GC backends by d-netto · Pull Request #51788 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/51788).