Operating system error: Not enough space Out of memory

Numerical methods and mathematical models of Elmer
Post Reply
Kankreu
Posts: 23
Joined: 04 Dec 2012, 08:44
Antispam: Yes

Operating system error: Not enough space Out of memory

Post by Kankreu » 05 Mar 2013, 11:10

Dear All,
I'm having a memory problem on my vista machine.
It's a 64bits with Elmer 32bits (5884).
I'm running a model of 42k (quadratic) elements with GCR solver.
I get the following error:

Code: Select all

StressSolve:
StressSolve: --------------------------------------------------
StressSolve: Solving displacements from linear elasticity model
StressSolve: --------------------------------------------------
StressSolve:  Displacemet iteration:            1
StressSolve: Starting assembly...
StressSolve: Assembly done
StressSolve: Set boundaries done
Operating system error: Not enough space
Out of memory
I had 2Gb on my machine and I added 4Gb extra but it doesn't help.
At the moment of the crash, Elmer is using only 870Mb of RAM.

I installed the 64bit version of Elmer and there it works with 2.5Gb of RAM.

So I'm just wondering if the error occurs at the moment of the allocation of the memory and this is why -even I'm below the 2Gb limit- it crashes?

thanks,
Kankreu

raback
Site Admin
Posts: 3697
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Operating system error: Not enough space Out of memory

Post by raback » 05 Mar 2013, 12:02

Hi Kankreu

Perhaps you could share you sif file. The amount of memory used depends on the type of linear solver. The direct solvers and higher order ILUs as preconditioner use the most memory.

-Peter

Kankreu
Posts: 23
Joined: 04 Dec 2012, 08:44
Antispam: Yes

Re: Operating system error: Not enough space Out of memory

Post by Kankreu » 05 Mar 2013, 17:22

Hello Peter,

here is the file.

In fact I'm doing a study of the different solvers and precond from a performance p.o.v. (RAM usage/time) on a given problem.
You can also find attached my results so far (knowing that it was with Elmer 32bits, linear model and all ilu5-9 failed).

Thanks for your help,
K.
Attachments
Book1.7z
comparison results.
(230.86 KiB) Downloaded 140 times
quad.7z
sif file
(1.05 KiB) Downloaded 124 times

raback
Site Admin
Posts: 3697
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Operating system error: Not enough space Out of memory

Post by raback » 06 Mar 2013, 01:09

Hi

Nice rigorous studies you've been doing there. Perhaps I can suggest some more memory efficient strategies.

You might replace ILUn by BILUn. These make incomplete LU decomposition but only consider the diagonal blocks. Thus in 3D the BILU uses 1/3 of the memory of ILU, I think.

Now there are even some better strategies. Block preconditioning seems to perform very well. You can study the Linear system settings under test case BlockLinElast3, for example. The optimal solver for the individual blocks is some multigrid solver. Of those the clustering version (CMG) uses the least memory. The attached picture shows the promise of block preconditioning.

If you would have a massively parallel machine at your disposal. FETI (finite element tearing and interconnect) would be a good option. It is a domain decomposition method that under certain assumptions has a guaranteed convergence. The memory usage is quite large but if you can distribute the problem over many processors you can solver large problems. Good scalability has been demonstrated up to thousands of cores with problems size of ~1e8 unknowns.

-Peter
Attachments
Block preconditioning vs. Krylov method
Block preconditioning vs. Krylov method
ElasticBlockPrec.png (87.66 KiB) Viewed 3698 times

Kankreu
Posts: 23
Joined: 04 Dec 2012, 08:44
Antispam: Yes

Re: Operating system error: Not enough space Out of memory

Post by Kankreu » 06 Mar 2013, 11:05

Hi Peter,

Thanks for your reply.
I was just reading yesterday some of your posts where you mention this BILU(n).
From what I have seen, even with my 32bit version, I never had a memory issue during the preconditioning. Only during the iteration but of course, we could also see that the preconditioning is affecting the memory (for the same selected solver). I must say that I didn't read a lot (yet) about the different precond and solvers (even though I'm quite familiar with FEA solvers as I worked for CAE company before).

Of course I'm really interested in clustering, but I would like 1st to have a pretty good understanding of the IL/solvers -before going to multigrid- as I can choose my problem size.

I'm familiar with FETI and METIS decompositions, we used them in one of our solver for acoustic FEM mesh.

My planning is to update the excel sheet with the results of the 64bits version instead of the 32. I'll probably also extend it to the BILU preconds.
I'm also wondering if the problem size might affect the ranking (what if my model is 10, 40 or 100k elements).

My 2nd test will be (for a given precond/solver) to make a mesh (linear/quad) size convergence analysis. I'll study the convergence of the max stress as a function of the mesh size and monitor the time/memory usage. As I'm using a simple loaded clamp beam, I can compare to the analytical results.

If you are interested, I'll transfer you the results.

Greetings,
K.

raback
Site Admin
Posts: 3697
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Operating system error: Not enough space Out of memory

Post by raback » 06 Mar 2013, 11:35

Hi Kankreu

The solver and preconditioning greatly depends on the problem size. The standard Krylov methods will have a scaling O(~1.5) with the problem size in 3D, and O(~1.8) in 2D. Only multilevel methods can grow almost linearly with problem size. This is why we hope to advance those stategies. The block preconditioners allow that the problem is split into parts for which multigrid methods work ideally. Therefore I would warmly recommend you to try those strategies as well. You could get a totally different performance.

You might use higher order elemenets also by just setting in Solver section

Code: Select all

Element = p:2
-Peter

Kankreu
Posts: 23
Joined: 04 Dec 2012, 08:44
Antispam: Yes

Re: Operating system error: Not enough space Out of memory

Post by Kankreu » 06 Mar 2013, 12:46

Hi Peter,

I realized that the block preconditioners don't requires parallel solving (even it's probably the intention?).
I start getting confused between parallelization (which I know) and BP or what you call "multilvel methods".
Can you tell me more (or references) about the CMG/SGS?
I'll try to include that in my testings (it will make a lot of tests...)

apparently, I should also repeat my study for different mesh sizes.

I'm confused with Multigrid. When I tested all combinations, I realized that multigrid required more info or should be linked to multigrid method.

Code: Select all

ERROR:: AddEquationSolution: > MG Levels < must be defined for geometric multigrid!
Am I wrong thinking that multigrid method means parallel computation? or can it be used in serial (like the BP then).

thanks,
K.

PS: If you look in my case.sif, I'm already using quad elements.
This is how I'm planning to make the comparison for the same mesh size with different order.

raback
Site Admin
Posts: 3697
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Operating system error: Not enough space Out of memory

Post by raback » 06 Mar 2013, 13:59

Hi

Multigrid does not imply parallelism. In fact of the different multigrid options there are several native ones to Elmer that are limited to serial computation except for GMG (geometric multigrid). There are interfaces to BoomerAMG (via Hypre) and ML (Trilinos) but these are only available in parallel. So I can understand confusion.

In Elmer the "cmg" is basically "unsmoothed agglomeration multigrid" and "sgs" refers to "symmetric gauss-seidel" as a smoother. These are used for each block separately hence the term "block preconditioning". Even this does not imply parallelism. These BP is best used for some Krylov method that directly minimizes the residual (GMRES, GCR).

-Peter

Kankreu
Posts: 23
Joined: 04 Dec 2012, 08:44
Antispam: Yes

Re: Operating system error: Not enough space Out of memory

Post by Kankreu » 11 Mar 2013, 11:15

Thanks Peter,

I'll give them a try when I'll have time to complete my study.
I'll also try to find some doc. If not, I'll contact you back.

Greetings,
K

Post Reply