Using ElmerSolver_mpi on a Multi-body Problem

General discussion about Elmer
Post Reply
gschrank
Posts: 17
Joined: 21 Aug 2016, 05:40
Antispam: Yes

Using ElmerSolver_mpi on a Multi-body Problem

Post by gschrank »

Good evening!

I'm still fairly new to Elmer and FEM in general. I've been putting together a custom, non-linear solver module for a niche application I have, and I have run into an issue with running parallel computations with a multi-body in Elmer. I'd like to ask if it is possible in principle to run a multi-body problem in Elmer in parallel.

Physically, the problem will study laser absorption (the custom solver) in a flowing gas mixture. I've constructed a relatively complicated geometry/mesh using a combination of AutoDesk Inventor and Salome. The geometry is a 3D rendering of a component used in the process I'm studying. I've drawn a cylinder running through this geometry, and this cylinder represents the domain over which my custom solver will operate, i.e. the laser beam. On the rest of the geometry, I have navier-stokes and diffusion solver operating. Eventually, the custom solver will take information from the diffusion solver and use it as an input.

I have an Xeon i7 workstation that can handle the problem with in parallel or single thread. However, the speed up I gain using parallel computation is attractive. The problem will solve the navier-stokes and diffusion problems. But when it gets to the custom solver, I get thrown an error: there is no matrix. I know the custom solver can operate independently on a single body cylinder using a parallel computation.

I seem to recall reading somewhere on the forum that this was a known limitation; ElmerGrid -metis does not respect multi-body boundaries. I'd like to confirm that this is what is causing the problem. I'd also like to ask if it is the only barrier to trying to get these solvers to run on the geometry I described in a parallel computation. That is, if I'm able to figure out some way to partition the mesh, independent of ElmerGrid, in such away that the body boundaries are respected, will ElmerSolver_mpi be able to handle a computation in which a solver is active on one body but not active on the other? If ElmerSolver_mpi cannot do this, is there a simple way to cause Elmer to abort the mpi run after the diffusion solver and seamlessly complete the custom solver computation on a single thread so that I can get the speed-up for at least part of the computation?
raback
Site Admin
Posts: 4812
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Using ElmerSolver_mpi on a Multi-body Problem

Post by raback »

Hi

By multibody I guess you mean cases where there are number of bodies with different equations? This should be standard feature of Elmer.

Also the BCs associated to this kind of case should in general be ok. For example standard Neumann and Dirichlet conditions can be cut by the mesh partitioning without any problems. However, there are some more exotic conditions that do not allow this. Most importantly currently BCs utilizing mortar finite elements must recide in one partition (or in a few partitions among which halo is shared). This could happen if you have nonconforming contact BCs, periodic BCs, etc. Also diffuse grey radiation is problematic in the same way.

I would think that usually the "no matrix" case should be handled with the library features. A solver should not be called in a partition where there are no active elements for that case.

Also if you have a custom solver and compute there something locally remember that if you want to have some local operators (e.g. min/max) you may have to take care of the parallel reduction also.

-Peter
gschrank
Posts: 17
Joined: 21 Aug 2016, 05:40
Antispam: Yes

Re: Using ElmerSolver_mpi on a Multi-body Problem

Post by gschrank »

Peter:

By multi-body, I do mean different bodies connected on the same mesh with different equations. I prepare these meshes using Salome to define the different bodies.

Thank you so much for your quick reply and letting me know that what I am attempting is at least possible in the framework of ElmerSolver.

I've moved the basis of my custom module from the ModelPDE to AdvectionReaction. The equation I'm attempting to solve is essentially a sourceless Advection-Reaction Equation with a non-linear gamma term. I'd originally attempted this with the ModelPDE as a template, and that was causing problems with the "No Matrix" error when running in parallel. After changing the basis of the custom solver to the AdvectionReaction solver, I now get SEG FAULT errors when attempting to solve in parallel. I understand that the AdvectionReaction Solver might be a better basis for this due to it's reliance on DG.

I am using the -halo flag when partitioning my geometries with ElmerGrid. When I use this flag, I get a number of warnings of the form "Warning: side element ### of type ### is halo but nodes are not in partition :## ##."

It's unclear whether or not the problem is caused by not properly preparing my mesh or a problem with my custom solver. Could you offer any insight?

Thank you again!
raback
Site Admin
Posts: 4812
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Using ElmerSolver_mpi on a Multi-body Problem

Post by raback »

Hi

Unfortunately I cannot much help you without a minimalistic case which would demonstrate the problem. I just run a problem with multiple bodies such that some partition was totally void of any solvers without any problems.

-Peter
gschrank
Posts: 17
Joined: 21 Aug 2016, 05:40
Antispam: Yes

Re: Using ElmerSolver_mpi on a Multi-body Problem

Post by gschrank »

Peter:

Thanks so much. I understand. Maybe if you could help me understand first if the behaviour of ElmerGrid when using the -halo flag with metis is normal.

I've tried running the ElmerGrid on one of the elmerice examples that comes bundled with the source code. Specifically, under

Code: Select all

 /elmerfem/elmerice/examples/Test_MshGlacierDEM/Partitioned
I've gone through the README.txt for this example. When I use ElmerGrid to partition the mesh with the command

Code: Select all

ElmerGrid 2 2 teterousse -halo -metis 6 4
I get the following output:

Code: Select all

Warning: side element 61 of type 202 is halo but nodes are not in partition: 1 2
   part  elements   nodes      shared   bc elems orphan   indirect
   1     157        67         36       13       0        0       
   2     150        67         29       16       0        0       
Warning: side element 55 of type 202 is halo but nodes are not in partition: 1 2
   3     148        66         28       14       0        0       
Warning: side element 13 of type 202 is halo but nodes are not in partition: 1 2
Warning: side element 67 of type 202 is halo but nodes are not in partition: 1 2
   4     129        66         17       21       0        0       
   5     143        63         29       15       0        0       
   6     155        63         39       9        0        0       
Nodes needed in maximum 1 boundary elements
I get a similar set of warnings regarding side elements when I run ElmerGrid on my on meshes, and I'm wondering
(a) Is this normal behaviour for ElmerGrid when using -halo with -metis (or any partitioning algorithm)?
(b) What does this warning mean?
(c) Could whatever this warning indicts interfere with Elmer's ability to use the Discontinuous Galerkin method with modules such as Advection-Reaction.

I should note that I'm using the latest release (8.2) from the git tree and compiling it on an Ubuntu 14.04 machine.

Thank you so much. I really appreciate it.
raback
Site Admin
Posts: 4812
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Using ElmerSolver_mpi on a Multi-body Problem

Post by raback »

Hi

I made a fix to the -halo saving of ElmerGrid needed for DG solvers. You might try again with the current devel branch.

-Peter
Post Reply