Good evening!
I'm still fairly new to Elmer and FEM in general. I've been putting together a custom, nonlinear solver module for a niche application I have, and I have run into an issue with running parallel computations with a multibody in Elmer. I'd like to ask if it is possible in principle to run a multibody problem in Elmer in parallel.
Physically, the problem will study laser absorption (the custom solver) in a flowing gas mixture. I've constructed a relatively complicated geometry/mesh using a combination of AutoDesk Inventor and Salome. The geometry is a 3D rendering of a component used in the process I'm studying. I've drawn a cylinder running through this geometry, and this cylinder represents the domain over which my custom solver will operate, i.e. the laser beam. On the rest of the geometry, I have navierstokes and diffusion solver operating. Eventually, the custom solver will take information from the diffusion solver and use it as an input.
I have an Xeon i7 workstation that can handle the problem with in parallel or single thread. However, the speed up I gain using parallel computation is attractive. The problem will solve the navierstokes and diffusion problems. But when it gets to the custom solver, I get thrown an error: there is no matrix. I know the custom solver can operate independently on a single body cylinder using a parallel computation.
I seem to recall reading somewhere on the forum that this was a known limitation; ElmerGrid metis does not respect multibody boundaries. I'd like to confirm that this is what is causing the problem. I'd also like to ask if it is the only barrier to trying to get these solvers to run on the geometry I described in a parallel computation. That is, if I'm able to figure out some way to partition the mesh, independent of ElmerGrid, in such away that the body boundaries are respected, will ElmerSolver_mpi be able to handle a computation in which a solver is active on one body but not active on the other? If ElmerSolver_mpi cannot do this, is there a simple way to cause Elmer to abort the mpi run after the diffusion solver and seamlessly complete the custom solver computation on a single thread so that I can get the speedup for at least part of the computation?
Using ElmerSolver_mpi on a Multibody Problem

 Site Admin
 Posts: 3498
 Joined: 22 Aug 2009, 11:57
 Antispam: Yes
 Location: Espoo, Finland
 Contact:
Re: Using ElmerSolver_mpi on a Multibody Problem
Hi
By multibody I guess you mean cases where there are number of bodies with different equations? This should be standard feature of Elmer.
Also the BCs associated to this kind of case should in general be ok. For example standard Neumann and Dirichlet conditions can be cut by the mesh partitioning without any problems. However, there are some more exotic conditions that do not allow this. Most importantly currently BCs utilizing mortar finite elements must recide in one partition (or in a few partitions among which halo is shared). This could happen if you have nonconforming contact BCs, periodic BCs, etc. Also diffuse grey radiation is problematic in the same way.
I would think that usually the "no matrix" case should be handled with the library features. A solver should not be called in a partition where there are no active elements for that case.
Also if you have a custom solver and compute there something locally remember that if you want to have some local operators (e.g. min/max) you may have to take care of the parallel reduction also.
Peter
By multibody I guess you mean cases where there are number of bodies with different equations? This should be standard feature of Elmer.
Also the BCs associated to this kind of case should in general be ok. For example standard Neumann and Dirichlet conditions can be cut by the mesh partitioning without any problems. However, there are some more exotic conditions that do not allow this. Most importantly currently BCs utilizing mortar finite elements must recide in one partition (or in a few partitions among which halo is shared). This could happen if you have nonconforming contact BCs, periodic BCs, etc. Also diffuse grey radiation is problematic in the same way.
I would think that usually the "no matrix" case should be handled with the library features. A solver should not be called in a partition where there are no active elements for that case.
Also if you have a custom solver and compute there something locally remember that if you want to have some local operators (e.g. min/max) you may have to take care of the parallel reduction also.
Peter
Re: Using ElmerSolver_mpi on a Multibody Problem
Peter:
By multibody, I do mean different bodies connected on the same mesh with different equations. I prepare these meshes using Salome to define the different bodies.
Thank you so much for your quick reply and letting me know that what I am attempting is at least possible in the framework of ElmerSolver.
I've moved the basis of my custom module from the ModelPDE to AdvectionReaction. The equation I'm attempting to solve is essentially a sourceless AdvectionReaction Equation with a nonlinear gamma term. I'd originally attempted this with the ModelPDE as a template, and that was causing problems with the "No Matrix" error when running in parallel. After changing the basis of the custom solver to the AdvectionReaction solver, I now get SEG FAULT errors when attempting to solve in parallel. I understand that the AdvectionReaction Solver might be a better basis for this due to it's reliance on DG.
I am using the halo flag when partitioning my geometries with ElmerGrid. When I use this flag, I get a number of warnings of the form "Warning: side element ### of type ### is halo but nodes are not in partition :## ##."
It's unclear whether or not the problem is caused by not properly preparing my mesh or a problem with my custom solver. Could you offer any insight?
Thank you again!
By multibody, I do mean different bodies connected on the same mesh with different equations. I prepare these meshes using Salome to define the different bodies.
Thank you so much for your quick reply and letting me know that what I am attempting is at least possible in the framework of ElmerSolver.
I've moved the basis of my custom module from the ModelPDE to AdvectionReaction. The equation I'm attempting to solve is essentially a sourceless AdvectionReaction Equation with a nonlinear gamma term. I'd originally attempted this with the ModelPDE as a template, and that was causing problems with the "No Matrix" error when running in parallel. After changing the basis of the custom solver to the AdvectionReaction solver, I now get SEG FAULT errors when attempting to solve in parallel. I understand that the AdvectionReaction Solver might be a better basis for this due to it's reliance on DG.
I am using the halo flag when partitioning my geometries with ElmerGrid. When I use this flag, I get a number of warnings of the form "Warning: side element ### of type ### is halo but nodes are not in partition :## ##."
It's unclear whether or not the problem is caused by not properly preparing my mesh or a problem with my custom solver. Could you offer any insight?
Thank you again!

 Site Admin
 Posts: 3498
 Joined: 22 Aug 2009, 11:57
 Antispam: Yes
 Location: Espoo, Finland
 Contact:
Re: Using ElmerSolver_mpi on a Multibody Problem
Hi
Unfortunately I cannot much help you without a minimalistic case which would demonstrate the problem. I just run a problem with multiple bodies such that some partition was totally void of any solvers without any problems.
Peter
Unfortunately I cannot much help you without a minimalistic case which would demonstrate the problem. I just run a problem with multiple bodies such that some partition was totally void of any solvers without any problems.
Peter
Re: Using ElmerSolver_mpi on a Multibody Problem
Peter:
Thanks so much. I understand. Maybe if you could help me understand first if the behaviour of ElmerGrid when using the halo flag with metis is normal.
I've tried running the ElmerGrid on one of the elmerice examples that comes bundled with the source code. Specifically, under
I've gone through the README.txt for this example. When I use ElmerGrid to partition the mesh with the command
I get the following output:
I get a similar set of warnings regarding side elements when I run ElmerGrid on my on meshes, and I'm wondering
(a) Is this normal behaviour for ElmerGrid when using halo with metis (or any partitioning algorithm)?
(b) What does this warning mean?
(c) Could whatever this warning indicts interfere with Elmer's ability to use the Discontinuous Galerkin method with modules such as AdvectionReaction.
I should note that I'm using the latest release (8.2) from the git tree and compiling it on an Ubuntu 14.04 machine.
Thank you so much. I really appreciate it.
Thanks so much. I understand. Maybe if you could help me understand first if the behaviour of ElmerGrid when using the halo flag with metis is normal.
I've tried running the ElmerGrid on one of the elmerice examples that comes bundled with the source code. Specifically, under
Code: Select all
/elmerfem/elmerice/examples/Test_MshGlacierDEM/Partitioned
Code: Select all
ElmerGrid 2 2 teterousse halo metis 6 4
Code: Select all
Warning: side element 61 of type 202 is halo but nodes are not in partition: 1 2
part elements nodes shared bc elems orphan indirect
1 157 67 36 13 0 0
2 150 67 29 16 0 0
Warning: side element 55 of type 202 is halo but nodes are not in partition: 1 2
3 148 66 28 14 0 0
Warning: side element 13 of type 202 is halo but nodes are not in partition: 1 2
Warning: side element 67 of type 202 is halo but nodes are not in partition: 1 2
4 129 66 17 21 0 0
5 143 63 29 15 0 0
6 155 63 39 9 0 0
Nodes needed in maximum 1 boundary elements
(a) Is this normal behaviour for ElmerGrid when using halo with metis (or any partitioning algorithm)?
(b) What does this warning mean?
(c) Could whatever this warning indicts interfere with Elmer's ability to use the Discontinuous Galerkin method with modules such as AdvectionReaction.
I should note that I'm using the latest release (8.2) from the git tree and compiling it on an Ubuntu 14.04 machine.
Thank you so much. I really appreciate it.

 Site Admin
 Posts: 3498
 Joined: 22 Aug 2009, 11:57
 Antispam: Yes
 Location: Espoo, Finland
 Contact:
Re: Using ElmerSolver_mpi on a Multibody Problem
Hi
I made a fix to the halo saving of ElmerGrid needed for DG solvers. You might try again with the current devel branch.
Peter
I made a fix to the halo saving of ElmerGrid needed for DG solvers. You might try again with the current devel branch.
Peter