## ElmerGrid: Strongly different number of elements in partition mesh

Numerical methods and mathematical models of Elmer
mb5
Posts: 17
Joined: 10 Jun 2017, 18:07
Antispam: Yes

### ElmerGrid: Strongly different number of elements in partition mesh

Hallo,

I have a Model (rotating machine) with about 250.000 elements which I want to devide in partitions for mpi calculation. I have boundaries to connect, which can be seen in the attached picture. These boundaries get connected with the parameter "-connect 1 43" with ElmerGrid. With this I got 4 mesh partitions (in the attached picture, too). This works and the simulation show correct results. My Problem is, that the number of elements/nodes are very different in the mesh partitions. So I got a bad parallel scaling.

4 parts:

Code: Select all

``````ElmerGrid 2 2 mesh -metis 4 3 -connect 1 4 3 -partdual
...
part  elements   nodes      shared   bc elems indirect
1     41600      33715      9062     12800    0
2     68429      39077      2763     8821     0
3     68426      42445      3354     8274     0
4     68425      48143      3084     7153     0
Nodes needed in maximum 3 boundary elements
``````
2 parts:

Code: Select all

``````ElmerGrid 2 2 mesh -metis 2 3 -connect 1 4 3 -partdual
...
part  elements   nodes      shared   bc elems indirect
1     41600      33705      9072     12800    0
2     205280     129675     8442     24248    0
Nodes needed in maximum 3 boundary elements
``````
I have an extruded mesh, so I could use "partconnect". But with this it is getting much worse:

Code: Select all

``````ElmerGrid 2 2 mesh -metis 4 3 -connect 1 4 3 -partdual -partconnect 2
...
part  elements   nodes      shared   bc elems indirect
1     20800      16694      5713     6400     0
2     20800      16695      5712     6400     0
3     102477     71175      4435     10980    0
4     102803     58816      3961     13268    0
Nodes needed in maximum 3 boundary elements
``````

It seems, that ElmerGrid creates the first partition with all elements/nodes which belongs to the boundary and all other elements/nodes belongs to other partitions.
But I want, that the first partition (with the boundary) get more elements to get a similar number of elements in each partition. Is this possible?

Best regards
Martin
Attachments
Model.png (140.97 KiB) Viewed 485 times

raback
Posts: 3416
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

### Re: ElmerGrid: Strongly different number of elements in partition mesh

Hi

Yes, when you have mortar BCs then the current implementation does not work unless they are in the same partition. Hence the "-connect" flag.

You could extend the connected volume with "-partlayers n" where n is number of layers, e.g. 2. However, I suspect that it wont help much.

What if you study the number of iterations needed for parallel / serial task. Is the solution of the linear system partly to blame for a poorer scalability. And when you say that your scalability os "poor" how is it?

We are usually working with much larger numbers of cores when performing parallel runs. Then it is not usually a problem that a one single partition has less of a work than the others. Usually it is problematic is if one partition has more work than the others. For rotating problems with mortar BCs the "-partconnect" flag was a remedy for that case. Really haven't spent too much effort for this. The "-partlayers" is quite coarse since the number of elements grows in steps.

-Peter

mb5
Posts: 17
Joined: 10 Jun 2017, 18:07
Antispam: Yes

### Re: ElmerGrid: Strongly different number of elements in partition mesh

Hi,

I've done some tests with different settings. The attached picture (the lower one) shows the number of elements in each parts and the speed up compared to a serial simulation. Because of the nonlinear calculation with many time steps (electrical machine) i summed up all linear iterations of the simulation (BiCGstabl). In some cases the number of iterations increase compared to the serial simulation. So when optimizing the partition meshing by myself I got equal element numbers. But this isn't a convenient for automated model generation.

As expected I got the best scale with a mesh with similar number of elements in each partition. The best speed up ist 1.83 for 4 CPU Cores. Could there be a problem with my mpi? I got a message from the solver:
[[13601,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
Host: *****

Another transport will be used instead, although this may result in
lower performance.

You are absolute right. The number of cores for the simulation is quiet small, because my Model isn't that big. My task is to optimize electrical machines. Therefore I have run hundreds of simulations for different designs for example. In my opinion there are three methods to do that
1. Run as many simulations as CPU cores available -> good scaling, but to few memory
• Run all simulation successively but on all CPU cores -> maybe bad scaling
• Combination of 1 and 2: Use only a small number of CPU cores per simulation but run many simulation at the same time -> good usage of the CPUs?
I've tried to make a sketch of that in the attached picture (the upper one). I thought number 3 could be a good way. Whats your opinion?

Best regards,
Martin
Attachments
serial_parallel.png (93.78 KiB) Viewed 469 times
bench.png (13.99 KiB) Viewed 469 times

raback
Posts: 3416
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

### Re: ElmerGrid: Strongly different number of elements in partition mesh

Hi

Yes, for your setup the strategy seems logical.

Do you have electrical circuits in your system, or is it simply driven by BCs? The electrical circuits add parallel constraints to the system which may make the scaling worse.

I would expect speedup of about ~3. It is not perfect since due to mortars etc. but should be not this bad either. Unfortutanely I'm no expert on MPI setup. Trying out on different platform might be usefull for comparison.

I guess you tried with empty machine. If you have large jobs they may influence each other if the memory is constrained.

-Peter

mb5
Posts: 17
Joined: 10 Jun 2017, 18:07
Antispam: Yes

### Re: ElmerGrid: Strongly different number of elements in partition mesh

Hi,

I have no circuits in the model, only magnetization and current density.

Hmm, I ran these benchmark at the same time. So there was cpu load on twenty cores (my machine have 40 cores + 40 Hyperthreading cores). Maybe I should repeat the benchmark. I will try to use my desktop computer with an other operating system. A speedup of 3 would be great.
Thanks for your hints and thanks for developing ELMER!

Regards
Martin