Single-core discretization, multi-core solver?

Numerical methods and mathematical models of Elmer
Post Reply
adamo
Posts: 19
Joined: 02 Feb 2017, 00:19
Antispam: Yes

Single-core discretization, multi-core solver?

Post by adamo »

Dear Elmer community,

I am currently facing with two problems which are difficult to parallelize on the geometrical level (by partitioning) but could still benefit from parallelization. The first is radiative heat transfer (you cannot parallelize Gebhardt Factor calculation and even if you did, it might be quite inefficient) and the second requires adaptive mesh refinement.

So there is one thing I have been wondering about - do you know/see a way to make Elmer perform the matrix composition on a single core and then only run the solver in parallel? In a custom code that I worked with years ago, we did discretization in serial, composed the linear system matrix and then fed it to MUMPS, which took care of the parallelization itself. Commercial COMSOL Multiphysics also seems to utilize this strategy, rather than the partitioning.

Now I do realize the obvious problem - I am running a single ElmerSolver process but I would like MUMPS or another solver to work with several processes. It seems that this is not currently available in Elmer but I would be very interested to hear the opinion of the developers, if they think it is feasible, and if so, where to begin :-).

Many thanks

Adam
raback
Site Admin
Posts: 4812
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Single-core discretization, multi-core solver?

Post by raback »

Hi Adam,

There have been a lot of developments to multithread Elmer. Unfortunately this is a long road so it has taken some time. The challenge with multithreading is that the changes need to be implemented in many places, and if it is not done everywhere then there are easily bottle-necks. Elmer has always been a MPI code and we consider this still the main strategy. However, as the number of cores grow it makes sense to use hybrid strategies using OpenMP within node. Using only multithreading sets some severe limitations to the number of cores that can be used. Elmer is developed at a supercomputer center so we should not be shy of thousands of cores.

Splitting the mesh results to natural parallelization. All other steps except for the solution are trivially parallel in FEM. The best OpenMP parallelization is now comparable to MPI within node. However, the radiation factor computation is not parallelized. I would think that if you turn the number of threads to be larger than one, and choose a multithreaded linear solver (e.g. MKL Pardiso) you should at least have significant boost from serial computation.

-Peter
adamo
Posts: 19
Joined: 02 Feb 2017, 00:19
Antispam: Yes

Re: Single-core discretization, multi-core solver?

Post by adamo »

Hi Peter,

thanks a lot for the quick answer, I will try compiling Elmer with Pardiso and SuperLU and post my experience here.

Best

Adam
adamo
Posts: 19
Joined: 02 Feb 2017, 00:19
Antispam: Yes

Re: Single-core discretization, multi-core solver?

Post by adamo »

Dear Peter, could you please advise me, where to find instructions on how to build Elmer with Pardiso. I have not been able to find it.

Many thanks

Adam
Post Reply