Parallel Operation

High Performance Computing with Elmer
Post Reply
willsprocket
Posts: 33
Joined: 18 Jun 2022, 05:32
Antispam: Yes

Parallel Operation

Post by willsprocket »

I have a Windows PC with dual Xeons (each with 14 cores/28 logical cores). It seems if I specify more than 16 processors, ElmerSolver fails. Is there another setting I need to change?
Last edited by willsprocket on 28 Jun 2022, 16:36, edited 1 time in total.
kevinarden
Posts: 2237
Joined: 25 Jan 2019, 01:28
Antispam: Yes

Re: Parallel Operation

Post by kevinarden »

By fail what is the specific reason reported?
raback
Site Admin
Posts: 4812
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Parallel Operation

Post by raback »

Hi,

Threading is supported partly. The main method for parallel performance is mpi. There are Elmer packages with mpi also for Windows.

OpenMP only works with shared memory and therefore is limited in scalabillity.

-Peter
kevinarden
Posts: 2237
Joined: 25 Jan 2019, 01:28
Antispam: Yes

Re: Parallel Operation

Post by kevinarden »

Cores and threads are 2 different things in hardware and programing. However if you have 2 cpu with 14 core each then you should have 28 cores or logical processors that MPI can use. To verify this go to task manager, performance tab, got to the bottom and it should say how cores and how many logical processors are available.
willsprocket
Posts: 33
Joined: 18 Jun 2022, 05:32
Antispam: Yes

Re: Parallel Operation

Post by willsprocket »

For steady-state
The errors start after 23 cores. It gives an error in allocating 24 bytes for a matrix in a piece of Fortran code.

For transient
The errors start after 20 cores. Error: AllocateElementVector: Unable to Allocate structured 699624 element array

The PC has 28 cores and 56 logical processors.
kevinarden
Posts: 2237
Joined: 25 Jan 2019, 01:28
Antispam: Yes

Re: Parallel Operation

Post by kevinarden »

Sounds like RAM allocation issues. If it is one machine with shared memory than each processor has to have so much RAM to use. I believe parallel increases the memory demand because each core has a certain amount basic usage. How much RAM does the machine have? If you use the task manager, performance tab, there is a plot available to monitor memory usage.
Dmitrij
Posts: 4
Joined: 16 Jul 2019, 01:41
Antispam: Yes

Re: Parallel Operation

Post by Dmitrij »

Hi,
I also encountered parallel calculation problem by using MS MPI when threads number exceed a certain value. In example project AcousticWaves it is 12 threads, below this value Elmer calculates normally, but above this value MS MPI/Elmer gives error. While the same project with Intel MPI runs without errors at any threads number, but runtime is considerably slower that that in the case of MS MPI. See attached logs.


Hardware Cisco server with 4 socket CPU 4xE7-8880v3 with 512GB RAM(OS Win Server 2019)
Attachments
microsoft_mpi_64cores_log_error.txt
gives error
(32 KiB) Downloaded 154 times
microsoft_mpi_10cores_log.txt
works fine
(11.49 KiB) Downloaded 165 times
intel_mpi_10cores_log.txt
works, but slower than MS MPI
(47.59 KiB) Downloaded 158 times
Dmitrij
Posts: 4
Joined: 16 Jul 2019, 01:41
Antispam: Yes

Re: Parallel Operation

Post by Dmitrij »

intel MPI 64 thread log
Attachments
intel_mpi_64c_log.txt
(266.56 KiB) Downloaded 147 times
kevinarden
Posts: 2237
Joined: 25 Jan 2019, 01:28
Antispam: Yes

Re: Parallel Operation

Post by kevinarden »

There is a difference between cores and threads. We seem to be using the terms interchangeable here. I believe Elmer is programmed for cores, not threads.

https://www.temok.com/blog/cores-vs-threads/
Dmitrij
Posts: 4
Joined: 16 Jul 2019, 01:41
Antispam: Yes

Re: Parallel Operation

Post by Dmitrij »

Yes, I mean threads, not cores. The system have 64 active physical cores (HT off). Number of threads in Elmer was changed only.
Post Reply