Mulling over mpi and per element calculations

General discussion about Elmer
Post Reply
spacedout
Posts: 177
Joined: 30 Mar 2020, 23:27
Antispam: Yes

Mulling over mpi and per element calculations

Post by spacedout »

Hi everybody

Since I now understand that all processors keep a separate copy of the data space as well, I realize they will duplicate assignments on field variables in all existing Elmer solvers in the manner:

DO t=1,Solver % NumberOfActiveElements
! get element info
Element => GetActiveElement(t)
n = GetElementNOFNodes()

Potential(1:n) = .......

END DO

which is a waste of time let alone the fact that numerical approximations to a PDE are computed on a nodal basis and that a node belonging to an interior edge will result in Potential(i) for the same ith node being assigned exactly twice in the above do loop.

Can anybody cheer me up on this disappointing note ?

Marc
kevinarden
Posts: 2237
Joined: 25 Jan 2019, 01:28
Antispam: Yes

Re: Mulling over mpi and per element calculations

Post by kevinarden »

It is open source. You can download the whole code, change it as you desire. Compile it and run it. If you like the results you can propose it back to the Elmer team as improvements. This is not possible in commercial codes. That alone cheers me up.
raback
Site Admin
Posts: 4812
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Mulling over mpi and per element calculations

Post by raback »

Hi

The fact that nodes are shared over the interface is not different in Elmer compared to any other FEM code using MPI. What would be the alternative? To always fetch data from elsewhere when you need it? If the code is closed source you just cannot verify what they do but I would bet that all MPI codes do exactly likewise. This makes it possible to do the finite element assembly without any communication. And it is communication that kills the performance. The idea is to have so large partitions that the fraction of shared nodes is small compared to the not shared nodes, say 10-50 thousand elements is usually enough.

-Peter
spacedout
Posts: 177
Joined: 30 Mar 2020, 23:27
Antispam: Yes

Re: Mulling over mpi and per element calculations

Post by spacedout »

I didn't me to offend anybody. I was just hoping to get a confirmation of my suspicion that the major computational cost in ElmerSolver is by far the solution of

Ax = b

which itself can be iterated several times in the non linear case. The mere assignment of source variables is insignificant by comparison. Thus, ElmerSolver_mpi concentrates on communicating information across partitions interfaces but for the solution of linear equations only. And this is hidden from the user unless he wants to delve deeper into the code rather than just perusing through the individual solvers. It won't matter much if source variable assignments are duplicated for every processor.

So, no hard feelings towards Elmer on my part.
Post Reply