ElmerSolver_mpi stuck in a loop

General discussion about Elmer
raback
Site Admin
Posts: 4173
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: ElmerSolver_mpi stuck in a loop

Post by raback »

Hi,

Ok, I had a closer look at your sif.

This is fully expected. You block one of the two partitions from going into the VTU output. However, the VTU output needs to be a parallel process since otherwise it cannot write the .PVTU file. It needs to communicate what partitions are active in order to write it. And it cannot communicate when the other partition never even starts the code. MPI commands always do some kind of syncronization and there everybody for which the communicator is active should participate.

What do you want to achieve with this?

-Peter
spacedout
Posts: 103
Joined: 30 Mar 2020, 23:27
Antispam: Yes

Re: ElmerSolver_mpi stuck in a loop

Post by spacedout »

I further simplified volt.F90 by shortening it to

SUBROUTINE AirElectrodePotential( Model,Solver,dt,TransientSimulation )
USE DefUtils

TYPE(Solver_t) :: Solver
TYPE(Model_t) :: Model

REAL(KIND=dp) :: dt
LOGICAL :: TransientSimulation

TYPE(Variable_t), POINTER :: vtufgVar

vtufgVar => VariableGet( Solver % Mesh % Variables, 'vtuflag' )

IF( ParEnv % MyPe == 0 )vtufgVar % Values(1) = 1.0

Print *, "My processor is", ParEnv % MyPe, " the vtu flag ", vtufgVar % Values(1)

END SUBROUTINE AirElectrodePotential

and case.sif to

Header
Mesh DB "." "rect"
End

Simulation
Coordinate System = "Cartesian 2D"
Coordinate Mapping(3) = 1 2 3
Simulation Type = Transient
TimeStepping Method = BDF
BDF Order = 1
Timestep Intervals(1) = 12
Timestep Size = 1
Output Intervals(1) = 100
Output File = "case.result"
Output File Cycle = 2 ! save results to files 0,1,0,1,.. (when 2)

Binary Output = Logical True ! ascii format is the default so select binary as it saves disk space

Output Global Variables = True

Max Output Level = 20
Max Output Partition = 2

End

Constants
Permittivity of Vacuum = 8.8542e-12
End

Body 1
Target Bodies(1) = 1
Name = "Air"
Equation = 1
Material = 1
Body Force = 1
Initial Condition = 1
End

Equation 1
Name = "Poisson"
Active Solvers(1) = 1
End

Material 1
Name = "Material 1"

Heat Conductivity = .026 ! W/m/K air

End

Initial Condition 1
vtuflag = Real -1.0
Potential = Real 0.0
End


Boundary Condition 1
Target Boundaries(1) = 1
Name = "FarField"
Potential = Real 0.0
End


Body Force 1
Name = "BodyForce 1"
Charge Density = 0.
End


Solver 1
Equation = "Poisson"
Variable = "Potential"
Variable DOFs = 1

Exported Variable 1 = -global vtuflag

Exec Solver = Always
Procedure = "volt" "AirElectrodePotential"
End


So only one solver is now involved. The results clearly show that

vtufgVar % Values(1) remains equal to -1 for ParEnv % MyPe == 1

which indicate there is a separate global variable for each processor.

Therefore one should never have conditions like

IF( ParEnv % MyPe /= 0 )....

for global variables. It is most likely that in

Solver 2
Exec Condition = Equals vtuflag
Equation = "result vtu"
Procedure = "ResultOutputSolve" "ResultOutputSolver"
Output File Name = "parav"
Vtu Format = Logical True
Single Precision = Logical True ! double precision is the default
Scalar Field 1 = String Potential
End

ResultOutputSolver is called for a given processor that is waiting for vtuflag to be 1 for all processors before saving its own assigned partition of field variable Potential.

Likewise for field variables: 3 days ago I showed I was intending to execute

DO i=1,Model % NumberOfNodes
j = fehdPerm(i)
IF( j == 0 ) CYCLE
Potential(j) = .......
END DO

[or, instead I could use something like

DO t=1,Solver % NumberOfActiveElements
! get element info
Element => GetActiveElement(t)
n = GetElementNOFNodes()

Potential(1:n) = .......

END DO

the latter concept being part of all Elmer solvers .]

Again no condition like

IF( ParEnv % MyPe /= 0 )....

should come into play. A pity though since all processors execute exactly the same code.
spacedout
Posts: 103
Joined: 30 Mar 2020, 23:27
Antispam: Yes

Re: ElmerSolver_mpi stuck in a loop

Post by spacedout »

Sorry Peter but I overlooked the second page in the posts to this thread and so I was not aware that you had replied, Anyhow my last post confirms your last post. Now you ask What is it I am trying to achieve ? Well, everything was OK in my solvers until I started adding a

IF( ParEnv % MyPe /= 0 )RETURN

condition. I only did this because I was trying to avoid duplicating code execution. But now I know I cannot use this condition for global variables. However, I take back what I said about field variables on my last post:

IF( ParEnv % MyPe == 0 )THEN
DO i=1,Model % NumberOfNodes
j = fehdPerm(i)
IF( j == 0 ) CYCLE
DO k=1,DIM
FEHD(DIM*(j-1)+k) = 0.0
END DO
END DO
ENDIF

Luckily, I can the use the condition as shown above because ResultOutputSolver does not freeze the program. Which is great since I wanted to avoid unnecessary duplicating of code execution.

Marc
raback
Site Admin
Posts: 4173
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: ElmerSolver_mpi stuck in a loop

Post by raback »

Hi Marc,

Yes, MPI tasks must have redundant copies of the "global" variables such as time. The reason is that it would be expensive to communicate it every time it is used, and that would add a lot of new code.

Instead if global variables must be the same then the user must take action. The simplest MPI commands (hidden) that you see are ParallelReduction() calls where you can have a variable min/max/sum communicated to each instance of it.

Here you did set a value for the flag but didn't communicate it to others so they could not perform as expected.

-Peter
spacedout
Posts: 103
Joined: 30 Mar 2020, 23:27
Antispam: Yes

Re: ElmerSolver_mpi stuck in a loop

Post by spacedout »

I\m a yoyo. I'm coming back to my second to last post: we cannot have

IF( ParEnv % MyPe == 0 )
field variable assignment code
ENDIF

i.e. all processors must execute exactly the same field variable assignment code which unfortunately wastes a lot of time. I proved this statement in the following easy way:

case.sif excerpt

Initial Condition 1

Potential = Real 3.3

End

volt.F90 excerpt

IF( ParEnv % MyPe == 0 )THEN
DO i=1,Model % NumberOfNodes
j = PotentialPerm(i)
IF( j == 0 ) CYCLE
Potential(j) = 7.7
END DO
ENDIF

DO i=1,Model % NumberOfNodes
j = PotentialPerm(i)
IF( j == 0 ) CYCLE
Print *, "My processor is", ParEnv % MyPe, " potential = ", Potential(j)
END DO


Guess what ? Potential(.) is now 7.7 for ParEnv % MyPe = 0 but it retains its initial value 3.3 for
ParEnv % MyPe =1 ! Which means each processor has its own distinct field variable space.
raback
Site Admin
Posts: 4173
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: ElmerSolver_mpi stuck in a loop

Post by raback »

Hi

The process that divides the mesh into independent sections is partitioning. It is the basic operational principle of MPI that each partition works with their own data. This is so called distributed memory model (as opposed to shared memory model).

-Peter
spacedout
Posts: 103
Joined: 30 Mar 2020, 23:27
Antispam: Yes

Re: ElmerSolver_mpi stuck in a loop

Post by spacedout »

In the same spirit, I found this on StackOverFlow:

QUOTE

MPI array syncronization

Asked 11 years, 5 months ago

Each of your two MPI tasks is working on a different copy of the array. You need to explicitly merge the two arrays using something like MPI_Send() and MPI_Recv() or one of the more complex MPI functions.

MPI is a distributed memory programming model, not shared memory like OpenMP or threads.

ENDQUOTE

This could have saved me a lot of useless testing, especially "Each of your two MPI tasks is working on a different copy of the array"

I'm thinking of writing some code along those lines:

IF( ParEnv % MyPe == 0 )THEN
DO i=1,Model % NumberOfNodes
j = PotentialPerm(i)
IF( j == 0 ) CYCLE
Potential(j) = whatever
MPI_Send() ! ? somehow send potential to all processors
END DO
ENDIF

to avoid duplicating the same task. Mind you, I am not sure if I will be wasting more time communicating with other processors than just letting them perform the same Potential(j) assignment.

So my next step is to get familiar with the openMPI routines (like for example MPI_Send() ) but I guess I have to look up their documentation as it is not part of the Elmer manuals.
raback
Site Admin
Posts: 4173
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: ElmerSolver_mpi stuck in a loop

Post by raback »

Hi

Here are some MPI intro videos from our colleagues at CSC:
https://www.youtube.com/watch?v=BPSgXQ9aUXY

-Peter
Post Reply