Hi,
Ok, I had a closer look at your sif.
This is fully expected. You block one of the two partitions from going into the VTU output. However, the VTU output needs to be a parallel process since otherwise it cannot write the .PVTU file. It needs to communicate what partitions are active in order to write it. And it cannot communicate when the other partition never even starts the code. MPI commands always do some kind of syncronization and there everybody for which the communicator is active should participate.
What do you want to achieve with this?
-Peter
ElmerSolver_mpi stuck in a loop
Re: ElmerSolver_mpi stuck in a loop
I further simplified volt.F90 by shortening it to
SUBROUTINE AirElectrodePotential( Model,Solver,dt,TransientSimulation )
USE DefUtils
TYPE(Solver_t) :: Solver
TYPE(Model_t) :: Model
REAL(KIND=dp) :: dt
LOGICAL :: TransientSimulation
TYPE(Variable_t), POINTER :: vtufgVar
vtufgVar => VariableGet( Solver % Mesh % Variables, 'vtuflag' )
IF( ParEnv % MyPe == 0 )vtufgVar % Values(1) = 1.0
Print *, "My processor is", ParEnv % MyPe, " the vtu flag ", vtufgVar % Values(1)
END SUBROUTINE AirElectrodePotential
and case.sif to
Header
Mesh DB "." "rect"
End
Simulation
Coordinate System = "Cartesian 2D"
Coordinate Mapping(3) = 1 2 3
Simulation Type = Transient
TimeStepping Method = BDF
BDF Order = 1
Timestep Intervals(1) = 12
Timestep Size = 1
Output Intervals(1) = 100
Output File = "case.result"
Output File Cycle = 2 ! save results to files 0,1,0,1,.. (when 2)
Binary Output = Logical True ! ascii format is the default so select binary as it saves disk space
Output Global Variables = True
Max Output Level = 20
Max Output Partition = 2
End
Constants
Permittivity of Vacuum = 8.8542e-12
End
Body 1
Target Bodies(1) = 1
Name = "Air"
Equation = 1
Material = 1
Body Force = 1
Initial Condition = 1
End
Equation 1
Name = "Poisson"
Active Solvers(1) = 1
End
Material 1
Name = "Material 1"
Heat Conductivity = .026 ! W/m/K air
End
Initial Condition 1
vtuflag = Real -1.0
Potential = Real 0.0
End
Boundary Condition 1
Target Boundaries(1) = 1
Name = "FarField"
Potential = Real 0.0
End
Body Force 1
Name = "BodyForce 1"
Charge Density = 0.
End
Solver 1
Equation = "Poisson"
Variable = "Potential"
Variable DOFs = 1
Exported Variable 1 = -global vtuflag
Exec Solver = Always
Procedure = "volt" "AirElectrodePotential"
End
So only one solver is now involved. The results clearly show that
vtufgVar % Values(1) remains equal to -1 for ParEnv % MyPe == 1
which indicate there is a separate global variable for each processor.
Therefore one should never have conditions like
IF( ParEnv % MyPe /= 0 )....
for global variables. It is most likely that in
Solver 2
Exec Condition = Equals vtuflag
Equation = "result vtu"
Procedure = "ResultOutputSolve" "ResultOutputSolver"
Output File Name = "parav"
Vtu Format = Logical True
Single Precision = Logical True ! double precision is the default
Scalar Field 1 = String Potential
End
ResultOutputSolver is called for a given processor that is waiting for vtuflag to be 1 for all processors before saving its own assigned partition of field variable Potential.
Likewise for field variables: 3 days ago I showed I was intending to execute
DO i=1,Model % NumberOfNodes
j = fehdPerm(i)
IF( j == 0 ) CYCLE
Potential(j) = .......
END DO
[or, instead I could use something like
DO t=1,Solver % NumberOfActiveElements
! get element info
Element => GetActiveElement(t)
n = GetElementNOFNodes()
Potential(1:n) = .......
END DO
the latter concept being part of all Elmer solvers .]
Again no condition like
IF( ParEnv % MyPe /= 0 )....
should come into play. A pity though since all processors execute exactly the same code.
SUBROUTINE AirElectrodePotential( Model,Solver,dt,TransientSimulation )
USE DefUtils
TYPE(Solver_t) :: Solver
TYPE(Model_t) :: Model
REAL(KIND=dp) :: dt
LOGICAL :: TransientSimulation
TYPE(Variable_t), POINTER :: vtufgVar
vtufgVar => VariableGet( Solver % Mesh % Variables, 'vtuflag' )
IF( ParEnv % MyPe == 0 )vtufgVar % Values(1) = 1.0
Print *, "My processor is", ParEnv % MyPe, " the vtu flag ", vtufgVar % Values(1)
END SUBROUTINE AirElectrodePotential
and case.sif to
Header
Mesh DB "." "rect"
End
Simulation
Coordinate System = "Cartesian 2D"
Coordinate Mapping(3) = 1 2 3
Simulation Type = Transient
TimeStepping Method = BDF
BDF Order = 1
Timestep Intervals(1) = 12
Timestep Size = 1
Output Intervals(1) = 100
Output File = "case.result"
Output File Cycle = 2 ! save results to files 0,1,0,1,.. (when 2)
Binary Output = Logical True ! ascii format is the default so select binary as it saves disk space
Output Global Variables = True
Max Output Level = 20
Max Output Partition = 2
End
Constants
Permittivity of Vacuum = 8.8542e-12
End
Body 1
Target Bodies(1) = 1
Name = "Air"
Equation = 1
Material = 1
Body Force = 1
Initial Condition = 1
End
Equation 1
Name = "Poisson"
Active Solvers(1) = 1
End
Material 1
Name = "Material 1"
Heat Conductivity = .026 ! W/m/K air
End
Initial Condition 1
vtuflag = Real -1.0
Potential = Real 0.0
End
Boundary Condition 1
Target Boundaries(1) = 1
Name = "FarField"
Potential = Real 0.0
End
Body Force 1
Name = "BodyForce 1"
Charge Density = 0.
End
Solver 1
Equation = "Poisson"
Variable = "Potential"
Variable DOFs = 1
Exported Variable 1 = -global vtuflag
Exec Solver = Always
Procedure = "volt" "AirElectrodePotential"
End
So only one solver is now involved. The results clearly show that
vtufgVar % Values(1) remains equal to -1 for ParEnv % MyPe == 1
which indicate there is a separate global variable for each processor.
Therefore one should never have conditions like
IF( ParEnv % MyPe /= 0 )....
for global variables. It is most likely that in
Solver 2
Exec Condition = Equals vtuflag
Equation = "result vtu"
Procedure = "ResultOutputSolve" "ResultOutputSolver"
Output File Name = "parav"
Vtu Format = Logical True
Single Precision = Logical True ! double precision is the default
Scalar Field 1 = String Potential
End
ResultOutputSolver is called for a given processor that is waiting for vtuflag to be 1 for all processors before saving its own assigned partition of field variable Potential.
Likewise for field variables: 3 days ago I showed I was intending to execute
DO i=1,Model % NumberOfNodes
j = fehdPerm(i)
IF( j == 0 ) CYCLE
Potential(j) = .......
END DO
[or, instead I could use something like
DO t=1,Solver % NumberOfActiveElements
! get element info
Element => GetActiveElement(t)
n = GetElementNOFNodes()
Potential(1:n) = .......
END DO
the latter concept being part of all Elmer solvers .]
Again no condition like
IF( ParEnv % MyPe /= 0 )....
should come into play. A pity though since all processors execute exactly the same code.
Re: ElmerSolver_mpi stuck in a loop
Sorry Peter but I overlooked the second page in the posts to this thread and so I was not aware that you had replied, Anyhow my last post confirms your last post. Now you ask What is it I am trying to achieve ? Well, everything was OK in my solvers until I started adding a
IF( ParEnv % MyPe /= 0 )RETURN
condition. I only did this because I was trying to avoid duplicating code execution. But now I know I cannot use this condition for global variables. However, I take back what I said about field variables on my last post:
IF( ParEnv % MyPe == 0 )THEN
DO i=1,Model % NumberOfNodes
j = fehdPerm(i)
IF( j == 0 ) CYCLE
DO k=1,DIM
FEHD(DIM*(j-1)+k) = 0.0
END DO
END DO
ENDIF
Luckily, I can the use the condition as shown above because ResultOutputSolver does not freeze the program. Which is great since I wanted to avoid unnecessary duplicating of code execution.
Marc
IF( ParEnv % MyPe /= 0 )RETURN
condition. I only did this because I was trying to avoid duplicating code execution. But now I know I cannot use this condition for global variables. However, I take back what I said about field variables on my last post:
IF( ParEnv % MyPe == 0 )THEN
DO i=1,Model % NumberOfNodes
j = fehdPerm(i)
IF( j == 0 ) CYCLE
DO k=1,DIM
FEHD(DIM*(j-1)+k) = 0.0
END DO
END DO
ENDIF
Luckily, I can the use the condition as shown above because ResultOutputSolver does not freeze the program. Which is great since I wanted to avoid unnecessary duplicating of code execution.
Marc
-
- Site Admin
- Posts: 4823
- Joined: 22 Aug 2009, 11:57
- Antispam: Yes
- Location: Espoo, Finland
- Contact:
Re: ElmerSolver_mpi stuck in a loop
Hi Marc,
Yes, MPI tasks must have redundant copies of the "global" variables such as time. The reason is that it would be expensive to communicate it every time it is used, and that would add a lot of new code.
Instead if global variables must be the same then the user must take action. The simplest MPI commands (hidden) that you see are ParallelReduction() calls where you can have a variable min/max/sum communicated to each instance of it.
Here you did set a value for the flag but didn't communicate it to others so they could not perform as expected.
-Peter
Yes, MPI tasks must have redundant copies of the "global" variables such as time. The reason is that it would be expensive to communicate it every time it is used, and that would add a lot of new code.
Instead if global variables must be the same then the user must take action. The simplest MPI commands (hidden) that you see are ParallelReduction() calls where you can have a variable min/max/sum communicated to each instance of it.
Here you did set a value for the flag but didn't communicate it to others so they could not perform as expected.
-Peter
Re: ElmerSolver_mpi stuck in a loop
I\m a yoyo. I'm coming back to my second to last post: we cannot have
IF( ParEnv % MyPe == 0 )
field variable assignment code
ENDIF
i.e. all processors must execute exactly the same field variable assignment code which unfortunately wastes a lot of time. I proved this statement in the following easy way:
case.sif excerpt
Initial Condition 1
Potential = Real 3.3
End
volt.F90 excerpt
IF( ParEnv % MyPe == 0 )THEN
DO i=1,Model % NumberOfNodes
j = PotentialPerm(i)
IF( j == 0 ) CYCLE
Potential(j) = 7.7
END DO
ENDIF
DO i=1,Model % NumberOfNodes
j = PotentialPerm(i)
IF( j == 0 ) CYCLE
Print *, "My processor is", ParEnv % MyPe, " potential = ", Potential(j)
END DO
Guess what ? Potential(.) is now 7.7 for ParEnv % MyPe = 0 but it retains its initial value 3.3 for
ParEnv % MyPe =1 ! Which means each processor has its own distinct field variable space.
IF( ParEnv % MyPe == 0 )
field variable assignment code
ENDIF
i.e. all processors must execute exactly the same field variable assignment code which unfortunately wastes a lot of time. I proved this statement in the following easy way:
case.sif excerpt
Initial Condition 1
Potential = Real 3.3
End
volt.F90 excerpt
IF( ParEnv % MyPe == 0 )THEN
DO i=1,Model % NumberOfNodes
j = PotentialPerm(i)
IF( j == 0 ) CYCLE
Potential(j) = 7.7
END DO
ENDIF
DO i=1,Model % NumberOfNodes
j = PotentialPerm(i)
IF( j == 0 ) CYCLE
Print *, "My processor is", ParEnv % MyPe, " potential = ", Potential(j)
END DO
Guess what ? Potential(.) is now 7.7 for ParEnv % MyPe = 0 but it retains its initial value 3.3 for
ParEnv % MyPe =1 ! Which means each processor has its own distinct field variable space.
-
- Site Admin
- Posts: 4823
- Joined: 22 Aug 2009, 11:57
- Antispam: Yes
- Location: Espoo, Finland
- Contact:
Re: ElmerSolver_mpi stuck in a loop
Hi
The process that divides the mesh into independent sections is partitioning. It is the basic operational principle of MPI that each partition works with their own data. This is so called distributed memory model (as opposed to shared memory model).
-Peter
The process that divides the mesh into independent sections is partitioning. It is the basic operational principle of MPI that each partition works with their own data. This is so called distributed memory model (as opposed to shared memory model).
-Peter
Re: ElmerSolver_mpi stuck in a loop
In the same spirit, I found this on StackOverFlow:
QUOTE
MPI array syncronization
Asked 11 years, 5 months ago
Each of your two MPI tasks is working on a different copy of the array. You need to explicitly merge the two arrays using something like MPI_Send() and MPI_Recv() or one of the more complex MPI functions.
MPI is a distributed memory programming model, not shared memory like OpenMP or threads.
ENDQUOTE
This could have saved me a lot of useless testing, especially "Each of your two MPI tasks is working on a different copy of the array"
I'm thinking of writing some code along those lines:
IF( ParEnv % MyPe == 0 )THEN
DO i=1,Model % NumberOfNodes
j = PotentialPerm(i)
IF( j == 0 ) CYCLE
Potential(j) = whatever
MPI_Send() ! ? somehow send potential to all processors
END DO
ENDIF
to avoid duplicating the same task. Mind you, I am not sure if I will be wasting more time communicating with other processors than just letting them perform the same Potential(j) assignment.
So my next step is to get familiar with the openMPI routines (like for example MPI_Send() ) but I guess I have to look up their documentation as it is not part of the Elmer manuals.
QUOTE
MPI array syncronization
Asked 11 years, 5 months ago
Each of your two MPI tasks is working on a different copy of the array. You need to explicitly merge the two arrays using something like MPI_Send() and MPI_Recv() or one of the more complex MPI functions.
MPI is a distributed memory programming model, not shared memory like OpenMP or threads.
ENDQUOTE
This could have saved me a lot of useless testing, especially "Each of your two MPI tasks is working on a different copy of the array"
I'm thinking of writing some code along those lines:
IF( ParEnv % MyPe == 0 )THEN
DO i=1,Model % NumberOfNodes
j = PotentialPerm(i)
IF( j == 0 ) CYCLE
Potential(j) = whatever
MPI_Send() ! ? somehow send potential to all processors
END DO
ENDIF
to avoid duplicating the same task. Mind you, I am not sure if I will be wasting more time communicating with other processors than just letting them perform the same Potential(j) assignment.
So my next step is to get familiar with the openMPI routines (like for example MPI_Send() ) but I guess I have to look up their documentation as it is not part of the Elmer manuals.
-
- Site Admin
- Posts: 4823
- Joined: 22 Aug 2009, 11:57
- Antispam: Yes
- Location: Espoo, Finland
- Contact:
Re: ElmerSolver_mpi stuck in a loop
Hi
Here are some MPI intro videos from our colleagues at CSC:
https://www.youtube.com/watch?v=BPSgXQ9aUXY
-Peter
Here are some MPI intro videos from our colleagues at CSC:
https://www.youtube.com/watch?v=BPSgXQ9aUXY
-Peter