Page 1 of 1
Reload Existing Simulation Results with MPI
Posted: 26 Sep 2018, 14:16
by Plo
Hello,
I am trying to reload existing simulation results. While I run the task in serial, everything is great. Problem occurs when I try to run it in parallel (MPI on Windows 7) It seems there is a problem with reading the restart file.
Here is a piece of my sif file:
Code: Select all
Solver 3
Exec Solver = Before Timestep
Equation = "Reload Data"
Procedure = "ReloadData" "ReloadSolution"
Reload Solution File = "MyFile.result"
End
And error:
Code: Select all
SingleSolver: Attempting to call solver
SingleSolver: Solver Equation string is: reload data
ERROR:: LoadRestartFile: =======================================
ERROR:: LoadRestartFile: =======================================
ERROR:: LoadRestartFile:
ERROR:: LoadRestartFile:
ERROR:: LoadRestartFile: =======================================
ERROR:: LoadRestartFile: Could not open file "././MyFile.result"
ERROR:: LoadRestartFile: Could not open file "././MyFile.result"
ERROR:: LoadRestartFile:
ERROR:: LoadRestartFile: No restart possible!
ERROR:: LoadRestartFile: No restart possible!
ERROR:: LoadRestartFile: Could not open file "././MyFile.result"
ERROR:: LoadRestartFile:
ERROR:: LoadRestartFile:
ERROR:: LoadRestartFile: No restart possible!
ERROR:: LoadRestartFile: =======================================
ERROR:: LoadRestartFile: =======================================
ERROR:: LoadRestartFile:
ERROR:: LoadRestartFile: =======================================
ReloadSolution: Continuous reading disabled
ReloadSolution: Reading old solution from file MyFile.result
ReloadSolution: Opening file: ./MyFile.result
ReloadSolution: Loading Timestep: 1
LoadRestartFile:
LoadRestartFile: --------------------------------------------
LoadRestartFile: Reading data from file: ./MyFile.result
ERROR:: LoadRestartFile: =======================================
ERROR:: LoadRestartFile:
ERROR:: LoadRestartFile: Could not open file "././MyFile.result"
ERROR:: LoadRestartFile: No restart possible!
ERROR:: LoadRestartFile:
ERROR:: LoadRestartFile: =======================================
I am not sure, if I am doing something wrong, or it is a bug. Or are there any restriction in reloading simulation results in MPI run?
I am using Elmer 8.2.
- Plo
Re: Reload Existing Simulation Results with MPI
Posted: 26 Sep 2018, 14:32
by mzenker
Hi,
it seems that ElmerSolver does not find your results file.
Trivial question: are you sure that your directory structure is the same in both serial and parallel cases, and that the results file is really there?
I have no experience in parallel execution of Elmer, but I remember vaguely that it creates directories to hold the results for the single runs. Maybe it is due to that?
HTH,
Matthias
Re: Reload Existing Simulation Results with MPI
Posted: 26 Sep 2018, 15:24
by Plo
Hi,
I created result file using
Output File = MyFile.result in
Simulation block in previous Elmer run. In the case of the serial run the file
MyFile.result is created in the working directory. In the case of parallel run, four files
MyFile.result.0, ……,
MyFile.result.4 are created. In both cases, the files are not empty. Really contains required data.
Both cases I run from the same directory, same sif file. Only one thing has changed – the Elmer execution command (
ElmerSolver case.sif vs.
....mpirun -np 4 ElmerSolver_mpi....).
I have tried to solve this for two weeks. No success yet
-Plo
Re: Reload Existing Simulation Results with MPI
Posted: 26 Sep 2018, 15:42
by mzenker
Hmmm... maybe the four results files have to be merged in order to create a single restart file?
I think someone with more knowledge/experience in parallel running has to take over...
Matthias
Re: Reload Existing Simulation Results with MPI
Posted: 02 Oct 2018, 02:52
by Plo
Hi,
Thank you for your advice. I have written python script to merge files into one single one. This time the result file is found and header is read, but another error appears.
Code: Select all
……
LoadRestartFile: Total number of dofs to load: 12
LoadRestartFile: Reading time sequence: 1.000E+00
LoadRestartFile: Reading timestep: 1
At line 3479 of file c:/ElmerBuild/src/elmer/elmerfem/fem/src/ModelDescription.F90 (unit = 30, file = '././myfile_merge.result')
Fortran runtime error: Bad integer for item 1 in list input
Thank you. I will continue with finding solution
Re: Reload Existing Simulation Results with MPI
Posted: 02 Oct 2018, 10:36
by raback
Hi
About the restart.
Restart should be possible both in serial and in parallel. The historical limitation has been that you need to start from the same mesh. The list of variables (and solvers) may be different.
Only quite recently (June this year) the restart was augmented such that we may have mesh-specific restart and employing mesh2mesh interpolation can effectively perform restart from a nonconfoming mesh as well. This even works in parallel.
There are many test cases that use the restart, for example diffuser* and NaturalConvectionRestart. These are not parallel, but just tested the latter one also in parallel. For the nonconforming restart there are many test cases named NonconformingRestart*, one of them also in parallel.
As you are using quite on old version your only option is to use exactly the same mesh and partitioning when doing restart. Perhaps this was not the case?
-Peter
Re: Reload Existing Simulation Results with MPI
Posted: 05 Oct 2018, 13:21
by Plo
Hi Peter,
Thank you for your reply. I have the same mesh for both runs.
Originally I was trying to reload data using subroutine “ReloadSolutin” in module “ReloadData” (according Elmer Models Manual, page 251). In this case it was successful for serial run only.
I tried to use “Restart File” keyword in the Simulation block instead of ReloadSolition module. And your right! IT WORKS in parallel as well
Plo
Re: Reload Existing Simulation Results with MPI
Posted: 05 Oct 2018, 23:18
by raback
Hi Plo
I think the "ReloadSolution" solver idea was that you can load a precomputed solution for each timestep.
-Peter