Reload Existing Simulation Results with MPI

Clearly defined bug reports and their fixes
Post Reply
Plo
Posts: 4
Joined: 26 Sep 2018, 14:06
Antispam: Yes

Reload Existing Simulation Results with MPI

Post by Plo »

Hello,
I am trying to reload existing simulation results. While I run the task in serial, everything is great. Problem occurs when I try to run it in parallel (MPI on Windows 7) It seems there is a problem with reading the restart file.

Here is a piece of my sif file:

Code: Select all

Solver 3
  Exec Solver = Before Timestep
  Equation =  "Reload Data"
  Procedure = "ReloadData" "ReloadSolution"
  Reload Solution File =  "MyFile.result"
End
And error:

Code: Select all

SingleSolver: Attempting to call solver
SingleSolver: Solver Equation string is: reload data
ERROR:: LoadRestartFile: =======================================
ERROR:: LoadRestartFile: =======================================
ERROR:: LoadRestartFile: 
ERROR:: LoadRestartFile: 
ERROR:: LoadRestartFile: =======================================
ERROR:: LoadRestartFile: Could not open file "././MyFile.result"
ERROR:: LoadRestartFile: Could not open file "././MyFile.result"
ERROR:: LoadRestartFile: 
ERROR:: LoadRestartFile: No restart possible!
ERROR:: LoadRestartFile: No restart possible!
ERROR:: LoadRestartFile: Could not open file "././MyFile.result"
ERROR:: LoadRestartFile: 
ERROR:: LoadRestartFile: 
ERROR:: LoadRestartFile: No restart possible!
ERROR:: LoadRestartFile: =======================================
ERROR:: LoadRestartFile: =======================================
ERROR:: LoadRestartFile: 
ERROR:: LoadRestartFile: =======================================
ReloadSolution: Continuous reading disabled
ReloadSolution:  Reading old solution from file MyFile.result
ReloadSolution:  Opening file: ./MyFile.result
ReloadSolution:  Loading Timestep:            1
LoadRestartFile: 
LoadRestartFile: --------------------------------------------
LoadRestartFile: Reading data from file: ./MyFile.result
ERROR:: LoadRestartFile: =======================================
ERROR:: LoadRestartFile: 
ERROR:: LoadRestartFile: Could not open file "././MyFile.result"
ERROR:: LoadRestartFile: No restart possible!
ERROR:: LoadRestartFile: 
ERROR:: LoadRestartFile: =======================================
I am not sure, if I am doing something wrong, or it is a bug. Or are there any restriction in reloading simulation results in MPI run?
I am using Elmer 8.2.

- Plo
mzenker
Posts: 1999
Joined: 07 Dec 2009, 11:49
Location: Germany

Re: Reload Existing Simulation Results with MPI

Post by mzenker »

Hi,

it seems that ElmerSolver does not find your results file.
Trivial question: are you sure that your directory structure is the same in both serial and parallel cases, and that the results file is really there?
I have no experience in parallel execution of Elmer, but I remember vaguely that it creates directories to hold the results for the single runs. Maybe it is due to that?

HTH,
Matthias
Plo
Posts: 4
Joined: 26 Sep 2018, 14:06
Antispam: Yes

Re: Reload Existing Simulation Results with MPI

Post by Plo »

Hi,
I created result file using Output File = MyFile.result in Simulation block in previous Elmer run. In the case of the serial run the file MyFile.result is created in the working directory. In the case of parallel run, four files MyFile.result.0, ……, MyFile.result.4 are created. In both cases, the files are not empty. Really contains required data.
Both cases I run from the same directory, same sif file. Only one thing has changed – the Elmer execution command (ElmerSolver case.sif vs. ....mpirun -np 4 ElmerSolver_mpi....).
I have tried to solve this for two weeks. No success yet :(

-Plo
mzenker
Posts: 1999
Joined: 07 Dec 2009, 11:49
Location: Germany

Re: Reload Existing Simulation Results with MPI

Post by mzenker »

Hmmm... maybe the four results files have to be merged in order to create a single restart file?

I think someone with more knowledge/experience in parallel running has to take over...

Matthias
Plo
Posts: 4
Joined: 26 Sep 2018, 14:06
Antispam: Yes

Re: Reload Existing Simulation Results with MPI

Post by Plo »

Hi,
Thank you for your advice. I have written python script to merge files into one single one. This time the result file is found and header is read, but another error appears.

Code: Select all

……
LoadRestartFile: Total number of dofs to load: 12
LoadRestartFile: Reading time sequence:    1.000E+00
LoadRestartFile: Reading timestep: 1
At line 3479 of file c:/ElmerBuild/src/elmer/elmerfem/fem/src/ModelDescription.F90 (unit = 30, file = '././myfile_merge.result')
Fortran runtime error: Bad integer for item 1 in list input
Thank you. I will continue with finding solution
raback
Site Admin
Posts: 4812
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Reload Existing Simulation Results with MPI

Post by raback »

Hi

About the restart.

Restart should be possible both in serial and in parallel. The historical limitation has been that you need to start from the same mesh. The list of variables (and solvers) may be different.

Only quite recently (June this year) the restart was augmented such that we may have mesh-specific restart and employing mesh2mesh interpolation can effectively perform restart from a nonconfoming mesh as well. This even works in parallel.

There are many test cases that use the restart, for example diffuser* and NaturalConvectionRestart. These are not parallel, but just tested the latter one also in parallel. For the nonconforming restart there are many test cases named NonconformingRestart*, one of them also in parallel.

As you are using quite on old version your only option is to use exactly the same mesh and partitioning when doing restart. Perhaps this was not the case?

-Peter
Plo
Posts: 4
Joined: 26 Sep 2018, 14:06
Antispam: Yes

Re: Reload Existing Simulation Results with MPI

Post by Plo »

Hi Peter,
Thank you for your reply. I have the same mesh for both runs.
Originally I was trying to reload data using subroutine “ReloadSolutin” in module “ReloadData” (according Elmer Models Manual, page 251). In this case it was successful for serial run only.
I tried to use “Restart File” keyword in the Simulation block instead of ReloadSolition module. And your right! IT WORKS in parallel as well :D

Plo
raback
Site Admin
Posts: 4812
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Reload Existing Simulation Results with MPI

Post by raback »

Hi Plo

I think the "ReloadSolution" solver idea was that you can load a precomputed solution for each timestep.

-Peter
Post Reply