Parallel run problem

[bbosak]

Hi,

I have tried to test my parallel compilation of Elmer but I've found the following problem when I wanted run heateq-par:

mpiexec -np 4 /opt/app/fem/elmer/bin/ElmerSolver_mpi
ELMER SOLVER (v 6.0) STARTED AT: 2010/08/23 10:46:36
ELMER SOLVER (v 6.0) STARTED AT: 2010/08/23 10:46:36
ELMER SOLVER (v 6.0) STARTED AT: 2010/08/23 10:46:36
ELMER SOLVER (v 6.0) STARTED AT: 2010/08/23 10:46:36
ParCommInit: Initialize #PEs: 4
MAIN:
MAIN: ==========================================
MAIN: E L M E R S O L V E R S T A R T I N G
MAIN: Library version: 6.0 (Rev: 4579)
MAIN: Running in parallel using 4 tasks.
MAIN: ==========================================
MAIN:
MAIN:
MAIN: -----------------------
MAIN: Reading Model ...
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
ERROR:: GetMatrixFormat: Only MUMPS direct solver implemented in parallel!
MAIN: Done
MAIN: -----------------------
ERROR:: GetMatrixFormat: Only MUMPS direct solver implemented in parallel!
ERROR:: GetMatrixFormat: Only MUMPS direct solver implemented in parallel!
ERROR:: GetMatrixFormat: Only MUMPS direct solver implemented in parallel!
--------------------------------------------------------------------------
mpiexec has exited due to process rank 1 with PID 6520 on
node xxx.xxx.xxx.xxx exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpiexec (as reported here).

I really don't know why this occurs: I think that I have done everything correctly (accordingly to the available scripts).

Thanks for any help.

Cheers,
Bartek

[raback]

Hi, It seems you've chosen a "direct" solver but of them only MUMPS works in parallel (if compiled). Try rather some "iterative" linear solvers. -Peter

[bbosak]

Thanks for your fast reply.

Stupid question: How can I set and use the "iterative" linear solvers?

I am using the following installation script:

Code: Select all

#!/bin/sh -f                                                                                                                                                                                                                                                    
export CC="mpicc"                                                                                                                                                                                                                                               
export CXX="mpic++"                                                                                                                                                                                                                                             
export FC="mpif90"                                                                                                                                                                                                                                              
export F77="mpif90"                                                                                                                                                                                                                                             
                                                                                                                                                                                                                                                                
export ELMER_INSTALL="/opt/app/fem/elmer"                                                                                                                                                                                                                       
export ELMER_HOME="/usr/app/fem/elmer"                                                                                                                                                                                                                          
export MPI_HOME="/usr/local/mpi/openmpi/1.4.2/gcc/4.4"                                                                                                                                                                                                          
export PATH="/usr/local/mpi/openmpi/1.4.2/gcc/4.4/bin:$PATH"                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                
modules="matc umfpack mathlibs meshgen2d eio hutiter fem elmergrid"                                                                                                                                                                                             
                                                                                                                                                                                                                                                                
##### configure, build and install #########                                                                                                                                                                                                                    
 for m in $modules; do                                                                                                                                                                                                                                          
   echo "module $m"                                                                                                                                                                                                                                             
   echo "###############"                                                                                                                                                                                                                                       
   ##### parallel #######                                                                                                                                                                                                                                       
  cd $m ; make distclean; ./configure --enable-debug --prefix=$ELMER_INSTALL --with-mpi=yes --with-mpi-dir=$MPI_HOME  && make install && cd ..                                                                                                                  
done

[raback]

It's set in the command file of ElmerSolver i.e. the *.sif file.

Code: Select all

Linear System Solver = iterative 

[bbosak]

Thanks a lot! It works now;)

Best Regards,
Bartek

[udeng85]

Hi Peter,

I have the same problem with Bartek. Does this mean that I can't use direct implementation of MUMPS with Elmer? I tried running Elmer example, fsi_beam, and the following is the error produced:

Code: Select all

ELMER SOLVER (v 6.0) STARTED AT: 2010/11/09 10:05:50
ELMER SOLVER (v 6.0) STARTED AT: 2010/11/09 10:05:50
ELMER SOLVER (v 6.0) STARTED AT: 2010/11/09 10:05:50
ELMER SOLVER (v 6.0) STARTED AT: 2010/11/09 10:05:50
ELMER SOLVER (v 6.0) STARTED AT: 2010/11/09 10:05:50
ELMER SOLVER (v 6.0) STARTED AT: 2010/11/09 10:05:50
ELMER SOLVER (v 6.0) STARTED AT: 2010/11/09 10:05:50
ELMER SOLVER (v 6.0) STARTED AT: 2010/11/09 10:05:50
ParCommInit:  Initialize #PEs:            8
MAIN:
MAIN: ==========================================
MAIN:  E L M E R  S O L V E R  S T A R T I N G
MAIN:  Library version: 6.0 (Rev: exported)
MAIN:  Running in parallel using 8 tasks.
MAIN:  HYPRE library linked in.
MAIN:  MUMPS library linked in.
MAIN: ==========================================
MAIN:
MAIN:
MAIN: -----------------------
MAIN: Reading Model ...
TEST CASE 1
Loading user function library: [FsiStuff]...[YOUNGS]
Loading user function library: [FsiStuff]...[INFLOW]
TEST CASE 1
Loading user function library: [ElasticSolve]...[ElasticSolver_Init0]
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
MAIN: Done
MAIN: -----------------------
TEST CASE 1
TEST CASE 1
Loading user function library: [ElasticSolve]...[ElasticSolver_Init]
ERROR:: GetMatrixFormat: Only MUMPS direct solver implemented in parallel!

Thank you.

-Aizat-

[raback]

Well at least you have to define

Code: Select all

Linear System Direct Method = mumps

since that's the only direct solver available in parallel, if you wan't to use direct solvers...