Compiling Elmer parallel version (Ubuntu 12.04)

From Elmer Wiki
Jump to: navigation, search

This wiki page is created specifically for compiling parallel version of Elmer to Ubuntu 12.04. It is intended for beginners, so the instructions are very detailled. The instructions have been tested with only two installations: use with caution.

The installation path is "/opt/elmer" and the sourcefiles are downloaded to "/opt/elmerfem".

Change directory to "/opt/":

 cd /opt 

Make directories called "elmer" and "elmerfem". The "/opt" directory requires superuser priviledges:

sudo su
mkdir elmer
mkdir elmerfem 

If your Ubuntu installation does not contain "subversion" package, install it with

apt-get install subversion

Now we can download the "elmerfem" source files

svn checkout http://svn.code.sf.net/p/elmerfem/code/trunk elmerfem

This will take a few minutes. Change directory to "/opt/elmerfem":

cd elmerfem

Open a new file called "compile" with some editor. For example gedit:

gedit compile

and write the following commands in it

#!/bin/sh -f

export CC="mpicc"
export CXX="mpicxx"
export FC="mpif90"
export F77="mpif90"

export ELMER_INSTALL="/opt/elmer/"


export CFLAGS="-O3 -march=x86-64  -ftree-vectorize -funroll-loops -ffast-math"
export CXXFLAGS="-O3 -march=x86-64  -ftree-vectorize -funroll-loops -ffast-math"
export FCFLAGS="-O3  -march=x86-64  -ftree-vectorize -funroll-loops -ffast-math"
export F90FLAGS="-O3  -march=x86-64  -ftree-vectorize -funroll-loops -ffast-math"
export F77FLAGS="-O3 -march=x86-64  -ftree-vectorize -funroll-loops -ffast-math"
export FFLAGS="-O3  -march=x86-64  -ftree-vectorize -funroll-loops -ffast-math"

export FCPPFLAGS="$FCPPFLAGS -DHAVE_MUMPS"
export LDFLAGS="-ldmumps -lmumps_common -lpord -lscalapack -lblacs -lmpi_f77 -lmpi -lHYPRE"

modules="matc umfpack mathlibs meshgen2d eio hutiter fem elmergrid"
par1="mathlibs"
par2="fem"

##### configure, build and install #########
for m in $modules; do
  echo "module $m"
  echo "###############"

  cd $m 
  make distclean

  if [ $m = $par1 ] || [ $m = $par2 ]; then
    echo "This module needs special attention due to the parallelisation."
    ##### parallel #######
    ./configure --prefix=$ELMER_INSTALL --with-mpi-dir="/usr/lib/openmpi/" --with-64bits=yes --with-hypre="-lHYPRE"
  else
    ##### no special treatment #######
    ./configure --prefix=$ELMER_INSTALL --with-64bits=yes
  fi

make install && cd ..

done


Now we will have to make sure that the correct libraries are installed in the Ubuntu installation. The required libraries are found in the "LDFLAGS" variable. For example -ldmumps that refers to a file called "libdmumps.so".

You can try to locate it in your machine by writing

locate libdmumps

propably at this point it will not be found. Let us install the needed libraries and programs

apt-get install gfortran libopenmpi1.3 libopenmpi-dev libmumps-4.9.2 libmumps-dev libhypre-2.4.0 libhypre-dev g++

Now if we try to locate "libdmumps" (before the search, we need to update the database)

updatedb
locate dmumps
/usr/lib/libdmumps-4.9.2.so

We see that it is "libdmumps-4.9.2.so". We can make a symbolic link that is called "libdmumps.so". If you already have a "libdmumps.so", don't do the following:

ln -s /usr/lib/libdmumps-4.9.2.so /usr/lib/libdmumps.so

Similarly you should find every library mentioned in the script. For example in this case, in addition to the previous, the following was necessary

ln -s /usr/lib/libmumps_common-4.9.2.so /usr/lib/libmumps_common.so
ln -s /usr/lib/libpord-4.9.2.so /usr/lib/libpord.so
ln -s /usr/lib/libscalapack-openmpi.so.1 /usr/lib/libscalapack.so
ln -s /usr/lib/libblacs-openmpi.so.1 /usr/lib/libblacs.so
ln -s /usr/lib/libHYPRE-2.4.0.so /usr/lib/libHYPRE.so

Now we are ready to run the compilation. However, the compile script file needs to be give priviledges to run.

chmod u+x compile
./compile

Now Parallel Elmerfem should be in "/opt/elmer" and we can exit from the root terminal.

exit

Note: If you have several Elmers installed and you prefer to use the one installed in "/opt/elmer", you can edit your $PATH environment variable.

To make the change permanent: open your ".bashrc" file:

cd
gedit .bashrc

And write at the end of the file:

PATH=/opt/elmer/bin:$PATH

Now if you restart your terminal, you will be able to run the Elmer you have in /opt/elmer.