SIGABRT exception

Clearly defined bug reports and their fixes
Post Reply
Franz Pichler
Posts: 196
Joined: 29 Sep 2011, 12:25
Antispam: Yes

SIGABRT exception

Post by Franz Pichler »

Hi everybody,

i have a bug that occured the last two or three runs of my programs that i think i haven't seen before.
I start a parallel run of elmer that execute s as expected and after some hours during a random timestep i get this

Code: Select all

....
SaveScalars: Performed 4 reduction operations
SaveScalars: Found 4 values to save in total
SaveScalars: Performed 1 parallel reduction operations
SaveScalars:  Saving values to file Time.dat
SaveScalars: Defining: connectionarea = area: pot 1 over bc 1 : mpi_sum
SaveScalars: All done
SaveScalars: -----------------------------------------
END TEST CASE 1: Target NRM=221.28738
END TEST CASE 1: Target NRM=221.28738
*** glibc detected *** ElmerSolver_mpi: free(): invalid pointer: 0x000000000129f3a8 ***
*** glibc detected *** ElmerSolver_mpi: free(): invalid pointer: 0x000000000176c938 ***
======= Backtrace: =========
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x76d76)[0x7f2acea53d76]
/lib/x86_64-linux-gnu/libc.so.6(+0x76d76)[0x7f7de0155d76]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f2acea58aac]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f7de015aaac]
/usr/lib/libelmersolver-7.0.so(__meshutils_MOD_releasevariablelist+0x3e2)[0x7f2acf9ae987]
/usr/lib/libelmersolver-7.0.so(__meshutils_MOD_releasevariablelist+0x3e2)[0x7f7de10b0987]
/usr/lib/libelmersolver-7.0.so(__meshutils_MOD_releasemesh+0x22)[0x7f2acf9acff6]
/usr/lib/libelmersolver-7.0.so(__meshutils_MOD_releasemesh+0x22)[0x7f7de10aeff6]
/usr/lib/libelmersolver-7.0.so(+0x42c57b)[0x7f2acfb7657b]
/usr/lib/libelmersolver-7.0.so(+0x42c57b)[0x7f7de127857b]
/usr/lib/libelmersolver-7.0.so(+0x428b8e)[0x7f2acfb72b8e]
/usr/lib/libelmersolver-7.0.so(+0x428b8e)[0x7f7de1274b8e]
/usr/lib/libelmersolver-7.0.so(elmersolver_+0x2f48)[0x7f2acfb76168]
ElmerSolver_mpi[0x4010ee]
ElmerSolver_mpi[0x40140f]
/usr/lib/libelmersolver-7.0.so(elmersolver_+0x2f48)[0x7f7de1278168]
ElmerSolver_mpi[0x4010ee]
ElmerSolver_mpi[0x40140f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7f2ace9fbead]
ElmerSolver_mpi[0x400d49]
======= Memory map: ========
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7f7de00fdead]
ElmerSolver_mpi[0x400d49]
======= Memory map: ========
00400000-00402000 r-xp 00000000 68:01 2550840                            /usr/bin/ElmerSolver_mpi
00601000-00602000 rw-p 00001000 68:01 2550840                            /usr/bin/ElmerSolver_mpi
00f2f000-02ce6000 rw-p 00000000 00:00 0                                  [heap]
7f2ac0000000-7f2ac0021000 rw-p 00000000 00:00 0 
7f2ac0021000-7f2ac4000000 ---p 00000000 00:00 0 
7f2ac52da000-7f2ac52e3000 r-xp 00000000 00:17 6197445                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/TabSolver.so
7f2ac52e3000-7f2ac54e2000 ---p 00009000 00:17 6197445                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/TabSolver.so
7f2ac54e2000-7f2ac54e3000 rw-p 00008000 00:17 6197445                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/TabSolver.so
7f2ac54e3000-7f2ac551b000 r-xp 00000000 00:17 6197444                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/mySaveData.so
7f2ac551b000-7f2ac571b000 ---p 00038000 00:17 6197444                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/mySaveData.so
7f2ac571b000-7f2ac571d000 rw-p 00038000 00:17 6197444                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/mySaveData.so
7f2ac571d000-7f2ac5724000 rw-p 00000000 00:00 0 
7f2ac5724000-7f2ac575e000 r-xp 00000000 00:17 6197443                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myResultOutputSolve.so
7f2ac575e000-7f2ac595e000 ---p 0003a000 00:17 6197443                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myResultOutputSolve.so
7f2ac595e000-7f2ac595f000 rw-p 0003a000 00:17 6197443                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myResultOutputSolve.so
7f2ac595f000-7f2ac5969000 r-xp 00000000 00:17 6197412                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myAdaptiveSolver.so
7f2ac5969000-7f2ac5b68000 ---p 0000a000 00:17 6197412                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myAdaptiveSolver.so
7f2ac5b68000-7f2ac5b69000 rw-p 00009000 00:17 6197412                    /cluster/00400000-00402000 r-xp 00000000 68:01 2550840                            /usr/bin/ElmerSolver_mpi
00601000-00602000 rw-p 00001000 68:01 2550840                            /usr/bin/ElmerSolver_mpi
01403000-03272000 rw-p 00000000 00:00 0                                  [heap]
7f7dd0000000-7f7dd0021000 rw-p 00000000 00:00 0 
7f7dd0021000-7f7dd4000000 ---p 00000000 00:00 0 
7f7dd69dc000-7f7dd69e5000 r-xp 00000000 00:17 6197445                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/TabSolver.so
7f7dd69e5000-7f7dd6be4000 ---p 00009000 00:17 6197445                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/TabSolver.so
7f7dd6be4000-7f7dd6be5000 rw-p 00008000 00:17 6197445                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/TabSolver.so
7f7dd6be5000-7f7dd6c1d000 r-xp 00000000 00:17 6197444                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/mySaveData.so
7f7dd6c1d000-7f7dd6e1d000 ---p 00038000 00:17 6197444                    /cluster/pichler/Somabat_EC_Multpichler/Somabat_EC_Multiscale_Model_V1.2/myAdaptiveSolver.so
7f2ac5b69000-7f2ac5b70000 r-xp 00000000 00:17 6197421                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myBCSolver.so
7f2ac5b70000-7f2ac5d6f000 ---p 00007000 00:17 6197421                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myBCSolver.so
7f2ac5d6f000-7f2ac5d70000 rw-p 00006000 00:17 6197421                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myBCSolver.so
7f2ac5d70000-7f2ac5d77000 r-xp 00000000 00:17 6197440                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myHeatSolver.so
7f2ac5d77000-7f2ac5f76000 ---p 00007000 00:17 6197440                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myHeatSolver.so
7f2ac5f76000-7f2ac5f77000 rw-p 00006000 00:17 6197440                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myHeatSolver.so
7f2ac5f77000-7f2ac5f83000 r-xp 00000000 00:17 6197419                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/CoupledDiffuiscale_Model_V1.2/mySaveData.so
7f7dd6e1d000-7f7dd6e1f000 rw-p 00038000 00:17 6197444                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/mySaveData.so
7f7dd6e1f000-7f7dd6e26000 rw-p 00000000 00:00 0 
7f7dd6e26000-7f7dd6e60000 r-xp 00000000 00:17 6197443                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myResultOutputSolve.so
7f7dd6e60000-7f7dd7060000 ---p 0003a000 00:17 6197443                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myResultOutputSolve.so
7f7dd7060000-7f7dd7061000 rw-p 0003a000 00:17 6197443                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myResultOutputSolve.so
7f7dd7061000-7f7dd706b000 r-xp 00000000 00:17 6197412                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myAdaptiveSolver.so
7f7dd706b000-7f7dd726a000 ---p 0000a000 00:17 6197412                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myAdaptiveSolver.so
7f7dd726a000-7f7dd726b000 rw-p 00009000 00:17 6197412                    /cluster/sion.so
7f2ac5f83000-7f2ac6183000 ---p 0000c000 00:17 6197419                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/CoupledDiffusion.so
7f2ac6183000-7f2ac6184000 rw-p 0000c000 00:17 6197419                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/CoupledDiffusion.so
7f2ac6184000-7f2ac6194000 r-xp 00000000 00:17 6197437                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/MS_Comm_Interpolation.so
7f2ac6194000-7f2ac6393000 ---p 00010000 00:17 6197437                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/MS_Comm_Interpolation.so
7f2ac6393000-7f2ac6394000 rw-p 0000f000 00:17 6197437                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/MS_Comm_Interpolation.so
7f2ac6394000-7f2ac63a1000 r-xp 00000000 68:01 2722877                    /usr/lib/openmpi/lib/openmpi/mca_osc_rdma.so
7f2ac63a1000-7f2ac65a1000 ---p 0000d000 68:01 2722877                    /usr/lib/openmpi/lib/openmpi/mca_osc_rdma.so
7f2ac65a1000-7f2ac65a2000 rw-p 0000d000 68:01 2722pichler/Somabat_EC_Multiscale_Model_V1.2/myAdaptiveSolver.so
7f7dd726b000-7f7dd7272000 r-xp 00000000 00:17 6197421                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myBCSolver.so
7f7dd7272000-7f7dd7471000 ---p 00007000 00:17 6197421                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myBCSolver.so
7f7dd7471000-7f7dd7472000 rw-p 00006000 00:17 6197421                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myBCSolver.so
7f7dd7472000-7f7dd7479000 r-xp 00000000 00:17 6197440                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myHeatSolver.so
7f7dd7479000-7f7dd7678000 ---p 00007000 00:17 6197440                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myHeatSolver.so
7f7dd7678000-7f7dd7679000 rw-p 00006000 00:17 6197440                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myHeatSolver.so
7f7dd7679000-7f7dd7685000 r-xp 00000000 00:17 6197419                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/CoupledDiffu877                    /usr/lib/openmpi/lib/openmpi/mca_osc_rdma.so
7f2ac65a2000-7f2ac65ac000 r-xp 00000000 68:01 2722842                    /usr/lib/openmpi/lib/openmpi/mca_osc_pt2pt.so
7f2ac65ac000-7f2ac67ab000 ---p 0000a000 68:01 2722842                    /usr/lib/openmpi/lib/openmpi/mca_osc_pt2pt.so
7f2ac67ab000-7f2ac67ad000 rw-p 00009000 68:01 2722842                    /usr/lib/openmpi/lib/openmpi/mca_osc_pt2pt.so
7f2ac67ad000-7f2ac67c6000 r-xp 00000000 68:01 2722835                    /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so
7f2ac67c6000-7f2ac69c6000 ---p 00019000 68:01 2722835                    /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so
7f2ac69c6000-7f2ac69c7000 rw-p 00019000 68:01 2722835                    /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so
7f2ac69c7000-7f2ac69ca000 r-xp 00000000 68:01 2722886                    /usr/lib/openmpi/lib/openmpi/mca_coll_sync.so
7f2ac69ca000-7f2ac6bc9000 ---p 00003000 68:01 2722886                    /usr/lib/openmpi/lib/openmpi/mca_coll_sync.so
7sion.so
7f7dd7685000-7f7dd7885000 ---p 0000c000 00:17 6197419                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/CoupledDiffusion.so
7f7dd7885000-7f7dd7886000 rw-p 0000c000 00:17 6197419                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/CoupledDiffusion.so
7f7dd7886000-7f7dd7896000 r-xp 00000000 00:17 6197437                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/MS_Comm_Interpolation.so
7f7dd7896000-7f7dd7a95000 ---p 00010000 00:17 6197437                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/MS_Comm_Interpolation.so
7f7dd7a95000-7f7dd7a96000 rw-p 0000f000 00:17 6197437                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/MS_Comm_Interpolation.so
7f7dd7a96000-7f7dd7aa3000 r-xp 00000000 68:01 2722877                    /usr/lib/openmpi/lib/openmpi/mca_osc_rdma.so
7f7dd7aa3000-7f7dd7ca3000 ---p 0000d000 68:01 2722877                    /usr/lib/openmpi/lib/openmpi/mca_osc_rdma.so
7f7dd7ca3000-7f7dd7ca4000 rw-p 0000d000 68:01 2722f2ac6bc9000-7f2ac6bca000 rw-p 00002000 68:01 2722886                    /usr/lib/openmpi/lib/openmpi/mca_coll_sync.so
7f2ac6bca000-7f2ac6bcf000 r-xp 00000000 68:01 2722838                    /usr/lib/openmpi/lib/openmpi/mca_coll_sm.so
7f2ac6bcf000-7f2ac6dce000 ---p 00005000 68:01 2722838                    /usr/lib/openmpi/lib/openmpi/mca_coll_sm.so
7f2ac6dce000-7f2ac6dcf000 rw-p 00004000 68:01 2722838                    /usr/lib/openmpi/lib/openmpi/mca_coll_sm.so
7f2ac6dcf000-7f2ac6dd1000 r-xp 00000000 68:01 2722843                    /usr/lib/openmpi/lib/openmpi/mca_coll_self.so
7f2ac6dd1000-7f2ac6fd0000 ---p 00002000 68:01 2722843                    /usr/lib/openmpi/lib/openmpi/mca_coll_self.so
7f2ac6fd0000-7f2ac6fd1000 rw-p 00001000 68:01 2722843                    /usr/lib/openmpi/lib/openmpi/mca_coll_self.so
7f2ac6fd1000-7f2ac6fd4000 r-xp 00000000 68:01 2722894                    /usr/lib/openmpi/lib/openmpi/mca_coll_inter.so
7f2ac6fd4000-7f2ac71d3000 ---p 00003000 68:01 2722894                    /usr/877                    /usr/lib/openmpi/lib/openmpi/mca_osc_rdma.so
7f7dd7ca4000-7f7dd7cae000 r-xp 00000000 68:01 2722842                    /usr/lib/openmpi/lib/openmpi/mca_osc_pt2pt.so
7f7dd7cae000-7f7dd7ead000 ---p 0000a000 68:01 2722842                    /usr/lib/openmpi/lib/openmpi/mca_osc_pt2pt.so
7f7dd7ead000-7f7dd7eaf000 rw-p 00009000 68:01 2722842                    /usr/lib/openmpi/lib/openmpi/mca_osc_pt2pt.so
7f7dd7eaf000-7f7dd7ec8000 r-xp 00000000 68:01 2722835                    /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so
7f7dd7ec8000-7f7dd80c8000 ---p 00019000 68:01 2722835                    /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so
7f7dd80c8000-7f7dd80c9000 rw-p 00019000 68:01 2722835                    /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so
7f7dd80c9000-7f7dd80cc000 r-xp 00000000 68:01 2722886                    /usr/lib/openmpi/lib/openmpi/mca_coll_sync.so
7f7dd80cc000-7f7dd82cb000 ---p 00003000 68:01 2722886                    /usr/lib/openmpi/lib/openmpi/mca_coll_sync.so
7f7dd82cb000-7f7dd82cc000 rw-p 00002000 68:01 2722886                    /usr/lib/openmpi/lib/openmpi/mca_coll_sync.so
7f7dd82cc000-7f7dd82d1000 r-xp 00000000 68:01 2722838                    /usr/lib/openmpi/lib/openmpi/mca_coll_sm.so
7f7dd82d1000-7f7dd84d0000 ---p 00005000 68:01 2722838                    /usr/lib/openmpi/lib/openmpi/mca_coll_sm.so
7f7dd84d0000-7f7dd84d1000 rw-p 00004000 68:01 2722838                    /usr/lib/openmpi/lib/openmpi/mca_coll_sm.so
7f7dd84d1000-7f7dd84d3000 r-xp 00000000 68:01 2722843                    /usr/lib/openmpi/lib/openmpi/mca_coll_self.so
7f7dd84d3000-7f7dd86d2000 ---p 00002000 68:01 2722843                    /usr/lib/openmpi/lib/openmpi/mca_coll_self.so
7f7dd86d2000-7f7dd86d3000 rw-p 00001000 68:01 2722843                    /usr/lib/openmpi/lib/openmpi/mca_coll_self.so
7f7dd86d3000-7f7dd86d6000 r-xp 00000000 68:01 2722894                    /usr/lib/openmpi/lib/openmpi/mca_coll_inter.so
7f7dd86d6000-7f7dd88d5000 ---p 00003000 68:01 2722894                    /usr/lib/openmpi/lib/openmpi/mca_coll_inter.so
7f2ac71d3000-7f2ac71d4000 rw-p 00002000 68:01 2722894                    /usr/lib/openmpi/lib/openmpi/mca_coll_inter.so
7f2ac71d4000-7f2ac71d8000 r-xp 00000000 68:01 2722897                    /usr/lib/openmpi/lib/openmpi/mca_coll_hierarch.so
7f2ac71d8000-7f2ac73d8000 ---p 00004000 68:01 2722897                    /usr/lib/openmpi/lib/openmpi/mca_coll_hierarch.so
7f2ac73d8000-7f2ac73d9000 rw-p 00004000 68:01 2722897                    /usr/lib/openmpi/lib/openmpi/mca_coll_hierarch.so
7f2ac73d9000-7f2ac73e1000 r-xp 00000000 68:01 2722844                    /usr/lib/openmpi/lib/openmpi/mca_coll_basic.so
7f2ac73e1000-7f2ac75e0000 ---p 00008000 68:01 2722844                    /usr/lib/openmpi/lib/openmpi/mca_coll_basic.so
7f2ac75e0000-7f2ac75e1000 rw-p 00007000 68:01 2722844                    /usr/lib/openmpi/lib/openmpi/mca_coll_basic.so
7f2ac75e1000-7f2ac75ec000 r-xp 00000000 68:01 2722847                    /usr/lib/openmpi/lib/openmpi/mca_btl_tcp.so
7f2ac75ec000-7f2ac77eb000 ---p 0000b000 68:01 2722847                    /usr/lib/openmpi/lib/openmpi/mca_btl_tcp.so
7f2ac77eb000-7f2ac77ed000 rw-p 0000a000 68:01 2722847                    /usr/lib/openmpi/lib/openmpi/mca_btl_tcp.so
7f2ac77ed000-7f2ac786d000 rw-p 00000000 00:00 0 
7f2ac786d000-7f2ac7870000 r-xp 00000000 68:01 2722857                    /usr/lib/openmpi/lib/openmpi/mca_btl_self.so
7f2ac7870000-7f2ac7a6f000 ---p 00003000 68:01 2722857                    /usr/lib/openmpi/lib/openmpi/mca_btl_self.so
7f2ac7a6f000-7f2ac7a70000 rw-p 00002000 68:01 2722857                    /usr/lib/openmpi/lib/openmpi/mca_btl_self.so
7f2ac7a70000-7f2ac7a74000 r-xp 00000000 68:01 2722864                    /usr/lib/openmpi/lib/openmpi/mca_bml_r2.so
7f2ac7a74000-7f2ac7c73000 ---p 00004000 68:01 2722864                    /usr/lib/openmpi/lib/openmpi/mca_bml_r2.so
7f2ac7c73000-7f2ac7c74000 rw-p 00003000 68:01 2722864                    /usr/lib/openmpi/lib/openmpi/mca_bml_r2.so
7f2ac7c75000-7f2ac7c78000 r-xp 00000000 00:17 6197442  lib/openmpi/lib/openmpi/mca_coll_inter.so
7f7dd88d5000-7f7dd88d6000 rw-p 00002000 68:01 2722894                    /usr/lib/openmpi/lib/openmpi/mca_coll_inter.so
7f7dd88d6000-7f7dd88da000 r-xp 00000000 68:01 2722897                    /usr/lib/openmpi/lib/openmpi/mca_coll_hierarch.so
7f7dd88da000-7f7dd8ada000 ---p 00004000 68:01 2722897                    /usr/lib/openmpi/lib/openmpi/mca_coll_hierarch.so
7f7dd8ada000-7f7dd8adb000 rw-p 00004000 68:01 2722897                    /usr/lib/openmpi/lib/openmpi/mca_coll_hierarch.so
7f7dd8adb000-7f7dd8ae3000 r-xp 00000000 68:01 2722844                    /usr/lib/openmpi/lib/openmpi/mca_coll_basic.so
7f7dd8ae3000-7f7dd8ce2000 ---p 00008000 68:01 2722844                    /usr/lib/openmpi/lib/openmpi/mca_coll_basic.so
7f7dd8ce2000-7f7dd8ce3000 rw-p 00007000 68:01 2722844                    /usr/lib/openmpi/lib/openmpi/mca_coll_basic.so
7f7dd8ce3000-7f7dd8cee000 r-xp 00000000 68:01 2722847                    /usr/lib/openmpi/lib/openmpi/mca_btl_tcp.so
7f7dd8cee000-7f7                  /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myInitialGuesser.so
7f2ac7c78000-7f2ac7e77000 ---p 00003000 00:17 6197442                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myInitialGuesser.so
7f2ac7e77000-7f2ac7e78000 rw-p 00002000 00:17 6197442                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myInitialGuesser.so
7f2ac7e78000-7f2ac7e8d000 r-xp 00000000 68:01 2722846                    /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so
7f2ac7e8d000-7f2ac808c000 ---p 00015000 68:01 2722846                    /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so
7f2ac808c000-7f2ac808e000 rw-p 00014000 68:01 2722846                    /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so
7f2ac80ac000-7f2ac80ad000 r-xp 00000000 00:17 6197441                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myHeatSource.so
7f2ac80ad000-7f2ac82ac000 ---p 00001000 00:17 6197441                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myHeatSource.so
7f2ac82ac000-7f2ac82add8eed000 ---p 0000b000 68:01 2722847                    /usr/lib/openmpi/lib/openmpi/mca_btl_tcp.so
7f7dd8eed000-7f7dd8eef000 rw-p 0000a000 68:01 2722847                    /usr/lib/openmpi/lib/openmpi/mca_btl_tcp.so
7f7dd8eef000-7f7dd8f6f000 rw-p 00000000 00:00 0 
7f7dd8f6f000-7f7dd8f72000 r-xp 00000000 68:01 2722857                    /usr/lib/openmpi/lib/openmpi/mca_btl_self.so
7f7dd8f72000-7f7dd9171000 ---p 00003000 68:01 2722857                    /usr/lib/openmpi/lib/openmpi/mca_btl_self.so
7f7dd9171000-7f7dd9172000 rw-p 00002000 68:01 2722857                    /usr/lib/openmpi/lib/openmpi/mca_btl_self.so
7f7dd9172000-7f7dd9176000 r-xp 00000000 68:01 2722864                    /usr/lib/openmpi/lib/openmpi/mca_bml_r2.so
7f7dd9176000-7f7dd9375000 ---p 00004000 68:01 2722864                    /usr/lib/openmpi/lib/openmpi/mca_bml_r2.so
7f7dd9375000-7f7dd9376000 rw-p 00003000 68:01 2722864                    /usr/lib/openmpi/lib/openmpi/mca_bml_r2.so
7f7dd9377000-7f7dd937a000 r-xp 00000000 00:17 6197442  d000 rw-p 00000000 00:17 6197441                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myHeatSource.so
7f2ac82ad000-7f2ac82b2000 r-xp 00000000 68:01 2722891                    /usr/lib/openmpi/lib/openmpi/mca_dpm_orte.so
7f2ac82b2000-7f2ac84b1000 ---p 00005000 68:01 2722891                    /usr/lib/openmpi/lib/openmpi/mca_dpm_orte.so
7f2ac84b1000-7f2ac84b2000 rw-p 00004000 68:01 2722891                    /usr/lib/openmpi/lib/openmpi/mca_dpm_orte.so
7f2ac84b2000-7f2ac84b5000 r-xp 00000000 68:01 2722862                    /usr/lib/openmpi/lib/openmpi/mca_pubsub_orte.so
7f2ac84b5000-7f2ac86b4000 ---p 00003000 68:01 2722862                    /usr/lib/openmpi/lib/openmpi/mca_pubsub_orte.so
7f2ac86b4000-7f2ac86b5000 rw-p 00002000 68:01 2722862                    /usr/lib/openmpi/lib/openmpi/mca_pubsub_orte.so
7f2ac86b5000-7f2ac86b7000 r-xp 00000000 68:01 2722827                    /usr/lib/openmpi/lib/libmca_common_sm.so.1.0.2
7f2ac86b7000-7f2ac88b7000 ---p 00002000 68:01 2722827                                  /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myInitialGuesser.so
7f7dd937a000-7f7dd9579000 ---p 00003000 00:17 6197442                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myInitialGuesser.so
7f7dd9579000-7f7dd957a000 rw-p 00002000 00:17 6197442                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myInitialGuesser.so
7f7dd957a000-7f7dd958f000 r-xp 00000000 68:01 2722846                    /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so
7f7dd958f000-7f7dd978e000 ---p 00015000 68:01 2722846                    /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so
7f7dd978e000-7f7dd9790000 rw-p 00014000 68:01 2722846                    /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so
7f7dd97ae000-7f7dd97af000 r-xp 00000000 00:17 6197441                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myHeatSource.so
7f7dd97af000-7f7dd99ae000 ---p 00001000 00:17 6197441                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myHeatSource.so
7f7dd99ae000-7f7dd99a    /usr/lib/openmpi/lib/libmca_common_sm.so.1.0.2
7f2ac88b7000-7f2ac88b8000 rw-p 00002000 68:01 2722827                    /usr/lib/openmpi/lib/libmca_common_sm.so.1.0.2
7f2ac88b8000-7f2ac88ba000 r-xp 00000000 68:01 2722845                    /usr/lib/openmpi/lib/openmpi/mca_mpool_sm.so
7f2ac88ba000-7f2ac8ab9000 ---p 00002000 68:01 2722845                    /usr/lib/openmpi/lib/openmpi/mca_mpool_sm.so
7f2ac8ab9000-7f2ac8aba000 rw-p 00001000 68:01 2722845                    /usr/lib/openmpi/lib/openmpi/mca_mpool_sm.so
7f2ac8aba000-7f2ac8abd000 r-xp 00000000 68:01 2722851                    /usr/lib/openmpi/lib/openmpi/mca_mpool_rdma.so
7f2ac8abd000-7f2ac8cbc000 ---p 00003000 68:01 2722851                    /usr/lib/openmpi/lib/openmpi/mca_mpool_rdma.so
7f2ac8cbc000-7f2ac8cbd000 rw-p 00002000 68:01 2722851                    /usr/lib/openmpi/lib/openmpi/mca_mpool_rdma.so
7f2ac8cbd000-7f2ac8cbe000 r-xp 00000000 68:01 2722889                    /usr/lib/openmpi/lib/openmpi/mca_mpool_fake.so
7f2ac8cbe000-7f2ac8f000 rw-p 00000000 00:17 6197441                    /cluster/pichler/Somabat_EC_Multiscale_Model_V1.2/myHeatSource.so
7f7dd99af000-7f7dd99b4000 r-xp 00000000 68:01 2722891                    /usr/lib/openmpi/lib/openmpi/mca_dpm_orte.so
7f7dd99b4000-7f7dd9bb3000 ---p 00005000 68:01 2722891                    /usr/lib/openmpi/lib/openmpi/mca_dpm_orte.so
7f7dd9bb3000-7f7dd9bb4000 rw-p 00004000 68:01 2722891                    /usr/lib/openmpi/lib/openmpi/mca_dpm_orte.so
7f7dd9bb4000-7f7dd9bb7000 r-xp 00000000 68:01 2722862                    /usr/lib/openmpi/lib/openmpi/mca_pubsub_orte.so
7f7dd9bb7000-7f7dd9db6000 ---p 00003000 68:01 2722862                    /usr/lib/openmpi/lib/openmpi/mca_pubsub_orte.so
7f7dd9db6000-7f7dd9db7000 rw-p 00002000 68:01 2722862                    /usr/lib/openmpi/lib/openmpi/mca_pubsub_orte.so
7f7dd9db7000-7f7dd9db9000 r-xp 00000000 68:01 2722827                    /usr/lib/openmpi/lib/libmca_common_sm.so.1.0.2
7f7dd9db9000-7f7dd9fb9000 ---p 00002000 68:01 2722827                ebd000 ---p 00001000 68:01 2722889                    /usr/lib/openmpi/lib/openmpi/mca_mpool_fake.so
7f2ac8ebd000-7f2ac8ebe000 rw-p 00000000 68:01 2722889                    /usr/lib/openmpi/lib/openmpi/mca_mpool_fake.so
7f2ac8ebe000-7f2ac8ec1000 r-xp 00000000 68:01 2722879                    /usr/lib/openmpi/lib/openmpi/mca_rcache_vma.so
7f2ac8ec1000-7f2ac90c0000 ---p 00003000 68:01 2722879                    /usr/lib/openmpi/lib/openmpi/mca_rcache_vma.so
7f2ac90c0000-7f2ac90c1000 rw-p 00002000 68:01 2722879                    /usr/lib/openmpi/lib/openmpi/mca_rcache_vma.so
7f2ac90c1000-7f2ac90c3000 r-xp 00000000 68:01 2722861                    /usr/lib/openmpi/lib/openmpi/mca_allocator_basic.so
7f2ac90c3000-7f2ac92c2000 ---p 00002000 68:01 2722861                    /usr/lib/openmpi/lib/openmpi/mca_allocator_basic.so
7f2ac92c2000-7f2ac92c3000 rw-p 00001000 68:01 2722861                    /usr/lib/openmpi/lib/openmpi/mca_allocator_basic.so
7f2ac92c3000-7f2ac92ce000 r-xp 00000000 68:01 1893139                   /usr/lib/openmpi/lib/libmca_common_sm.so.1.0.2
7f7dd9fb9000-7f7dd9fba000 rw-p 00002000 68:01 2722827                    /usr/lib/openmpi/lib/libmca_common_sm.so.1.0.2
7f7dd9fba000-7f7dd9fbc000 r-xp 00000000 68:01 2722845                    /usr/lib/openmpi/lib/openmpi/mca_mpool_sm.so
7f7dd9fbc000-7f7dda1bb000 ---p 00002000 68:01 2722845                    /usr/lib/openmpi/lib/openmpi/mca_mpool_sm.so
7f7dda1bb000-7f7dda1bc000 rw-p 00001000 68:01 2722845                    /usr/lib/openmpi/lib/openmpi/mca_mpool_sm.so
7f7dda1bc000-7f7dda1bf000 r-xp 00000000 68:01 2722851                    /usr/lib/openmpi/lib/openmpi/mca_mpool_rdma.so
7f7dda1bf000-7f7dda3be000 ---p 00003000 68:01 2722851                    /usr/lib/openmpi/lib/openmpi/mca_mpool_rdma.so
7f7dda3be000-7f7dda3bf000 rw-p 00002000 68:01 2722851                    /usr/lib/openmpi/lib/openmpi/mca_mpool_rdma.so
7f7dda3bf000-7f7dda3c0000 r-xp 00000000 68:01 2722889                    /usr/lib/openmpi/lib/openmpi/mca_mpool_fake.so
7f7dda3c0000-7f7dda     /lib/x86_64-linux-gnu/libnss_files-2.13.so
7f2ac92ce000-7f2ac94cd000 ---p 0000b000 68:01 1893139                    /lib/x86_64-linux-gnu/libnss_files-2.13.so
7f2ac94cd000-7f2ac94ce000 r--p 0000a000 68:01 1893139                    /lib/x86_64-linux-gnu/libnss_files-2.13.so
7f2ac94ce000-7f2ac94cf000 rw-p 0000b000 68:01 1893139                    /lib/x86_64-linux-gnu/libnss_files-2.13.so
7f2ac94cf000-7f2ac94d9000 r-xp 00000000 68:01 1893135                    /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7f2ac94d9000-7f2ac96d8000 ---p 0000a000 68:01 1893135                    /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7f2ac96d8000-7f2ac96d9000 r--p 00009000 68:01 1893135                    /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7f2ac96d9000-7f2ac96da000 rw-p 0000a000 68:01 1893135                    /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7f2ac96da000-7f2ac96e1000 r-xp 00000000 68:01 1893128                    /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7f2ac96e1000-7f2ac98e0000 ---p 00007000 68:01 1893128  5bf000 ---p 00001000 68:01 2722889                    /usr/lib/openmpi/lib/openmpi/mca_mpool_fake.so
7f7dda5bf000-7f7dda5c0000 rw-p 00000000 68:01 2722889                    /usr/lib/openmpi/lib/openmpi/mca_mpool_fake.so
7f7dda5c0000-7f7dda5c3000 r-xp 00000000 68:01 2722879                    /usr/lib/openmpi/lib/openmpi/mca_rcache_vma.so
7f7dda5c3000-7f7dda7c2000 ---p 00003000 68:01 2722879                    /usr/lib/openmpi/lib/openmpi/mca_rcache_vma.so
7f7dda7c2000-7f7dda7c3000 rw-p 00002000 68:01 2722879                    /usr/lib/openmpi/lib/openmpi/mca_rcache_vma.so
7f7dda7c3000-7f7dda7c5000 r-xp 00000000 68:01 2722861                    /usr/lib/openmpi/lib/openmpi/mca_allocator_basic.so
7f7dda7c5000-7f7dda9c4000 ---p 00002000 68:01 2722861                    /usr/lib/openmpi/lib/openmpi/mca_allocator_basic.so
7f7dda9c4000-7f7dda9c5000 rw-p 00001000 68:01 2722861                    /usr/lib/openmpi/lib/openmpi/mca_allocator_basic.so
7f7dda9c5000-7f7dda9d0000 r-xp 00000000 68:01 1893139                                 /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7f2ac98e0000-7f2ac98e1000 r--p 00006000 68:01 1893128                    /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7f2ac98e1000-7f2ac98e2000 rw-p 00007000 68:01 1893128                    /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7f2ac98e2000-7f2ac98e5000 r-xp 00000000 68:01 2722836                    /usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so
7f2ac98e5000-7f2ac9ae4000 ---p 00003000 68:01 2722836                    /usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so
7f2ac9ae4000-7f2ac9ae5000 rw-p 00002000 68:01 2722836                    /usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so
7f2ac9ae5000-7f2ac9ae9000 r-xp 00000000 68:01 2722873                    /usr/lib/openmpi/lib/openmpi/mca_routed_binomial.so
7f2ac9ae9000-7f2ac9ce9000 ---p 00004000 68:01 2722873                    /usr/lib/openmpi/lib/openmpi/mca_routed_binomial.so
7f2ac9ce9000-7f2ac9cea000 rw-p 00004000 68:01 2722873                    /usr/lib/openmpi/lib/openmpi/mca_routed_binomial.so
7f2ac9cea000-7f2ac9cf8000 r-xp 00000000 68:01 2722892                    /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so
7f2ac9cf8000-7f2ac9ef8000 ---p 0000e000 68:01 2722892                    /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so
7f2ac9ef8000-7f2ac9efa000 rw-p 0000e000 68:01 2722892                    /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so
7f2ac9efa000-7f2ac9eff000 r-xp 00000000 68:01 2722860                    /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so
7f2ac9eff000-7f2aca0ff000 ---p 00005000 68:01 2722860                    /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so
7f2aca0ff000-7f2aca100000 rw-p 00005000 68:01 2722860                    /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so
7f2aca101000-7f2aca103000 r-xp 00000000 68:01 2722866                    /usr/lib/openmpi/lib/openmpi/mca_allocator_bucket.so
7f2aca103000-7f2aca302000 ---p 00002000 68:01 2722866                    /usr/lib/openmpi/lib/openmpi/mca_allocator_bucket.so
7f2aca302000-7f2aca303000 rw-p 00001000 68:01 2722866            /lib/x86_64-linux-gnu/libnss_files-2.13.so
7f7dda9d0000-7f7ddabcf000 ---p 0000b000 68:01 1893139                    /lib/x86_64-linux-gnu/libnss_files-2.13.so
7f7ddabcf000-7f7ddabd0000 r--p 0000a000 68:01 1893139                    /lib/x86_64-linux-gnu/libnss_files-2.13.so
7f7ddabd0000-7f7ddabd1000 rw-p 0000b000 68:01 1893139                    /lib/x86_64-linux-gnu/libnss_files-2.13.so
7f7ddabd1000-7f7ddabdb000 r-xp 00000000 68:01 1893135                    /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7f7ddabdb000-7f7ddadda000 ---p 0000a000 68:01 1893135                    /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7f7ddadda000-7f7ddaddb000 r--p 00009000 68:01 1893135                    /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7f7ddaddb000-7f7ddaddc000 rw-p 0000a000 68:01 1893135                    /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7f7ddaddc000-7f7ddade3000 r-xp 00000000 68:01 1893128                    /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7f7ddade3000-7f7ddafe2000 ---p 00007000 68:01 1893128               /usr/lib/openmpi/lib/openmpi/mca_allocator_bucket.so
7f2aca303000-7f2aca306000 r-xp 00000000 68:01 2722868                    /usr/lib/openmpi/lib/openmpi/mca_ess_env.so
7f2aca306000-7f2aca505000 ---p 00003000 68:01 2722868                    /usr/lib/openmpi/lib/openmpi/mca_ess_env.so
7f2aca505000-7f2aca506000 rw-p 00002000 68:01 2722868                    /usr/lib/openmpi/lib/openmpi/mca_ess_env.so
7f2aca506000-7f2aca508000 r-xp 00000000 68:01 2722837                    /usr/lib/openmpi/lib/openmpi/mca_crs_none.so
7f2aca508000-7f2aca707000 ---p 00002000 68:01 2722837                    /usr/lib/openmpi/lib/openmpi/mca_crs_none.so
7f2aca707000-7f2aca708000 rw-p 00001000 68:01 2722837                    /usr/lib/openmpi/lib/openmpi/mca_crs_none.so
7f2aca708000-7f2aca709000 r-xp 00000000 68:01 2722876                    /usr/lib/openmpi/lib/openmpi/mca_carto_auto_detect.so
7f2aca709000-7f2aca909000 ---p 00001000 68:01 2722876                    /usr/lib/openmpi/lib/openmpi/mca_carto_auto_detect.so                  /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7f7ddafe2000-7f7ddafe3000 r--p 00006000 68:01 1893128                    /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7f7ddafe3000-7f7ddafe4000 rw-p 00007000 68:01 1893128                    /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7f7ddafe4000-7f7ddafe7000 r-xp 00000000 68:01 2722836                    /usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so
7f7ddafe7000-7f7ddb1e6000 ---p 00003000 68:01 2722836                    /usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so
7f7ddb1e6000-7f7ddb1e7000 rw-p 00002000 68:01 2722836                    /usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so
7f7ddb1e7000-7f7ddb1eb000 r-xp 00000000 68:01 2722873                    /usr/lib/openmpi/lib/openmpi/mca_routed_binomial.so
7f7ddb1eb000-7f7ddb3eb000 ---p 00004000 68:01 2722873                    /usr/lib/openmpi/lib/openmpi/mca_routed_binomial.so
7f7ddb3eb000-7f7ddb3ec000 rw-p 00004000 68:01 2722873                    /usr/lib/openmpi/lib/openmpi/mca_routed_bi
7f2aca909000-7f2aca90a000 rw-p 00001000 68:01 2722876                    /usr/lib/openmpi/lib/openmpi/mca_carto_auto_detect.so
7f2aca90a000-7f2aca90f000 r-xp 00000000 68:01 2722872                    /usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so
7f2aca90f000-7f2acab0e000 ---p 00005000 68:01 2722872                    /usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so
7f2acab0e000-7f2acab0f000 rw-p 00004000 68:01 2722872                    /usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so
7f2acab0f000-7f2acab10000 r-xp 00000000 68:01 2015407                    /usr/lib/libblacsCinit-openmpi.so.1.1
7f2acab10000-7f2acad0f000 ---p 00001000 68:01 2015407                    /usr/lib/libblacsCinit-openmpi.so.1.1
7f2acad0f000-7f2acad10000 rw-p 00000000 68:01 2015407                    /usr/lib/libblacsCinit-openmpi.so.1.1
7f2acad10000-7f2acad3e000 r-xp 00000000 68:01 2015408                    /usr/lib/libblacs-openmpi.so.1.1
7f2acad3e000-7f2acaf3d000 ---p 0002e000 68:01 2015408                    /usr/lib/nomial.so
7f7ddb3ec000-7f7ddb3fa000 r-xp 00000000 68:01 2722892                    /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so
7f7ddb3fa000-7f7ddb5fa000 ---p 0000e000 68:01 2722892                    /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so
7f7ddb5fa000-7f7ddb5fc000 rw-p 0000e000 68:01 2722892                    /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so
7f7ddb5fc000-7f7ddb601000 r-xp 00000000 68:01 2722860                    /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so
7f7ddb601000-7f7ddb801000 ---p 00005000 68:01 2722860                    /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so
7f7ddb801000-7f7ddb802000 rw-p 00005000 68:01 2722860                    /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so
7f7ddb803000-7f7ddb805000 r-xp 00000000 68:01 2722866                    /usr/lib/openmpi/lib/openmpi/mca_allocator_bucket.so
7f7ddb805000-7f7ddba04000 ---p 00002000 68:01 2722866                    /usr/lib/openmpi/lib/openmpi/mca_allocator_bucket.so
7f7ddba04000-7f7ddba05000 rw-p 00001000 68:01 2722866       libblacs-openmpi.so.1.1
7f2acaf3d000-7f2acaf3e000 rw-p 0002d000 68:01 2015408                    /usr/lib/libblacs-openmpi.so.1.1
7f2acaf3e000-7f2acb5fb000 r-xp 00000000 68:01 2015440                    /usr/lib/libscalapack-openmpi.so.1.8.0
7f2acb5fb000-7f2acb7fb000 ---p 006bd000 68:01 2015440                    /usr/lib/libscalapack-openmpi.so.1.8.0
7f2acb7fb000-7f2acb7fe000 rw-p 006bd000 68:01 2015440                    /usr/lib/libscalapack-openmpi.so.1.8.0
7f2acb7fe000-7f2acb851000 rw-p 00000000 00:00 0 
7f2acb851000-7f2acb88f000 r-xp 00000000 68:01 2015444                    /usr/lib/libmumps_common-4.10.0.so
7f2acb88f000-7f2acb890000 rw-p 0003e000 68:01 2015444                    /usr/lib/libmumps_common-4.10.0.so
7f2acb890000-7f2acb891000 rw-p 00000000 00:00 0 
7f2acb891000-7f2acb8a8000 r-xp 00000000 68:01 1893143                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7f2acb8a8000-7f2acbaa7000 ---p 00017000 68:01 1893143                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7f2acbaa7000-7f2             /usr/lib/openmpi/lib/openmpi/mca_allocator_bucket.so
7f7ddba05000-7f7ddba08000 r-xp 00000000 68:01 2722868                    /usr/lib/openmpi/lib/openmpi/mca_ess_env.so
7f7ddba08000-7f7ddbc07000 ---p 00003000 68:01 2722868                    /usr/lib/openmpi/lib/openmpi/mca_ess_env.so
7f7ddbc07000-7f7ddbc08000 rw-p 00002000 68:01 2722868                    /usr/lib/openmpi/lib/openmpi/mca_ess_env.so
7f7ddbc08000-7f7ddbc0a000 r-xp 00000000 68:01 2722837                    /usr/lib/openmpi/lib/openmpi/mca_crs_none.so
7f7ddbc0a000-7f7ddbe09000 ---p 00002000 68:01 2722837                    /usr/lib/openmpi/lib/openmpi/mca_crs_none.so
7f7ddbe09000-7f7ddbe0a000 rw-p 00001000 68:01 2722837                    /usr/lib/openmpi/lib/openmpi/mca_crs_none.so
7f7ddbe0a000-7f7ddbe0b000 r-xp 00000000 68:01 2722876                    /usr/lib/openmpi/lib/openmpi/mca_carto_auto_detect.so
7f7ddbe0b000-7f7ddc00b000 ---p 00001000 68:01 2722876                    /usr/lib/openmpi/lib/openmpi/mca_carto_auto_detect.soacbaa8000 r--p 00016000 68:01 1893143                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7f2acbaa8000-7f2acbaa9000 rw-p 00017000 68:01 1893143                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7f2acbaa9000-7f2acbaad000 rw-p 00000000 00:00 0 
7f2acbaad000-7f2acbaaf000 r-xp 00000000 68:01 1893147                    /lib/x86_64-linux-gnu/libutil-2.13.so
7f2acbaaf000-7f2acbcae000 ---p 00002000 68:01 1893147                    /lib/x86_64-linux-gnu/libutil-2.13.so
7f2acbcae000-7f2acbcaf000 r--p 00001000 68:01 1893147                    /lib/x86_64-linux-gnu/libutil-2.13.so
7f2acbcaf000-7f2acbcb0000 rw-p 00002000 68:01 1893147                    /lib/x86_64-linux-gnu/libutil-2.13.so
7f2acbcb0000-7f2acbcc5000 r-xp 00000000 68:01 1893133                    /lib/x86_64-linux-gnu/libnsl-2.13.so
7f2acbcc5000-7f2acbec4000 ---p 00015000 68:01 1893133                    /lib/x86_64-linux-gnu/libnsl-2.13.so
7f2acbec4000-7f2acbec5000 r--p 00014000 68:01 1893133                    /lib/x86_64-linux-gnu/libn
7f7ddc00b000-7f7ddc00c000 rw-p 00001000 68:01 2722876                    /usr/lib/openmpi/lib/openmpi/mca_carto_auto_detect.so
7f7ddc00c000-7f7ddc011000 r-xp 00000000 68:01 2722872                    /usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so
7f7ddc011000-7f7ddc210000 ---p 00005000 68:01 2722872                    /usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so
7f7ddc210000-7f7ddc211000 rw-p 00004000 68:01 2722872                    /usr/lib/openmpi/lib/openmpi/mca_paffinity_linux.so
7f7ddc211000-7f7ddc212000 r-xp 00000000 68:01 2015407                    /usr/lib/libblacsCinit-openmpi.so.1.1
7f7ddc212000-7f7ddc411000 ---p 00001000 68:01 2015407                    /usr/lib/libblacsCinit-openmpi.so.1.1
7f7ddc411000-7f7ddc412000 rw-p 00000000 68:01 2015407                    /usr/lib/libblacsCinit-openmpi.so.1.1
7f7ddc412000-7f7ddc440000 r-xp 00000000 68:01 2015408                    /usr/lib/libblacs-openmpi.so.1.1
7f7ddc440000-7f7ddc63f000 ---p 0002e000 68:01 2015408                    /usr/lib/sl-2.13.so
7f2acbec5000-7f2acbec6000 rw-p 00015000 68:01 1893133                    /lib/x86_64-linux-gnu/libnsl-2.13.so
7f2acbec6000-7f2acbec8000 rw-p 00000000 00:00 0 
7f2acbec8000-7f2acbf19000 r-xp 00000000 68:01 2722825                    /usr/lib/openmpi/lib/libopen-pal.so.0.0.0
7f2acbf19000-7f2acc119000 ---p 00051000 68:01 2722825                    /usr/lib/openmpi/lib/libopen-pal.so.0.0.0
7f2acc119000-7f2acc11c000 rw-p 00051000 68:01 2722825                    /usr/lib/openmpi/lib/libopen-pal.so.0.0.0
7f2acc11c000-7f2acc11f000 rw-p 00000000 00:00 0 
7f2acc11f000-7f2acc169000 r-xp 00000000 68:01 2722901                    /usr/lib/openmpi/lib/libopen-rte.so.0.0.0
7f2acc169000-7f2acc368000 ---p 0004a000 68:01 2722901                    /usr/lib/openmpi/lib/libopen-rte.so.0.0.0
7f2acc368000-7f2acc36b000 rw-p 00049000 68:01 2722901                    /usr/lib/openmpi/lib/libopen-rte.so.0.0.0
7f2acc36b000-7f2acc36d000 rw-p 00000000 00:00 0 
7f2acc36d000-7f2acc385000 r-xp 00000000 68:01 2722902             libblacs-openmpi.so.1.1
7f7ddc63f000-7f7ddc640000 rw-p 0002d000 68:01 2015408                    /usr/lib/libblacs-openmpi.so.1.1
7f7ddc640000-7f7ddccfd000 r-xp 00000000 68:01 2015440                    /usr/lib/libscalapack-openmpi.so.1.8.0
7f7ddccfd000-7f7ddcefd000 ---p 006bd000 68:01 2015440                    /usr/lib/libscalapack-openmpi.so.1.8.0
7f7ddcefd000-7f7ddcf00000 rw-p 006bd000 68:01 2015440                    /usr/lib/libscalapack-openmpi.so.1.8.0
7f7ddcf00000-7f7ddcf53000 rw-p 00000000 00:00 0 
7f7ddcf53000-7f7ddcf91000 r-xp 00000000 68:01 2015444                    /usr/lib/libmumps_common-4.10.0.so
7f7ddcf91000-7f7ddcf92000 rw-p 0003e000 68:01 2015444                    /usr/lib/libmumps_common-4.10.0.so
7f7ddcf92000-7f7ddcf93000 rw-p 00000000 00:00 0 
7f7ddcf93000-7f7ddcfaa000 r-xp 00000000 68:01 1893143                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7f7ddcfaa000-7f7ddd1a9000 ---p 00017000 68:01 1893143                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7f7ddd1a9000-7f7       /usr/lib/openmpi/lib/libmpi_cxx.so.0.0.1
7f2acc385000-7f2acc584000 ---p 00018000 68:01 2722902                    /usr/lib/openmpi/lib/libmpi_cxx.so.0.0.1
7f2acc584000-7f2acc587000 rw-p 00017000 68:01 2722902                    /usr/lib/openmpi/lib/libmpi_cxx.so.0.0.1
7f2acc587000-7f2acc61d000 r-xp 00000000 68:01 2722899                    /usr/lib/openmpi/lib/libmpi.so.0.0.4
7f2acc61d000-7f2acc81d000 ---p 00096000 68:01 2722899                    /usr/lib/openmpi/lib/libmpi.so.0.0.4
7f2acc81d000-7f2acc82f000 rw-p 00096000 68:01 2722899                    /usr/lib/openmpi/lib/libmpi.so.0.0.4
7f2acc82f000-7f2acc839000 rw-p 00000000 00:00 0 
7f2acc839000-7f2acc870000 r-xp 00000000 68:01 2722823                    /usr/lib/openmpi/lib/libmpi_f77.so.0.0.3
7f2acc870000-7f2acca70000 ---p 00037000 68:01 2722823                    /usr/lib/openmpi/lib/libmpi_f77.so.0.0.3
7f2acca70000-7f2acca71000 rw-p 00037000 68:01 2722823                    /usr/lib/openmpi/lib/libmpi_f77.so.0.0.3
7f2acca71000-7f2accb59000 rddd1aa000 r--p 00016000 68:01 1893143                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7f7ddd1aa000-7f7ddd1ab000 rw-p 00017000 68:01 1893143                    /lib/x86_64-linux-gnu/libpthread-2.13.so
.....
....
7f2acfebd000-7f2acfebe000 rw-p 00006000 68:01 2015337                    /usr/lib/libHYPRE_seq_mv-2.8.0b.so
7f2acfebe000-7f2acfebf000 rw-p 00000000 0000de000-7f7de00df000 rw-p 00003000 68:01 1893137                    /lib/x86_64-linux-gnu/libdl-2.13.so
7f7de00df000-7f7de025f000 r-xp 00000000 68:01 1893123                    /lib/x86_64-linux-gnu/libc-2.13.so
7f7de025f000-7f7de045f000 ---p 00180000 68:01 1893123                    /lib/x86_64-linux-gnu/libc-2.13.so
7f7de045f000-7f7de0463000 r--p 00180000 68:01 1893123                    /lib/x86_64-linux-gnu/libc-2.13.so
7f7de0463000-7f7de0464000 rw-p 00184000 68:01 1893123                    /lib/x86_64-linux-gnu/libc-2.13.so
7f7de0464000-7f7de0469000 rw-p 00000000 00:00 0 
7f7de0469000-7f7de049e000 r-xp 00000000 68:01 2607552                    /usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0
7f7de049e000-7f7de069d000 ---p 00035000 68:01 2607552                    /usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0
7f7de069d000-7f7de069e000 rw-p 00034000 68:01 2607552                    /usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0
7f7de069e000-7f7de06b3000 r-xp 00000000 68:01 1893347                    /lib/x:00 0 
7f2acfebf000-7f2acfed5000 r-xp 00000000 68:01 2015317                    /usr/lib/libHYPRE_krylov-2.8.0b.so
7f2acfed5000-7f2acfed6000 rw-p 00016000 68:01 2015317                    /usr/lib/libHYPRE_krylov-2.8.0b.so
7f2acfed6000-7f2acfedb000 r-xp 00000000 68:01 2015329                    /usr/lib/libHYPRE_multivector-2.8.0b.so
7f2acfedb000-7f2acfedc000 rw-p 00004000 68:01 2015329                    /usr/lib/libHYPRE_multivector-2.8.0b.so
7f2acfedc000-7f2acfedf000 rw-p 00000000 00:00 0 
7f2acfedf000-7f2acfeeb000 r-xp 00000000 68:01 2015319                    /usr/lib/libHYPRE_utilities-2.8.0b.so
7f2acfeeb000-7f2acfeec000 rw-p 0000b000 68:01 2015319                    /usr/lib/libHYPRE_utilities-2.8.0b.so
7f2acfeec000-7f2acff0c000 r-xp 00000000 68:01 2015330                    /usr/lib/libHYPRE_parcsr_mv-2.8.0b.so
7f2acff0c000-7f2acff0d000 rw-p 00020000 68:01 2015330                    /usr/lib/libHYPRE_parcsr_mv-2.8.0b.so
7f2acff0d000-7f2acff10000 rw-p 00000000 00:00 0 
7f2acff10000-7f2acff1e000 r-xp 0086_64-linux-gnu/libgcc_s.so.1
7f7de06b3000-7f7de08b3000 ---p 00015000 68:01 1893347                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f7de08b3000-7f7de08b4000 rw-p 00015000 68:01 1893347                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f7de08b4000-7f7de0935000 r-xp 00000000 68:01 1893130                    /lib/x86_64-linux-gnu/libm-2.13.so
7f7de0935000-7f7de0b34000 ---p 00081000 68:01 1893130                    /lib/x86_64-linux-gnu/libm-2.13.so
7f7de0b34000-7f7de0b35000 r--p 00080000 68:01 1893130                    /lib/x86_64-linux-gnu/libm-2.13.so
7f7de0b35000-7f7de0b36000 rw-p 00081000 68:01 1893130                    /lib/x86_64-linux-gnu/libm-2.13.so
7f7de0b36000-7f7de0c4a000 r-xp 00000000 68:01 2607554                    /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0
7f7de0c4a000-7f7de0e4a000 ---p 00114000 68:01 2607554                    /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0
7f7de0e4a000-7f7de0e4c000 rw-p 00114000 68:01 2607554                    /usr/lib/x86_64-linux-gnu/libgf000000 68:01 2015333                    /usr/lib/libHYPRE_IJ_mv-2.8.0b.so
7f2acff1e000-7f2acff1f000 rw-p 0000d000 68:01 2015333                    /usr/lib/libHYPRE_IJ_mv-2.8.0b.so
7f2acff1f000-7f2acff2f000 r-xp 00000000 68:01 2015322                    /usr/lib/libHYPRE_ParaSails-2.8.0b.so
7f2acff2f000-7f2acff30000 rw-p 0000f000 68:01 2015322                    /usr/lib/libHYPRE_ParaSails-2.8.0b.so
7f2acff30000-7f2acff31000 rw-p 00000000 00:00 0 
7f2acff31000-7f2acfff2000 r-xp 00000000 68:01 2015318                    /usr/lib/libHYPRE_parcsr_ls-2.8.0b.so
7f2acfff2000-7f2acfff5000 rw-p 000c1000 68:01 2015318                    /usr/lib/libHYPRE_parcsr_ls-2.8.0b.so
7f2acfff5000-7f2ad002d000 r-xp 00000000 68:01 2015321                    /usr/lib/libHYPRE_Euclid-2.8.0b.so
7f2ad002d000-7f2ad002e000 rw-p 00037000 68:01 2015321                    /usr/lib/libHYPRE_Euclid-2.8.0b.so
7f2ad002e000-7f2ad003c000 rw-p 00000000 00:00 0 
7f2ad0054000-7f2ad0056000 rw-p 00000000 00:00 0 
7f2ad0056000-7f2ad0057000 r--p 0001fortran.so.3.0.0
7f7de0e4c000-7f7de1311000 r-xp 00000000 68:01 1998872                    /usr/lib/libelmersolver-7.0.so
7f7de1311000-7f7de1511000 ---p 004c5000 68:01 1998872                    /usr/lib/libelmersolver-7.0.so
7f7de1511000-7f7de152c000 rw-p 004c5000 68:01 1998872                    /usr/lib/libelmersolver-7.0.so
7f7de152c000-7f7de1539000 rw-p 00000000 00:00 0 
7f7de1539000-7f7de1559000 r-xp 00000000 68:01 1893148                    /lib/x86_64-linux-gnu/ld-2.13.so
7f7de1568000-7f7de156f000 rw-p 00000000 00:00 0 
7f7de156f000-7f7de1583000 r-xp 00000000 68:01 2015446                    /usr/lib/libpord-4.10.0.so
7f7de1583000-7f7de1584000 rw-p 00013000 68:01 2015446                    /usr/lib/libpord-4.10.0.so
7f7de1584000-7f7de1585000 rw-p 00000000 00:00 0 
7f7de1585000-7f7de1594000 r-xp 00000000 68:01 2015328                    /usr/lib/libHYPRE_DistributedMatrixPilutSolver-2.8.0b.so
7f7de1594000-7f7de1595000 rw-p 0000e000 68:01 2015328                    /usr/lib/libHYPRE_DistributedMatrixPilut000 68:01 1893148                    /lib/x86_64-linux-gnu/ld-2.13.so
7f2ad0057000-7f2ad0058000 rw-p 00020000 68:01 1893148                    /lib/x86_64-linux-gnu/ld-2.13.so
7f2ad0058000-7f2ad0059000 rw-p 00000000 00:00 0 
7fffe0c27000-7fffe0c41000 rw-p 00000000 00:00 0                          [stack]
7fffe0df6000-7fffe0df7000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
Solver-2.8.0b.so
7f7de1595000-7f7de1596000 r-xp 00000000 68:01 2015325                    /usr/lib/libHYPRE_MatrixMatrix-2.8.0b.so
7f7de1596000-7f7de1597000 rw-p 00001000 68:01 2015325                    /usr/lib/libHYPRE_MatrixMatrix-2.8.0b.so
7f7de1597000-7f7de1598000 rw-p 00000000 00:00 0 
7f7de1598000-7f7de159b000 r-xp 00000000 68:01 2015323                    /usr/lib/libHYPRE_DistributedMatrix-2.8.0b.so
7f7de159b000-7f7de159c000 rw-p 00002000 68:01 2015323                    /usr/lib/libHYPRE_DistributedMatrix-2.8.0b.so
7f7de159c000-7f7de15b7000 r-xp 00000000 68:01 2015320                    /usr/lib/libHYPRE_parcsr_block_mv-2.8.0b.so
7f7de15b7000-7f7de15b8000 rw-p 0001a000 68:01 2015320                    /usr/lib/libHYPRE_parcsr_block_mv-2.8.0b.so
7f7de15b8000-7f7de15bf000 r-xp 00000000 68:01 2015337                    /usr/lib/libHYPRE_seq_mv-2.8.0b.so
7f7de15bf000-7f7de15c0000 rw-p 00006000 68:01 2015337                    /usr/lib/libHYPRE_seq_mv-2.8.0b.so
7f7de15c0000-7f7de15c1000 rw-p 00000000 00:00 0 
7f7de15c1000-7f7de15d7000 r-xp 00000000 68:01 2015317                    /usr/lib/libHYPRE_krylov-2.8.0b.so
7f7de15d7000-7f7de15d8000 rw-p 00016000 68:01 2015317                    /usr/lib/libHYPRE_krylov-2.8.0b.so
7f7de15d8000-7f7de15dd000 r-xp 00000000 68:01 2015329                    /usr/lib/libHYPRE_multivector-2.8.0b.so
7f7de15dd000-7f7de15de000 rw-p 00004000 68:01 2015329                    /usr/lib/libHYPRE_multivector-2.8.0b.so
7f7de15de000-7f7de15e1000 rw-p 00000000 00:00 0 
7f7de15e1000-7f7de15ed000 r-xp 00000000 68:01 2015319                    /usr/lib/libHYPRE_utilities-2.8.0b.so
7f7de15ed000-7f7de15ee000 rw-p 0000b000 68:01 2015319                    /usr/lib/libHYPRE_utilities-2.8.0b.so
7f7de15ee000-7f7de160e000 r-xp 00000000 68:01 2015330                    /usr/lib/libHYPRE_parcsr_mv-2.8.0b.so
7f7de160e000-7f7de160f000 rw-p 00020000 68:01 2015330                    /usr/lib/libHYPRE_parcsr_mv-2.8.0b.so
7f7de160f000-7f7de1612000 rw-p 00000000 00:00 0 
7f7de1612000-7f7de1620000 r-xp 00000000 68:01 2015333                    /usr/lib/libHYPRE_IJ_mv-2.8.0b.so
7f7de1620000-7f7de1621000 rw-p 0000d000 68:01 2015333                    /usr/lib/libHYPRE_IJ_mv-2.8.0b.so
7f7de1621000-7f7de1631000 r-xp 00000000 68:01 2015322                    /usr/lib/libHYPRE_ParaSails-2.8.0b.so
7f7de1631000-7f7de1632000 rw-p 0000f000 68:01 2015322                    /usr/lib/libHYPRE_ParaSails-2.8.0b.so
7f7de1632000-7f7de1633000 rw-p 00000000 00:00 0 
7f7de1633000-7f7de16f4000 r-xp 00000000 68:01 2015318                    /usr/lib/libHYPRE_parcsr_ls-2.8.0b.so
7f7de16f4000-7f7de16f7000 rw-p 000c1000 68:01 2015318                    /usr/lib/libHYPRE_parcsr_ls-2.8.0b.so
7f7de16f7000-7f7de172f000 r-xp 00000000 68:01 2015321                    /usr/lib/libHYPRE_Euclid-2.8.0b.so
7f7de172f000-7f7de1730000 rw-p 00037000 68:01 2015321                    /usr/lib/libHYPRE_Euclid-2.8.0b.so
7f7de1730000-7f7de173e000 rw-p 00000000 00:00 0 
7f7de1756000-7f7de1758000 rw-p 00000000 00:00 0 
7f7de1758000-7f7de1759000 r--p 0001f000 68:01 1893148                    /lib/x86_64-linux-gnu/ld-2.13.so
7f7de1759000-7f7de175a000 rw-p 00020000 68:01 1893148                    /lib/x86_64-linux-gnu/ld-2.13.so
7f7de175a000-7f7de175b000 rw-p 00000000 00:00 0 
7fffa0976000-7fffa098f000 rw-p 00000000 00:00 0                          [stack]
7fffa09b7000-7fffa09b8000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
MAIN: 
MAIN: -------------------------------------
MAIN:  Time: 277/1000000   686.67016152200677
MAIN:  Estimated time left: 11129 hours.
MAIN: -------------------------------------
MAIN: 

I just left the save scalars lines there to show that it just happens right after the last solver has executed.
I had to cut out some lines in the middle of the output (...) because the forum post is constraint to max 600000 characters.
The last lines are the very end of the output.
Any hints are highly appreciated,
bye Franz
Post Reply