Hypre ILU Preconditioner

Franz Pichler · Post by **Franz Pichler** » 12 Jun 2012, 19:02

Hello,

I am not definitly sure if i make the mistake or if its in elmer or hypre but,

I tested the heateq-par example in the last version (6.2) and also in the new version (7.0):

I used

HYPRE
with
ILU0 Preconditioning

THen i start that in serial and then parallel for 4 processors

What i observe is
For serial i need 10 iterations. (naturally this is without hypre)
for parallel i need 37

Now if i change the preconditioner it really help the sderial version

for ilu8 i only need 1 iteration, basically i have inverted the matrix.
the parallel run doesnt change tho.
it sais it uses ilu8 preconditioner but still needs 37 iterations (with same residuals)

I have the same behaviour for different examples and number of processors.

Now with Elmer7.0 i can change to CG which behaves perfectly. The better the preconditioner the less iterations are needed. also in parallel.

So there are my two questions:
1) could someone check if my results for the heateq-par example with hypre and ilun preconditioning are the same for everyone?

2) if yes, could it be that the bicgstab does not get the preconditioned system? because the preconditioner has no effect.

thanks, and as always, if this is my mistake i am sorry

best regards franz

raback · Post by **raback** » 13 Jun 2012, 23:15

Hi Franz

Now the problem with parallel preconditioners is a hard one. In Elmer the ILUn is performed only partitionwise. The reason for this is that ILU is highly sequential and doing that really in parallel results to some performance bottle-necks. Of course the partitionwise ILUn is not the same as the true ILUn which may be seen often in worse convergence performance.

Now I've understood that ILUn in Hypre package should really be a true one. I don't know how they do it, but I've seen some results that show good scaling of the overall problem. Could it be that they still make some compromises.

Now there are some preconditioners that are exactly the same in parallel. In Elmer these are currently "none" (which is actually diagonal as scaling is performed on default) and Vanka. Using these should give the same number of iterations as the serial problem. With these cheaper precontioners you may spent some resources for the Krylov method. My personal favorite is "BiCGStabl" with "BicgStabl Polynomial Degree". Also Parasails of Hypre should be the same in parallel.

Ultimate it would be nice to get the multigrid field also sorted. There is a older link to Boomer AMG of Hypre, and Jonas recently wrote a link for ML of Trilinos. Elmer itself includes only geometric multigrid in parallel but it would be nice to parallellize the unsmoothed agglomoration AMG. To get some order to the multiscale business and the different options would be nice. This becomes increasingly important as for the vector valued problems the block preconditioners bear a great promise. To maximize their outcome they should be combined with multilevel methods for the individual components.

-Peter

Franz Pichler · Post by **Franz Pichler** » 14 Jun 2012, 14:25

Hello,

I see your point but the thing is the preconditioning has no effect at all not just a to little effect.

This is the sif file i use:

Code: Select all

Header
  CHECK KEYWORDS Warn
  Mesh DB "." "Mesh"
  Include Path ""
  Results Directory ""
End

Simulation
  Max Output Level = 3
  Coordinate System = "Cartesian 2D"
  Coordinate Mapping(3) = 1 2 3

  Simulation Type = "Steady State"
  Steady State Max Iterations = 1
  Output Intervals = 1

  Solver Input File = "TempDist.sif"
! Output File = "TempDist.dat"
!  Post File = "TempDist.ep"
End

Constants
  Gravity(4) = 0 -1 0 9.82
  Stefan Boltzmann = 5.67e-08
End

Body 1
  Name = "Body1"

  Body Force = 1
  Equation = 1
  Material = 1
End

Equation 1
  Name = "Equation1"

  Heat Equation = True
End

Solver 1
  Exec Solver = "Always"
  Equation = "Heat Equation"
  Variable = "Temperature"
  Variable Dofs = 1
  Linear System Solver = "Iterative"
  Linear System Use Hypre=True
  Linear System Iterative Method = "BiCGStab"
!  Linear System Iterative Method=CG
  Linear System Max Iterations = 350
  Linear System Convergence Tolerance = 1.0e-08
  Linear System Abort Not Converged = True
  Linear System Preconditioning = "ILU2"
  Linear System Residual Output = 1
  Steady State Convergence Tolerance = 1.0e-05
  Stabilize = True
  Nonlinear System Convergence Tolerance = 1.0e-05
  Nonlinear System Max Iterations = 1
  Nonlinear System Newton After Iterations = 3
  Nonlinear System Newton After Tolerance = 1.0e-02
  Nonlinear System Relaxation Factor = 1.0
  Adaptive Error Limit = 0.1
  Adaptive Max Change = 2
  Adaptive Coarsening = Logical True
End

Material 1
  Name = "Material1"

  Density = 1
  Heat Conductivity = 1
End

Body Force 1
  Name = "BodyForce1"

  Heat Source = 1
End

Boundary Condition 1
  Name = "Constraint1"
  Target Boundaries(6) = 1 2 3 4 5 6

  Temperature = 0
End

$fprintf( stderr, "TEST CASE 1\n");
RUN
$fprintf( stderr, "END TEST CASE 1: Target NRM=0.768016492512E-01\n ");

!End Of File

and here is the output for ILU0

Code: Select all

ELMER SOLVER (v 7.0) STARTED AT: 2012/06/14 13:17:16
ELMER SOLVER (v 7.0) STARTED AT: 2012/06/14 13:17:16
ELMER SOLVER (v 7.0) STARTED AT: 2012/06/14 13:17:16
ELMER SOLVER (v 7.0) STARTED AT: 2012/06/14 13:17:16
ParCommInit:  Initialize #PEs:            4
MAIN:
MAIN: =============================================================
MAIN: ElmerSolver finite element software, Welcome!
MAIN: This program is free software licensed under (L)GPL
MAIN: Copyright 1st April 1995 - , CSC - IT Center for Science Ltd.
MAIN: Webpage http://www.csc.fi/elmer, Email elmeradm@csc.fi
MAIN: Library version: 7.0 (Rev: 5739)
MAIN:  Running in parallel using 4 tasks.
MAIN:  HYPRE library linked in.
MAIN: =============================================================
MAIN:
MAIN:
MAIN: -------------------------------------
MAIN: Reading Model: TempDist.sif
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
MAIN:
MAIN: -------------------------------------
MAIN:  Steady state iteration:            1
MAIN: -------------------------------------
MAIN:
SParIterSolver: Hypre: BiCGStab
SParIterSolver: Preconditioner: ILU0
HYPRE Setup
SolveHypre: using ILU0
create preconditioner...setup time: 0.000350952
HYPRE Solve
L2 norm of b: 5.171692e-01
Initial L2 norm of residual: 5.171692e-01
=============================================

Iters     resid.norm     conv.rate  rel.res.norm
-----    ------------    ---------- ------------
    1    4.939721e-01    0.955146   9.551460e-01
    2    3.589250e-01    0.726610   6.940185e-01
    3    2.465940e-01    0.687035   4.768149e-01
    4    1.607513e-01    0.651887   3.108293e-01
    5    1.068923e-01    0.664955   2.066874e-01
    6    7.846537e-02    0.734060   1.517209e-01
    7    5.768143e-02    0.735120   1.115330e-01
    8    3.339941e-02    0.579032   6.458120e-02
    9    2.104473e-02    0.630093   4.069215e-02
   10    1.439604e-02    0.684069   2.783624e-02
   11    1.047469e-02    0.727609   2.025389e-02
   12    7.709063e-03    0.735971   1.490627e-02
   13    5.674539e-03    0.736087   1.097231e-02
   14    3.637802e-03    0.641074   7.034065e-03
   15    2.033213e-03    0.558913   3.931428e-03
   16    9.605813e-04    0.472445   1.857383e-03
   17    4.506560e-04    0.469149   8.713898e-04
   18    2.237595e-04    0.496520   4.326621e-04
   19    1.458862e-04    0.651978   2.820860e-04
   20    1.161499e-04    0.796168   2.245879e-04
   21    9.252050e-05    0.796561   1.788979e-04
   22    7.140786e-05    0.771806   1.380745e-04
   23    5.486561e-05    0.768341   1.060883e-04
   24    4.408417e-05    0.803494   8.524129e-05
   25    3.585032e-05    0.813224   6.932029e-05
   26    2.834224e-05    0.790571   5.480264e-05
   27    2.067883e-05    0.729612   3.998465e-05
   28    1.374008e-05    0.664452   2.656786e-05
   29    7.011713e-06    0.510311   1.355787e-05
   30    2.345912e-06    0.334571   4.536064e-06
   31    7.398701e-07    0.315387   1.430615e-06
   32    3.341533e-07    0.451638   6.461199e-07
   33    1.821325e-07    0.545057   3.521719e-07
   34    7.945760e-08    0.436263   1.536395e-07
   35    1.927780e-08    0.242617   3.727561e-08
   36    2.812546e-08    1.458956   5.438348e-08
   37    3.243349e-09    0.115317   6.271350e-09


Final L2 norm of residual: 3.243349e-09

solve time: 0.00244713
ComputeChange: NS (ITER=1) (NRM,RELC): ( 0.79838843E-01  2.0000000     ) :: heat equation
ComputeChange: SS (ITER=1) (NRM,RELC): ( 0.79838843E-01  2.0000000     ) :: heat equation
END TEST CASE 1: Target NRM=0.768016492512E-01
 END TEST CASE 1: Target NRM=0.768016492512E-01
 END TEST CASE 1: Target NRM=0.768016492512E-01
 ElmerSolver: *** Elmer Solver: ALL DONE ***
END TEST CASE 1: Target NRM=0.768016492512E-01
 ElmerSolver: The end
SOLVER TOTAL TIME(CPU,REAL):         0.13        1.14
ELMER SOLVER FINISHED AT: 2012/06/14 13:17:17

Here is the output i change to ILU1

Code: Select all

ELMER SOLVER (v 7.0) STARTED AT: 2012/06/14 13:18:24
ELMER SOLVER (v 7.0) STARTED AT: 2012/06/14 13:18:24
ELMER SOLVER (v 7.0) STARTED AT: 2012/06/14 13:18:24
ELMER SOLVER (v 7.0) STARTED AT: 2012/06/14 13:18:24
ParCommInit:  Initialize #PEs:            4
MAIN:
MAIN: =============================================================
MAIN: ElmerSolver finite element software, Welcome!
MAIN: This program is free software licensed under (L)GPL
MAIN: Copyright 1st April 1995 - , CSC - IT Center for Science Ltd.
MAIN: Webpage http://www.csc.fi/elmer, Email elmeradm@csc.fi
MAIN: Library version: 7.0 (Rev: 5739)
MAIN:  Running in parallel using 4 tasks.
MAIN:  HYPRE library linked in.
MAIN: =============================================================
MAIN:
MAIN:
MAIN: -------------------------------------
MAIN: Reading Model: TempDist.sif
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
MAIN:
MAIN: -------------------------------------
MAIN:  Steady state iteration:            1
MAIN: -------------------------------------
MAIN:
SParIterSolver: Hypre: BiCGStab
SParIterSolver: Preconditioner: ILU1
HYPRE Setup
SolveHypre: using ILU1
create preconditioner...setup time: 0.000415802
HYPRE Solve
L2 norm of b: 5.171692e-01
Initial L2 norm of residual: 5.171692e-01
=============================================

Iters     resid.norm     conv.rate  rel.res.norm
-----    ------------    ---------- ------------
    1    4.939721e-01    0.955146   9.551460e-01
    2    3.589250e-01    0.726610   6.940185e-01
    3    2.465940e-01    0.687035   4.768149e-01
    4    1.607513e-01    0.651887   3.108293e-01
    5    1.068923e-01    0.664955   2.066874e-01
    6    7.846537e-02    0.734060   1.517209e-01
    7    5.768143e-02    0.735120   1.115330e-01
    8    3.339941e-02    0.579032   6.458120e-02
    9    2.104473e-02    0.630093   4.069215e-02
   10    1.439604e-02    0.684069   2.783624e-02
   11    1.047469e-02    0.727609   2.025389e-02
   12    7.709063e-03    0.735971   1.490627e-02
   13    5.674539e-03    0.736087   1.097231e-02
   14    3.637802e-03    0.641074   7.034065e-03
   15    2.033213e-03    0.558913   3.931428e-03
   16    9.605813e-04    0.472445   1.857383e-03
   17    4.506560e-04    0.469149   8.713898e-04
   18    2.237595e-04    0.496520   4.326621e-04
   19    1.458862e-04    0.651978   2.820860e-04
   20    1.161499e-04    0.796168   2.245879e-04
   21    9.252050e-05    0.796561   1.788979e-04
   22    7.140786e-05    0.771806   1.380745e-04
   23    5.486561e-05    0.768341   1.060883e-04
   24    4.408417e-05    0.803494   8.524129e-05
   25    3.585032e-05    0.813224   6.932029e-05
   26    2.834224e-05    0.790571   5.480264e-05
   27    2.067883e-05    0.729612   3.998465e-05
   28    1.374008e-05    0.664452   2.656786e-05
   29    7.011713e-06    0.510311   1.355787e-05
   30    2.345912e-06    0.334571   4.536064e-06
   31    7.398701e-07    0.315387   1.430615e-06
   32    3.341533e-07    0.451638   6.461199e-07
   33    1.821325e-07    0.545057   3.521719e-07
   34    7.945760e-08    0.436263   1.536395e-07
   35    1.927780e-08    0.242617   3.727561e-08
   36    2.812546e-08    1.458956   5.438348e-08
   37    3.243349e-09    0.115317   6.271350e-09


Final L2 norm of residual: 3.243349e-09

solve time: 0.00151515
ComputeChange: NS (ITER=1) (NRM,RELC): ( 0.79838843E-01  2.0000000     ) :: heat equation
ComputeChange: SS (ITER=1) (NRM,RELC): ( 0.79838843E-01  2.0000000     ) :: heat equation
END TEST CASE 1: Target NRM=0.768016492512E-01
 END TEST CASE 1: Target NRM=0.768016492512E-01
 END TEST CASE 1: Target NRM=0.768016492512E-01
 END TEST CASE 1: Target NRM=0.768016492512E-01
 ElmerSolver: *** Elmer Solver: ALL DONE ***
ElmerSolver: The end
SOLVER TOTAL TIME(CPU,REAL):         0.12        1.13
ELMER SOLVER FINISHED AT: 2012/06/14 13:18:25

Here for ILU2

Code: Select all


ELMER SOLVER (v 7.0) STARTED AT: 2012/06/14 13:18:59
ELMER SOLVER (v 7.0) STARTED AT: 2012/06/14 13:18:59
ELMER SOLVER (v 7.0) STARTED AT: 2012/06/14 13:18:59
ELMER SOLVER (v 7.0) STARTED AT: 2012/06/14 13:18:59
ParCommInit:  Initialize #PEs:            4
MAIN:
MAIN: =============================================================
MAIN: ElmerSolver finite element software, Welcome!
MAIN: This program is free software licensed under (L)GPL
MAIN: Copyright 1st April 1995 - , CSC - IT Center for Science Ltd.
MAIN: Webpage http://www.csc.fi/elmer, Email elmeradm@csc.fi
MAIN: Library version: 7.0 (Rev: 5739)
MAIN:  Running in parallel using 4 tasks.
MAIN:  HYPRE library linked in.
MAIN: =============================================================
MAIN:
MAIN:
MAIN: -------------------------------------
MAIN: Reading Model: TempDist.sif
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
TEST CASE 1
MAIN:
MAIN: -------------------------------------
MAIN:  Steady state iteration:            1
MAIN: -------------------------------------
MAIN:
TEST CASE 1
TEST CASE 1
SParIterSolver: Hypre: BiCGStab
SParIterSolver: Preconditioner: ILU2
HYPRE Setup
SolveHypre: using ILU2
create preconditioner...setup time: 0.000448942
HYPRE Solve
L2 norm of b: 5.171692e-01
Initial L2 norm of residual: 5.171692e-01
=============================================

Iters     resid.norm     conv.rate  rel.res.norm
-----    ------------    ---------- ------------
    1    4.939721e-01    0.955146   9.551460e-01
    2    3.589250e-01    0.726610   6.940185e-01
    3    2.465940e-01    0.687035   4.768149e-01
    4    1.607513e-01    0.651887   3.108293e-01
    5    1.068923e-01    0.664955   2.066874e-01
    6    7.846537e-02    0.734060   1.517209e-01
    7    5.768143e-02    0.735120   1.115330e-01
    8    3.339941e-02    0.579032   6.458120e-02
    9    2.104473e-02    0.630093   4.069215e-02
   10    1.439604e-02    0.684069   2.783624e-02
   11    1.047469e-02    0.727609   2.025389e-02
   12    7.709063e-03    0.735971   1.490627e-02
   13    5.674539e-03    0.736087   1.097231e-02
   14    3.637802e-03    0.641074   7.034065e-03
   15    2.033213e-03    0.558913   3.931428e-03
   16    9.605813e-04    0.472445   1.857383e-03
   17    4.506560e-04    0.469149   8.713898e-04
   18    2.237595e-04    0.496520   4.326621e-04
   19    1.458862e-04    0.651978   2.820860e-04
   20    1.161499e-04    0.796168   2.245879e-04
   21    9.252050e-05    0.796561   1.788979e-04
   22    7.140786e-05    0.771806   1.380745e-04
   23    5.486561e-05    0.768341   1.060883e-04
   24    4.408417e-05    0.803494   8.524129e-05
   25    3.585032e-05    0.813224   6.932029e-05
   26    2.834224e-05    0.790571   5.480264e-05
   27    2.067883e-05    0.729612   3.998465e-05
   28    1.374008e-05    0.664452   2.656786e-05
   29    7.011713e-06    0.510311   1.355787e-05
   30    2.345912e-06    0.334571   4.536064e-06
   31    7.398701e-07    0.315387   1.430615e-06
   32    3.341533e-07    0.451638   6.461199e-07
   33    1.821325e-07    0.545057   3.521719e-07
   34    7.945760e-08    0.436263   1.536395e-07
   35    1.927780e-08    0.242617   3.727561e-08
   36    2.812546e-08    1.458956   5.438348e-08
   37    3.243349e-09    0.115317   6.271350e-09


Final L2 norm of residual: 3.243349e-09

solve time: 0.00160813
ComputeChange: NS (ITER=1) (NRM,RELC): ( 0.79838843E-01  2.0000000     ) :: heat equation
ComputeChange: SS (ITER=1) (NRM,RELC): ( 0.79838843E-01  2.0000000     ) :: heat equation
END TEST CASE 1: Target NRM=0.768016492512E-01
 END TEST CASE 1: Target NRM=0.768016492512E-01
 END TEST CASE 1: Target NRM=0.768016492512E-01
 END TEST CASE 1: Target NRM=0.768016492512E-01
 ElmerSolver: *** Elmer Solver: ALL DONE ***
ElmerSolver: The end
SOLVER TOTAL TIME(CPU,REAL):         0.12        1.14
ELMER SOLVER FINISHED AT: 2012/06/14 13:19:00

So as you can see somehow the preconditioner is recognized, because it says ILu0 ILU1 or ILu2 in the output. But the iterations and also the residuals do not change at all. So what i believe is that the system does not really get preconditioned but is the same all the time.

best regards
Franz

raback · Post by **raback** » 15 Jun 2012, 22:31

Hi Franz

Indeed there was a bug. It's the tiniest of bugs: one missing asterisk (*)

Juha fixed it today on svn. Thank you for reporting!

-Peter

Elmer Discussion Forum

Hypre ILU Preconditioner

Hypre ILU Preconditioner

Re: Hypre ILU Preconditioner

Re: Hypre ILU Preconditioner

Re: Hypre ILU Preconditioner