Since the last message was getting already too long, I'll append here
separately some older discussion on elliptic solvers, on which I never 
really got any feedback. That's the problem with long emails. We really
need to talk more often.

Bernd



On Tue, 17 Feb 1998, Wai-Mo Suen wrote:

>  Bernd, 
>   
>  Dan, a grad student of Paul Saylor in the CS department of UI, 
>  will be coming to St. Louis tomorrow, to work with people 
>  here on elliptic solver.  One thing that we are planning on have him 
>  look at is the computer science aspect of BAM (as we talked about when 
>  we saw each other in Potsdam).  Dan is familar with multi-grid methods, 
>  but not GR, nor Cactus features (e.g., PUGH). 
>   
>  Any suggestions on what he should be looking at?  How well separated is 
>  the multi-grid routine used in BAM with other things specific to PUGH 
and
>  GR?  Malcolm, Ed and Mark would be able to help him on PUGH and GR, but 
>  it would be good to have a clean piece cut out for him, and specific 
>  instructions on what he should be working on.  What about we ask him 
>  to look at multi-grid part of BAM with the maximal 
>  slicing equation as 
>  a specific example, and see what he can do to improve the speed? 
>  Is there anything we have to do before hand to separate out the 
multi-grid
>  solver part of BAM for this purpose? 
>   
>  We also plan on working with him on the PETSc routine and investigate 
>  the various options in it for solving the 
>  maximal equation.  He is said to be familar with PETSc.   
>  Any comment (Paul?) ? 
>   
>  Wai-Mo 
>   
>   


It would be great if Dan works with us on BAM_Elliptic and PETSc!

About BAM_Elliptic:  
By construction (and its history) BAM_Elliptic is very cleanly separated
from PUGH. This means that to experiment with different multigrid
algorithms
for maximal slicing, one would never have to look at PUGH.
On the other hand, if one focusses on optimizing the parallelization, then
one has to look at PUGH and BAM_Elliptic at the same time.

Here a few interesting things to do. Let's make maximally sliced
Schwarzschild our basic test case.

- Optimize the parallelization
  I never systematically studied scaling, where the bottlenecks are etc.
  We know that there is a bottleneck on the T3E, as reported by Mark,
  and I can confirm that going from 32 to 64 processors actually increased
  the run time. (Making the multigrid stack permanent should help.)

Getting BAM_Elliptic to be useful on the T3E I would call our most
pressing problem.


- Implement a direct exact solver for the coarsest grid
  One could even call PETSc for that. The fewer levels of coarsening
  are needed before the exact solver is called, the better for
  communication overhead. 

- Implement linear multigrid
  In case you didn't know, BAM_Elliptic solves the linear maximal slicing
  equation with the FAS scheme which is designed to handle non-linear
  equations and non-uniform meshes (e.g. BoxInBox). Should give a
  significant speedup depending on communication vs computation costs.	

- Use BAM_Elliptic as a preconditioner for PETSc
  Multigrid is great to get you down below the truncation error, 
  but to beat the error down to machine precision there are better
  techniques.

The last three projects are good examples for things that take
less than a day to implement (but of course many more days to evaluate),
and require very little knowledge of PUGH or the physics involved. 

Here are a couple of nitty gritty PUGH/BAM detail things that
aren't really hard but you really have to go sit down and concentrate:

- Octants with stencil width 2
  Stupid border width counting. I still didn't get around to make a
  serious attempt (i.e. stop reading cactus mail for a few days :)

- Other processor numbers than 2^n
  Again, just a few messy grid decomposition headaches.


This essentially completes my technical BAM_Elliptic to-do list. Dan or
anybody else would be very welcome to contribute to that. 
Ed E., how are those vector elliptic equations coming along? Which reminds
me:

- We should have a uniform way to set up the elliptic equations.
  Currently, some coefficients are computed centrally like Mlinear for
  maximal slicing, but e.g. the covariant Laplacian is computed in five
  different places. Also, Friedbert suggested that the matrix
  generation for PETSc could be automated.



About PETSc:  

Basically, I would say we have to try harder to coordinate our ellitpic
solver effort, and Wai-Mo's mail is just about that. I am not attached to
BAM_Elliptic for life. In particular, anytime that, say, the NASA
Grand Challenge wants to have BAM_Elliptic as a public code, that would
be fine with me. The way I see it, BAM_Elliptic gives the group
a rather simple and rather efficient multigrid solver that is completely
under our control. E.g., the multigrid broke for steep gradients in strong
Brill wave data. An hour later everything was working again. 

PETSc has all the advantages and disadvantages of a big professional
package. It doesn't do (yet?) all the things we would like it to do. For
now it is slower on small machines like the AEI Origin. When PETSc broke
on the ZIB T3E, it took quite some time to find a work around, which sort
of worked, but the real problem was out of our control. On the other hand, 
we don't have to maintain PETSc itself, only the interface, and in
principle PETSc offers many more options for elliptic solves.


In conclusion, it would be great if Dan works with us on BAM_Elliptic and
PETSc. All we need is a clear cut task list of what Dan, Friedbert, and 
EdE would like to work on. 

Best wishes,
Bernd


PS: I gave Dan AFS access to thorn_BAM_Elliptic. A simple parameter file
fitting Paul's minimal thorn list should be bam_bhmax.par.  
I think I'll have to start calling Ed E. "Eddy" ... :)

Created for the WUGRAV CoCoBoard Page Projects Page.

Created by The CoCoBoard.