On Jul 27, 2004, at 1:55 PM, martin wood wrote:
> I can confirm the findings that .../examples/fault fails to perform
> as advertised. The program will hang if more than ONE node is killed
> and consequently fails to be anywhere close to fault tolerant.
It is possible that this is the case. Of course, that says nothing
about the MPI layer's fault tolerance - it is quite possible we have a
bug in the example ;). We will take a look at the symptom you reported
and see if we can pin down the problem.
> Fault tolerance, which may be one of the most important behaviour
> needs of a parallel system, seems to be lacking at this time in the
> LAM implementation. There may be some ways to get this example to run
> properly, but at this time it is not obvious.
While you may consider fault tolerance to be the most important
behavior of a parallel system, the MPI standard says nothing about
fault tolerance. It says quite a bit that makes real fault tolerance
difficult to implement, especially while retaining the high performance
most of our users find to be the most important behavior in a parallel
environment. So at present, the fault tolerance in LAM/MPI is pretty
minimal. But much better than most other high performance MPI
implementation out there.
There are a number of different options for fault tolerance in MPI -
LAM/MPI offer minimal fault tolerance with full MPI conformance. MPI
Software Technologies (currently Verari Systems, I believe) is
developing a redundancy approach to fault tolerance. The University of
Tennessee, Knoxville has a project called FT/MPI that has taken great
pains to provide guaranteed fault semantics in an MPI-like environment.
FT/MPI requires some non-conferment hooks to achieve fault tolerance,
but it gets the job done.
We are, of course, open to new ideas on fault tolerance in an MPI
environment. But it isn't easy, unless you are willing to throw away
the MPI part :).
Brian
|
