One thing on swap: if you're not peaked out on your actual RAM, the OS
might still decide to use some swap files. Even if it's not from the
solver process and it's one of the other low-level apps you mentioned, if
it swaps on the processor your solver is on, even a tiny bit, your
performance will decrease a lot.
Also, if CG solver = Conjugate Gradient, you have a very fast algorithm
compared to say a direct solver, so you might be loading the processor in
shorter spikes which is harder for the kernel to schedule. With a
dual-processor, you could also be losing a bit because the kernel has to
schedule the parallel parts of the solver, so you might be running part
parallel and part sequential (while one waits for the other to finish) in
terms of execution sequence. 128KB to move on your larger problems is not
very much, even on 100Mbit, and it sounds like you have short bursts of
activity rather than a steady load. If you can do some runs with nothing
but the solver loaded, you'll get a better idea.
Damien
> On Nov 21, 2005, at 9:14 AM, Angel Tsankov wrote:
>
>> In fact, the issue of swapping was the first that came to my mind as
>> well. However, I also doubt that this is the reason.
>> It seems to me that other activities in the system could contribute to
>> the longer execution times when both copies of the program execeutee
>> on the same node. In this case both CPU are busy executing not only
>> the CG solver but other applications as well. In the other case one of
>> the CPUs in each node executes the CG solver while the other CPU is
>> free to execute other applications.
>> I've noticed, however, that the activity of other applications is very
>> low. Moreover, the solver has been executed 5 times for each matrix
>> size (and each type of communication) and the times are almost the
>> same for each of the 5 runs. This way I make sure the times are
>> accurate.
>> Another possible reason is that ther is that more communications are
>> overlapped with computations in the case of larger systems.
>> Of course, other ideas are welcome.
>
> These are all definite possibilities. Unfortunately, it's something
> that only you, the application developer, can reasonably answer.
> Caching effects can also have a big impact (I think that was mentioned
> in a prior mail).
>
>> And finally, I just wonder if communication time depends on the amount
>> of data to be transferred when shared memory is used.
>
> Yes, message size definitely affects communication time, regardless of
> the communication media (i.e., even with shared memory).
>
> --
> {+} Jeff Squyres
> {+} The Open MPI Project
> {+} http://www.open-mpi.org/
>
> _______________________________________________
> This list is archived at http://www.lam-mpi.org/MailArchives/lam/
>
|
