>
> The sysv timing is perhaps the only one that makes sense. But it does
> show that the Origin is about 10-15 times slower than your Altrix
> machine (but there's a lot of other factors here -- I assume the
> machines were idle, you didn't oversubscribe the CPUs, etc.).
>
> But you're right; clearly, a 3 second job on the Altrix should not
> take over an hour on the Origin. It certainly suggests a tcp problem,
> but anything is suspect for a difference that egregious. A few random
> questions to look into: Are there any messages in the system logs that
> indicate that things were going wrong? Can you run tcp ping-pong
> tests (without MPI) to verify bandwidth and latency (e.g., Netpipe)?
> Was someone else using the Origin machine at the time? Was memory
> full, and therefore continually swapping?
Yes the machine was and is idle, and the test was run just after a
reboot (just to be sure that it wasnt some evil process
hanging around). I repeated the test using netpipe which reveals that
the tcp communcation on its own seems to be all happy
but the lam mpi tcp communication is still playing up as soon as we run
into packet size larger than 128K (it also looks like netpipe
gives up on the long transmission times ?! and bails out earlier than it
would under normal conditions).
I take that the performace drop happens when we go from short message
(which are send in one block) to long messages which
are send in multiple blocks.
I have checked the SYSLOG, but didnt find anything of interest.
I will start a clean build of lam itself and see whether that make any
differences
FYI the IRIX version is 6.5.19, compilers are mipsPRO 7.41.
I do acknowledge that it might be a bit hard problem to debug from remote,
but well life is not supposed to be easy ;-)
best regards
-Morten
-Morten
1 0.059002 0.00012931
2 0.117840 0.00012949
3 0.178272 0.00012839
4 0.235918 0.00012936
6 0.356727 0.00012832
8 0.470894 0.00012962
12 0.700596 0.00013068
13 0.749968 0.00013225
16 0.935270 0.00013052
19 1.142463 0.00012688
21 1.217070 0.00013164
24 1.392007 0.00013154
27 1.571100 0.00013111
29 1.700666 0.00013010
32 1.840901 0.00013262
35 2.037280 0.00013107
45 2.602766 0.00013191
48 2.780584 0.00013170
51 2.956342 0.00013162
61 3.538747 0.00013151
64 3.728407 0.00013096
67 3.885831 0.00013155
93 5.385064 0.00013176
96 5.568783 0.00013152
99 5.741599 0.00013155
125 7.243096 0.00013167
128 7.351171 0.00013284
131 7.603037 0.00013145
189 10.693712 0.00013484
192 10.907770 0.00013429
195 10.898973 0.00013650
253 14.007918 0.00013780
256 14.116519 0.00013836
259 14.487493 0.00013639
381 20.563630 0.00014136
384 21.018134 0.00013939
387 20.784881 0.00014205
509 27.369711 0.00014189
512 27.717293 0.00014093
515 27.516779 0.00014279
765 43.265934 0.00013490
768 39.988551 0.00014653
771 41.431724 0.00014197
1021 53.219589 0.00014637
1024 52.564821 0.00014863
1027 53.910289 0.00014534
1533 78.114289 0.00014973
1536 79.437648 0.00014752
1539 77.795646 0.00015093
2045 103.880072 0.00015019
2048 102.026667 0.00015315
2051 94.669357 0.00016529
3069 127.811288 0.00018320
3072 133.006394 0.00017621
3075 130.535123 0.00017972
4093 165.089086 0.00018915
4096 168.962063 0.00018495
4099 172.290149 0.00018151
6141 204.354378 0.00022927
6144 205.111277 0.00022853
6147 202.817448 0.00023123
8189 252.993808 0.00024695
8192 254.829019 0.00024526
8195 261.521139 0.00023907
12285 339.599732 0.00027599
12288 328.113968 0.00028572
12291 330.240941 0.00028395
16381 386.685607 0.00032320
16384 336.827238 0.00037111
16387 307.021089 0.00040721
24573 343.081601 0.00054645
24576 366.497993 0.00051160
24579 361.307722 0.00051901
32765 433.475903 0.00057668
32768 439.097901 0.00056935
32771 390.173497 0.00064080
49149 458.248616 0.00081828
49152 402.566349 0.00093152
49155 386.122565 0.00097125
65533 408.163884 0.00122494
65536 451.997985 0.00110620
65539 393.518481 0.00127065
98301 493.659817 0.00151922
98304 447.115515 0.00167742
98307 441.775917 0.00169775
131069 404.534977 0.00247192
131072 431.340798 0.00231835
131075 399.590803 0.00250262
196605 413.837914 0.00362455
196608 428.062195 0.00350416
196611 406.576846 0.00368940
262141 405.345475 0.00493401
262144 443.990865 0.00450460
262147 410.888715 0.00486755
393213 399.781282 0.00750405
393216 433.620635 0.00691849
393219 398.223760 0.00753351
524285 416.821563 0.00959638
524288 413.838409 0.00966561
524291 404.398283 0.00989130
786429 411.769206 0.01457121
786432 403.144214 0.01488301
786435 397.730813 0.01508564
1048573 411.846084 0.01942468
1048576 411.019066 0.01946382
1048579 401.534056 0.01992365
1572861 414.542603 0.02894751
1572864 417.043406 0.02877398
1572867 414.577761 0.02894517
2097149 419.162148 0.03817133
2097152 419.261385 0.03816235
2097155 417.582328 0.03831585
3145725 415.635593 0.05774283
3145728 421.787725 0.05690066
3145731 415.286300 0.05779151
4194301 418.573985 0.07644999
4194304 423.633417 0.07553701
4194307 421.350478 0.07594633
6291453 425.220133 0.11288265
6291456 420.882738 0.11404602
6291459 424.613539 0.11304402
8388605 423.526323 0.15111216
8388608 429.352925 0.14906152
8388611 422.263122 0.15156432
1 0.040418 0.00018876
2 0.083025 0.00018379
3 0.124726 0.00018351
4 0.168151 0.00018149
6 0.250187 0.00018297
8 0.333433 0.00018305
12 0.492510 0.00018589
13 0.533353 0.00018596
16 0.657776 0.00018558
19 0.779622 0.00018593
21 0.844744 0.00018966
24 0.992372 0.00018451
27 1.105468 0.00018634
29 1.211030 0.00018270
32 1.321045 0.00018481
35 1.439750 0.00018547
45 1.873855 0.00018322
48 2.017247 0.00018154
51 2.114138 0.00018405
61 2.592621 0.00017951
64 2.638914 0.00018503
67 2.788539 0.00018331
93 3.945521 0.00017983
96 3.921084 0.00018679
99 4.021827 0.00018780
125 5.158893 0.00018486
128 5.222675 0.00018699
131 5.490159 0.00018204
189 7.766668 0.00018566
192 7.949084 0.00018428
195 7.899036 0.00018834
253 10.080122 0.00019149
256 10.160014 0.00019224
259 10.211649 0.00019351
381 14.780873 0.00019666
384 15.030214 0.00019492
387 15.143223 0.00019498
509 20.057410 0.00019361
512 19.990780 0.00019540
515 20.031052 0.00019615
765 29.598739 0.00019719
768 29.428112 0.00019911
771 30.026648 0.00019590
1021 38.937280 0.00020006
1024 38.683829 0.00020196
1027 39.104381 0.00020037
1533 56.120056 0.00020841
1536 56.164056 0.00020865
1539 56.551574 0.00020763
2045 70.021400 0.00022282
2048 69.958035 0.00022335
2051 69.354870 0.00022562
3069 97.228705 0.00024082
3072 97.108788 0.00024135
3075 97.715637 0.00024009
4093 122.639182 0.00025463
4096 124.803793 0.00025039
4099 122.781528 0.00025470
6141 166.974182 0.00028059
6144 165.833619 0.00028266
6147 165.888008 0.00028271
8189 200.543904 0.00031154
8192 187.071587 0.00033410
8195 188.048088 0.00033248
12285 265.855143 0.00035255
12288 271.505927 0.00034530
12291 256.775361 0.00036519
16381 313.260780 0.00039896
16384 328.449393 0.00038058
16387 287.023354 0.00043558
24573 361.423012 0.00051872
24576 377.584588 0.00049658
24579 411.183468 0.00045606
32765 398.200117 0.00062777
32768 431.430728 0.00057947
32771 382.022590 0.00065447
49149 402.350097 0.00093197
49152 369.486876 0.00101492
49155 366.049557 0.00102451
65533 445.029607 0.00112347
65536 412.537277 0.00121201
65539 295.062180 0.00169464
98301 367.645506 0.00203995
98304 352.314490 0.00212878
98307 343.947773 0.00218063
131069 467.118244 0.00214074
131072 369.012114 0.00270994
131075 419.769986 0.00238231
196605 0.921810 1.62720823
196608 0.899265 1.66802867
196611 0.899213 1.66815031
|
