I'm having a problem starting lamboot. I'm using ssh keys and can ssh to the
machines without incident. But lamboot still will not execute.
I have ssh keys setup on the machines and am using the command
$lamboot -d machinefile.all
I watch var/log/messages on the guest machine I recieve:
sshd session opened for user parallel
sshd session closed for user parallel
sshd session opened for user parallel
lamd started (7.1.1) uid 501, gid 501
lamd kernel initialized
sshd session closed for user parallel
While watching /var/log/messages on the host machine I recieve:
lamd: started (7.1.1), uid 503, gid 503
lamd: kernel: initialized
When I execute $lamboot -d machinefile.all I recieve:
n-1<10600> ssi:boot:open: opening
n-1<10600> ssi:boot:open: opening boot module globus
n-1<10600> ssi:boot:open: opened boot module globus
n-1<10600> ssi:boot:open: opening boot module rsh
n-1<10600> ssi:boot:open: opened boot module rsh
n-1<10600> ssi:boot:open: opening boot module slurm
n-1<10600> ssi:boot:open: opened boot module slurm
n-1<10600> ssi:boot:select: initializing boot module globus
n-1<10600> ssi:boot:globus: globus-job-run not found, globus boot will not
run
n-1<10600> ssi:boot:select: boot module not available: globus
n-1<10600> ssi:boot:select: initializing boot module rsh
n-1<10600> ssi:boot:rsh: module initializing
n-1<10600> ssi:boot:rsh:agent: /usr/bin/ssh -x -a
n-1<10600> ssi:boot:rsh:username: <same>
n-1<10600> ssi:boot:rsh:verbose: 1000
n-1<10600> ssi:boot:rsh:algorithm: linear
n-1<10600> ssi:boot:rsh:no_n: 0
n-1<10600> ssi:boot:rsh:no_profile: 0
n-1<10600> ssi:boot:rsh:fast: 0
n-1<10600> ssi:boot:rsh:ignore_stderr: 0
n-1<10600> ssi:boot:rsh:priority: 10
n-1<10600> ssi:boot:select: boot module available: rsh, priority: 10
n-1<10600> ssi:boot:select: initializing boot module slurm
n-1<10600> ssi:boot:slurm: not running under SLURM
n-1<10600> ssi:boot:select: boot module not available: slurm
n-1<10600> ssi:boot:select: finalizing boot module globus
n-1<10600> ssi:boot:globus: finalizing
n-1<10600> ssi:boot:select: closing boot module globus
n-1<10600> ssi:boot:select: finalizing boot module slurm
n-1<10600> ssi:boot:slurm: finalizing
n-1<10600> ssi:boot:select: closing boot module slurm
n-1<10600> ssi:boot:select: selected boot module rsh
LAM 7.1.1/MPI 2 C++/ROMIO - Indiana University
n-1<10600> ssi:boot:base: looking for boot schema in following directories:
n-1<10600> ssi:boot:base: <current directory>
n-1<10600> ssi:boot:base: $TROLLIUSHOME/etc
n-1<10600> ssi:boot:base: $LAMHOME/etc
n-1<10600> ssi:boot:base: /etc/lam
n-1<10600> ssi:boot:base: looking for boot schema file:
n-1<10600> ssi:boot:base: machinefile.all
n-1<10600> ssi:boot:base: found boot schema: machinefile.all
n-1<10600> ssi:boot:rsh: found the following hosts:
n-1<10600> ssi:boot:rsh: n0 168.158.222.80 (cpu=1)
n-1<10600> ssi:boot:rsh: n1 168.158.222.152 (cpu=1)
n-1<10600> ssi:boot:rsh: resolved hosts:
n-1<10600> ssi:boot:rsh: n0 168.158.222.80 --> 168.158.222.80 (origin)
n-1<10600> ssi:boot:rsh: n1 168.158.222.152 --> 168.158.222.152
n-1<10600> ssi:boot:rsh: starting RTE procs
n-1<10600> ssi:boot:base:linear: starting
n-1<10600> ssi:boot:base:server: opening server TCP socket
n-1<10600> ssi:boot:base:server: opened port 32856
n-1<10600> ssi:boot:base:linear: booting n0 (168.158.222.80)
n-1<10600> ssi:boot:rsh: starting lamd on (168.158.222.80)
n-1<10600> ssi:boot:rsh: starting on n0 (168.158.222.80): hboot -t -c
lam-conf.lamd -d -I -H 168.158.222.80 -P 32856 -n 0 -o 0
n-1<10600> ssi:boot:rsh: launching locally
hboot: performing tkill
hboot: tkill -d
tkill: setting prefix to (null)
tkill: setting suffix to (null)
tkill: got killname back: /tmp/lam-parallel_at_Solar1202.localdomain
/lam-killfile
tkill: removing socket file ...
tkill: socket file: /tmp/lam-parallel_at_Solar1202.localdomain
/lam-kernel-socketd
tkill: removing IO daemon socket file ...
tkill: IO daemon socket file: /tmp/lam-parallel_at_Solar1202.localdomain
/lam-io-socket
tkill: f_kill = "/tmp/lam-parallel_at_Solar1202.localdomain/lam-killfile"
tkill: nothing to kill: "/tmp/lam-parallel_at_Solar1202.localdomain
/lam-killfile"
hboot: booting...
hboot: fork /usr/bin/lamd
hboot: attempting to execute
n-1<10603> ssi:boot:open: opening
n-1<10603> ssi:boot:open: opening boot module globus
n-1<10603> ssi:boot:open: opened boot module globus
n-1<10603> ssi:boot:open: opening boot module rsh
n-1<10603> ssi:boot:open: opened boot module rsh
n-1<10603> ssi:boot:open: opening boot module slurm
n-1<10603> ssi:boot:open: opened boot module slurm
n-1<10603> ssi:boot:select: initializing boot module globus
n-1<10603> ssi:boot:globus: globus-job-run not found, globus boot will not
run
n-1<10603> ssi:boot:select: boot module not available: globus
n-1<10603> ssi:boot:select: initializing boot module rsh
n-1<10603> ssi:boot:rsh: module initializing
n-1<10603> ssi:boot:rsh:agent: /usr/bin/ssh -x -a
n-1<10603> ssi:boot:rsh:username: <same>
n-1<10603> ssi:boot:rsh:verbose: 1000
n-1<10603> ssi:boot:rsh:algorithm: linear
n-1<10603> ssi:boot:rsh:no_n: 0
n-1<10603> ssi:boot:rsh:no_profile: 0
n-1<10603> ssi:boot:rsh:fast: 0
n-1<10603> ssi:boot:rsh:ignore_stderr: 0
n-1<10603> ssi:boot:rsh:priority: 10
n-1<10603> ssi:boot:select: boot module available: rsh, priority: 10
n-1<10603> ssi:boot:select: initializing boot module slurm
n-1<10603> ssi:boot:slurm: not running under SLURM
n-1<10603> ssi:boot:select: boot module not available: slurm
n-1<10603> ssi:boot:select: finalizing boot module globus
n-1<10603> ssi:boot:globus: finalizing
n-1<10603> ssi:boot:select: closing boot module globus
n-1<10603> ssi:boot:select: finalizing boot module slurm
n-1<10603> ssi:boot:slurm: finalizing
n-1<10603> ssi:boot:select: closing boot module slurm
n-1<10603> ssi:boot:select: selected boot module rsh
n-1<10603> ssi:boot:send_lamd: getting node ID from command line
n-1<10603> ssi:boot:send_lamd: getting agent haddr from command line
n-1<10603> ssi:boot:send_lamd: getting agent port from command line
n-1<10603> ssi:boot:send_lamd: getting node ID from command line
n-1<10603> ssi:boot:send_lamd: connecting to 168.158.222.80:32856, node id 0
n-1<10603> ssi:boot:send_lamd: sending dli_port 33037
[1] 10603 lamd -H 168.158.222.80 -P 32856 -n 0 -o 0 -d
n-1<10600> ssi:boot:rsh: successfully launched on n0 (168.158.222.80)
n-1<10600> ssi:boot:base:server: expecting connection from finite list
n-1<10600> ssi:boot:base:server: got connection from 168.158.222.80
n-1<10600> ssi:boot:base:server: this connection is expected (n0)
n-1<10600> ssi:boot:base:server: remote lamd is at 168.158.222.80:33037
n-1<10600> ssi:boot:base:linear: booting n1 (168.158.222.152)
n-1<10600> ssi:boot:rsh: starting lamd on (168.158.222.152)
n-1<10600> ssi:boot:rsh: starting on n1 (168.158.222.152): hboot -t -c
lam-conf.lamd -d -s -I "-H 168.158.222.80 -P 32856 -n 1 -o 0"
n-1<10600> ssi:boot:rsh: launching remotely
n-1<10600> ssi:boot:rsh: attempting to execute: /usr/bin/ssh -x -a
168.158.222.152 -n 'echo $SHELL'
n-1<10600> ssi:boot:rsh: remote shell /bin/bash
n-1<10600> ssi:boot:rsh: attempting to execute: /usr/bin/ssh -x -a
168.158.222.152 -n hboot -t -c lam-conf.lamd -d -s -I '"-H 168.158.222.80 -P
32856 -n 1 -o 0"'
tkill: setting prefix to (null)
tkill: setting suffix to (null)
tkill: got killname back: /tmp/lam-parallel_at_Solar1204.localdomain
/lam-killfile
tkill: removing socket file ...
tkill: socket file: /tmp/lam-parallel_at_Solar1204.localdomain
/lam-kernel-socketd
tkill: removing IO daemon socket file ...
tkill: IO daemon socket file: /tmp/lam-parallel_at_Solar1204.localdomain
/lam-io-socket
tkill: f_kill = "/tmp/lam-parallel_at_Solar1204.localdomain/lam-killfile"
tkill: nothing to kill: "/tmp/lam-parallel_at_Solar1204.localdomain
/lam-killfile"
hboot: performing tkill
hboot: tkill -d
hboot: booting...
hboot: fork /usr/bin/lamd
[1] 15798 lamd -H 168.158.222.80 -P 32856 -n 1 -o 0 -d
n-1<10600> ssi:boot:rsh: successfully launched on n1 (168.158.222.152)
n-1<10600> ssi:boot:base:server: expecting connection from finite list
-----------------------------------------------------------------------------
The lamboot agent timed out while waiting for the newly-booted process
to call back and indicated that it had successfully booted.
*** PLEASE READ THIS ENTIRE MESSAGE, FOLLOW ITS SUGGESTIONS, AND
*** CONSULT THE "BOOTING LAM" SECTION OF THE LAM/MPI FAQ
*** (http://www.lam-mpi.org/faq/) BEFORE POSTING TO THE LAM/MPI USER'S
*** MAILING LIST.
As far as LAM could tell, the remote process started properly, but
then never called back. Possible reasons that this may happen:
- There are network filters between the lamboot agent host and
the remote host such that communication on random TCP ports
is blocked
- Network routing from the remote host to the local host isn't
properly configured (this is uncommon)
You can check these things by watching the output from "lamboot -d".
1. On the command line for hboot, there are two important parameters:
one is the IP address of where the lamboot agent was invoked, the
other is the port number that the lamboot agent is expecting the
newly-booted process to call back on (this will be a random
integer).
2. Manually login to the remote machine and try to telnet to the port
indicated on the hboot command line. For example,
telnet <ipnumber> <portnumber>
If all goes well, you should get a "Connection refused" error. If
you get any other kind of error, it could indicate either of the
two conditions above. Consult with your system/network
administrator.
-----------------------------------------------------------------------------
n-1<10600> ssi:boot:base:server: failed to connect to remote lamd!
n-1<10600> ssi:boot:base:server: closing server socket
n-1<10600> ssi:boot:base:linear: aborted!
n-1<10616> ssi:boot:open: opening
n-1<10616> ssi:boot:open: opening boot module globus
n-1<10616> ssi:boot:open: opened boot module globus
n-1<10616> ssi:boot:open: opening boot module rsh
n-1<10616> ssi:boot:open: opened boot module rsh
n-1<10616> ssi:boot:open: opening boot module slurm
n-1<10616> ssi:boot:open: opened boot module slurm
n-1<10616> ssi:boot:select: initializing boot module globus
n-1<10616> ssi:boot:globus: globus-job-run not found, globus boot will not
run
n-1<10616> ssi:boot:select: boot module not available: globus
n-1<10616> ssi:boot:select: initializing boot module rsh
n-1<10616> ssi:boot:rsh: module initializing
n-1<10616> ssi:boot:rsh:agent: /usr/bin/ssh -x -a
n-1<10616> ssi:boot:rsh:username: <same>
n-1<10616> ssi:boot:rsh:verbose: 1000
n-1<10616> ssi:boot:rsh:algorithm: linear
n-1<10616> ssi:boot:rsh:no_n: 0
n-1<10616> ssi:boot:rsh:no_profile: 0
n-1<10616> ssi:boot:rsh:fast: 0
n-1<10616> ssi:boot:rsh:ignore_stderr: 0
n-1<10616> ssi:boot:rsh:priority: 10
n-1<10616> ssi:boot:select: boot module available: rsh, priority: 10
n-1<10616> ssi:boot:select: initializing boot module slurm
n-1<10616> ssi:boot:slurm: not running under SLURM
n-1<10616> ssi:boot:select: boot module not available: slurm
n-1<10616> ssi:boot:select: finalizing boot module globus
n-1<10616> ssi:boot:globus: finalizing
n-1<10616> ssi:boot:select: closing boot module globus
n-1<10616> ssi:boot:select: finalizing boot module slurm
n-1<10616> ssi:boot:slurm: finalizing
n-1<10616> ssi:boot:select: closing boot module slurm
n-1<10616> ssi:boot:select: selected boot module rsh
n-1<10616> ssi:boot:base: looking for boot schema in following directories:
n-1<10616> ssi:boot:base: <current directory>
n-1<10616> ssi:boot:base: $TROLLIUSHOME/etc
n-1<10616> ssi:boot:base: $LAMHOME/etc
n-1<10616> ssi:boot:base: /etc/lam
n-1<10616> ssi:boot:base: looking for boot schema file:
n-1<10616> ssi:boot:base: machinefile.all
n-1<10616> ssi:boot:base: found boot schema: machinefile.all
n-1<10616> ssi:boot:rsh: found the following hosts:
n-1<10616> ssi:boot:rsh: n0 168.158.222.80 (cpu=1)
n-1<10616> ssi:boot:rsh: n1 168.158.222.152 (cpu=1)
n-1<10616> ssi:boot:rsh: resolved hosts:
n-1<10616> ssi:boot:rsh: n0 168.158.222.80 --> 168.158.222.80 (origin)
n-1<10616> ssi:boot:rsh: n1 168.158.222.152 --> 168.158.222.152
n-1<10616> ssi:boot:rsh: starting RTE procs
n-1<10616> ssi:boot:base:linear: starting
n-1<10616> ssi:boot:base:linear: booting n0 (168.158.222.80)
n-1<10616> ssi:boot:rsh: starting wipe on (168.158.222.80)
n-1<10616> ssi:boot:rsh: starting on n0 (168.158.222.80): tkill -d
n-1<10616> ssi:boot:rsh: launching locally
tkill: setting prefix to (null)
tkill: setting suffix to (null)
tkill: got killname back: /tmp/lam-parallel_at_Solar1202.localdomain
/lam-killfile
tkill: removing socket file ...
tkill: socket file: /tmp/lam-parallel_at_Solar1202.localdomain
/lam-kernel-socketd
tkill: removing IO daemon socket file ...
tkill: IO daemon socket file: /tmp/lam-parallel_at_Solar1202.localdomain
/lam-io-socket
tkill: f_kill = "/tmp/lam-parallel_at_Solar1202.localdomain/lam-killfile"
tkill: killing LAM...
tkill: killing PID (SIGHUP) 10603 ...
tkill: killed
tkill: all finished
n-1<10616> ssi:boot:rsh: successfully launched on n0 (168.158.222.80)
n-1<10616> ssi:boot:base:linear: booting n1 (168.158.222.152)
n-1<10616> ssi:boot:rsh: starting wipe on (168.158.222.152)
n-1<10616> ssi:boot:rsh: starting on n1 (168.158.222.152): tkill -d
n-1<10616> ssi:boot:rsh: launching remotely
n-1<10616> ssi:boot:rsh: attempting to execute: /usr/bin/ssh -x -a
168.158.222.152 -n 'echo $SHELL'
n-1<10616> ssi:boot:rsh: remote shell /bin/bash
n-1<10616> ssi:boot:rsh: attempting to execute: /usr/bin/ssh -x -a
168.158.222.152 -n tkill -d
tkill: setting prefix to (null)
tkill: setting suffix to (null)
tkill: got killname back: /tmp/lam-parallel_at_Solar1204.localdomain
/lam-killfile
tkill: removing socket file ...
tkill: socket file: /tmp/lam-parallel_at_Solar1204.localdomain
/lam-kernel-socketd
tkill: removing IO daemon socket file ...
tkill: IO daemon socket file: /tmp/lam-parallel_at_Solar1204.localdomain
/lam-io-socket
tkill: f_kill = "/tmp/lam-parallel_at_Solar1204.localdomain/lam-killfile"
tkill: killing LAM...
tkill: killing PID (SIGHUP) 15812 ...
tkill: killed
tkill: all finished
n-1<10616> ssi:boot:rsh: successfully launched on n1 (168.158.222.152)
n-1<10616> ssi:boot:base:linear: finished
n-1<10616> ssi:boot:rsh: all RTE procs started
n-1<10616> ssi:boot:rsh: finalizing
n-1<10616> ssi:boot: Closing
lamboot did NOT complete successfully
[parallel_at_Solar1202 ~]$
|
