I tried to use BLCR with MPICH3. However, it seems not to work. I compile the blcr in CentOS and `make test` show not fail tests. Then, I compile mpich
with BLCR. The information is shown as follows,
$ mpichversion
MPICH Version:
3.1.2
MPICH Release date:
Mon Jul 21 16:00:21 CDT 2014
MPICH Device:
ch3:nemesis
MPICH configure:
--prefix=/home/test/develop/mpich3-blcr --with-device=ch3:nemesis CFLAGS=-fPIC --enable-checkpointing --with-blcr=/home/test/develop/blcr-0.8.5 --with-hydra-ckpointlib=blcr
MPICH CC:
gcc -fPIC -O2
MPICH CXX:
g++ -O2
MPICH F77:
gfortran -O2
MPICH FC:
gfortran -O2
After that, I compile my application like this
$ mpicc mpiblcr.c -o mpiblcr -lcr
When I firstly run the application, it seems ok to make the checkpoint files, such as context-num0-0-0.
$ mpiexec -ckpointlib blcr -ckpoint-prefix `pwd` -ckpoint-interval 2 -n 2 ./mpiblcr
5411) Step 0
5410) Step 0
5410) Step 1
5411) Step 1
[proxy:0:0@node1] requesting checkpoint
[proxy:0:0@node1] checkpoint completed
5410) Step 2
However, when I try to restart the process with checkpoint, it hangs and thereis no information printed.
$ mpiexec -ckpointlib blcr -ckpoint-prefix `pwd` -n 2 -ckpoint-num 1
The pstree shows the pmi start application process
©À©¤sshd©¤©Ð©¤3*[sshd©¤©¤©¤sshd©¤©¤©¤bash]
©¦ ©À©¤sshd©¤©¤©¤sshd©¤©¤©¤bash©¤©¤©¤mpiexec©¤©¤©¤hydra_pmi_proxy©¤©¤©¤mpiblcr
©¦ ©¸©¤sshd©¤©¤©¤sshd©¤©¤©¤bash©¤©¤©¤pstree
and `ps aux` shows the process is defunct
$ ps aux | grep osu_bw
test 15290 0.0 0.0 0 0 ? Z 21:44 0:00 [mpiblcr] <defunct>
I don't know how to identify this problem. I also see someone had the same problem like me several years ago
#1144. But, there are no solutions.