May 2017 - discuss - mailman3.cels.anl.gov

ULFM revoke doesn't work properly?
by Nils-Arne Dreier 18 May '17

18 May '17

Dear MPICH community. I'm currently playing around with the ULFM features for fault-tolerance. I know that these features are experimental, but want to discuss the following example. I used the following minimal example, which deadlock for all nonzero ranks at MPI_Ssend. For my understanding MPI_Ssend should return MPI_ERR_REVOKED, shouldn't it? If i substitute MPI_Ssend with MPI_Send. All ranks reach the MPIX_Comm_shrink command but then deadlock. I observed very rarly that the shrink succeed, but can't determine the reason. #include <iostream> #include <mpi.h> void checkMPIresult(int result){ if(result!=MPI_SUCCESS){ int len; char msg[MPI_MAX_ERROR_STRING]; MPI_Error_string(result, msg, &len); std::cout << msg << std::endl; } } int main(int argc, char** argv){ checkMPIresult(MPI_Init(&argc,&argv)); MPI_Comm comm, new_comm; checkMPIresult(MPI_Comm_dup(MPI_COMM_WORLD,&comm)); MPI_Comm_set_errhandler(comm, MPI_ERRORS_RETURN); int rank = -1; checkMPIresult(MPI_Comm_rank(comm,&rank)); int error = 1; if(rank==0){ std::cout << "revoking..." << std::endl; checkMPIresult(MPIX_Comm_revoke(comm)); error = 0; }else{ checkMPIresult(MPI_Ssend(&error,1,MPI_INT,0,0,comm)); } std::cout << rank << "\twaiting for agree..." << std::endl; checkMPIresult(MPIX_Comm_agree(comm,&error)); std::cout << rank << "\tagreed on " << error << std::endl; checkMPIresult(MPIX_Comm_shrink(comm,&new_comm)); std::cout << rank << "\tcomm shrinked" << std::endl; checkMPIresult(MPI_Comm_free(&new_comm)); checkMPIresult(MPI_Comm_free(&comm)); checkMPIresult(MPI_Finalize()); return 0; } I compile with mpicxx -std=c++11 -pthread mpich-shrink.cc -o mpich-shrink and run with mpirun -n 4 --disable-auto-cleanup ./mpich-shrink I used the recent master branch: The output of mpirun --version is: HYDRA build details: Version: 3.3a2 Release Date: unreleased development copy CC: gcc CXX: g++ F77: gfortran F90: gfortran Configure options: '--disable-option-checking' '--prefix=/home/nils/tmp/mpich-git-install' '--enable-error-checking=all' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -I/home/nils/tmp/mpich/src/mpl/include -I/home/nils/tmp/mpich/src/mpl/include -I/home/nils/tmp/mpich/src/openpa/src -I/home/nils/tmp/mpich/src/openpa/src -D_REENTRANT -I/home/nils/tmp/mpich/src/mpi/romio/include' 'MPLLIBNAME=mpl' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Checkpointing libraries available: blcr Demux engines available: poll select Thank you Nils _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH on ArchLinux
by lucas.drei＠web.de 15 May '17

15 May '17

_______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH on ArchLinux
by Guo, Yanfei 11 May '17

11 May '17

Hi Lucas, I just did a quick try on my archlinux box and I could not reproduce the problem. Can you provide more detail about the setup such as the number of nodes in your test? How to environment and environment variable is setup? Yanfei Guo Postdoctoral Researcher MCS Division, ANL On 5/11/17, 9:10 AM, "lucas.drei(a)web.de" <lucas.drei(a)web.de> wrote: Dear mpich, I experimented with MPICH and followed your install-guide. I use ArchLinux, SSH and NFS among other things. After the installation of mpich3.2 i tried to execute the example from your guide: "mpiexec -f machinefile hostname". The command is executed and the terminal prints the hostname. But after the use of mpiexec several problems occur. When I try to use nano, the text editor closes immediately. But beneath the options(Save,help,exit....) it prints "SIGHUB or SIGTERM recived". When I try to use vi, the text editor prints repeatedly "AT E0F". Im also unable to install applications. Only after a reboot or exit, and then re-login im able to use a text editor or install an application again. When i tried the same thing with raspbian it seems to work. So maybe it could be an ArchLinux specific problem? What could be the problem? Thank you Lucas _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPICH on ArchLinux
by lucas.drei＠web.de 11 May '17

11 May '17

_______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Process Group Collision for multiple clients from different host machines having same pid with MPI_Comm_accept
by Min Si 03 May '17

03 May '17

Hi Hirak, I created a ticket for this fix. Thanks for pointing out this problem. https://github.com/pmodels/mpich/issues/2619 Min On 5/3/17 11:02 AM, Balaji, Pavan wrote: > Yup. +1. > > Note that this needs to be fixed in Hydra as well, which essentially does the same thing. > > -- Pavan > >> On May 1, 2017, at 11:52 PM, Roy, Hirak <Hirak_Roy(a)mentor.com> wrote: >> >> Hi Min, >> >> I found the following piece of code in MPICH source : src/pm/util/pmiserv.c : >> >> /* We include the pid of the PMI server as a way to allow multiple >> PMI servers to coexist. This is needed to support connect/accept >> operations when multiple mpiexec's are used, and the KVS space >> is served directly by mpiexec (it should really have the >> hostname as well, just to avoid getting the same pid on two >> different hosts, but this is probably good enough for most >> uses) */ >> >> MPIU_Snprintf( (char *)(kvs->kvsname), MAXNAMELEN, "kvs_%d_%d", >> (int)getpid(), kvsnum++ ); >> kvs->pairs = 0; >> kvs->lastByIdx = 0; >> kvs->lastIdx = -1; >> >> >> and in src/pm/hydra/pm/pmiserv/common.c >> >> >> HYD_status HYD_pmcd_pmi_allocate_kvs(struct HYD_pmcd_pmi_kvs ** kvs, int pgid) >> { >> HYD_status status = HYD_SUCCESS; >> >> HYDU_FUNC_ENTER(); >> HYDU_MALLOC(*kvs, struct HYD_pmcd_pmi_kvs *, sizeof(struct HYD_pmcd_pmi_kvs), status); >> HYDU_snprintf((*kvs)->kvsname, PMI_MAXKVSLEN, "kvs_%d_%d", (int) getpid(), pgid); >> (*kvs)->key_pair = NULL; >> >> fn_exit: >> HYDU_FUNC_EXIT(); >> return status; >> >> fn_fail: >> goto fn_exit; >> } >> >> >> I think hostname should be added to kvsname to make it unique when accept/connect are being done from multiple processes which are not invoked by the same mpiexec. >> >> >> Thanks, >> Hirak >> >> >> >> >> >> Hi Hirak, >> >> Before look into PMI, it would be good to first make sure if this is a >> problem in your server-client code, or in the dynamic process part of >> MPICH code. Could you please reproduce this issue with a simple program >> and give it to us ? >> >> One thing I noticed is that the server program is multithreaded. Are you >> using multiple threads to accept client connection ? Anyway, a >> reproducer program will be great. >> >> Please also try to use the latest MPICH release and see if it happens. >> >> In summary, it would be great if you can send us the following files. >> - A reproducer program >> - MPICH's config.log (you can find it in the directory where you build >> MPICH) >> >> Thanks, >> Min >> >> On 4/14/17 1:15 AM, Roy, Hirak wrote: >>> >>> Dear MPICH team, >>> >>> We use MPICH for a server-client application. We use MPICH-3.0.4 with >>> sock channel. >>> In this application there is one server and 100 clients. >>> Each client is launched independently in different host-machines using >>> individual-wrapper scripts. (we explicitly use : mpiexec -n 1 ) >>> >>> The server is multithreaded and it uses MPI_Comm_accept (on >>> MPI_COMM_SELF) and clients use MPI_Comm_connect to connect. >>> We have observed the following issue after all the clients connect to >>> server : >>> if we send message to a client (lets say 'm'), it reaches >>> unexpectedly to some other client (lets say 'n'). { server sends the >>> message using the communicator returned by accept call }. This happens >>> randomly in one out of 5-6 runs. >>> >>> On further looking into MPICH code, we found that >>> 1) There is a collsion of pg (process-group) of two processes (m and >>> n) after mpi-comm-accept >>> 2) As a result of (1), comm->vc are same (for m and n, although comm >>> are different). It seems that the unique string (something like >>> kva_<int>_int) is not unique for such two processes. 'm' and 'n' >>> processes are running in different host-machine and they have the same >>> pid. The kva string looked like kva_pid_rank. >>> >>> >>> We have the following questions : >>> 1) Have we built MPICH with some kind of incorrect >>> configuration (hydra configuration at the end of the email) ? >>> 2) Are we using incorrect process-manager or configuration and that is >>> why there is a possible collision of process-groups? >>> 3) What is the purpose of process group sharing/uniquifying? If there >>> is no real reason for this, could it be disabled or will something >>> else rely on the id string being unique? >>> 4) If there are no other work-around, what could be done to make the >>> id string unique? Add the host-name? Will everything else be ok with this? >>> >>> >>> It would be good if you can let us know if there is any workaround for >>> this issue or not. >>> >>> >>> Thanks, >>> Hirak Roy >>> >>> HYDRA build details: >>> CXX: no -O3 -fPIC >>> F77: no >>> F90: no >>> Configure options: '--disable-option-checking' >>> '--prefix=/home/hroy/local/mpich-3.0.4/linux_x86_64' '--disable-f77' >>> '--disable-fc' '--disable-f90modules' '--disable-cxx' >>> '--enable-fast=nochkmsg' '--enable-fast=notiming' >>> '--enable-fast=ndebug' '--enable-fast=O3' '--with-device=ch3:sock' >>> 'CFLAGS=-O3 -fPIC -O3' 'CXXFLAGS=-O3 -fPIC' >>> 'CC=/u/prod/gnu/gcc/20121129/gcc-4.5.0-linux_x86_64/bin/gcc' 'LDFLAGS= >>> ' '--cache-file=/dev/null' '--srcdir=.' 'LIBS=-lrt -lpthread' >>> 'CPPFLAGS= -I/home/hroy/tools/mpich/mpich-3.0.4/src/mpl/include >>> -I/home/hroy/tools/mpich/mpich-3.0.4/src/mpl/include >>> -I/home/hroy/tools/mpich/mpich-3.0.4/src/openpa/src >>> -I/home/hroy/tools/mpich/mpich-3.0.4/src/openpa/src >>> -I/home/hroy/tools/mpich/mpich-3.0.4/src/mpi/romio/include' >>> Process Manager: pmi >>> Launchers available: ssh rsh fork slurm ll lsf >>> sge manual persist >>> Topology libraries available: hwloc >>> Resource management kernels available: user slurm ll lsf sge pbs >>> cobalt >>> Checkpointing libraries available: >>> Demux engines available: poll select >>> >>> >>> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Process Group Collision for multiple clients from different host machines having same pid with MPI_Comm_accept
by Balaji, Pavan 03 May '17

03 May '17

Yup. +1. Note that this needs to be fixed in Hydra as well, which essentially does the same thing. -- Pavan > On May 1, 2017, at 11:52 PM, Roy, Hirak <Hirak_Roy(a)mentor.com> wrote: > > Hi Min, > > I found the following piece of code in MPICH source : src/pm/util/pmiserv.c : > > /* We include the pid of the PMI server as a way to allow multiple > PMI servers to coexist. This is needed to support connect/accept > operations when multiple mpiexec's are used, and the KVS space > is served directly by mpiexec (it should really have the > hostname as well, just to avoid getting the same pid on two > different hosts, but this is probably good enough for most > uses) */ > > MPIU_Snprintf( (char *)(kvs->kvsname), MAXNAMELEN, "kvs_%d_%d", > (int)getpid(), kvsnum++ ); > kvs->pairs = 0; > kvs->lastByIdx = 0; > kvs->lastIdx = -1; > > > and in src/pm/hydra/pm/pmiserv/common.c > > > HYD_status HYD_pmcd_pmi_allocate_kvs(struct HYD_pmcd_pmi_kvs ** kvs, int pgid) > { > HYD_status status = HYD_SUCCESS; > > HYDU_FUNC_ENTER(); > HYDU_MALLOC(*kvs, struct HYD_pmcd_pmi_kvs *, sizeof(struct HYD_pmcd_pmi_kvs), status); > HYDU_snprintf((*kvs)->kvsname, PMI_MAXKVSLEN, "kvs_%d_%d", (int) getpid(), pgid); > (*kvs)->key_pair = NULL; > > fn_exit: > HYDU_FUNC_EXIT(); > return status; > > fn_fail: > goto fn_exit; > } > > > I think hostname should be added to kvsname to make it unique when accept/connect are being done from multiple processes which are not invoked by the same mpiexec. > > > Thanks, > Hirak > > > > > > Hi Hirak, > > Before look into PMI, it would be good to first make sure if this is a > problem in your server-client code, or in the dynamic process part of > MPICH code. Could you please reproduce this issue with a simple program > and give it to us ? > > One thing I noticed is that the server program is multithreaded. Are you > using multiple threads to accept client connection ? Anyway, a > reproducer program will be great. > > Please also try to use the latest MPICH release and see if it happens. > > In summary, it would be great if you can send us the following files. > - A reproducer program > - MPICH's config.log (you can find it in the directory where you build > MPICH) > > Thanks, > Min > > On 4/14/17 1:15 AM, Roy, Hirak wrote: > > > > > > Dear MPICH team, > > > > We use MPICH for a server-client application. We use MPICH-3.0.4 with > > sock channel. > > In this application there is one server and 100 clients. > > Each client is launched independently in different host-machines using > > individual-wrapper scripts. (we explicitly use : mpiexec -n 1 ) > > > > The server is multithreaded and it uses MPI_Comm_accept (on > > MPI_COMM_SELF) and clients use MPI_Comm_connect to connect. > > We have observed the following issue after all the clients connect to > > server : > > if we send message to a client (lets say 'm'), it reaches > > unexpectedly to some other client (lets say 'n'). { server sends the > > message using the communicator returned by accept call }. This happens > > randomly in one out of 5-6 runs. > > > > On further looking into MPICH code, we found that > > 1) There is a collsion of pg (process-group) of two processes (m and > > n) after mpi-comm-accept > > 2) As a result of (1), comm->vc are same (for m and n, although comm > > are different). It seems that the unique string (something like > > kva_<int>_int) is not unique for such two processes. 'm' and 'n' > > processes are running in different host-machine and they have the same > > pid. The kva string looked like kva_pid_rank. > > > > > > We have the following questions : > > 1) Have we built MPICH with some kind of incorrect > > configuration (hydra configuration at the end of the email) ? > > 2) Are we using incorrect process-manager or configuration and that is > > why there is a possible collision of process-groups? > > 3) What is the purpose of process group sharing/uniquifying? If there > > is no real reason for this, could it be disabled or will something > > else rely on the id string being unique? > > 4) If there are no other work-around, what could be done to make the > > id string unique? Add the host-name? Will everything else be ok with this? > > > > > > It would be good if you can let us know if there is any workaround for > > this issue or not. > > > > > > Thanks, > > Hirak Roy > > > > HYDRA build details: > > CXX: no -O3 -fPIC > > F77: no > > F90: no > > Configure options: '--disable-option-checking' > > '--prefix=/home/hroy/local/mpich-3.0.4/linux_x86_64' '--disable-f77' > > '--disable-fc' '--disable-f90modules' '--disable-cxx' > > '--enable-fast=nochkmsg' '--enable-fast=notiming' > > '--enable-fast=ndebug' '--enable-fast=O3' '--with-device=ch3:sock' > > 'CFLAGS=-O3 -fPIC -O3' 'CXXFLAGS=-O3 -fPIC ' > > 'CC=/u/prod/gnu/gcc/20121129/gcc-4.5.0-linux_x86_64/bin/gcc' 'LDFLAGS= > > ' '--cache-file=/dev/null' '--srcdir=.' 'LIBS=-lrt -lpthread ' > > 'CPPFLAGS= -I/home/hroy/tools/mpich/mpich-3.0.4/src/mpl/include > > -I/home/hroy/tools/mpich/mpich-3.0.4/src/mpl/include > > -I/home/hroy/tools/mpich/mpich-3.0.4/src/openpa/src > > -I/home/hroy/tools/mpich/mpich-3.0.4/src/openpa/src > > -I/home/hroy/tools/mpich/mpich-3.0.4/src/mpi/romio/include' > > Process Manager: pmi > > Launchers available: ssh rsh fork slurm ll lsf > > sge manual persist > > Topology libraries available: hwloc > > Resource management kernels available: user slurm ll lsf sge pbs > > cobalt > > Checkpointing libraries available: > > Demux engines available: poll select > > > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Need perspective on use of atomic I/O
by Latham, Robert J. 02 May '17

02 May '17

On Sun, 2017-04-30 at 17:16 -0700, Ted Dunning wrote: > > I am trying to assess how well a high-performance file system that I > work on will support MPI programming for some customers we have. In > particular, I am looking at MPI I/O. > > The technical aspects of the assessment are pretty simple. Collective > and direct I/O should work great, but the locking apparently required > for atomic mode I/O will be very, very expensive. > > That leads to the community questions of how important these trade- > offs will be. And thus, I land here on this list. > > I have a few questions that folk here probably can answer off the top > of their head: > > 1) how often is atomic mode I/O used? It seems that most I/O will > involve non-overlapping writes and thus atomic mode will be > irrelevant. But what is the real sense from people who actually write > this kind of code? atomic mode is rarely used in practice and sometimes not even supported by the underlying file system. Where needed, applications are in a better position to know when to synchronize and coordinate. > 2) do commonly used MPI libraries implement atomic mode internally as > in [1], or do they require file system support? What kind of support? > the file system needs to support fcntl() locks. Lustre, for example, only supports fcntl() locks with a special mount option. > 3) how to file-system people usually go wrong in approaching MPI > applications? A lot of parallel file systems assume dumb clients, but MPI clients have a lot more information: they know, via the MPI communicator, who is participating in this I/O operation. They also follow looser consistency semantics. Whereas POSIX demands bytes written are visible immediately, MPI-IO semantics demand that bytes written are visible at synchronization points. POSIX is everywhere of course so one should provide a POSIX interface. Providing that interface on top of a more parallel I/O friendly interface is a lot easier than going the other way around. > > [1] http://www.mcs.anl.gov/~thakur/papers/atomic-mode.pdf hey, I worked on that! You have really done your homework. While we still think it's a good approach, we've never deployed it nor do I think anyone else has. Furthermore, a lot of the cleverness is no longer needed now that MPI-3 RMA supports fetch-and-increment. ==rob _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re : Process Group Collision for multiple clients from different host machines having same pid with MPI_Comm_accept
by Roy, Hirak 01 May '17

01 May '17

Hi Min, I found the following piece of code in MPICH source : src/pm/util/pmiserv.c : /* We include the pid of the PMI server as a way to allow multiple PMI servers to coexist. This is needed to support connect/accept operations when multiple mpiexec's are used, and the KVS space is served directly by mpiexec (it should really have the hostname as well, just to avoid getting the same pid on two different hosts, but this is probably good enough for most uses) */ MPIU_Snprintf( (char *)(kvs->kvsname), MAXNAMELEN, "kvs_%d_%d", (int)getpid(), kvsnum++ ); kvs->pairs = 0; kvs->lastByIdx = 0; kvs->lastIdx = -1; and in src/pm/hydra/pm/pmiserv/common.c HYD_status HYD_pmcd_pmi_allocate_kvs(struct HYD_pmcd_pmi_kvs ** kvs, int pgid) { HYD_status status = HYD_SUCCESS; HYDU_FUNC_ENTER(); HYDU_MALLOC(*kvs, struct HYD_pmcd_pmi_kvs *, sizeof(struct HYD_pmcd_pmi_kvs), status); HYDU_snprintf((*kvs)->kvsname, PMI_MAXKVSLEN, "kvs_%d_%d", (int) getpid(), pgid); (*kvs)->key_pair = NULL; fn_exit: HYDU_FUNC_EXIT(); return status; fn_fail: goto fn_exit; } I think hostname should be added to kvsname to make it unique when accept/connect are being done from multiple processes which are not invoked by the same mpiexec. Thanks, Hirak Hi Hirak, Before look into PMI, it would be good to first make sure if this is a problem in your server-client code, or in the dynamic process part of MPICH code. Could you please reproduce this issue with a simple program and give it to us ? One thing I noticed is that the server program is multithreaded. Are you using multiple threads to accept client connection ? Anyway, a reproducer program will be great. Please also try to use the latest MPICH release and see if it happens. In summary, it would be great if you can send us the following files. - A reproducer program - MPICH's config.log (you can find it in the directory where you build MPICH) Thanks, Min On 4/14/17 1:15 AM, Roy, Hirak wrote: > > > Dear MPICH team, > > We use MPICH for a server-client application. We use MPICH-3.0.4 with > sock channel. > In this application there is one server and 100 clients. > Each client is launched independently in different host-machines using > individual-wrapper scripts. (we explicitly use : mpiexec -n 1 ) > > The server is multithreaded and it uses MPI_Comm_accept (on > MPI_COMM_SELF) and clients use MPI_Comm_connect to connect. > We have observed the following issue after all the clients connect to > server : > if we send message to a client (lets say 'm'), it reaches > unexpectedly to some other client (lets say 'n'). { server sends the > message using the communicator returned by accept call }. This happens > randomly in one out of 5-6 runs. > > On further looking into MPICH code, we found that > 1) There is a collsion of pg (process-group) of two processes (m and > n) after mpi-comm-accept > 2) As a result of (1), comm->vc are same (for m and n, although comm > are different). It seems that the unique string (something like > kva_<int>_int) is not unique for such two processes. 'm' and 'n' > processes are running in different host-machine and they have the same > pid. The kva string looked like kva_pid_rank. > > > We have the following questions : > 1) Have we built MPICH with some kind of incorrect > configuration (hydra configuration at the end of the email) ? > 2) Are we using incorrect process-manager or configuration and that is > why there is a possible collision of process-groups? > 3) What is the purpose of process group sharing/uniquifying? If there > is no real reason for this, could it be disabled or will something > else rely on the id string being unique? > 4) If there are no other work-around, what could be done to make the > id string unique? Add the host-name? Will everything else be ok with this? > > > It would be good if you can let us know if there is any workaround for > this issue or not. > > > Thanks, > Hirak Roy > > HYDRA build details: > CXX: no -O3 -fPIC > F77: no > F90: no > Configure options: '--disable-option-checking' > '--prefix=/home/hroy/local/mpich-3.0.4/linux_x86_64' '--disable-f77' > '--disable-fc' '--disable-f90modules' '--disable-cxx' > '--enable-fast=nochkmsg' '--enable-fast=notiming' > '--enable-fast=ndebug' '--enable-fast=O3' '--with-device=ch3:sock' > 'CFLAGS=-O3 -fPIC -O3' 'CXXFLAGS=-O3 -fPIC ' > 'CC=/u/prod/gnu/gcc/20121129/gcc-4.5.0-linux_x86_64/bin/gcc' 'LDFLAGS= > ' '--cache-file=/dev/null' '--srcdir=.' 'LIBS=-lrt -lpthread ' > 'CPPFLAGS= -I/home/hroy/tools/mpich/mpich-3.0.4/src/mpl/include > -I/home/hroy/tools/mpich/mpich-3.0.4/src/mpl/include > -I/home/hroy/tools/mpich/mpich-3.0.4/src/openpa/src > -I/home/hroy/tools/mpich/mpich-3.0.4/src/openpa/src > -I/home/hroy/tools/mpich/mpich-3.0.4/src/mpi/romio/include' > Process Manager: pmi > Launchers available: ssh rsh fork slurm ll lsf > sge manual persist > Topology libraries available: hwloc > Resource management kernels available: user slurm ll lsf sge pbs > cobalt > Checkpointing libraries available: > Demux engines available: poll select > > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH Connection to Self Rejected
by Melissa Romanus 01 May '17

01 May '17

Oh okay - thank you! -Melissa On Mon, May 1, 2017 at 3:29 PM Kenneth Raffenetti <raffenet(a)mcs.anl.gov> wrote: > Hi Melissa, > > Probably best to post this question to > mvapich-discuss(a)cse.ohio-state.edu and go from there. > > Thanks, > Ken > > On 05/01/2017 01:15 PM, Melissa Romanus wrote: > > Hello, > > > > I am experiencing issues on the SDSC Comet system when using the intel > > compilers with mvapich2. The scheduling system on Comet is slurm. It > > seems like the code is seg-faulting inside of MPI_Comm_dup, but prior to > > that, it seems like it is rejecting a connection request to "self" > > (i.e., same IP to same IP). > > > > The modules loaded are: > > > > ``` > > $ module list > > > > Currently Loaded Modulefiles: > > 1) intel/2013_sp1.2.144 2) mvapich2_ib/2.1 3) gnutools/2.69 > > ``` > > > > I am attempting to use the `ib0` interface. In my job script, I am > > launching 3 different applications. I am **not** using slurm > > `--multi-prog`. I am instead using 3 different `srun` commands. My job > > has to be launched this way. > > > > Using OpenMPI, I can set the MCA parameters to allow connections from > > `self` at the byte-transfer layer, i.e., `OMPI_MCA_btl="self,openib"` > > and specify to slurm that I would like to use `--mpi=pmi2`. > > > > I think the mvapich errors that I am experiencing stem from the fact > > that the "self" connection is rejected (i.e., node to itself). Is there > > a way to tell MVAPICH to allow the self connection? I think I want the > > `--with-device=ch3:nemesis:ib` command in some capacity, but I'm not > > sure if that would be enough to allow the connection from the node to > > itself. Is the self connection inherently a TCP connection? Or do I > > still need `--mpi=pmi2` for srun? Can I use srun or do I need to use > > `mpiexec` explicitly? > > > > Could this also be the cause of the error described by this FAQ? > > > https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_All_my… > . > > > > Any help you can provide is greatly appreciated. > > > > -Melissa > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH Connection to Self Rejected
by Kenneth Raffenetti 01 May '17

01 May '17

Hi Melissa, Probably best to post this question to mvapich-discuss(a)cse.ohio-state.edu and go from there. Thanks, Ken On 05/01/2017 01:15 PM, Melissa Romanus wrote: > Hello, > > I am experiencing issues on the SDSC Comet system when using the intel > compilers with mvapich2. The scheduling system on Comet is slurm. It > seems like the code is seg-faulting inside of MPI_Comm_dup, but prior to > that, it seems like it is rejecting a connection request to "self" > (i.e., same IP to same IP). > > The modules loaded are: > > ``` > $ module list > > Currently Loaded Modulefiles: > 1) intel/2013_sp1.2.144 2) mvapich2_ib/2.1 3) gnutools/2.69 > ``` > > I am attempting to use the `ib0` interface. In my job script, I am > launching 3 different applications. I am **not** using slurm > `--multi-prog`. I am instead using 3 different `srun` commands. My job > has to be launched this way. > > Using OpenMPI, I can set the MCA parameters to allow connections from > `self` at the byte-transfer layer, i.e., `OMPI_MCA_btl="self,openib"` > and specify to slurm that I would like to use `--mpi=pmi2`. > > I think the mvapich errors that I am experiencing stem from the fact > that the "self" connection is rejected (i.e., node to itself). Is there > a way to tell MVAPICH to allow the self connection? I think I want the > `--with-device=ch3:nemesis:ib` command in some capacity, but I'm not > sure if that would be enough to allow the connection from the node to > itself. Is the self connection inherently a TCP connection? Or do I > still need `--mpi=pmi2` for srun? Can I use srun or do I need to use > `mpiexec` explicitly? > > Could this also be the cause of the error described by this FAQ? > https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_All_my…. > > Any help you can provide is greatly appreciated. > > -Melissa > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0