January 2017 - discuss - mailman3.cels.anl.gov

Fwd: MPI_Info key for hostfile
by Sergio Iserte 18 Jan '17

18 Jan '17

Hello everybody, I need to determine the target hosts of an MPI_Comm_spawn by using a file. My solution has been to follow the same procedure of giving a host list, but changing the key: *MPI_Info_set(info, "hostfile", filename)* instead of* MPI_Info_set(info, "hosts", hostlist)* However, MPI_Comm_spawn seems to ignore "hostfile" key and the processes are spawned in the wrong hosts (the hosts set during the first mpiexec). Which key should I use? I'm working over MPICH 3.2 Best regards, -- Sergio Iserte - http://me.sergioiserte.info Research Assistant @ HPC&A Research Group Universitat Jaume I (UJI) *Castellón de la Plana (Spain)* _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH3 and Nagfor: Corrupts writing/IO?
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 17 Jan '17

17 Jan '17

Ken, I just tried your configure (with enable-cxx) and it seemed to work! Thanks! I suppose I should try and figure out what flag caused the corruption but it's working. Don't really want to poke the bear right now. Matt On 01/12/2017 04:46 PM, Kenneth Raffenetti wrote: > Can you check your compiler flags and make sure they are all necessary? > I was able to reproduce the error locally with your settings, but a more > default configuration works fine. I.e. > > ./configure --prefix=$PWD/i --disable-wrapper-rpath CC=gcc CXX=g++ > FC=nagfor F77=nagfor FCFLAGS=-mismatch FFLAGS=-mismatch > --enable-fortran=all > > Ken > > On 01/12/2017 10:54 AM, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND > APPLICATIONS INC] wrote: >> All, >> >> I've been having some "fun" recently trying to get an MPI stack built >> with nagfor 6.1. I've tried Open MPI and MVAPICH2 and failed to even >> build the MPI stack, SGI MPT doesn't like nagfor, and Intel MPI I'm >> guessing wouldn't either. >> >> So, I figured I'd go for MPICH3. And, lo and behold, building with >> nagfor 6.1 and gcc 5.3 (for CC and CXX) with: >> >>> ./configure >>> --prefix=$SWDEV/MPI/mpich/3.2/nagfor_6.1-gcc_5.3-nomismatchall \ >> --disable-wrapper-rpath CC=gcc CXX=g++ FC=nagfor F77=nagfor \ >> CFLAGS='-fpic -m64' CXXFLAGS='-fpic -m64' \ >> FCFLAGS='-PIC -abi=64' FFLAGS='-PIC -abi=64 -mismatch' \ >> --enable-fortran=all --enable-cxx >> >> I got something to build. Huzzah! >> >> I then tried the cpi test, it worked! It even detected I was on slurm >> according to mpirun -verbose. >> >> I then tried a simple Fortran 90 Hello world program and...crash: >> >>> (1211)(master) $ cat helloWorld.F90 >>> program hello_world >>> >>> use mpi >>> >>> implicit none >>> >>> integer :: comm >>> integer :: myid, npes, ierror >>> integer :: name_length >>> >>> character(len=MPI_MAX_PROCESSOR_NAME) :: processor_name >>> >>> call mpi_init(ierror) >>> >>> comm = MPI_COMM_WORLD >>> >>> call MPI_Comm_Rank(comm,myid,ierror) >>> call MPI_Comm_Size(comm,npes,ierror) >>> call MPI_Get_Processor_Name(processor_name,name_length,ierror) >>> >>> write (*,'(A,1X,I4,1X,A,1X,I4,1X,A,1X,A)') "Process", myid, "of", >>> npes, "is on", trim(processor_name) >>> >>> call MPI_Finalize(ierror) >>> >>> end program hello_world >>> (1212)(master) $ mpifort -o helloWorld.exe helloWorld.F90 >>> NAG Fortran Compiler Release 6.1(Tozai) Build 6113 >>> [NAG Fortran Compiler normal termination] >>> (1213)(master) $ mpirun -np 4 ./helloWorld.exe >>> srun.slurm: cluster configuration lacks support for cpu binding >>> Runtime Error: Buffer overflow on output >>> Program terminated by I/O error on unit 6 >>> (Output_Unit,Unformatted,Direct) >>> Runtime Error: Buffer overflow on output >>> Program terminated by I/O error on unit 6 >>> (Output_Unit,Unformatted,Direct) >>> Runtime Error: Buffer overflow on output >>> Program terminated by I/O error on unit Runtime Error: Buffer overflow >>> on output >>> Program terminated by I/O error on unit 6 >>> (Output_Unit,Unformatted,Direct) >>> 6 (Output_Unit,Unformatted,Direct) >>> >>> =================================================================================== >>> >>> >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> = PID 7642 RUNNING AT borgl189 >>> = EXIT CODE: 134 >>> = CLEANING UP REMAINING PROCESSES >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> =================================================================================== >>> >>> >>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) >>> This typically refers to a problem with your application. >>> Please see the FAQ page for debugging suggestions >> >> Weird. So, I decided to try something different, this program: >> >>> program main >>> implicit none >>> real :: a >>> a = 1240.0 >>> write (*,*) "Hello world", a >>> end program main >> >> Looks boring and is standard-compliant and nagfor likes it: >> >>> (1226) $ nagfor test.F90 && ./a.out >>> NAG Fortran Compiler Release 6.1(Tozai) Build 6113 >>> [NAG Fortran Compiler normal termination] >>> Hello world 1.2400000E+03 >> >> Looks correct. Now let's try mpifort: >> >>> (1232) $ mpifort test.F90 && ./a.out >>> NAG Fortran Compiler Release 6.1(Tozai) Build 6113 >>> [NAG Fortran Compiler normal termination] >>> Hello world >>> Segmentation fault (core dumped) >> >> You can't really see it here, but that "Hello world" is surrounded by LF >> characters. Like a literal LineFeed...and then it core dumps. >> >> Now let's try running with mpirun as well: >> >>> (1233) $ mpifort test.F90 && mpirun -np 1 ./a.out >>> NAG Fortran Compiler Release 6.1(Tozai) Build 6113 >>> [NAG Fortran Compiler normal termination] >>> srun.slurm: cluster configuration lacks support for cpu binding >>> >>> =================================================================================== >>> >>> >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> = PID 8520 RUNNING AT borgl189 >>> = EXIT CODE: 139 >>> = CLEANING UP REMAINING PROCESSES >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> =================================================================================== >>> >>> >>> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault >>> (signal 11) >>> This typically refers to a problem with your application. >>> Please see the FAQ page for debugging suggestions >> >> All righty then. >> >> Does anyone have advice for this? I'll fully accept I configured MPICH3 >> wrong as it's the first time in a while I've built MPICH (think MPICH2). >> But, still, I don't have any exciting flags. >> >> Matt >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH3 and Nagfor: Corrupts writing/IO?
by Kenneth Raffenetti 12 Jan '17

12 Jan '17

Can you check your compiler flags and make sure they are all necessary? I was able to reproduce the error locally with your settings, but a more default configuration works fine. I.e. ./configure --prefix=$PWD/i --disable-wrapper-rpath CC=gcc CXX=g++ FC=nagfor F77=nagfor FCFLAGS=-mismatch FFLAGS=-mismatch --enable-fortran=all Ken On 01/12/2017 10:54 AM, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] wrote: > All, > > I've been having some "fun" recently trying to get an MPI stack built > with nagfor 6.1. I've tried Open MPI and MVAPICH2 and failed to even > build the MPI stack, SGI MPT doesn't like nagfor, and Intel MPI I'm > guessing wouldn't either. > > So, I figured I'd go for MPICH3. And, lo and behold, building with > nagfor 6.1 and gcc 5.3 (for CC and CXX) with: > >> ./configure >> --prefix=$SWDEV/MPI/mpich/3.2/nagfor_6.1-gcc_5.3-nomismatchall \ > --disable-wrapper-rpath CC=gcc CXX=g++ FC=nagfor F77=nagfor \ > CFLAGS='-fpic -m64' CXXFLAGS='-fpic -m64' \ > FCFLAGS='-PIC -abi=64' FFLAGS='-PIC -abi=64 -mismatch' \ > --enable-fortran=all --enable-cxx > > I got something to build. Huzzah! > > I then tried the cpi test, it worked! It even detected I was on slurm > according to mpirun -verbose. > > I then tried a simple Fortran 90 Hello world program and...crash: > >> (1211)(master) $ cat helloWorld.F90 >> program hello_world >> >> use mpi >> >> implicit none >> >> integer :: comm >> integer :: myid, npes, ierror >> integer :: name_length >> >> character(len=MPI_MAX_PROCESSOR_NAME) :: processor_name >> >> call mpi_init(ierror) >> >> comm = MPI_COMM_WORLD >> >> call MPI_Comm_Rank(comm,myid,ierror) >> call MPI_Comm_Size(comm,npes,ierror) >> call MPI_Get_Processor_Name(processor_name,name_length,ierror) >> >> write (*,'(A,1X,I4,1X,A,1X,I4,1X,A,1X,A)') "Process", myid, "of", >> npes, "is on", trim(processor_name) >> >> call MPI_Finalize(ierror) >> >> end program hello_world >> (1212)(master) $ mpifort -o helloWorld.exe helloWorld.F90 >> NAG Fortran Compiler Release 6.1(Tozai) Build 6113 >> [NAG Fortran Compiler normal termination] >> (1213)(master) $ mpirun -np 4 ./helloWorld.exe >> srun.slurm: cluster configuration lacks support for cpu binding >> Runtime Error: Buffer overflow on output >> Program terminated by I/O error on unit 6 >> (Output_Unit,Unformatted,Direct) >> Runtime Error: Buffer overflow on output >> Program terminated by I/O error on unit 6 >> (Output_Unit,Unformatted,Direct) >> Runtime Error: Buffer overflow on output >> Program terminated by I/O error on unit Runtime Error: Buffer overflow >> on output >> Program terminated by I/O error on unit 6 >> (Output_Unit,Unformatted,Direct) >> 6 (Output_Unit,Unformatted,Direct) >> >> =================================================================================== >> >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 7642 RUNNING AT borgl189 >> = EXIT CODE: 134 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> >> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) >> This typically refers to a problem with your application. >> Please see the FAQ page for debugging suggestions > > Weird. So, I decided to try something different, this program: > >> program main >> implicit none >> real :: a >> a = 1240.0 >> write (*,*) "Hello world", a >> end program main > > Looks boring and is standard-compliant and nagfor likes it: > >> (1226) $ nagfor test.F90 && ./a.out >> NAG Fortran Compiler Release 6.1(Tozai) Build 6113 >> [NAG Fortran Compiler normal termination] >> Hello world 1.2400000E+03 > > Looks correct. Now let's try mpifort: > >> (1232) $ mpifort test.F90 && ./a.out >> NAG Fortran Compiler Release 6.1(Tozai) Build 6113 >> [NAG Fortran Compiler normal termination] >> Hello world >> Segmentation fault (core dumped) > > You can't really see it here, but that "Hello world" is surrounded by LF > characters. Like a literal LineFeed...and then it core dumps. > > Now let's try running with mpirun as well: > >> (1233) $ mpifort test.F90 && mpirun -np 1 ./a.out >> NAG Fortran Compiler Release 6.1(Tozai) Build 6113 >> [NAG Fortran Compiler normal termination] >> srun.slurm: cluster configuration lacks support for cpu binding >> >> =================================================================================== >> >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 8520 RUNNING AT borgl189 >> = EXIT CODE: 139 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> >> YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault >> (signal 11) >> This typically refers to a problem with your application. >> Please see the FAQ page for debugging suggestions > > All righty then. > > Does anyone have advice for this? I'll fully accept I configured MPICH3 > wrong as it's the first time in a while I've built MPICH (think MPICH2). > But, still, I don't have any exciting flags. > > Matt > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem using MPICH 3.2 compiled with --with-device=ch3:sock
by Halim Amer 12 Jan '17

12 Jan '17

Can you be more specific about "does not work"? If you are seeing error messages please forward them here. Also we need to know what your program does. A toy program that reproduces the problem would be very helpful. Halim www.mcs.anl.gov/~aamer On 12/21/16 4:24 AM, Anatoly G wrote: > Dear MPICH, > > I downloaded a MPICH 3.2 package. > > When I configure it using: > /./configure --prefix=/space/mpich3.2 --enable-error-checking=runtime > --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic > --enable-totalview --enable-static --disable-f77 --disable-fc > --enable-shared=no --disable-fortran/ > / > / > My test program (simple send/receive) works fine on single & multiple > computers. But each process activates "busy wait on each receive". > > I'd like to get program without busy wait. > > Than, I configured MPICH 3.2 with following: > /./configure > --prefix=/space/mpich3.2_no_busy_wait --enable-error-checking=runtime > --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic > --enable-totalview --enable-static --disable-f77 --disable-fc > --enable-shared=no --disable-fortran *--with-device=ch3:sock* > / > * > * > After build my test program with this mpi, my test works fine on single > computer two processes, but doesn't work on two computers (process per > computer) with same configuration. > > Can you please advice? > > Thank you, > Anatoly. > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] spurious lock ups on collective merge intercom
by Dmitriy Lyubimov 12 Jan '17

12 Jan '17

further debugging shows that it's not actually mergeIntercom that locks up but a pair of send/recv that two nodes decide to execute before MPI_intercom_merge. so the total snapshot of the situation is that everyone waits on mergeIntercom except for two processes that wait in send/recv respectively, while majority of others already have entered collective barrier. it would seem that this sort of assymetric logic would be acceptable, since the send/recv pair is balanced before the merge is to occur, but in practice it seems to lock up -- increasingly so as the number of participating processes increases. It almost like once collective barrier of certain cardinality is formed, point-to-point messages are not going thru any longer. If this scenario begets any ideas, please let me know. thank you! -Dmitriy On Wed, Jan 11, 2017 at 9:38 AM, Dmitriy Lyubimov <dlieu.7(a)gmail.com> wrote: > Maybe it has something to do with the fact that it is stepping thru JVM > JNI and that somehow screws threading model of MPI, although it is a single > threaded JVM process, and MPI mappings are known to have been done before > (e.g., openmpi had an effort towards that). > > Strange thing is that i never had lock up with # of processes under 120 > but something changes after that, the spurious condition becomes much more > common after that. By the time I am at 150 processes in the intercom, I am > almost certain to have a merge lock-up. > > > On Wed, Jan 11, 2017 at 9:34 AM, Dmitriy Lyubimov <dlieu.7(a)gmail.com> > wrote: > >> Thanks. >> it would not be easy for me to do immediately as i am using proprietary >> scala binding api for MPI. >> >> it would help me to know if there's a known problem like that in the >> past, or generally mergeIntercomm api is known to work on hundreds of >> processes. Sounds like there are no known issues with that. >> >> >> >> On Tue, Jan 10, 2017 at 11:53 PM, Oden, Lena <loden(a)anl.gov> wrote: >> >>> Hello Dmittiy, >>> >>> can you maybe create a simple example-program to reproduce this failure? >>> It is also often easier also to look at a code example to identify a >>> problem. >>> >>> Thanks, >>> Lena >>> > On Jan 11, 2017, at 2:45 AM, Dmitriy Lyubimov <dlieu.7(a)gmail.com> >>> wrote: >>> > >>> > Hello, >>> > >>> > (mpich 3.2) >>> > >>> > I have a scenario when i add a few extra processes do existing >>> intercom. >>> > >>> > it works as a simple loop -- >>> > (1) n processes accept on n-intercom >>> > (2) 1 process connects >>> > (3) intracom is merged into n+1 intercom, intracom and n-intercom are >>> closed >>> > (4) repeat 1-3 as needed. >>> > >>> > Occasionally, i observe that step 3 spuriously locks up (once i get in >>> the range of 100+ processes). From what i can tell, all processes in step 3 >>> are accounted for, and are waiting on the merge, but nothing happens. the >>> collective barrier locks up. >>> > >>> > I really have trouble resolving this issue, any ideas are appreciated! >>> > >>> > Thank you very much. >>> > -Dmitriy >>> > >>> > >>> > _______________________________________________ >>> > discuss mailing list discuss(a)mpich.org >>> > To manage subscription options or unsubscribe: >>> > https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPICH3 and Nagfor: Corrupts writing/IO?
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 12 Jan '17

12 Jan '17

All, I've been having some "fun" recently trying to get an MPI stack built with nagfor 6.1. I've tried Open MPI and MVAPICH2 and failed to even build the MPI stack, SGI MPT doesn't like nagfor, and Intel MPI I'm guessing wouldn't either. So, I figured I'd go for MPICH3. And, lo and behold, building with nagfor 6.1 and gcc 5.3 (for CC and CXX) with: > ./configure --prefix=$SWDEV/MPI/mpich/3.2/nagfor_6.1-gcc_5.3-nomismatchall \ --disable-wrapper-rpath CC=gcc CXX=g++ FC=nagfor F77=nagfor \ CFLAGS='-fpic -m64' CXXFLAGS='-fpic -m64' \ FCFLAGS='-PIC -abi=64' FFLAGS='-PIC -abi=64 -mismatch' \ --enable-fortran=all --enable-cxx I got something to build. Huzzah! I then tried the cpi test, it worked! It even detected I was on slurm according to mpirun -verbose. I then tried a simple Fortran 90 Hello world program and...crash: > (1211)(master) $ cat helloWorld.F90 > program hello_world > > use mpi > > implicit none > > integer :: comm > integer :: myid, npes, ierror > integer :: name_length > > character(len=MPI_MAX_PROCESSOR_NAME) :: processor_name > > call mpi_init(ierror) > > comm = MPI_COMM_WORLD > > call MPI_Comm_Rank(comm,myid,ierror) > call MPI_Comm_Size(comm,npes,ierror) > call MPI_Get_Processor_Name(processor_name,name_length,ierror) > > write (*,'(A,1X,I4,1X,A,1X,I4,1X,A,1X,A)') "Process", myid, "of", npes, "is on", trim(processor_name) > > call MPI_Finalize(ierror) > > end program hello_world > (1212)(master) $ mpifort -o helloWorld.exe helloWorld.F90 > NAG Fortran Compiler Release 6.1(Tozai) Build 6113 > [NAG Fortran Compiler normal termination] > (1213)(master) $ mpirun -np 4 ./helloWorld.exe > srun.slurm: cluster configuration lacks support for cpu binding > Runtime Error: Buffer overflow on output > Program terminated by I/O error on unit 6 (Output_Unit,Unformatted,Direct) > Runtime Error: Buffer overflow on output > Program terminated by I/O error on unit 6 (Output_Unit,Unformatted,Direct) > Runtime Error: Buffer overflow on output > Program terminated by I/O error on unit Runtime Error: Buffer overflow on output > Program terminated by I/O error on unit 6 (Output_Unit,Unformatted,Direct) > 6 (Output_Unit,Unformatted,Direct) > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 7642 RUNNING AT borgl189 > = EXIT CODE: 134 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) > This typically refers to a problem with your application. > Please see the FAQ page for debugging suggestions Weird. So, I decided to try something different, this program: > program main > implicit none > real :: a > a = 1240.0 > write (*,*) "Hello world", a > end program main Looks boring and is standard-compliant and nagfor likes it: > (1226) $ nagfor test.F90 && ./a.out > NAG Fortran Compiler Release 6.1(Tozai) Build 6113 > [NAG Fortran Compiler normal termination] > Hello world 1.2400000E+03 Looks correct. Now let's try mpifort: > (1232) $ mpifort test.F90 && ./a.out > NAG Fortran Compiler Release 6.1(Tozai) Build 6113 > [NAG Fortran Compiler normal termination] > Hello world > Segmentation fault (core dumped) You can't really see it here, but that "Hello world" is surrounded by LF characters. Like a literal LineFeed...and then it core dumps. Now let's try running with mpirun as well: > (1233) $ mpifort test.F90 && mpirun -np 1 ./a.out > NAG Fortran Compiler Release 6.1(Tozai) Build 6113 > [NAG Fortran Compiler normal termination] > srun.slurm: cluster configuration lacks support for cpu binding > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 8520 RUNNING AT borgl189 > = EXIT CODE: 139 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11) > This typically refers to a problem with your application. > Please see the FAQ page for debugging suggestions All righty then. Does anyone have advice for this? I'll fully accept I configured MPICH3 wrong as it's the first time in a while I've built MPICH (think MPICH2). But, still, I don't have any exciting flags. Matt -- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] spurious lock ups on collective merge intercom
by Dmitriy Lyubimov 11 Jan '17

11 Jan '17

Maybe it has something to do with the fact that it is stepping thru JVM JNI and that somehow screws threading model of MPI, although it is a single threaded JVM process, and MPI mappings are known to have been done before (e.g., openmpi had an effort towards that). Strange thing is that i never had lock up with # of processes under 120 but something changes after that, the spurious condition becomes much more common after that. By the time I am at 150 processes in the intercom, I am almost certain to have a merge lock-up. On Wed, Jan 11, 2017 at 9:34 AM, Dmitriy Lyubimov <dlieu.7(a)gmail.com> wrote: > Thanks. > it would not be easy for me to do immediately as i am using proprietary > scala binding api for MPI. > > it would help me to know if there's a known problem like that in the past, > or generally mergeIntercomm api is known to work on hundreds of processes. > Sounds like there are no known issues with that. > > > > On Tue, Jan 10, 2017 at 11:53 PM, Oden, Lena <loden(a)anl.gov> wrote: > >> Hello Dmittiy, >> >> can you maybe create a simple example-program to reproduce this failure? >> It is also often easier also to look at a code example to identify a >> problem. >> >> Thanks, >> Lena >> > On Jan 11, 2017, at 2:45 AM, Dmitriy Lyubimov <dlieu.7(a)gmail.com> >> wrote: >> > >> > Hello, >> > >> > (mpich 3.2) >> > >> > I have a scenario when i add a few extra processes do existing intercom. >> > >> > it works as a simple loop -- >> > (1) n processes accept on n-intercom >> > (2) 1 process connects >> > (3) intracom is merged into n+1 intercom, intracom and n-intercom are >> closed >> > (4) repeat 1-3 as needed. >> > >> > Occasionally, i observe that step 3 spuriously locks up (once i get in >> the range of 100+ processes). From what i can tell, all processes in step 3 >> are accounted for, and are waiting on the merge, but nothing happens. the >> collective barrier locks up. >> > >> > I really have trouble resolving this issue, any ideas are appreciated! >> > >> > Thank you very much. >> > -Dmitriy >> > >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] spurious lock ups on collective merge intercom
by Dmitriy Lyubimov 11 Jan '17

11 Jan '17

Thanks. it would not be easy for me to do immediately as i am using proprietary scala binding api for MPI. it would help me to know if there's a known problem like that in the past, or generally mergeIntercomm api is known to work on hundreds of processes. Sounds like there are no known issues with that. On Tue, Jan 10, 2017 at 11:53 PM, Oden, Lena <loden(a)anl.gov> wrote: > Hello Dmittiy, > > can you maybe create a simple example-program to reproduce this failure? > It is also often easier also to look at a code example to identify a > problem. > > Thanks, > Lena > > On Jan 11, 2017, at 2:45 AM, Dmitriy Lyubimov <dlieu.7(a)gmail.com> wrote: > > > > Hello, > > > > (mpich 3.2) > > > > I have a scenario when i add a few extra processes do existing intercom. > > > > it works as a simple loop -- > > (1) n processes accept on n-intercom > > (2) 1 process connects > > (3) intracom is merged into n+1 intercom, intracom and n-intercom are > closed > > (4) repeat 1-3 as needed. > > > > Occasionally, i observe that step 3 spuriously locks up (once i get in > the range of 100+ processes). From what i can tell, all processes in step 3 > are accounted for, and are waiting on the merge, but nothing happens. the > collective barrier locks up. > > > > I really have trouble resolving this issue, any ideas are appreciated! > > > > Thank you very much. > > -Dmitriy > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] FW: Potential MPICH problem
by Richard Warren 11 Jan '17

11 Jan '17

Hi Rob, I did put together an initial simplified test case to attempt to reproduce the issue, but that does code does NOT fail! I’ve since focused my efforts not so much on the file closing operations, but other collective operations that take place PRIOR to the close. I suspect that some operation(s) that precede the file close are responsible for potentially getting things out-of-sync and hence we observe the failure “down-stream”. At this time, I’m spending my time attempting to trace these other collective operations in order to gain a better understanding what’s happening. The code that we’re testing has not been released yet, so I’m not convinced that you could reproduce it directly from the currently available HDF5 and CGNS downloads. I could send you the actual test, since it links to the static libraries (.a) for both HDF5 and CGNS. Would that be of interest to you? Thanks, Richard On 1/11/17, 11:02 AM, "Rob Latham" <robl(a)mcs.anl.gov> wrote: On 12/29/2016 08:28 AM, Richard Warren wrote: > > > Hi All, > > I’m writing to get some advice and possibly report a bug. The > circumstances are that we are currently working on updating HDF5 > functionality and have run into an issue running a parallel test of a > CFD code (benchmark.hdf) from the CGNS code base > <https://github.com/CGNS/CGNS.git>. I’ve debugged enough to see that > our failure occurs during a call to MPI_File_set_view, with the failure > signature as follows: i'm having a heck of a time building CGNS. When I make cgns, cgnsconvert is failing to find the hdf5 symbols despite me telling cmake where to find libhdf5. did you get that test case you mentioned? ==rob > > > > [brtnfld@jelly] ~/scratch/CGNS/CGNS/src/ptests % mpirun -n 2 benchmark_hdf5 > > Fatal error in PMPI_Barrier: Message truncated, error stack: > > PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed > > MPIR_Barrier_impl(337)..............: Failure during collective > > MPIR_Barrier_impl(330)..............: > > MPIR_Barrier(294)...................: > > MPIR_Barrier_intra(151).............: > > barrier_smp_intra(111)..............: > > MPIR_Bcast_impl(1462)...............: > > MPIR_Bcast(1486)....................: > > MPIR_Bcast_intra(1295)..............: > > MPIR_Bcast_binomial(241)............: > > MPIC_Recv(352)......................: > > MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes > received but buffer size is 1 > > [cli_1]: aborting job: > > Fatal error in PMPI_Barrier: Message truncated, error stack: > > PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed > > MPIR_Barrier_impl(337)..............: Failure during collective > > MPIR_Barrier_impl(330)..............: > > MPIR_Barrier(294)...................: > > MPIR_Barrier_intra(151).............: > > barrier_smp_intra(111)..............: > > MPIR_Bcast_impl(1462)...............: > > MPIR_Bcast(1486)....................: > > MPIR_Bcast_intra(1295)..............: > > MPIR_Bcast_binomial(241)............: > > MPIC_Recv(352)......................: > > MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes > received but buffer size is 1 > > benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: > H5F_close: Assertion `f->file_id > 0' failed. > > Fatal error in PMPI_Allgather: Unknown error class, error stack: > > PMPI_Allgather(1002)......................: > MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, > rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed > > MPIR_Allgather_impl(842)..................: > > MPIR_Allgather(801).......................: > > MPIR_Allgather_intra(216).................: > > MPIC_Sendrecv(475)........................: > > MPIC_Wait(243)............................: > > MPIDI_CH3i_Progress_wait(239).............: an error occurred while > handling an event returned by MPIDU_Sock_Wait() > > MPIDI_CH3I_Progress_handle_sock_event(451): > > MPIDU_Socki_handle_read(649)..............: connection failure > (set=0,sock=1,errno=104:Connection reset by peer) > > [cli_0]: aborting job: > > Fatal error in PMPI_Allgather: Unknown error class, error stack: > > PMPI_Allgather(1002)......................: > MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, > rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed > > MPIR_Allgather_impl(842)..................: > > MPIR_Allgather(801).......................: > > MPIR_Allgather_intra(216).................: > > MPIC_Sendrecv(475)........................: > > MPIC_Wait(243)............................: > > MPIDI_CH3i_Progress_wait(239).............: an error occurred while > handling an event returned by MPIDU_Sock_Wait() > > MPIDI_CH3I_Progress_handle_sock_event(451): > > MPIDU_Socki_handle_read(649)..............: connection failure > (set=0,sock=1,errno=104:Connection reset by peer) > > benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: > H5F_close: Assertion `f->file_id > 0’ failed. > > > > Please note that the above trace was the original stacktrace which > appears to utilize sockets, though I’ve reproduced the same problem by > running on an SMP with shared memory. > > > > While it’s not definitive that the issue has anything to do with the > above stack trace, the very same benchmark runs perfectly well utilizing > PHDF5 built with OpenMPI. My own testing is with MPICH version 3.2 > available from your download site and with OpenMPI 2.0.1 (also their > latest download). Both MPI releases were built from source on my Fedora > 25 Linux distribution using GCC 6.2.1 20160916 (Red Hat 6.2.1-2). > > > > Given that the synchronous calls into MPI_File_set_view appear to be > coded correctly AND that there isn’t much in the way of input parameters > that would cause problems (other than incorrect coding), we tend to > believe that the internal message queues between processes may somehow > be corrupted. This impression is strengthened by the fact that our > recent codebase changes (which are unrelated to the actual calls to > MPI_File_set_view) may have introduced this issue. Note too, that the > code paths to MPI_File_set_view have been taken many times previously > and those function calls have all succeeded. > > > > Are there any suggestions out there as to how to further debug this > potential corruption issue? > > Many thanks, > > Richard A. Warren > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] FW: Potential MPICH problem
by Rob Latham 11 Jan '17

11 Jan '17

On 12/29/2016 08:28 AM, Richard Warren wrote: > > > Hi All, > > I’m writing to get some advice and possibly report a bug. The > circumstances are that we are currently working on updating HDF5 > functionality and have run into an issue running a parallel test of a > CFD code (benchmark.hdf) from the CGNS code base > <https://github.com/CGNS/CGNS.git>. I’ve debugged enough to see that > our failure occurs during a call to MPI_File_set_view, with the failure > signature as follows: i'm having a heck of a time building CGNS. When I make cgns, cgnsconvert is failing to find the hdf5 symbols despite me telling cmake where to find libhdf5. did you get that test case you mentioned? ==rob > > > > [brtnfld@jelly] ~/scratch/CGNS/CGNS/src/ptests % mpirun -n 2 benchmark_hdf5 > > Fatal error in PMPI_Barrier: Message truncated, error stack: > > PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed > > MPIR_Barrier_impl(337)..............: Failure during collective > > MPIR_Barrier_impl(330)..............: > > MPIR_Barrier(294)...................: > > MPIR_Barrier_intra(151).............: > > barrier_smp_intra(111)..............: > > MPIR_Bcast_impl(1462)...............: > > MPIR_Bcast(1486)....................: > > MPIR_Bcast_intra(1295)..............: > > MPIR_Bcast_binomial(241)............: > > MPIC_Recv(352)......................: > > MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes > received but buffer size is 1 > > [cli_1]: aborting job: > > Fatal error in PMPI_Barrier: Message truncated, error stack: > > PMPI_Barrier(430)...................: MPI_Barrier(comm=0x84000006) failed > > MPIR_Barrier_impl(337)..............: Failure during collective > > MPIR_Barrier_impl(330)..............: > > MPIR_Barrier(294)...................: > > MPIR_Barrier_intra(151).............: > > barrier_smp_intra(111)..............: > > MPIR_Bcast_impl(1462)...............: > > MPIR_Bcast(1486)....................: > > MPIR_Bcast_intra(1295)..............: > > MPIR_Bcast_binomial(241)............: > > MPIC_Recv(352)......................: > > MPIDI_CH3U_Request_unpack_uebuf(608): Message truncated; 4 bytes > received but buffer size is 1 > > benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: > H5F_close: Assertion `f->file_id > 0' failed. > > Fatal error in PMPI_Allgather: Unknown error class, error stack: > > PMPI_Allgather(1002)......................: > MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, > rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed > > MPIR_Allgather_impl(842)..................: > > MPIR_Allgather(801).......................: > > MPIR_Allgather_intra(216).................: > > MPIC_Sendrecv(475)........................: > > MPIC_Wait(243)............................: > > MPIDI_CH3i_Progress_wait(239).............: an error occurred while > handling an event returned by MPIDU_Sock_Wait() > > MPIDI_CH3I_Progress_handle_sock_event(451): > > MPIDU_Socki_handle_read(649)..............: connection failure > (set=0,sock=1,errno=104:Connection reset by peer) > > [cli_0]: aborting job: > > Fatal error in PMPI_Allgather: Unknown error class, error stack: > > PMPI_Allgather(1002)......................: > MPI_Allgather(sbuf=0x7ffdfdaf9b10, scount=1, MPI_LONG_LONG_INT, > rbuf=0x1d53ed8, rcount=1, MPI_LONG_LONG_INT, comm=0xc4000002) failed > > MPIR_Allgather_impl(842)..................: > > MPIR_Allgather(801).......................: > > MPIR_Allgather_intra(216).................: > > MPIC_Sendrecv(475)........................: > > MPIC_Wait(243)............................: > > MPIDI_CH3i_Progress_wait(239).............: an error occurred while > handling an event returned by MPIDU_Sock_Wait() > > MPIDI_CH3I_Progress_handle_sock_event(451): > > MPIDU_Socki_handle_read(649)..............: connection failure > (set=0,sock=1,errno=104:Connection reset by peer) > > benchmark_hdf5: /mnt/hdf/brtnfld/hdf5/trunk/hdf5/src/H5Fint.c:1465: > H5F_close: Assertion `f->file_id > 0’ failed. > > > > Please note that the above trace was the original stacktrace which > appears to utilize sockets, though I’ve reproduced the same problem by > running on an SMP with shared memory. > > > > While it’s not definitive that the issue has anything to do with the > above stack trace, the very same benchmark runs perfectly well utilizing > PHDF5 built with OpenMPI. My own testing is with MPICH version 3.2 > available from your download site and with OpenMPI 2.0.1 (also their > latest download). Both MPI releases were built from source on my Fedora > 25 Linux distribution using GCC 6.2.1 20160916 (Red Hat 6.2.1-2). > > > > Given that the synchronous calls into MPI_File_set_view appear to be > coded correctly AND that there isn’t much in the way of input parameters > that would cause problems (other than incorrect coding), we tend to > believe that the internal message queues between processes may somehow > be corrupted. This impression is strengthened by the fact that our > recent codebase changes (which are unrelated to the actual calls to > MPI_File_set_view) may have introduced this issue. Note too, that the > code paths to MPI_File_set_view have been taken many times previously > and those function calls have all succeeded. > > > > Are there any suggestions out there as to how to further debug this > potential corruption issue? > > Many thanks, > > Richard A. Warren > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0