August 2017 - discuss - mailman3.cels.anl.gov

Need help to run hybrid code.
by Pasha Pashaei 28 Aug '17

28 Aug '17

Dear friends I am going to run a hybrid MPI+OPENMP code. As you can see in the below I played with the number of threads in various cases while running my main code with (Openmpi,Mpich,Mpich2). As you can see in OpenMPi and Mpich it seems that openmp did not work at all as Total time did not change considerably. But in Mpich2 Total computational time increased with increasing the number of threads. It could be because of using virtual threads instead of physical threads(or you said that over-subscribing). Hybrid code result (MPI + OpenMP) : Your suggestions: mvapich mpirun -np 4 -genv OMP_NUM_THREADS 1 --bind-to hwthread:1 ./pjet.gfortran > output.txt Total time = 7.290E+02 mpirun -np 4 -genv OMP_NUM_THREADS 8 --bind-to hwthread:8 ./pjet.gfortran > output.txt Total time = 4.940E+02 mpirun -np 4 -genv OMP_NUM_THREADS 8 --bind-to hwthread:8 -map-by hwthread:8 ./pjet.gfortran > output.txt Total time = 4.960E+02 mpirun -np 4 -genv OMP_NUM_THREADS 16 --bind-to hwthread:16 ./pjet.gfortran > output.txt Total time = 4.502E+02 mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to core:16 -map-by core:16 ./pjet.gfortran > output.txt Total time = 4.628E+02 Pervios commands OpenMPI 1.8.1 mpirun -np 4 -x OMP_NUM_THREADS=1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt Total time = 4.475E+02 mpirun -np 4 -x OMP_NUM_THREADS=8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt Total time = 4.525E+02 mpirun -np 4 -x OMP_NUM_THREADS=16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt Total time = 4.611E+02 mvapich mpirun -np 4 -genv OMP_NUM_THREADS 1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt Total time = 4.441E+02 mpirun -np 4 -genv OMP_NUM_THREADS 4 -bind-to socket -map-by socket ./pjet.gfortran > output.txt Total time = 4.535E+02 mpirun -np 4 -genv OMP_NUM_THREADS 8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt Total time = 4.552E+02 mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt Total time = 4.591E+02 mvapich2 mpirun -np 4 -genv OMP_NUM_THREADS 1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt Total time = 4.935E+02 mpirun -np 4 -genv OMP_NUM_THREADS 4 -bind-to socket -map-by socket ./pjet.gfortran > output.txt Total time = 5.562E+02 mpirun -np 4 -genv OMP_NUM_THREADS 8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt Total time = 6.392E+02 mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt Total time = 8.170E+02 Then I used a simple "hybrid.f90" code and its result which I used to check whether the computer can recognize correct value of cores and threads or not. It showed that the correct values in all three in different cases. here is its result: Starting omp_dotprod_hybrid. Using 4 Cores... Core 3 using 16 threads Core 0 using 16 threads Core 2 using 16 threads Core 1 using 16 threads Core 1 thread 0 partial sum = 0.0000000000000000 Core 3 thread 0 partial sum = 0.0000000000000000 Core 1 thread 4 partial sum = 0.0000000000000000 Core 3 thread 7 partial sum = 200.00000000000000 Core 1 thread 8 partial sum = 200.00000000000000 Core 3 thread 9 partial sum = 200.00000000000000 Core 1 thread 11 partial sum = 200.00000000000000 Core 3 thread 3 partial sum = 200.00000000000000 Core 1 thread 2 partial sum = 0.0000000000000000 Core 3 thread 5 partial sum = 0.0000000000000000 Core 1 thread 3 partial sum = 200.00000000000000 Core 3 thread 2 partial sum = 0.0000000000000000 Core 1 thread 13 partial sum = 200.00000000000000 Core 3 thread 12 partial sum = 200.00000000000000 Core 1 thread 1 partial sum = 200.00000000000000 Core 3 thread 1 partial sum = 200.00000000000000 Core 3 thread 8 partial sum = 0.0000000000000000 Core 1 thread 7 partial sum = 0.0000000000000000 Core 3 thread 11 partial sum = 200.00000000000000 Core 1 thread 15 partial sum = 0.0000000000000000 Core 3 thread 15 partial sum = 0.0000000000000000 Core 1 thread 10 partial sum = 200.00000000000000 Core 1 thread 9 partial sum = 200.00000000000000 Core 3 thread 13 partial sum = 0.0000000000000000 Core 1 thread 5 partial sum = 200.00000000000000 Core 3 thread 6 partial sum = 0.0000000000000000 Core 3 thread 4 partial sum = 0.0000000000000000 Core 1 thread 6 partial sum = 0.0000000000000000 Core 3 thread 14 partial sum = 200.00000000000000 Core 1 thread 12 partial sum = 0.0000000000000000 Core 3 thread 10 partial sum = 200.00000000000000 Core 0 thread 0 partial sum = 0.0000000000000000 Core 0 thread 14 partial sum = 200.00000000000000 Core 0 thread 8 partial sum = 200.00000000000000 Core 0 thread 7 partial sum = 0.0000000000000000 Core 0 thread 15 partial sum = 200.00000000000000 Core 0 thread 5 partial sum = 200.00000000000000 Core 0 thread 9 partial sum = 200.00000000000000 Core 0 thread 11 partial sum = 0.0000000000000000 Core 0 thread 10 partial sum = 200.00000000000000 Core 0 thread 6 partial sum = 200.00000000000000 Core 0 thread 3 partial sum = 200.00000000000000 Core 0 thread 4 partial sum = 0.0000000000000000 Core 0 thread 2 partial sum = 0.0000000000000000 Core 0 thread 13 partial sum = 0.0000000000000000 Core 0 thread 12 partial sum = 0.0000000000000000 Core 0 thread 1 partial sum = 0.0000000000000000 Core 0 partial sum = 1600.0000000000000 Core 2 thread 3 partial sum = 0.0000000000000000 Core 2 thread 15 partial sum = 0.0000000000000000 Core 2 thread 0 partial sum = 0.0000000000000000 Core 2 thread 2 partial sum = 200.00000000000000 Core 2 thread 4 partial sum = 0.0000000000000000 Core 2 thread 5 partial sum = 0.0000000000000000 Core 2 thread 9 partial sum = 200.00000000000000 Core 2 thread 7 partial sum = 0.0000000000000000 Core 2 thread 14 partial sum = 200.00000000000000 Core 2 thread 8 partial sum = 200.00000000000000 Core 2 thread 12 partial sum = 200.00000000000000 Core 2 thread 10 partial sum = 200.00000000000000 Core 2 thread 6 partial sum = 200.00000000000000 Core 2 thread 1 partial sum = 0.0000000000000000 Core 2 thread 13 partial sum = 0.0000000000000000 Core 2 thread 11 partial sum = 200.00000000000000 Core 2 partial sum = 1600.0000000000000 Core 3 partial sum = 1600.0000000000000 Core 1 thread 14 partial sum = 0.0000000000000000 Core 1 partial sum = 1600.0000000000000 Done. Hybrid version: global sum = 6400.0000000000000 Please tell me If I should check something. I still getting nowhere. Best regards Pasha Pashaei _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] looking for Torque/PBS mailing list of forum
by Peter Biermann 24 Aug '17

24 Aug '17

Thank you, Halim, I know that this list is not for batch systems. But since message passing is usually done within some batch system, this is the closest I could find to ask people who are familiar with batch systems. I am not asking for a solution with my PBS/Torque problem here, but I am still looking for some mailing list or forum where those system are on-topic. torqueusers-owner(a)supercluster.org doesn't reply to messages, and besides, torqueusers seems pretty dead (about 1 question/month). Thanks Peter <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virenfrei. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> On Mon, Aug 21, 2017 at 4:46 PM, Halim Amer <aamer(a)anl.gov> wrote: > If you have trouble subscribing to the Torque mailing list then you should > contact the admins of that list (torqueusers-owner(a)supercluster.org on > the link you provided). This mailing list is for MPICH, not for PBS/Torque; > we cannot do much to help you. > > Halim > www.mcs.anl.gov/~aamer > > On 8/17/17 12:45 AM, Peter Biermann wrote: > >> Dear MPI users, >> I have a problem with the Torque/PBS system that I'd like to discuss on >> an appropriate mailing list or forum. Can you recommend somewhere to >> discuss such topics? >> I tried http://www.clusterresources.com/mailman/listinfo/torqueusers, >> but it says "You must supply a valid email address." after I filled in my >> name and email. >> >> Thanks >> Peter >> >> <https://www.avast.com/sig-email?utm_medium=email&utm_source >> =link&utm_campaign=sig-email&utm_content=webmail> Virenfrei. >> www.avast.com <https://www.avast.com/sig-ema >> il?utm_medium=email&utm_source=link&utm_campaign=sig-email& >> utm_content=webmail> >> >> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich-master: couldn't build libmpicxx.la with Sun C on Linux
by Chandler 23 Aug '17

23 Aug '17

Siegmar Gross wrote on 08/23/2017 01:51 AM: > Unfortunately I wasn't successful. > > loki mpich-master_2017-08-23_Linux.x86_64.64_cc_sun03 130 tail -13 > log.make.Linux.x86_64.64_cc > "./src/include/mpichconf.h", line 1132: Warning (Anachronism): Attempt > to redefine __restrict__ without using #undef. > 1 Warning(s) detected. > CXXLD lib/libmpicxx.la > CC: -library=Crun cannot be used with -std=c++03. To use this library > you need to switch to -std=sun03 > CC: -library=Cstd cannot be used with -std=c++03. To use this library > you need to switch to -std=sun03 > Makefile:9429: recipe for target 'lib/libmpicxx.la' failed > make[2]: *** [lib/libmpicxx.la] Error 1 > make[2]: Leaving directory CXXLD sounds like the linking phase, so maybe you need to add "-std=sun03" to the LDFLAGS parameter as well? sorry i don't know much about compiling.... _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Binding CPU
by Kenneth Raffenetti 23 Aug '17

23 Aug '17

I think you meant to send to the Open MPI user list. Ken On 08/23/2017 09:56 AM, Pasha Pashaei wrote: > Dear Friends, > > The problem that I faced is I could not bind CPUs yet to test my hybrid > code. > I tried various methods which I found on internet, but the main > problem is as we have different version of Linux and MPI, > there is no general solution for doing that. > Any solutions are specific for their computers and operating systems. > The last and proper solution that I found is following: > module load gcc/5.2.1 > module load openmpi-x86_64 > export OMP_SCHEDULE="dynamic,200" > export OMP_NUM_THREADS=32 > export OMP_PLACES=threads > export OMP_PROC_BIND=spread > numactl --all > numactl -N 0,1 > dbind.txt > numactl -C 0-15,32-47 > numactl -C 1-31,48-63 > numactl --show > dcpu.txt > mpirun -np 2 --map-by ppr:32:socket:pe=2 ./pjet.gfortran > output.txt > > Although it did not face with any error, It did not bind CPU, I do not > know why? > Can you please tell me that am I in right direction? do you have any > recommendation for that? > I am using module (Open MPI) 1.8.1, > Right now, I do not face with any errors. Thus I tried to find a proper > way to bind cpu for running the code in this new condition. > I changed NUMA setting and played with mpirun flags(as you saw above), > but it seems that openmp is not working in this condition. > As computational time do not vary in different cases. (it did not reduce > or change at all even if assuming export OMP_NUM_THREADS=16 or export > OMP_NUM_THREADS=1) > Can you please tell me what else can I do for solving that? > I would appreciate your help and thank in advance. > Thank you in advance > Best regards. > Pasha Pashaei > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] About an error while using mpi i/o collectives : "Error in ADIOI_Calc_aggregator(): rank_index(1)..."
by Thakur, Rajeev 23 Aug '17

23 Aug '17

> Just curious: Is/Was there any specific reason for this constraint? So that it is easier for implementations to implement. Rajeev > On Aug 23, 2017, at 12:35 AM, pramod kumbhar <pramod.s.kumbhar(a)gmail.com> wrote: > > Hi Rajeev, Rob, > > Thanks for clarification. I see etype and filetype specification in the standard mention "monotonically nondecreasing" constraint. > Just curious: Is/Was there any specific reason for this constraint? > > @Rob: the error message I posted in the first email triggered from the application violating this constraint. Depending on the number of ranks and offsets order, the application deadlock, crashes etc. > > Thanks, > > Pramod > > > On Wed, Aug 23, 2017 at 3:07 AM, Latham, Robert J. <robl(a)mcs.anl.gov> wrote: > On Tue, 2017-08-22 at 22:27 +0000, Thakur, Rajeev wrote: > > Yes, displacements for the filetype must be in “monotonically > > nondecreasing order”. > > ... which sounds pretty restrictive, but there is no constraint on > memory types. Folks work around this by shuffling the memory addresses > to match the ascending file offsets. > > ==rob > > > > > Rajeev > > > > > On Aug 22, 2017, at 3:05 PM, pramod kumbhar <pramod.s.kumbhar@gmail > > > .com> wrote: > > > > > > Hi Rob, > > > > > > Thanks! Below is not exactly same issue/error but related : > > > > > > While constructing derived datatype (filetype used for set_view), > > > do we need displacements / offsets to be in ascending order? > > > I mean, suppose I am creating derived datatype using > > > MPI_Type_create_hindexed (or mpi struct) with length/displacements > > > as: > > > > > > blocklengths[0] = 8; > > > blocklengths[1] = 231670; > > > blocklengths[2] = 116606; > > > > > > displacements[0] = 0; > > > displacements[1] = 8; > > > displacements[2] = 231678; > > > > > > Above displacements are in ascending order. Suppose I shuffle order > > > bit: > > > > > > blocklengths[0] = 8; > > > blocklengths[1] = 116606; > > > blocklengths[2] = 231670; > > > > > > displacements[0] = 0; > > > displacements[1] = 231678; > > > displacements[2] = 8; > > > > > > It's still the same but while specifying block-lengths/offsets I > > > changed the order. (resultant file will have data in different oder > > > but that's ignored here) > > > Isn't this a valid specification? This second example results in a > > > segfault (in ADIO_GEN_WriteStrided / Coll). > > > > > > I quickly wrote attached program, let me know if I have missed > > > anything obvious here. > > > > > > Regards, > > > Pramod > > > > > > p.s. you can compile & run as: > > > > > > Not working => mpicxx test.cpp && mpirun -n 2 ./a.out > > > Working =>. mpicxx test.cpp -DUSE_ORDER && mpirun -n 2 ./a.out > > > > > > > > > > > > On Tue, Aug 22, 2017 at 5:25 PM, Latham, Robert J. <robl(a)mcs.anl.go > > > v> wrote: > > > On Mon, 2017-08-21 at 17:45 +0200, pramod kumbhar wrote: > > > > Dear All, > > > > > > > > In one of our application I am seeing following error while using > > > > collective call MPI_File_write_all : > > > > > > > > Error in ADIOI_Calc_aggregator(): rank_index(1) >= fd->hints- > > > > > cb_nodes (1) fd_size=102486061 off=102486469 > > > > > > > > Non collective version works fine. > > > > > > > > While looking at callstack I came across below comment in mpich- > > > > 3.2/src/mpi/romio/adio/common/ad_aggregate.c : > > > > > > > > /* we index into fd_end with rank_index, and fd_end was > > > > allocated > > > > to be no > > > > * bigger than fd->hins->cb_nodes. If we ever violate that, > > > > we're > > > > * overrunning arrays. Obviously, we should never ever hit > > > > this > > > > abort */ > > > > if (rank_index >= fd->hints->cb_nodes || rank_index < 0) { > > > > FPRINTF(stderr, "Error in ADIOI_Calc_aggregator(): > > > > rank_index(%d) >= fd->hints->cb_nodes (%d) fd_size=%lld > > > > off=%lld\n", > > > > rank_index,fd->hints->cb_nodes,fd_size,off); > > > > MPI_Abort(MPI_COMM_WORLD, 1); > > > > } > > > > > > > > I am going to look into application and see if there is an issue > > > > with > > > > offset overflow. But looking at above comment ("Obviously, we > > > > should > > > > never ever hit this abort ") I thought should ask if there is any > > > > obvious thing I am missing. > > > > > > that's my comment. The 'rank_index' array is allocated based on > > > the > > > 'cb_nodes' hint. I definitely would like to know more about how > > > the > > > code is manipulating rank_index, cb_nodes, and fd_end . > > > > > > If there is a reduced test case you can send me, that will be a > > > huge > > > help. > > > > > > ==rob > > > > > > > > > > > Regards, > > > > Pramod > > > > > > > > p.s. I will provide reproducer after looking into this more > > > > carefully. > > > > _______________________________________________ > > > > discuss mailing list discuss(a)mpich.org > > > > To manage subscription options or unsubscribe: > > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > <test.cpp>_______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Binding CPU
by Pasha Pashaei 23 Aug '17

23 Aug '17

Dear Friends, The problem that I faced is I could not bind CPUs yet to test my hybrid code. I tried various methods which I found on internet, but the main problem is as we have different version of Linux and MPI, there is no general solution for doing that. Any solutions are specific for their computers and operating systems. The last and proper solution that I found is following: module load gcc/5.2.1 module load openmpi-x86_64 export OMP_SCHEDULE="dynamic,200" export OMP_NUM_THREADS=32 export OMP_PLACES=threads export OMP_PROC_BIND=spread numactl --all numactl -N 0,1 > dbind.txt numactl -C 0-15,32-47 numactl -C 1-31,48-63 numactl --show > dcpu.txt mpirun -np 2 --map-by ppr:32:socket:pe=2 ./pjet.gfortran > output.txt Although it did not face with any error, It did not bind CPU, I do not know why? Can you please tell me that am I in right direction? do you have any recommendation for that? I am using module (Open MPI) 1.8.1, Right now, I do not face with any errors. Thus I tried to find a proper way to bind cpu for running the code in this new condition. I changed NUMA setting and played with mpirun flags(as you saw above), but it seems that openmp is not working in this condition. As computational time do not vary in different cases. (it did not reduce or change at all even if assuming export OMP_NUM_THREADS=16 or export OMP_NUM_THREADS=1) Can you please tell me what else can I do for solving that? I would appreciate your help and thank in advance. Thank you in advance Best regards. Pasha Pashaei _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich-master: couldn't build libmpicxx.la with Sun C on Linux
by Siegmar Gross 23 Aug '17

23 Aug '17

Hi, > OK good luck! also be sure to post your updates to the list... i'm sorry that i > replied directly to you instead of the list.... > > Siegmar Gross wrote on 08/22/2017 05:31 AM: >> >> I used the wrong compiler! "CC" allows "-std=sun03". I will try to >> build it once more. Unfortunately I wasn't successful. loki mpich-master_2017-08-23_Linux.x86_64.64_cc_sun03 129 head -2 log.configure.Linux.x86_64.64_cc Configuring MPICH version 3.3a2 with '--prefix=/usr/local/mpich-master_64_cc' '--libdir=/usr/local/mpich-master_64_cc/lib64' '--includedir=/usr/local/mpich-master_64_cc/include64' 'CC=cc' 'CXX=CC' 'F77=f77' 'FC=f95' 'CFLAGS=-m64' 'CXXFLAGS=-m64 -std=sun03' 'FFLAGS=-m64' 'FCFLAGS=-m64' 'LDFLAGS=-m64 -L/usr/lib64 -R/usr/lib64' '--enable-fortran=ye' '--enable-cxx' '--enable-romio' '--enable-debuginfo' '--enable-smpcoll' '--enable-threads=multiple' '--with-thread-package=posix' '--enable-shared' Running on system: Linux loki 4.4.74-92.29-default #1 SMP Thu Jun 29 13:06:32 UTC 2017 (561ddb1) x86_64 x86_64 x86_64 GNU/Linux loki mpich-master_2017-08-23_Linux.x86_64.64_cc_sun03 130 tail -13 log.make.Linux.x86_64.64_cc "./src/include/mpichconf.h", line 1132: Warning (Anachronism): Attempt to redefine __restrict__ without using #undef. 1 Warning(s) detected. CXXLD lib/libmpicxx.la CC: -library=Crun cannot be used with -std=c++03. To use this library you need to switch to -std=sun03 CC: -library=Cstd cannot be used with -std=c++03. To use this library you need to switch to -std=sun03 Makefile:9429: recipe for target 'lib/libmpicxx.la' failed make[2]: *** [lib/libmpicxx.la] Error 1 make[2]: Leaving directory '/export2/src/mpich-master/mpich-master_2017-08-23_Linux.x86_64.64_cc_sun03' Makefile:34562: recipe for target 'all-recursive' failed make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/export2/src/mpich-master/mpich-master_2017-08-23_Linux.x86_64.64_cc_sun03' Makefile:9242: recipe for target 'all' failed make: *** [all] Error 2 loki mpich-master_2017-08-23_Linux.x86_64.64_cc_sun03 131 Thank you very much for your suggestion and kind regards Siegmar _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] About an error while using mpi i/o collectives : "Error in ADIOI_Calc_aggregator(): rank_index(1)..."
by pramod kumbhar 23 Aug '17

23 Aug '17

> Thanks for clarification. I see type and filetype specification in the > standard mention "monotonically nondecreasing" constraint. > I mean etype and filetype. -Pramod On Wed, Aug 23, 2017 at 3:07 AM, Latham, Robert J. <robl(a)mcs.anl.gov> wrote: > >> On Tue, 2017-08-22 at 22:27 +0000, Thakur, Rajeev wrote: >> > Yes, displacements for the filetype must be in “monotonically >> > nondecreasing order”. >> >> ... which sounds pretty restrictive, but there is no constraint on >> memory types. Folks work around this by shuffling the memory addresses >> to match the ascending file offsets. >> >> ==rob >> >> > >> > Rajeev >> > >> > > On Aug 22, 2017, at 3:05 PM, pramod kumbhar <pramod.s.kumbhar@gmail >> > > .com> wrote: >> > > >> > > Hi Rob, >> > > >> > > Thanks! Below is not exactly same issue/error but related : >> > > >> > > While constructing derived datatype (filetype used for set_view), >> > > do we need displacements / offsets to be in ascending order? >> > > I mean, suppose I am creating derived datatype using >> > > MPI_Type_create_hindexed (or mpi struct) with length/displacements >> > > as: >> > > >> > > blocklengths[0] = 8; >> > > blocklengths[1] = 231670; >> > > blocklengths[2] = 116606; >> > > >> > > displacements[0] = 0; >> > > displacements[1] = 8; >> > > displacements[2] = 231678; >> > > >> > > Above displacements are in ascending order. Suppose I shuffle order >> > > bit: >> > > >> > > blocklengths[0] = 8; >> > > blocklengths[1] = 116606; >> > > blocklengths[2] = 231670; >> > > >> > > displacements[0] = 0; >> > > displacements[1] = 231678; >> > > displacements[2] = 8; >> > > >> > > It's still the same but while specifying block-lengths/offsets I >> > > changed the order. (resultant file will have data in different oder >> > > but that's ignored here) >> > > Isn't this a valid specification? This second example results in a >> > > segfault (in ADIO_GEN_WriteStrided / Coll). >> > > >> > > I quickly wrote attached program, let me know if I have missed >> > > anything obvious here. >> > > >> > > Regards, >> > > Pramod >> > > >> > > p.s. you can compile & run as: >> > > >> > > Not working => mpicxx test.cpp && mpirun -n 2 ./a.out >> > > Working =>. mpicxx test.cpp -DUSE_ORDER && mpirun -n 2 ./a.out >> > > >> > > >> > > >> > > On Tue, Aug 22, 2017 at 5:25 PM, Latham, Robert J. <robl(a)mcs.anl.go >> > > v> wrote: >> > > On Mon, 2017-08-21 at 17:45 +0200, pramod kumbhar wrote: >> > > > Dear All, >> > > > >> > > > In one of our application I am seeing following error while using >> > > > collective call MPI_File_write_all : >> > > > >> > > > Error in ADIOI_Calc_aggregator(): rank_index(1) >= fd->hints- >> > > > > cb_nodes (1) fd_size=102486061 off=102486469 >> > > > >> > > > Non collective version works fine. >> > > > >> > > > While looking at callstack I came across below comment in mpich- >> > > > 3.2/src/mpi/romio/adio/common/ad_aggregate.c : >> > > > >> > > > /* we index into fd_end with rank_index, and fd_end was >> > > > allocated >> > > > to be no >> > > > * bigger than fd->hins->cb_nodes. If we ever violate that, >> > > > we're >> > > > * overrunning arrays. Obviously, we should never ever hit >> > > > this >> > > > abort */ >> > > > if (rank_index >= fd->hints->cb_nodes || rank_index < 0) { >> > > > FPRINTF(stderr, "Error in ADIOI_Calc_aggregator(): >> > > > rank_index(%d) >= fd->hints->cb_nodes (%d) fd_size=%lld >> > > > off=%lld\n", >> > > > rank_index,fd->hints->cb_nodes,fd_size,off); >> > > > MPI_Abort(MPI_COMM_WORLD, 1); >> > > > } >> > > > >> > > > I am going to look into application and see if there is an issue >> > > > with >> > > > offset overflow. But looking at above comment ("Obviously, we >> > > > should >> > > > never ever hit this abort ") I thought should ask if there is any >> > > > obvious thing I am missing. >> > > >> > > that's my comment. The 'rank_index' array is allocated based on >> > > the >> > > 'cb_nodes' hint. I definitely would like to know more about how >> > > the >> > > code is manipulating rank_index, cb_nodes, and fd_end . >> > > >> > > If there is a reduced test case you can send me, that will be a >> > > huge >> > > help. >> > > >> > > ==rob >> > > >> > > > >> > > > Regards, >> > > > Pramod >> > > > >> > > > p.s. I will provide reproducer after looking into this more >> > > > carefully. >> > > > _______________________________________________ >> > > > discuss mailing list discuss(a)mpich.org >> > > > To manage subscription options or unsubscribe: >> > > > https://lists.mpich.org/mailman/listinfo/discuss >> > > >> > > _______________________________________________ >> > > discuss mailing list discuss(a)mpich.org >> > > To manage subscription options or unsubscribe: >> > > https://lists.mpich.org/mailman/listinfo/discuss >> > > >> > > <test.cpp>_______________________________________________ >> > > discuss mailing list discuss(a)mpich.org >> > > To manage subscription options or unsubscribe: >> > > https://lists.mpich.org/mailman/listinfo/discuss >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] About an error while using mpi i/o collectives : "Error in ADIOI_Calc_aggregator(): rank_index(1)..."
by pramod kumbhar 23 Aug '17

23 Aug '17

Hi Rajeev, Rob, Thanks for clarification. I see type and filetype specification in the standard mention "monotonically nondecreasing" constraint. Just curious: Is/Was there any specific reason for this constraint? @Rob: the error message I posted in the first email triggered from the application violating this constraint. Depending on the number of ranks and offsets order, the application deadlock, crashes etc. Thanks, Pramod On Wed, Aug 23, 2017 at 3:07 AM, Latham, Robert J. <robl(a)mcs.anl.gov> wrote: > On Tue, 2017-08-22 at 22:27 +0000, Thakur, Rajeev wrote: > > Yes, displacements for the filetype must be in “monotonically > > nondecreasing order”. > > ... which sounds pretty restrictive, but there is no constraint on > memory types. Folks work around this by shuffling the memory addresses > to match the ascending file offsets. > > ==rob > > > > > Rajeev > > > > > On Aug 22, 2017, at 3:05 PM, pramod kumbhar <pramod.s.kumbhar@gmail > > > .com> wrote: > > > > > > Hi Rob, > > > > > > Thanks! Below is not exactly same issue/error but related : > > > > > > While constructing derived datatype (filetype used for set_view), > > > do we need displacements / offsets to be in ascending order? > > > I mean, suppose I am creating derived datatype using > > > MPI_Type_create_hindexed (or mpi struct) with length/displacements > > > as: > > > > > > blocklengths[0] = 8; > > > blocklengths[1] = 231670; > > > blocklengths[2] = 116606; > > > > > > displacements[0] = 0; > > > displacements[1] = 8; > > > displacements[2] = 231678; > > > > > > Above displacements are in ascending order. Suppose I shuffle order > > > bit: > > > > > > blocklengths[0] = 8; > > > blocklengths[1] = 116606; > > > blocklengths[2] = 231670; > > > > > > displacements[0] = 0; > > > displacements[1] = 231678; > > > displacements[2] = 8; > > > > > > It's still the same but while specifying block-lengths/offsets I > > > changed the order. (resultant file will have data in different oder > > > but that's ignored here) > > > Isn't this a valid specification? This second example results in a > > > segfault (in ADIO_GEN_WriteStrided / Coll). > > > > > > I quickly wrote attached program, let me know if I have missed > > > anything obvious here. > > > > > > Regards, > > > Pramod > > > > > > p.s. you can compile & run as: > > > > > > Not working => mpicxx test.cpp && mpirun -n 2 ./a.out > > > Working =>. mpicxx test.cpp -DUSE_ORDER && mpirun -n 2 ./a.out > > > > > > > > > > > > On Tue, Aug 22, 2017 at 5:25 PM, Latham, Robert J. <robl(a)mcs.anl.go > > > v> wrote: > > > On Mon, 2017-08-21 at 17:45 +0200, pramod kumbhar wrote: > > > > Dear All, > > > > > > > > In one of our application I am seeing following error while using > > > > collective call MPI_File_write_all : > > > > > > > > Error in ADIOI_Calc_aggregator(): rank_index(1) >= fd->hints- > > > > > cb_nodes (1) fd_size=102486061 off=102486469 > > > > > > > > Non collective version works fine. > > > > > > > > While looking at callstack I came across below comment in mpich- > > > > 3.2/src/mpi/romio/adio/common/ad_aggregate.c : > > > > > > > > /* we index into fd_end with rank_index, and fd_end was > > > > allocated > > > > to be no > > > > * bigger than fd->hins->cb_nodes. If we ever violate that, > > > > we're > > > > * overrunning arrays. Obviously, we should never ever hit > > > > this > > > > abort */ > > > > if (rank_index >= fd->hints->cb_nodes || rank_index < 0) { > > > > FPRINTF(stderr, "Error in ADIOI_Calc_aggregator(): > > > > rank_index(%d) >= fd->hints->cb_nodes (%d) fd_size=%lld > > > > off=%lld\n", > > > > rank_index,fd->hints->cb_nodes,fd_size,off); > > > > MPI_Abort(MPI_COMM_WORLD, 1); > > > > } > > > > > > > > I am going to look into application and see if there is an issue > > > > with > > > > offset overflow. But looking at above comment ("Obviously, we > > > > should > > > > never ever hit this abort ") I thought should ask if there is any > > > > obvious thing I am missing. > > > > > > that's my comment. The 'rank_index' array is allocated based on > > > the > > > 'cb_nodes' hint. I definitely would like to know more about how > > > the > > > code is manipulating rank_index, cb_nodes, and fd_end . > > > > > > If there is a reduced test case you can send me, that will be a > > > huge > > > help. > > > > > > ==rob > > > > > > > > > > > Regards, > > > > Pramod > > > > > > > > p.s. I will provide reproducer after looking into this more > > > > carefully. > > > > _______________________________________________ > > > > discuss mailing list discuss(a)mpich.org > > > > To manage subscription options or unsubscribe: > > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > <test.cpp>_______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] About an error while using mpi i/o collectives : "Error in ADIOI_Calc_aggregator(): rank_index(1)..."
by Latham, Robert J. 22 Aug '17

22 Aug '17

On Tue, 2017-08-22 at 22:27 +0000, Thakur, Rajeev wrote: > Yes, displacements for the filetype must be in “monotonically > nondecreasing order”. ... which sounds pretty restrictive, but there is no constraint on memory types. Folks work around this by shuffling the memory addresses to match the ascending file offsets. ==rob > > Rajeev > > > On Aug 22, 2017, at 3:05 PM, pramod kumbhar <pramod.s.kumbhar@gmail > > .com> wrote: > > > > Hi Rob, > > > > Thanks! Below is not exactly same issue/error but related : > > > > While constructing derived datatype (filetype used for set_view), > > do we need displacements / offsets to be in ascending order? > > I mean, suppose I am creating derived datatype using > > MPI_Type_create_hindexed (or mpi struct) with length/displacements > > as: > > > > blocklengths[0] = 8; > > blocklengths[1] = 231670; > > blocklengths[2] = 116606; > > > > displacements[0] = 0; > > displacements[1] = 8; > > displacements[2] = 231678; > > > > Above displacements are in ascending order. Suppose I shuffle order > > bit: > > > > blocklengths[0] = 8; > > blocklengths[1] = 116606; > > blocklengths[2] = 231670; > > > > displacements[0] = 0; > > displacements[1] = 231678; > > displacements[2] = 8; > > > > It's still the same but while specifying block-lengths/offsets I > > changed the order. (resultant file will have data in different oder > > but that's ignored here) > > Isn't this a valid specification? This second example results in a > > segfault (in ADIO_GEN_WriteStrided / Coll). > > > > I quickly wrote attached program, let me know if I have missed > > anything obvious here. > > > > Regards, > > Pramod > > > > p.s. you can compile & run as: > > > > Not working => mpicxx test.cpp && mpirun -n 2 ./a.out > > Working =>. mpicxx test.cpp -DUSE_ORDER && mpirun -n 2 ./a.out > > > > > > > > On Tue, Aug 22, 2017 at 5:25 PM, Latham, Robert J. <robl(a)mcs.anl.go > > v> wrote: > > On Mon, 2017-08-21 at 17:45 +0200, pramod kumbhar wrote: > > > Dear All, > > > > > > In one of our application I am seeing following error while using > > > collective call MPI_File_write_all : > > > > > > Error in ADIOI_Calc_aggregator(): rank_index(1) >= fd->hints- > > > > cb_nodes (1) fd_size=102486061 off=102486469 > > > > > > Non collective version works fine. > > > > > > While looking at callstack I came across below comment in mpich- > > > 3.2/src/mpi/romio/adio/common/ad_aggregate.c : > > > > > > /* we index into fd_end with rank_index, and fd_end was > > > allocated > > > to be no > > > * bigger than fd->hins->cb_nodes. If we ever violate that, > > > we're > > > * overrunning arrays. Obviously, we should never ever hit > > > this > > > abort */ > > > if (rank_index >= fd->hints->cb_nodes || rank_index < 0) { > > > FPRINTF(stderr, "Error in ADIOI_Calc_aggregator(): > > > rank_index(%d) >= fd->hints->cb_nodes (%d) fd_size=%lld > > > off=%lld\n", > > > rank_index,fd->hints->cb_nodes,fd_size,off); > > > MPI_Abort(MPI_COMM_WORLD, 1); > > > } > > > > > > I am going to look into application and see if there is an issue > > > with > > > offset overflow. But looking at above comment ("Obviously, we > > > should > > > never ever hit this abort ") I thought should ask if there is any > > > obvious thing I am missing. > > > > that's my comment. The 'rank_index' array is allocated based on > > the > > 'cb_nodes' hint. I definitely would like to know more about how > > the > > code is manipulating rank_index, cb_nodes, and fd_end . > > > > If there is a reduced test case you can send me, that will be a > > huge > > help. > > > > ==rob > > > > > > > > Regards, > > > Pramod > > > > > > p.s. I will provide reproducer after looking into this more > > > carefully. > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > <test.cpp>_______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0