February 2015 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Memory alignment with MPI_Alloc_mem
by Marcin Zalewski 16 Feb '15

16 Feb '15

Thanks for your answer Jeff. On Mon, Feb 16, 2015 at 1:42 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > If you are going to suballocate from a slab yourself, you can handle > alignment yourself easy enough, no? Do I not understand what you mean > here. And what sort of alignment do you want? Are you trying to > align to 32/64 bytes because of AVX or some other x86 feature on Cray > XC30 or do you want page alignment? We have an implementation of a global heap using a custom memory allocator that depends on the heap being aligned to the maximum size of a block, which we want to be 32 bits where possible. We are able to achieve this with mmap by trying to allocate at addresses that are right for our heap. So, trying to allocate random memory and align to 32 bits may be difficult. > But what do you really want to achieve? While it is usually > beneficial to use pre-registered buffers on RDMA networks, good MPI > implementations have a page-registration cache. If, as you say, you > are suballocating from a slab, Cray MPI should have the backing pages > in the registration cache after you use them as MPI buffers. That's true, I did not think of registration cache. As you say below, if I use huge pages for our heap, there should not be a problem with the whole heap being cached. So probably the complexity of dealing with MPI_Alloc_mem is not worth it at all. > You can maximize the efficiency of the page registration cache by > using large pages. Search for intro_hugepages using 'man' or on the > Internet to learn the specifics of this. I suspect that using large > pages will induce much of the benefit you hoped to achieve with an > explicitly-registering MPI_Alloc_mem. > > If you really want to max out RDMA on Cray networks, you need to use > DMAPP. I have some simple examples and pointers to docs here: > https://github.com/jeffhammond/HPCInfo/tree/master/dmapp. I have more > examples other places that I'll migrate to that location if requested. > > If you're interested in portability, MPI-3 RMA is a good abstraction > for RDMA networks. Some implementations do a better job than others > at exposing this relationship. Cray MPI has a DMAPP back-end for RMA > now, although it is not active by default. You could also try Torsten > Hoefler's foMPI > [http://spcl.inf.ethz.ch/Research/Parallel_Programming/foMPI/] > > Best, > > Jeff > > On Sat, Feb 14, 2015 at 2:37 PM, Marcin Zalewski > <marcin.zalewski(a)gmail.com> wrote: >> I am using Cray MPT, and I would like to allocate a large region of >> memory from which, in turn, I will allocate buffers to be used with >> MPI. I am wondering if there is any benefit from allocating that heap >> with MPI_Alloc_mem. I would hope that it could be pre-registered for >> RDMA, speeding things up. However, I need this memory to have a >> specific alignment. Is there a general way in MPICH or maybe a >> specific way for MPT to request alignment with MPI_Alloc_mem? >> >> Thanks, >> Marcin >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Memory alignment with MPI_Alloc_mem
by Atchley, Scott 16 Feb '15

16 Feb '15

Jeff, I expect that he is concerned with GNI's 4-byte alignment requirement for both the address and length for RDMA Reads and the 8-byte alignment for atomics. Scott On Feb 16, 2015, at 1:42 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > If you are going to suballocate from a slab yourself, you can handle > alignment yourself easy enough, no? Do I not understand what you mean > here. And what sort of alignment do you want? Are you trying to > align to 32/64 bytes because of AVX or some other x86 feature on Cray > XC30 or do you want page alignment? > > But what do you really want to achieve? While it is usually > beneficial to use pre-registered buffers on RDMA networks, good MPI > implementations have a page-registration cache. If, as you say, you > are suballocating from a slab, Cray MPI should have the backing pages > in the registration cache after you use them as MPI buffers. > > You can maximize the efficiency of the page registration cache by > using large pages. Search for intro_hugepages using 'man' or on the > Internet to learn the specifics of this. I suspect that using large > pages will induce much of the benefit you hoped to achieve with an > explicitly-registering MPI_Alloc_mem. > > If you really want to max out RDMA on Cray networks, you need to use > DMAPP. I have some simple examples and pointers to docs here: > https://github.com/jeffhammond/HPCInfo/tree/master/dmapp. I have more > examples other places that I'll migrate to that location if requested. > > If you're interested in portability, MPI-3 RMA is a good abstraction > for RDMA networks. Some implementations do a better job than others > at exposing this relationship. Cray MPI has a DMAPP back-end for RMA > now, although it is not active by default. You could also try Torsten > Hoefler's foMPI > [http://spcl.inf.ethz.ch/Research/Parallel_Programming/foMPI/] > > Best, > > Jeff > > On Sat, Feb 14, 2015 at 2:37 PM, Marcin Zalewski > <marcin.zalewski(a)gmail.com> wrote: >> I am using Cray MPT, and I would like to allocate a large region of >> memory from which, in turn, I will allocate buffers to be used with >> MPI. I am wondering if there is any benefit from allocating that heap >> with MPI_Alloc_mem. I would hope that it could be pre-registered for >> RDMA, speeding things up. However, I need this memory to have a >> specific alignment. Is there a general way in MPICH or maybe a >> specific way for MPT to request alignment with MPI_Alloc_mem? >> >> Thanks, >> Marcin >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Memory alignment with MPI_Alloc_mem
by Jeff Hammond 16 Feb '15

16 Feb '15

If you are going to suballocate from a slab yourself, you can handle alignment yourself easy enough, no? Do I not understand what you mean here. And what sort of alignment do you want? Are you trying to align to 32/64 bytes because of AVX or some other x86 feature on Cray XC30 or do you want page alignment? But what do you really want to achieve? While it is usually beneficial to use pre-registered buffers on RDMA networks, good MPI implementations have a page-registration cache. If, as you say, you are suballocating from a slab, Cray MPI should have the backing pages in the registration cache after you use them as MPI buffers. You can maximize the efficiency of the page registration cache by using large pages. Search for intro_hugepages using 'man' or on the Internet to learn the specifics of this. I suspect that using large pages will induce much of the benefit you hoped to achieve with an explicitly-registering MPI_Alloc_mem. If you really want to max out RDMA on Cray networks, you need to use DMAPP. I have some simple examples and pointers to docs here: https://github.com/jeffhammond/HPCInfo/tree/master/dmapp. I have more examples other places that I'll migrate to that location if requested. If you're interested in portability, MPI-3 RMA is a good abstraction for RDMA networks. Some implementations do a better job than others at exposing this relationship. Cray MPI has a DMAPP back-end for RMA now, although it is not active by default. You could also try Torsten Hoefler's foMPI [http://spcl.inf.ethz.ch/Research/Parallel_Programming/foMPI/] Best, Jeff On Sat, Feb 14, 2015 at 2:37 PM, Marcin Zalewski <marcin.zalewski(a)gmail.com> wrote: > I am using Cray MPT, and I would like to allocate a large region of > memory from which, in turn, I will allocate buffers to be used with > MPI. I am wondering if there is any benefit from allocating that heap > with MPI_Alloc_mem. I would hope that it could be pre-registered for > RDMA, speeding things up. However, I need this memory to have a > specific alignment. Is there a general way in MPICH or maybe a > specific way for MPT to request alignment with MPI_Alloc_mem? > > Thanks, > Marcin > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpiexec hydra : signal handling SIGINT with popen
by Balaji, Pavan 15 Feb '15

15 Feb '15

Hirak, The signal is forwarded to the MPI processes. I don't know how "popen" works compared to the "system" call, to comment on why one is working and the other isn't. -- Pavan > On Feb 14, 2015, at 11:27 AM, Roy, Hirak <Hirak_Roy(a)mentor.com> wrote: > > Hi All, > > I am using a wrapper program which runs “mpiexec –n 5 actual_program” using a popen command. > [ This is the simplified version of running my program. Ideally there is a TCL layer which does these things]. > I have attached my program here. > > My actual program has installed SIGINT signal handler and takes some action before it exits. > It also writes lot of information on stdout and a file. > > In general, if I press CTRL+C, I expect SIGINT to be passed to all the underlying processes. > However, in my case, although the processes receive SIGINT, they are killed immediately. > > I have noticed that If I run using > system(“mpiexec –n 5 actual_program”), SIGINT is delivered and appropriate actions are taken. > > I am a bit confused here. The only thing I can relate is that there is somehow a SIGPIPE signal which might lead mpiexec to kill all subprocesses. > Can someone please have a look and let me know what is the issue here and what can be a done. > > Thanks, > Hirak > > // wrapper program > > #include <stdio.h> > #include <signal.h> > #include <string.h> > #include <stdlib.h> > > > > int main () > { > > const char* command = "mpiexec -n 5 ./mpiprog.out"; > #if 1 > FILE *fp = popen(command, "r"); > if (!fp) { > printf ("failed\n"); > } > while (!feof(fp)) { > char ch ; > fscanf(fp, "%c", &ch); > printf ("%c", ch); > } > > pclose(fp); > #else > system (command); > #endif > } > > //mpiprog.c > #include "mpi.h" > #include <signal.h> > #include <string.h> > #include <stdio.h> > #include <stdlib.h> > > void signalHandler(int signum, siginfo_t *info, void *ptr) > { > fprintf (stdout,"******** SIGINT *******************\n"); > MPI_Finalize(); > exit (0); > } > > int main(int argc, char** argv) > { > struct sigaction act; > memset(&act, 0, sizeof(act)); > act.sa_sigaction = signalHandler; > act.sa_flags = SA_SIGINFO; > sigaction(SIGINT, &act, NULL); > > > MPI_Init(&argc, &argv); > > > while (1) > { > > sleep (10); > printf ("---Hhahaha\n"); > } > > MPI_Finalize(); > } > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Memory alignment with MPI_Alloc_mem
by Balaji, Pavan 15 Feb '15

15 Feb '15

There's no generic way in MPICH to do that, but we could add an info argument for such hints. For the Cray MPT, you'll need to check with the Cray folks. We have a ticket requesting this feature: https://trac.mpich.org/projects/mpich/ticket/1526 -- Pavan > On Feb 14, 2015, at 4:37 PM, Marcin Zalewski <marcin.zalewski(a)gmail.com> wrote: > > I am using Cray MPT, and I would like to allocate a large region of > memory from which, in turn, I will allocate buffers to be used with > MPI. I am wondering if there is any benefit from allocating that heap > with MPI_Alloc_mem. I would hope that it could be pre-registered for > RDMA, speeding things up. However, I need this memory to have a > specific alignment. Is there a general way in MPICH or maybe a > specific way for MPT to request alignment with MPI_Alloc_mem? > > Thanks, > Marcin > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Memory alignment with MPI_Alloc_mem
by Marcin Zalewski 14 Feb '15

14 Feb '15

I am using Cray MPT, and I would like to allocate a large region of memory from which, in turn, I will allocate buffers to be used with MPI. I am wondering if there is any benefit from allocating that heap with MPI_Alloc_mem. I would hope that it could be pre-registered for RDMA, speeding things up. However, I need this memory to have a specific alignment. Is there a general way in MPICH or maybe a specific way for MPT to request alignment with MPI_Alloc_mem? Thanks, Marcin _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpiexec hydra : signal handling SIGINT with popen
by Roy, Hirak 14 Feb '15

14 Feb '15

Hi All, I am using a wrapper program which runs "mpiexec -n 5 actual_program" using a popen command. [ This is the simplified version of running my program. Ideally there is a TCL layer which does these things]. I have attached my program here. My actual program has installed SIGINT signal handler and takes some action before it exits. It also writes lot of information on stdout and a file. In general, if I press CTRL+C, I expect SIGINT to be passed to all the underlying processes. However, in my case, although the processes receive SIGINT, they are killed immediately. I have noticed that If I run using system("mpiexec -n 5 actual_program"), SIGINT is delivered and appropriate actions are taken. I am a bit confused here. The only thing I can relate is that there is somehow a SIGPIPE signal which might lead mpiexec to kill all subprocesses. Can someone please have a look and let me know what is the issue here and what can be a done. Thanks, Hirak // wrapper program #include <stdio.h> #include <signal.h> #include <string.h> #include <stdlib.h> int main () { const char* command = "mpiexec -n 5 ./mpiprog.out"; #if 1 FILE *fp = popen(command, "r"); if (!fp) { printf ("failed\n"); } while (!feof(fp)) { char ch ; fscanf(fp, "%c", &ch); printf ("%c", ch); } pclose(fp); #else system (command); #endif } //mpiprog.c #include "mpi.h" #include <signal.h> #include <string.h> #include <stdio.h> #include <stdlib.h> void signalHandler(int signum, siginfo_t *info, void *ptr) { fprintf (stdout,"******** SIGINT *******************\n"); MPI_Finalize(); exit (0); } int main(int argc, char** argv) { struct sigaction act; memset(&act, 0, sizeof(act)); act.sa_sigaction = signalHandler; act.sa_flags = SA_SIGINFO; sigaction(SIGINT, &act, NULL); MPI_Init(&argc, &argv); while (1) { sleep (10); printf ("---Hhahaha\n"); } MPI_Finalize(); } _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Issue in MPI_Win_get_attr(MPI_WIN_SIZE)
by Hajime Fujita 09 Feb '15

09 Feb '15

Ah, I see, I totally missed that sentence in the specification. Now it works fine if I interpret the returned value as a pointer. Thank you, Hajime > On Feb 9, 2015, at 1:09 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > > > > On Mon, Feb 9, 2015 at 12:29 PM, Hajime Fujita <hfujita(a)uchicago.edu> wrote: > Hello, > > I think I found an issue in MPI_Win_get_attr, when MPI_WIN_SIZE is specified. It is supposed to return the size of the window, but it seems to give a random number. > > Just wanted to confirm one thing: I think MPI_Win_get_attr is supposed to return a value of type MPI_Aint, when MPI_WIN_SIZE is specified. Is it correct? > > No. According to p415 of MPI-3.0 spec, in C bindings, MPI_Win_get_attr(..., MPI_WIN_SIZE, &size, ...) return in size "pointer" to the size of the window. In Fortran bindings, it returns a value instead. > > > > Test program: > See the attached file > > Result I got: > hfujita@lssg-lp-00:~/work/mpi_win_size_test$ ./win_size_test > Got size=140729394560728, while expecting 1024 > hfujita@lssg-lp-00:~/work/mpi_win_size_test$ ./win_size_test > Got size=139994694213336, while expecting 1024 > hfujita@lssg-lp-00:~/work/mpi_win_size_test$ ./win_size_test > Got size=139739628887768, while expecting 1024 > hfujita@lssg-lp-00:~/work/mpi_win_size_test$ ./win_size_test > Got size=140131810524888, while expecting 1024 > hfujita@lssg-lp-00:~/work/mpi_win_size_test$ ./win_size_test > Got size=139859819581144, while expecting 1024 > > Expected result: > If the MPI library works as I expected, the test program won’t show any message. > > MPICH version and configuration: > $ mpichversion > MPICH Version: 3.2a2 > MPICH Release date: unreleased development copy > MPICH Device: ch3:nemesis > MPICH configure: CC=gcc CXX=g++ FC=gfortran F77=gfortran CFLAGS=-O0 --prefix=/home/hfujita/opt/mpich-3.2 --disable-mpe --disable-romio --enable-g=all --enable-spawn --enable-maintainer-mode --enable-error-checking=all --with-pm=hydra --with-pmi=simple --disable-cxx --enable-fortran --enable-strict=noopt --disable-fast --disable-perftest --no-create --no-recursion > MPICH CC: gcc -O0 -Wall -Wextra -Wno-missing-field-initializers -Wmissing-prototypes -DGCC_WALL -Wno-unused-parameter -Wno-unused-label -Wshadow -Wmissing-declarations -Wno-long-long -Wundef -Wno-endif-labels -Wpointer-arith -Wbad-function-cast -Wwrite-strings -Wno-sign-compare -Wno-multichar -Wno-deprecated-declarations -Wpacked -Wnested-externs -Winvalid-pch -Wno-pointer-sign -Wvariadic-macros -Wno-format-zero-length -Wtype-limits -Werror-implicit-function-declaration -D_POSIX_C_SOURCE=200112L -std=c99 -D_STDC_C99= -g -O0 > MPICH CXX: no -g > MPICH F77: gfortran -g -O0 > MPICH FC: gfortran -g -O0 > > Git revision info: > I got the MPI source from git (http://git.mpich.org/mpich.git master) and the id is 7dfe2840 > > Platform: > Ubuntu 12.04 amd64 > > > Thanks, > Hajime > > -- > Hajime Fujita > Postdoctoral Scholar, Large-Scale Systems Group > Department of Computer Science, The University of Chicago > http://www.cs.uchicago.edu/people/hfujita > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Issue in MPI_Win_get_attr(MPI_WIN_SIZE)
by Junchao Zhang 09 Feb '15

09 Feb '15

On Mon, Feb 9, 2015 at 12:29 PM, Hajime Fujita <hfujita(a)uchicago.edu> wrote: > Hello, > > I think I found an issue in MPI_Win_get_attr, when MPI_WIN_SIZE is > specified. It is supposed to return the size of the window, but it seems to > give a random number. > > Just wanted to confirm one thing: I think MPI_Win_get_attr is supposed to > return a value of type MPI_Aint, when MPI_WIN_SIZE is specified. Is it > correct? > No. According to p415 of MPI-3.0 spec, in C bindings, MPI_Win_get_attr(..., MPI_WIN_SIZE, &size, ...) return in size "pointer" to the size of the window. In Fortran bindings, it returns a value instead. > > > Test program: > See the attached file > > Result I got: > hfujita@lssg-lp-00:~/work/mpi_win_size_test$ ./win_size_test > Got size=140729394560728, while expecting 1024 > hfujita@lssg-lp-00:~/work/mpi_win_size_test$ ./win_size_test > Got size=139994694213336, while expecting 1024 > hfujita@lssg-lp-00:~/work/mpi_win_size_test$ ./win_size_test > Got size=139739628887768, while expecting 1024 > hfujita@lssg-lp-00:~/work/mpi_win_size_test$ ./win_size_test > Got size=140131810524888, while expecting 1024 > hfujita@lssg-lp-00:~/work/mpi_win_size_test$ ./win_size_test > Got size=139859819581144, while expecting 1024 > > Expected result: > If the MPI library works as I expected, the test program won’t show any > message. > > MPICH version and configuration: > $ mpichversion > MPICH Version: 3.2a2 > MPICH Release date: unreleased development copy > MPICH Device: ch3:nemesis > MPICH configure: CC=gcc CXX=g++ FC=gfortran F77=gfortran CFLAGS=-O0 > --prefix=/home/hfujita/opt/mpich-3.2 --disable-mpe --disable-romio > --enable-g=all --enable-spawn --enable-maintainer-mode > --enable-error-checking=all --with-pm=hydra --with-pmi=simple --disable-cxx > --enable-fortran --enable-strict=noopt --disable-fast --disable-perftest > --no-create --no-recursion > MPICH CC: gcc -O0 -Wall -Wextra -Wno-missing-field-initializers > -Wmissing-prototypes -DGCC_WALL -Wno-unused-parameter -Wno-unused-label > -Wshadow -Wmissing-declarations -Wno-long-long -Wundef -Wno-endif-labels > -Wpointer-arith -Wbad-function-cast -Wwrite-strings -Wno-sign-compare > -Wno-multichar -Wno-deprecated-declarations -Wpacked -Wnested-externs > -Winvalid-pch -Wno-pointer-sign -Wvariadic-macros -Wno-format-zero-length > -Wtype-limits -Werror-implicit-function-declaration > -D_POSIX_C_SOURCE=200112L -std=c99 -D_STDC_C99= -g -O0 > MPICH CXX: no -g > MPICH F77: gfortran -g -O0 > MPICH FC: gfortran -g -O0 > > Git revision info: > I got the MPI source from git (http://git.mpich.org/mpich.git master) and > the id is 7dfe2840 > > Platform: > Ubuntu 12.04 amd64 > > > Thanks, > Hajime > > -- > Hajime Fujita > Postdoctoral Scholar, Large-Scale Systems Group > Department of Computer Science, The University of Chicago > http://www.cs.uchicago.edu/people/hfujita > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Issue in MPI_Win_get_attr(MPI_WIN_SIZE)
by Hajime Fujita 09 Feb '15

09 Feb '15

1 0