April 2019 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Debugging MPI Window
by Si, Min 17 Apr '19

17 Apr '19

Hi Bruce, If you want to check if the window object is the same, you can compare the window handle value. Just note that if you do win_create-win_free-win_create, the second window may be assigned with the same handle as the first one. Best regards, Min On 2019/04/17 17:32, Palmer, Bruce J via discuss wrote: Hi, Strictly for debugging purposes, is there a way in MPICH to determine if a MPI_Win created at point A in a code is the same window being accessed at point B? It doesn’t need to be portable, I’m just interested in using it to fix some bugs (that are probably not associated with MPI). Bruce Palmer Computer Scientist Pacific Northwest National Laboratory (509) 375-3899 _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Debugging MPI Window
by Palmer, Bruce J 17 Apr '19

17 Apr '19

Hi, Strictly for debugging purposes, is there a way in MPICH to determine if a MPI_Win created at point A in a code is the same window being accessed at point B? It doesn't need to be portable, I'm just interested in using it to fix some bugs (that are probably not associated with MPI). Bruce Palmer Computer Scientist Pacific Northwest National Laboratory (509) 375-3899

1 0

Re: [mpich-discuss] Reduction with MPI_CHAR
by Jeff Hammond 06 Apr '19

06 Apr '19

I’d create a GitHub issue for this. Checking for incorrect usage is a quality-of-implementation issue but I’m sure the devs want to fix this. It’s easy enough. Jeff On Fri, Apr 5, 2019 at 8:21 AM Zhang, Junchao via discuss <discuss(a)mpich.org> wrote: > MPI 3.1 Chapter 5.9.3 reads "MPI_CHAR, MPI_WCHAR, and MPI_CHARACTER (which > represent printable characters) cannot be used in reduction operations". > But the following code with MPICH does not abort. > > #include <mpi.h> > #include <stdlib.h> > int main(int argc, char* argv[]) > { > char a ='A',b; > MPI_Init(&argc,&argv); > if(MPI_Allreduce(&a,&b,1,MPI_CHAR,MPI_SUM,MPI_COMM_WORLD)) abort(); > MPI_Finalize(); > } > --Junchao Zhang > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Profiling MPICH Source Code
by Amer, Abdelhalim 05 Apr '19

05 Apr '19

I'll throw in a couple of options that are good at large scale: Open Source: HPCToolkit (http://hpctoolkit.org/) Commercial: Arm MAP Similar to Jeff's comment about the other tools, you need to build MPICH with debug symbols for these tools to give you useful information. Halim www.mcs.anl.gov/~aamer On 4/3/19 3:00 PM, Jeff Hammond via discuss wrote: > First, you need to compile MPICH with debug symbols. > https://wiki.mpich.org/mpich/index.php/Building_MPICH_For_Development has details. > > Second, use a profiler like Linux perf or Intel Vtune to see what the > hotspots are. > > Jeff > > On Wed, Apr 3, 2019 at 11:54 AM SANJAY KUMAR via discuss > <discuss(a)mpich.org <mailto:[email protected]>> wrote: > > Here at Indian Institute of Technology Kanpur, Computer Science > Department, we are trying to profile mpich source code itself. So as to > analyze the internal function calls made by the mpich functions like > MPI_Bcast etc. I have tried so many profilers like IPM, gprof to > compile the MPICH source code. But It turned out nothing works. I have > also so many flags but nothing worked out. Could you please tell me how > to profile the mpich source code itself. > > > Thanks > > Sanjay Kumar > > Masters of Technology > > Compute Science Department > > Indian Institute of Technology Kanpur > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > jeff.science(a)gmail.com <mailto:[email protected]> > http://jeffhammond.github.io/ > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

MPICH CONFIGURE OPTIONS
by SANJAY KUMAR 05 Apr '19

05 Apr '19

what does it mean by --enable-timing=runtime? Does this create logs and if it does where all these logs are stored? SANJAY KUMAR Master of Technology Department of Computer Science Indian Institute of Technology, Kanpur

1 0

Reduction with MPI_CHAR
by Zhang, Junchao 05 Apr '19

05 Apr '19

MPI 3.1 Chapter 5.9.3 reads "MPI_CHAR, MPI_WCHAR, and MPI_CHARACTER (which represent printable characters) cannot be used in reduction operations". But the following code with MPICH does not abort. #include <mpi.h> #include <stdlib.h> int main(int argc, char* argv[]) { char a ='A',b; MPI_Init(&argc,&argv); if(MPI_Allreduce(&a,&b,1,MPI_CHAR,MPI_SUM,MPI_COMM_WORLD)) abort(); MPI_Finalize(); } --Junchao Zhang

1 0

Re: [mpich-discuss] Profiling MPICH Source Code
by Jeff Hammond 03 Apr '19

03 Apr '19

First, you need to compile MPICH with debug symbols. https://wiki.mpich.org/mpich/index.php/Building_MPICH_For_Development has details. Second, use a profiler like Linux perf or Intel Vtune to see what the hotspots are. Jeff On Wed, Apr 3, 2019 at 11:54 AM SANJAY KUMAR via discuss <discuss(a)mpich.org> wrote: > Here at Indian Institute of Technology Kanpur, Computer Science > Department, we are trying to profile mpich source code itself. So as to > analyze the internal function calls made by the mpich functions like > MPI_Bcast etc. I have tried so many profilers like IPM, gprof to > compile the MPICH source code. But It turned out nothing works. I have > also so many flags but nothing worked out. Could you please tell me how > to profile the mpich source code itself. > > > Thanks > > Sanjay Kumar > > Masters of Technology > > Compute Science Department > > Indian Institute of Technology Kanpur > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Cannot catch std::bac_alloc?
by Zhen Wang 03 Apr '19

03 Apr '19

OK. After help from several forums, I think I understand the cause of the problem. As Jeff said, it has nothing to do with MPI. I do need 2 processes to trigger this issue though. Hui's comment also helps. Linux allows over committing. See here <https://www.kernel.org/doc/Documentation/vm/overcommit-accounting> and here <https://www.etalabs.net/overcommit.html>. In my case, say the machine has 32GB RAM, 2GB is already used and each MPI process is trying to allocate 8GB at a time. Then this happens: Initial memory usage: 2GB After the first memory allocation: 18GB In the second memory allocation, each MPI process thinks it has enough space for 8GB because of over committing, and writes to it. But that requires 34GB, exceeding RAM size. So the out of memory killer on Linux kills one of the MPI process (sends a SIGKILL signal), and the other MPI process receives a SIGTERM signal. This also explains my questions above. The reason this problem doesn't happen on Windows is Windows doesn't allow over commit <https://superuser.com/questions/1194263/will-microsoft-windows-10-overcommi…>. I don't know about Mac. Thanks again everyone. Best regards, Zhen On Wed, Apr 3, 2019 at 1:23 PM Joachim Protze via discuss <discuss(a)mpich.org> wrote: > Jeff, > > as I understand the original question, this is the intention of the > statement: reproducibly trigger an allocation exception, which should be > caught by the try-catch block. > > > Hui, > > As you will probably experience, your intended behavior will not be > reached with replacing the vector resize by allocation with new: > > std::vector<double*> a(100); > ... > a[i] = new double[1000000000]; > > The reason is, that new allocates virtual memory. You will see a > segfault, when you try to initialize the whole array and request > physical memory for the array. > The vector resize operation allocates the memory and immediately > initializes the memory. Therefore not the allocation itself fails, but > finally the access to the memory during initialization. > I have no explanation why you can see the bad_alloc when executing it > only on one rank. > > Best > Joachim > > > On 4/3/19 6:51 PM, Jeff Hammond via discuss wrote: > > This error has nothing to do with MPI. > > > > These two lines are the problem: > > > > std::vector<std::vector<double> > a(100); > > a[i].resize(1000000000); > > > > You are trying to allocate 1e9 std::vector<double>, which I assume is > > many gigabytes of memory. > > > > Jeff > > > > > > On Wed, Apr 3, 2019 at 8:58 AM Zhou, Hui via discuss <discuss(a)mpich.org > > <mailto:[email protected]>> wrote: > > > > > > I just tested on my Mac, the system keep popping out message asking > > me to force kill some application to free up memory, but the code never > > crashed, just stalled. So I guess the out-of-memory behavior is > > operating system dependent. My guess is when you have multiple process > > competing for memory, it may cause some race issue in the os that the OS > > returns a valid address but discover out-of memory later because the > > memory is stolen by another process. > > > > > > — > > > Hui Zhou > > > T: 630-252-3430 > > > > > > > > > > > > > > > > > > > > > > > > > > > On Apr 3, 2019, at 8:42 AM, Zhen Wang via discuss <discuss(a)mpich.org > > <mailto:[email protected]>> wrote: > > > > > > Hi, > > > > > > I have difficulty catching std::bac_alloc in an MPI environment. The > > code is attached. I'm uisng gcc 6.3 on SUSE Linux Enterprise Server 11 > > (x86_64). mpich is built from source. The commands are as follows: > > > > > > Build > > > g++ -I<mpich-3.3-opt/include> -L<mpich-3.3-opt/lib> -lmpi memory.cpp > > > > > > Run > > > <mpich-3.3-opt/bin/mpiexec> -n 2 a.out > > > > > > Output > > > 0 > > > 0 > > > 1 > > > 1 > > > > > > > > > =================================================================================== > > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > > = PID 16067 RUNNING AT <machine name> > > > = EXIT CODE: 9 > > > = CLEANING UP REMAINING PROCESSES > > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > > > > =================================================================================== > > > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9) > > > This typically refers to a problem with your application. > > > Please see the FAQ page for debugging suggestions > > > > > > If I uncomment the line //if (rank == 0), i.e., only rank 0 allocates > > memory, I'm able to catch bad_alloc as I expected. It seems that I am > > misunderstanding something. Could you please help? Thanks a lot. > > > > > > > > > Best regards, > > > Zhen > > > <memory.cpp>_______________________________________________ > > > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com <mailto:[email protected]> > > http://jeffhammond.github.io/ > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > Dipl.-Inf. Joachim Protze > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering > RWTH Aachen University > Seffenter Weg 23 > D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > protze(a)itc.rwth-aachen.de > www.itc.rwth-aachen.de > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Profiling MPICH Source Code
by Benson Muite 03 Apr '19

03 Apr '19

Might try HPC toolkit: http://hpctoolkit.org/ Are you compiling MPICH from source? Benson On 4/3/19 7:53 PM, SANJAY KUMAR via discuss wrote: > Here at Indian Institute of Technology Kanpur, Computer Science > Department, we are trying to profile mpich source code itself. So as > to analyze the internal function calls made by the mpich functions > like MPI_Bcast etc. I have tried so many profilers like IPM, gprof to > compile the MPICH source code. But It turned out nothing works. I have > also so many flags but nothing worked out. Could you please tell me > how to profile the mpich source code itself. > > > Thanks > > Sanjay Kumar > > Masters of Technology > > Compute Science Department > > Indian Institute of Technology Kanpur > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Profiling MPICH Source Code
by SANJAY KUMAR 03 Apr '19

03 Apr '19

Here at Indian Institute of Technology Kanpur, Computer Science Department, we are trying to profile mpich source code itself. So as to analyze the internal function calls made by the mpich functions like MPI_Bcast etc. I have tried so many profilers like IPM, gprof to compile the MPICH source code. But It turned out nothing works. I have also so many flags but nothing worked out. Could you please tell me how to profile the mpich source code itself. Thanks Sanjay Kumar Masters of Technology Compute Science Department Indian Institute of Technology Kanpur

1 0