Hello all, I'm running some largish finite element calculations at the moment (50 Million to 400 Million DoFs on up to 10,000 processors) using a code based on PETSc (obviously!) and while most of the simulations are working well, every now again I seem to run into a hang in the setup phase of the simulation. I've attached GDB several times and it seems to alway be hanging in PetscLayoutSetUp() during matrix creation. Here is the top of a stack trace showing what I mean: #0 0x00002aac9d86cef2 in opal_progress () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libopen-pal.so.0 #1 0x00002aac9d16a0c4 in ompi_request_default_wait_all () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libmpi.so.0 #2 0x00002aac9d1da9ee in ompi_coll_tuned_sendrecv_actual () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libmpi.so.0 #3 0x00002aac9d1e2716 in ompi_coll_tuned_allgather_intra_bruck () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libmpi.so.0 #4 0x00002aac9d1db439 in ompi_coll_tuned_allgather_intra_dec_fixed () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libmpi.so.0 #5 0x00002aac9d1827e6 in PMPI_Allgather () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libmpi.so.0 #6 0x0000000000508184 in PetscLayoutSetUp () #7 0x00000000005b9f39 in MatMPIAIJSetPreallocation_MPIAIJ () #8 0x00000000005c1317 in MatCreateMPIAIJ () As you can see, I'm currently using openMPI (even though I do have access to others) along with the intel compiler (this is a mostly C++ code). This problem doesn't exhibit itself on any smaller problems (we run TONS of runs all the time in the 10,000-5,000,000 DoF range on 1-3000 procs) and only seems to come up on these larger runs. I'm starting to suspect that it's an openMPI issue. Has anyone seen anything like this before? Here are some specs for my current environment PETSc 3.1-p8 (I know, I know....) OpenMPI 1.4.4 intel compilers 12.1.1 Modified Redhat with 2.6.18 Kernel QDR Infiniband Thanks for any help! Derek
On Tue, Feb 7, 2012 at 08:06, Derek Gaston <[email protected]> wrote:
Hello all,
I'm running some largish finite element calculations at the moment (50 Million to 400 Million DoFs on up to 10,000 processors) using a code based on PETSc (obviously!) and while most of the simulations are working well, every now again I seem to run into a hang in the setup phase of the simulation.
I've attached GDB several times and it seems to alway be hanging in PetscLayoutSetUp() during matrix creation. Here is the top of a stack trace showing what I mean:
#0 0x00002aac9d86cef2 in opal_progress () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libopen-pl.so.0 #1 0x00002aac9d16a0c4 in ompi_request_default_wait_all () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libmpi.so.0 #2 0x00002aac9d1da9ee in ompi_coll_tuned_sendrecv_actual () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libmpi.so.0 #3 0x00002aac9d1e2716 in ompi_coll_tuned_allgather_intra_bruck () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libmpi.so.0 #4 0x00002aac9d1db439 in ompi_coll_tuned_allgather_intra_dec_fixed () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libmpi.so.0 #5 0x00002aac9d1827e6 in PMPI_Allgather () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libmpi.so.0 #6 0x0000000000508184 in PetscLayoutSetUp () #7 0x00000000005b9f39 in MatMPIAIJSetPreallocation_MPIAIJ () #8 0x00000000005c1317 in MatCreateMPIAIJ ()
Are _all_ the processes making it here?
As you can see, I'm currently using openMPI (even though I do have access to others) along with the intel compiler (this is a mostly C++ code). This problem doesn't exhibit itself on any smaller problems (we run TONS of runs all the time in the 10,000-5,000,000 DoF range on 1-3000 procs) and only seems to come up on these larger runs.
I'm starting to suspect that it's an openMPI issue. Has anyone seen anything like this before?
Here are some specs for my current environment
PETSc 3.1-p8 (I know, I know....) OpenMPI 1.4.4 intel compilers 12.1.1 Modified Redhat with 2.6.18 Kernel QDR Infiniband
Thanks for any help!
Derek
On Mon, Feb 6, 2012 at 10:27 PM, Jed Brown <[email protected]> wrote:
Are _all_ the processes making it here?
Sigh. I knew someone was going to ask that ;-) I'll have to write a short script to grab the stack trace from every one of the 10,000 processes to see where they are and try to find any anomalies. Anyone have a script (or pieces of one) to do this that they wouldn't mind sharing? I did spot check quite a few and they were all in the same spot. Now here comes the weirdness: I left one of these processes attached in GDB for quite a while (10+ minutes) after the whole job had been hung for over an hour. When I noticed that I had left it attached I detached GDB and.... the job started right up! That is: it moved on past this problem! How is that for some weirdness. It might have just been coincidence... or maybe me stalling that process for a bit by attaching GDB nudged some communication in the right direction... I don't know. I know that's not terribly scientific. I'll have to wait until the next job hangs before I can do more inspection, but when (not if) that happens I'll post back with more info. Derek
On Tue, Feb 7, 2012 at 08:34, Derek Gaston <[email protected]> wrote:
On Mon, Feb 6, 2012 at 10:27 PM, Jed Brown <[email protected]> wrote:
Are _all_ the processes making it here?
Sigh. I knew someone was going to ask that ;-)
I'll have to write a short script to grab the stack trace from every one of the 10,000 processes to see where they are and try to find any anomalies. Anyone have a script (or pieces of one) to do this that they wouldn't mind sharing?
I did spot check quite a few and they were all in the same spot.
Now here comes the weirdness: I left one of these processes attached in GDB for quite a while (10+ minutes) after the whole job had been hung for over an hour. When I noticed that I had left it attached I detached GDB and.... the job started right up! That is: it moved on past this problem! How is that for some weirdness. It might have just been coincidence... or maybe me stalling that process for a bit by attaching GDB nudged some communication in the right direction... I don't know.
Hmm, progress semantics of MPI should ensure completion. Stalling the process with gdb should not change anything (assuming you weren't actually making changes with gdb). Can you run with MPICH2?
On Mon, Feb 6, 2012 at 11:20 PM, Jed Brown <[email protected]> wrote:
Hmm, progress semantics of MPI should ensure completion. Stalling the process with gdb should not change anything (assuming you weren't actually making changes with gdb). Can you run with MPICH2?
Ok - an update on this. I recompiled my whole stack with mvapich2... and it still is hanging in the same place: #0 0x00002b336a732f40 in PMI_Get_rank () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #1 0x00002b336a6bf453 in MPIDI_CH3I_MRAILI_Cq_poll () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #2 0x00002b336a675818 in MPIDI_CH3I_read_progress () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #3 0x00002b336a67485b in MPIDI_CH3I_Progress () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #4 0x00002b336a6bea96 in MPIC_Wait () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #5 0x00002b336a6be9db in MPIC_Sendrecv () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #6 0x00002b336a6be8aa in MPIC_Sendrecv_ft () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #7 0x00002b336a652db1 in MPIR_Allgather_intra_MV2 () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #8 0x00002b336a652965 in MPIR_Allgather_MV2 () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #9 0x00002b336a651846 in MPIR_Allgather_impl () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #10 0x00002b336a6517b1 in PMPI_Allgather () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #11 0x00000000004a1f23 in PetscLayoutSetUp () #12 0x000000000054e469 in MatMPIAIJSetPreallocation_MPIAIJ () #13 0x000000000055584a in MatCreateMPIAIJ () It's been hung there for about 35 minutes. This particular job has ~100 million DoFs with 512 MPI processes. Any ideas? Derek
On Tue, Feb 7, 2012 at 8:52 PM, Derek Gaston <[email protected]> wrote:
On Mon, Feb 6, 2012 at 11:20 PM, Jed Brown <[email protected]> wrote:
Hmm, progress semantics of MPI should ensure completion. Stalling the process with gdb should not change anything (assuming you weren't actually making changes with gdb). Can you run with MPICH2?
Ok - an update on this. I recompiled my whole stack with mvapich2... and it still is hanging in the same place:
#0 0x00002b336a732f40 in PMI_Get_rank () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #1 0x00002b336a6bf453 in MPIDI_CH3I_MRAILI_Cq_poll () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #2 0x00002b336a675818 in MPIDI_CH3I_read_progress () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #3 0x00002b336a67485b in MPIDI_CH3I_Progress () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #4 0x00002b336a6bea96 in MPIC_Wait () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #5 0x00002b336a6be9db in MPIC_Sendrecv () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #6 0x00002b336a6be8aa in MPIC_Sendrecv_ft () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #7 0x00002b336a652db1 in MPIR_Allgather_intra_MV2 () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #8 0x00002b336a652965 in MPIR_Allgather_MV2 () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #9 0x00002b336a651846 in MPIR_Allgather_impl () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #10 0x00002b336a6517b1 in PMPI_Allgather () from /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 #11 0x00000000004a1f23 in PetscLayoutSetUp () #12 0x000000000054e469 in MatMPIAIJSetPreallocation_MPIAIJ () #13 0x000000000055584a in MatCreateMPIAIJ ()
It's been hung there for about 35 minutes.
This particular job has ~100 million DoFs with 512 MPI processes. Any ideas?
Same question: are you sure every process is there. I will bet $10 there is at least one missing. Matt
Derek
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
On Tue, Feb 7, 2012 at 7:56 PM, Matthew Knepley <[email protected]> wrote:
Same question: are you sure every process is there. I will bet $10 there is at least one missing.
Ok - I'll get that info - I tried to compile PADB a moment ago... but it didn't compile. It configured just fine... but compile is a no-go. It doesn't look like too bad of an error, but I don't have time to fix it now so I'll just script something up with GDB. At least I got the processor count down for this job so it won't be too much data to sort through. Derek
Since the processor count is down, can you run it with debugging and see what the arguments to PetscLayout are? Dmitry On Tue, Feb 7, 2012 at 9:00 PM, Derek Gaston <[email protected]> wrote:
On Tue, Feb 7, 2012 at 7:56 PM, Matthew Knepley <[email protected]> wrote:
Same question: are you sure every process is there. I will bet $10 there is at least one missing.
Ok - I'll get that info - I tried to compile PADB a moment ago... but it didn't compile. It configured just fine... but compile is a no-go. It doesn't look like too bad of an error, but I don't have time to fix it now so I'll just script something up with GDB. At least I got the processor count down for this job so it won't be too much data to sort through.
Derek
On Tue, Feb 7, 2012 at 7:56 PM, Matthew Knepley <[email protected]> wrote:
Same question: are you sure every process is there. I will bet $10 there is at least one missing.
Well, of course you guys were right! I was able to write a short piece of python (pasted at the bottom of this email in case others find it interesting / useful) to look through the stack trace of every one of my processes in given job on the cluster and tell me which ones weren't in the place I was expecting them to be. Once I identified these processes I was able to ssh to the individual nodes and use gdb to attach to those processes and get stacktraces out to figure out what in the heck they were doing. Two things came up, both in a library our software depends on: 1. There was an inadvertent vector "localize" operation happening on the solution vector. This means that we were making a complete _local_ copy of the parallel solution vector to every processor! That would sometimes fail with 400 Million Dofs and 8000+ MPI ;-) 2. A small optimization problem having to do with threading and allocation / deallocation of small vectors. This was just slowing some of the nodes down so much that it would look like the processes had hung. After fixing up both of these things the jobs are now moving. Thanks for the suggestion ;-) Below is the python script I used to figure this stuff out. It works by calling the script with the job # of the job you want to analyze. It is set up for PBS, so if your cluster doesn't use PBS you'll have to disregard the top part where I'm just parsing out the list of nodes the job is running on. "fission-\d\d\d\d" is the regex pattern my cluster's nodes are named in. MatCreateMPIAIJ was what I was looking for in the stack trace (I wanted to see if every process had made it there). gastdr was my username, replace with yours. "marmot" was the name of the executable I was running. "bad_hosts" gets filled up with the number of processes owned by you on each node that have a stack trace containing the string you were looking for. Then at the end I analyzed that to see if it matched how many MPI per node I was running (in this case 4). Any host that had less than 4 processes on it that were where I was expecting them to be got spit out at the end. Then it was time to ssh to that node and attach to the processes and figure out what was going on. It's 3:30AM here right now, so you'll have to excuse some of the rough edges in the script. I really just hacked it together for myself but thought others might find some pieces useful from it. Oh, and yes, I did use os.popen()... after all these years I still find it more straightforward to use than any of the subprocesses stuff in Python. It has been deprecated for a _long_ time now... but I hope they never remove it ;-) Happy hunting all! Derek ------- import os import sys import re command = "qstat -n " + sys.argv[1] output = os.popen(command).readlines() regex = re.compile("(fission-\d\d\d\d)") hosts = [] for line in output: f = regex.findall(line) for i in f: hosts.append(i) matcreates = 0 bad_hosts = {} #host = hosts[0] for host in hosts: command = "ssh " + host + " \"ps aux | grep 'gastdr .*marmot' | grep -v grep | awk '{print \$2}' | xargs -I {} gdb --batch --pid={} -ex bt | grep 'MatCreateMPIAIJ' 2>/dev/null \"" lines = os.popen(command).readlines() for line in lines: if line.find("MatCreateMPIAIJ") != -1: matcreates = matcreates + 1 if host in bad_hosts: bad_hosts[host] += 1 else: bad_hosts[host] = 1 print bad_hosts print "Num matches: " + str(matcreates) print "Bad Hosts: " for host, num in bad_hosts.items(): if num != 4: print host
Good to hear you got it sorted out! I understand that PetscLayoutSetup() and VecScatterCreate() problems were unrelated, is that right? The PetscLayoutSetup() was hanging because some nodes took too long to enter the call -- they were waiting for mutex locks somewhere else; VecScatterCreate() was "hanging" due to the sheer number of indices in the IS (and building it as a PtoP, while it should have been MPI_ToAll)? Dmitry. On Wed, Feb 8, 2012 at 4:34 AM, Derek Gaston <[email protected]> wrote:
On Tue, Feb 7, 2012 at 7:56 PM, Matthew Knepley <[email protected]> wrote:
Same question: are you sure every process is there. I will bet $10 there is at least one missing.
Well, of course you guys were right! I was able to write a short piece of python (pasted at the bottom of this email in case others find it interesting / useful) to look through the stack trace of every one of my processes in given job on the cluster and tell me which ones weren't in the place I was expecting them to be. Once I identified these processes I was able to ssh to the individual nodes and use gdb to attach to those processes and get stacktraces out to figure out what in the heck they were doing.
Two things came up, both in a library our software depends on:
1. There was an inadvertent vector "localize" operation happening on the solution vector. This means that we were making a complete _local_ copy of the parallel solution vector to every processor! That would sometimes fail with 400 Million Dofs and 8000+ MPI ;-)
2. A small optimization problem having to do with threading and allocation / deallocation of small vectors. This was just slowing some of the nodes down so much that it would look like the processes had hung.
After fixing up both of these things the jobs are now moving.
Thanks for the suggestion ;-)
Below is the python script I used to figure this stuff out. It works by calling the script with the job # of the job you want to analyze. It is set up for PBS, so if your cluster doesn't use PBS you'll have to disregard the top part where I'm just parsing out the list of nodes the job is running on. "fission-\d\d\d\d" is the regex pattern my cluster's nodes are named in.
MatCreateMPIAIJ was what I was looking for in the stack trace (I wanted to see if every process had made it there). gastdr was my username, replace with yours. "marmot" was the name of the executable I was running. "bad_hosts" gets filled up with the number of processes owned by you on each node that have a stack trace containing the string you were looking for. Then at the end I analyzed that to see if it matched how many MPI per node I was running (in this case 4). Any host that had less than 4 processes on it that were where I was expecting them to be got spit out at the end. Then it was time to ssh to that node and attach to the processes and figure out what was going on.
It's 3:30AM here right now, so you'll have to excuse some of the rough edges in the script. I really just hacked it together for myself but thought others might find some pieces useful from it. Oh, and yes, I did use os.popen()... after all these years I still find it more straightforward to use than any of the subprocesses stuff in Python. It has been deprecated for a _long_ time now... but I hope they never remove it ;-)
Happy hunting all!
Derek
-------
import os import sys import re
command = "qstat -n " + sys.argv[1]
output = os.popen(command).readlines()
regex = re.compile("(fission-\d\d\d\d)")
hosts = []
for line in output: f = regex.findall(line) for i in f: hosts.append(i)
matcreates = 0 bad_hosts = {} #host = hosts[0] for host in hosts: command = "ssh " + host + " \"ps aux | grep 'gastdr .*marmot' | grep -v grep | awk '{print \$2}' | xargs -I {} gdb --batch --pid={} -ex bt | grep 'MatCreateMPIAIJ' 2>/dev/null \"" lines = os.popen(command).readlines() for line in lines: if line.find("MatCreateMPIAIJ") != -1: matcreates = matcreates + 1 if host in bad_hosts: bad_hosts[host] += 1 else: bad_hosts[host] = 1
print bad_hosts
print "Num matches: " + str(matcreates)
print "Bad Hosts: " for host, num in bad_hosts.items(): if num != 4: print host
El 07/02/2012, a las 06:34, Derek Gaston escribió:
On Mon, Feb 6, 2012 at 10:27 PM, Jed Brown <[email protected]> wrote:
Are _all_ the processes making it here?
Sigh. I knew someone was going to ask that ;-)
I'll have to write a short script to grab the stack trace from every one of the 10,000 processes to see where they are and try to find any anomalies. Anyone have a script (or pieces of one) to do this that they wouldn't mind sharing?
Try with PADB: http://padb.pittman.org.uk/ Jose
I did spot check quite a few and they were all in the same spot.
Now here comes the weirdness: I left one of these processes attached in GDB for quite a while (10+ minutes) after the whole job had been hung for over an hour. When I noticed that I had left it attached I detached GDB and.... the job started right up! That is: it moved on past this problem! How is that for some weirdness. It might have just been coincidence... or maybe me stalling that process for a bit by attaching GDB nudged some communication in the right direction... I don't know.
I know that's not terribly scientific. I'll have to wait until the next job hangs before I can do more inspection, but when (not if) that happens I'll post back with more info.
Derek
On Tue, Feb 7, 2012 at 12:45 AM, Jose E. Roman <[email protected]> wrote:
Try with PADB: http://padb.pittman.org.uk/ Jose
Thanks! That looks like just the thing I need! Derek
participants (5)
-
Derek Gaston -
Dmitry Karpeev -
Jed Brown -
Jose E. Roman -
Matthew Knepley