how to debug a stack overrun in Argobots
Hi all, There is a little bit of back story on https://github.com/pmodels/argobots/issues/93 , but make a long story short we have realized that we have some code that is overflowing the stack in Argobots. Many thanks to Shintaro for his help and insight or we may have never figured this out. We can work around the problem with `export ABT_THREAD_STACKSIZE=$((1024 * 1024))`. This not only fixes a Power8 test case for us, but also appears to solve a different frustrating, nonsensical segmentation fault that we've been chasing with a different code permutation on x86_64. Any suggestions on how to track down what's triggering this in our code or get a better idea of how much stack we need? We are using a considerable number of libraries, many of which are not maintained by us, so I don't even know where to start looking yet. My usual go to tool for this would be asan in gcc or clang, but I don't think that will work correctly with Argobots, and maybe there is a better solution anyway. thanks, -Phil
Just to follow up a little bit; I realized from looking at README.envvar just now that the default value of ABT_THREAD_STACKSIZE is 16K. That's almost certainly too low for us because we have ULTs that make calls into a variety of system libraries (including fairly big things like libfabric) that are beyond our control. It seems likely that we will have to run with a larger stack size, but I would still like to have a better understanding of where the problem paths are, and how much head room we really need, if anyone has suggestions. thanks! -Phil On 2019-02-21 15:31:53-05:00 Carns, Philip H. via discuss wrote: Hi all, There is a little bit of back story on https://github.com/pmodels/argobots/issues/93 , but make a long story short we have realized that we have some code that is overflowing the stack in Argobots. Many thanks to Shintaro for his help and insight or we may have never figured this out. We can work around the problem with `export ABT_THREAD_STACKSIZE=$((1024 * 1024))`. This not only fixes a Power8 test case for us, but also appears to solve a different frustrating, nonsensical segmentation fault that we've been chasing with a different code permutation on x86_64. Any suggestions on how to track down what's triggering this in our code or get a better idea of how much stack we need? We are using a considerable number of libraries, many of which are not maintained by us, so I don't even know where to start looking yet. My usual go to tool for this would be asan in gcc or clang, but I don't think that will work correctly with Argobots, and maybe there is a better solution anyway. thanks, -Phil
Hi Phil, I think we hit the same issue recently on the DAOS side and had to bump the stack size as well. Wangdi & Xuezhao should know more. Maybe a regression in ABT? Cheers, Johann From: "Carns, Philip H. via discuss" <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Thursday, 21 February 2019 at 15:50 To: "[email protected]" <[email protected]> Cc: "Carns, Philip H." <[email protected]> Subject: Re: [argobots-discuss] how to debug a stack overrun in Argobots Just to follow up a little bit; I realized from looking at README.envvar just now that the default value of ABT_THREAD_STACKSIZE is 16K. That's almost certainly too low for us because we have ULTs that make calls into a variety of system libraries (including fairly big things like libfabric) that are beyond our control. It seems likely that we will have to run with a larger stack size, but I would still like to have a better understanding of where the problem paths are, and how much head room we really need, if anyone has suggestions. thanks! -Phil On 2019-02-21 15:31:53-05:00 Carns, Philip H. via discuss wrote: Hi all, There is a little bit of back story on https://github.com/pmodels/argobots/issues/93 , but make a long story short we have realized that we have some code that is overflowing the stack in Argobots. Many thanks to Shintaro for his help and insight or we may have never figured this out. We can work around the problem with `export ABT_THREAD_STACKSIZE=$((1024 * 1024))`. This not only fixes a Power8 test case for us, but also appears to solve a different frustrating, nonsensical segmentation fault that we've been chasing with a different code permutation on x86_64. Any suggestions on how to track down what's triggering this in our code or get a better idea of how much stack we need? We are using a considerable number of libraries, many of which are not maintained by us, so I don't even know where to start looking yet. My usual go to tool for this would be asan in gcc or clang, but I don't think that will work correctly with Argobots, and maybe there is a better solution anyway. thanks, -Phil --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Hello All, Thank you for your reports! As one of the developers, I would like to summarize the current status of Argobots. As pointed out, Argobots, by default, uses 16KB for ULTs. At present, these three ways are relatively reasonable to work around, find, or solve this issue; 0. Use knowledge that, if Argobots is used alone, it should "typically" happens in ABT_finalize. The stack size is set by ABT_thread_attr_set_stacksize individually. 1. Use larger stacksize (e.g., ABT_THREAD_STACKSIZE=$((4 * 1024 * 1024)) for example) and see what happens. 2. [Uncertain] Use Valgrind with --enable-valgrind (although it is extremely slow, so not practical for large applications) For now, I think the workaround 1. (using larger stack size by default) is best among them. I haven't tried other tools, but I strongly believe that Argobots-unaware tools won't detect this problem; only Valgind can detect it. --- There are several issues: 1. Too small default stack size It might be too small to drive large system libraries (e.g., a ULT as a progress thread) I'm not sure how much it should be increased, or first of all, whether we should increase it. This does not solve the problem of the "silent stack corruption", though. In other words, if Argobots can detect stack overflow, users can change the value by increasing the default stack size or the stack size of a specific thread requiring large amount of stack. 2. Lack of stack overflow detection For example, the following two are often used; - Stack canaries (lazy but cheap) - mprotect (eager but expensive) I will create a GitHub issue for further details if detection is preferable. 3. Check if Valgrind works for this issue If --enable-valgrind is set, Argobots registers ULT's stacks to Valgrind. It should work but I haven't tested it yet. Another problem is that --enable-valgrind degrades performance of Argobots even if it is not run on Valgrind (see https://github.com/pmodels/argobots/issues/78). We would appreciate any feedback. Thank You, Shintaro Iwasaki ________________________________ From: Lombardi, Johann via discuss <[email protected]> Sent: Thursday, February 21, 2019 4:30 PM To: [email protected] Cc: Lombardi, Johann; Liu, Xuezhao; Wang, Di Subject: Re: [argobots-discuss] how to debug a stack overrun in Argobots Hi Phil, I think we hit the same issue recently on the DAOS side and had to bump the stack size as well. Wangdi & Xuezhao should know more. Maybe a regression in ABT? Cheers, Johann From: "Carns, Philip H. via discuss" <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Thursday, 21 February 2019 at 15:50 To: "[email protected]" <[email protected]> Cc: "Carns, Philip H." <[email protected]> Subject: Re: [argobots-discuss] how to debug a stack overrun in Argobots Just to follow up a little bit; I realized from looking at README.envvar just now that the default value of ABT_THREAD_STACKSIZE is 16K. That's almost certainly too low for us because we have ULTs that make calls into a variety of system libraries (including fairly big things like libfabric) that are beyond our control. It seems likely that we will have to run with a larger stack size, but I would still like to have a better understanding of where the problem paths are, and how much head room we really need, if anyone has suggestions. thanks! -Phil On 2019-02-21 15:31:53-05:00 Carns, Philip H. via discuss wrote: Hi all, There is a little bit of back story on https://github.com/pmodels/argobots/issues/93 , but make a long story short we have realized that we have some code that is overflowing the stack in Argobots. Many thanks to Shintaro for his help and insight or we may have never figured this out. We can work around the problem with `export ABT_THREAD_STACKSIZE=$((1024 * 1024))`. This not only fixes a Power8 test case for us, but also appears to solve a different frustrating, nonsensical segmentation fault that we've been chasing with a different code permutation on x86_64. Any suggestions on how to track down what's triggering this in our code or get a better idea of how much stack we need? We are using a considerable number of libraries, many of which are not maintained by us, so I don't even know where to start looking yet. My usual go to tool for this would be asan in gcc or clang, but I don't think that will work correctly with Argobots, and maybe there is a better solution anyway. thanks, -Phil --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Thanks Shintaro. It seems like a minimum we will have to use 1. We can't control our component scope enough to get away with 0 generally. I'm interested in following up on 2, or techniques like it, later on. Maybe for regression testing. I'll have to circle back to it a little later, though. Fortunately for our use case we have one particular library that is used in most of our builds where we can centralize some of these parameters. I'm setting this (and another parameter we have been using) in this subroutine: https://xgitlab.cels.anl.gov/sds/margo/blob/master/src/margo.c#L180 This works fine (setting stack size to 1M). My first guess was to set it to 4M to be even more conservative until we understand the issue better, but that caused a different crash almost immediately when I tried that: (gdb) where #0 __GI___libc_free (mem=0x7ffff72d9000) at malloc.c:3085 #1 0x00007ffff7fb16d8 in ABTI_sched_free ( p_sched=p_sched@entry=0x55555555bbb0) at ../../src/sched/sched.c:747 #2 0x00007ffff7fa5795 in ABTI_sched_discard_and_free (p_sched=0x55555555bbb0) at ../../src/include/abti_sched.h:51 #3 ABTI_xstream_set_main_sched (p_xstream=p_xstream@entry=0x55555555ba10, p_sched=0x55555555bfb0) at ../../src/stream.c:1721 #4 0x00007ffff7fa5f09 in ABT_xstream_set_main_sched (xstream=0x55555555ba10, sched=<optimized out>) at ../../src/stream.c:811 #5 0x00007ffff7fc1ee0 in margo_init_opt (addr_str=0x7fffffffc90a "na+sm://", mode=1, hg_init_info=0x0, use_progress_thread=0, rpc_thread_count=-1) at ../src/margo.c:248 #6 0x0000555555556aad in main (argc=2, argv=0x7fffffffc488) at ../examples/margo-example-server.c:39 I suspect that line number just happened to get caught in the crossfire of a memory corruption but I haven't investigated. At any rate, is there a practical (and lower than 4M) limit to how big we can set the stack size? Or is there another tunable that must be boosted as well to allow for larger stacks? thanks! -Phil On 2019-02-21 17:55:56-05:00 Iwasaki, Shintaro via discuss wrote: Hello All, Thank you for your reports! As one of the developers, I would like to summarize the current status of Argobots. As pointed out, Argobots, by default, uses 16KB for ULTs. At present, these three ways are relatively reasonable to work around, find, or solve this issue; 0. Use knowledge that, if Argobots is used alone, it should "typically" happens in ABT_finalize. The stack size is set by ABT_thread_attr_set_stacksize individually. 1. Use larger stacksize (e.g., ABT_THREAD_STACKSIZE=$((4 * 1024 * 1024)) for example) and see what happens. 2. [Uncertain] Use Valgrind with --enable-valgrind (although it is extremely slow, so not practical for large applications) For now, I think the workaround 1. (using larger stack size by default) is best among them. I haven't tried other tools, but I strongly believe that Argobots-unaware tools won't detect this problem; only Valgind can detect it. --- There are several issues: 1. Too small default stack size It might be too small to drive large system libraries (e.g., a ULT as a progress thread) I'm not sure how much it should be increased, or first of all, whether we should increase it. This does not solve the problem of the "silent stack corruption", though. In other words, if Argobots can detect stack overflow, users can change the value by increasing the default stack size or the stack size of a specific thread requiring large amount of stack. 2. Lack of stack overflow detection For example, the following two are often used; - Stack canaries (lazy but cheap) - mprotect (eager but expensive) I will create a GitHub issue for further details if detection is preferable. 3. Check if Valgrind works for this issue If --enable-valgrind is set, Argobots registers ULT's stacks to Valgrind. It should work but I haven't tested it yet. Another problem is that --enable-valgrind degrades performance of Argobots even if it is not run on Valgrind (see https://github.com/pmodels/argobots/issues/78). We would appreciate any feedback. Thank You, Shintaro Iwasaki ________________________________ From: Lombardi, Johann via discuss <[email protected]> Sent: Thursday, February 21, 2019 4:30 PM To: [email protected] Cc: Lombardi, Johann; Liu, Xuezhao; Wang, Di Subject: Re: [argobots-discuss] how to debug a stack overrun in Argobots Hi Phil, I think we hit the same issue recently on the DAOS side and had to bump the stack size as well. Wangdi & Xuezhao should know more. Maybe a regression in ABT? Cheers, Johann From: "Carns, Philip H. via discuss" <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Thursday, 21 February 2019 at 15:50 To: "[email protected]" <[email protected]> Cc: "Carns, Philip H." <[email protected]> Subject: Re: [argobots-discuss] how to debug a stack overrun in Argobots Just to follow up a little bit; I realized from looking at README.envvar just now that the default value of ABT_THREAD_STACKSIZE is 16K. That's almost certainly too low for us because we have ULTs that make calls into a variety of system libraries (including fairly big things like libfabric) that are beyond our control. It seems likely that we will have to run with a larger stack size, but I would still like to have a better understanding of where the problem paths are, and how much head room we really need, if anyone has suggestions. thanks! -Phil On 2019-02-21 15:31:53-05:00 Carns, Philip H. via discuss wrote: Hi all, There is a little bit of back story on https://github.com/pmodels/argobots/issues/93 , but make a long story short we have realized that we have some code that is overflowing the stack in Argobots. Many thanks to Shintaro for his help and insight or we may have never figured this out. We can work around the problem with `export ABT_THREAD_STACKSIZE=$((1024 * 1024))`. This not only fixes a Power8 test case for us, but also appears to solve a different frustrating, nonsensical segmentation fault that we've been chasing with a different code permutation on x86_64. Any suggestions on how to track down what's triggering this in our code or get a better idea of how much stack we need? We are using a considerable number of libraries, many of which are not maintained by us, so I don't even know where to start looking yet. My usual go to tool for this would be asan in gcc or clang, but I don't think that will work correctly with Argobots, and maybe there is a better solution anyway. thanks, -Phil --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Hello, Thank you, Phil. I will create an GitHub issue about the the stack overflow detection once we understand all the problems. I'm happy to hear that margo worked with 1MB stack size. However, Argobots should work with 4MB stack size (which is the default Pthreads stack size on some machines) or larger as well; there should be no upper bound. A scheduler has its own stack size (tunable via ABT_SCHED_STACKSIZE) and I suspect it is related since the bug happens in ABT_xstream_set_main_sched. I will check this issue soon after some urgent tasks. (Note that scheduler stack size is independent of ULT stack size by design since it virtually affects the stack size of every tasklet execution) Thank You, Shintaro Iwasaki On Fri, Feb 22, 2019 at 9:57 AM Carns, Philip H. <[email protected]<mailto:[email protected]>> wrote: Thanks Shintaro. It seems like a minimum we will have to use 1. We can't control our component scope enough to get away with 0 generally. I'm interested in following up on 2, or techniques like it, later on. Maybe for regression testing. I'll have to circle back to it a little later, though. Fortunately for our use case we have one particular library that is used in most of our builds where we can centralize some of these parameters. I'm setting this (and another parameter we have been using) in this subroutine: https://xgitlab.cels.anl.gov/sds/margo/blob/master/src/margo.c#L180 This works fine (setting stack size to 1M). My first guess was to set it to 4M to be even more conservative until we understand the issue better, but that caused a different crash almost immediately when I tried that: (gdb) where #0 __GI___libc_free (mem=0x7ffff72d9000) at malloc.c:3085 #1 0x00007ffff7fb16d8 in ABTI_sched_free ( p_sched=p_sched@entry=0x55555555bbb0) at ../../src/sched/sched.c:747 #2 0x00007ffff7fa5795 in ABTI_sched_discard_and_free (p_sched=0x55555555bbb0) at ../../src/include/abti_sched.h:51 #3 ABTI_xstream_set_main_sched (p_xstream=p_xstream@entry=0x55555555ba10, p_sched=0x55555555bfb0) at ../../src/stream.c:1721 #4 0x00007ffff7fa5f09 in ABT_xstream_set_main_sched (xstream=0x55555555ba10, sched=<optimized out>) at ../../src/stream.c:811 #5 0x00007ffff7fc1ee0 in margo_init_opt (addr_str=0x7fffffffc90a "na+sm://", mode=1, hg_init_info=0x0, use_progress_thread=0, rpc_thread_count=-1) at ../src/margo.c:248 #6 0x0000555555556aad in main (argc=2, argv=0x7fffffffc488) at ../examples/margo-example-server.c:39 I suspect that line number just happened to get caught in the crossfire of a memory corruption but I haven't investigated. At any rate, is there a practical (and lower than 4M) limit to how big we can set the stack size? Or is there another tunable that must be boosted as well to allow for larger stacks? thanks! -Phil On 2019-02-21 17:55:56-05:00 Iwasaki, Shintaro via discuss wrote: Hello All, Thank you for your reports! As one of the developers, I would like to summarize the current status of Argobots. As pointed out, Argobots, by default, uses 16KB for ULTs. At present, these three ways are relatively reasonable to work around, find, or solve this issue; 0. Use knowledge that, if Argobots is used alone, it should "typically" happens in ABT_finalize. The stack size is set by ABT_thread_attr_set_stacksize individually. 1. Use larger stacksize (e.g., ABT_THREAD_STACKSIZE=$((4 * 1024 * 1024)) for example) and see what happens. 2. [Uncertain] Use Valgrind with --enable-valgrind (although it is extremely slow, so not practical for large applications) For now, I think the workaround 1. (using larger stack size by default) is best among them. I haven't tried other tools, but I strongly believe that Argobots-unaware tools won't detect this problem; only Valgind can detect it. --- There are several issues: 1. Too small default stack size It might be too small to drive large system libraries (e.g., a ULT as a progress thread) I'm not sure how much it should be increased, or first of all, whether we should increase it. This does not solve the problem of the "silent stack corruption", though. In other words, if Argobots can detect stack overflow, users can change the value by increasing the default stack size or the stack size of a specific thread requiring large amount of stack. 2. Lack of stack overflow detection For example, the following two are often used; - Stack canaries (lazy but cheap) - mprotect (eager but expensive) I will create a GitHub issue for further details if detection is preferable. 3. Check if Valgrind works for this issue If --enable-valgrind is set, Argobots registers ULT's stacks to Valgrind. It should work but I haven't tested it yet. Another problem is that --enable-valgrind degrades performance of Argobots even if it is not run on Valgrind (see https://github.com/pmodels/argobots/issues/78). We would appreciate any feedback. Thank You, Shintaro Iwasaki ________________________________ From: Lombardi, Johann via discuss <[email protected]<mailto:[email protected]>> Sent: Thursday, February 21, 2019 4:30 PM To: [email protected]<mailto:[email protected]> Cc: Lombardi, Johann; Liu, Xuezhao; Wang, Di Subject: Re: [argobots-discuss] how to debug a stack overrun in Argobots Hi Phil, I think we hit the same issue recently on the DAOS side and had to bump the stack size as well. Wangdi & Xuezhao should know more. Maybe a regression in ABT? Cheers, Johann From: "Carns, Philip H. via discuss" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, 21 February 2019 at 15:50 To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Cc: "Carns, Philip H." <[email protected]<mailto:[email protected]>> Subject: Re: [argobots-discuss] how to debug a stack overrun in Argobots Just to follow up a little bit; I realized from looking at README.envvar just now that the default value of ABT_THREAD_STACKSIZE is 16K. That's almost certainly too low for us because we have ULTs that make calls into a variety of system libraries (including fairly big things like libfabric) that are beyond our control. It seems likely that we will have to run with a larger stack size, but I would still like to have a better understanding of where the problem paths are, and how much head room we really need, if anyone has suggestions. thanks! -Phil On 2019-02-21 15:31:53-05:00 Carns, Philip H. via discuss wrote: Hi all, There is a little bit of back story on https://github.com/pmodels/argobots/issues/93 , but make a long story short we have realized that we have some code that is overflowing the stack in Argobots. Many thanks to Shintaro for his help and insight or we may have never figured this out. We can work around the problem with `export ABT_THREAD_STACKSIZE=$((1024 * 1024))`. This not only fixes a Power8 test case for us, but also appears to solve a different frustrating, nonsensical segmentation fault that we've been chasing with a different code permutation on x86_64. Any suggestions on how to track down what's triggering this in our code or get a better idea of how much stack we need? We are using a considerable number of libraries, many of which are not maintained by us, so I don't even know where to start looking yet. My usual go to tool for this would be asan in gcc or clang, but I don't think that will work correctly with Argobots, and maybe there is a better solution anyway. thanks, -Phil --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Hello, Phil, Thank you for your report. We could find a bug. The problem you reported seems to happen when `ABT_THREAD_STACKSIZE` is equal to `ABT_SCHED_STACKSIZE` (by default 4MB). For now, it can be avoided by setting a different value (e.g., `ABT_THREAD_STACKSIZE=$((4 * 1024 * 1024 + 64))`). If the problem persists, please let us know. Please see the GitHub issue (https://github.com/pmodels/argobots/issues/94) and the PR (https://github.com/pmodels/argobots/pull/95) for details. I note that an error might occur if quite large thread stack size is requested (e.g., `ABT_THREAD_STACKSIZE` > `ABTD_MEM_STACK_PAGE_SIZE`, which is by default 8MB). If really needed, please increase `ABTD_MEM_STACK_PAGE_SIZE` as well, though I believe 2MB ~ 4MB should be enough in most cases. The latter issue will be investigated and fixed in the future. Best Regards, Shintaro Iwasaki On Fri, Feb 22, 2019 at 10:34 AM Iwasaki, Shintaro via discuss <[email protected]<mailto:[email protected]>> wrote: Hello, Thank you, Phil. I will create an GitHub issue about the the stack overflow detection once we understand all the problems. I'm happy to hear that margo worked with 1MB stack size. However, Argobots should work with 4MB stack size (which is the default Pthreads stack size on some machines) or larger as well; there should be no upper bound. A scheduler has its own stack size (tunable via ABT_SCHED_STACKSIZE) and I suspect it is related since the bug happens in ABT_xstream_set_main_sched. I will check this issue soon after some urgent tasks. (Note that scheduler stack size is independent of ULT stack size by design since it virtually affects the stack size of every tasklet execution) Thank You, Shintaro Iwasaki On Fri, Feb 22, 2019 at 9:57 AM Carns, Philip H. <[email protected]<mailto:[email protected]>> wrote: Thanks Shintaro. It seems like a minimum we will have to use 1. We can't control our component scope enough to get away with 0 generally. I'm interested in following up on 2, or techniques like it, later on. Maybe for regression testing. I'll have to circle back to it a little later, though. Fortunately for our use case we have one particular library that is used in most of our builds where we can centralize some of these parameters. I'm setting this (and another parameter we have been using) in this subroutine: https://xgitlab.cels.anl.gov/sds/margo/blob/master/src/margo.c#L180 This works fine (setting stack size to 1M). My first guess was to set it to 4M to be even more conservative until we understand the issue better, but that caused a different crash almost immediately when I tried that: (gdb) where #0 __GI___libc_free (mem=0x7ffff72d9000) at malloc.c:3085 #1 0x00007ffff7fb16d8 in ABTI_sched_free ( p_sched=p_sched@entry=0x55555555bbb0) at ../../src/sched/sched.c:747 #2 0x00007ffff7fa5795 in ABTI_sched_discard_and_free (p_sched=0x55555555bbb0) at ../../src/include/abti_sched.h:51 #3 ABTI_xstream_set_main_sched (p_xstream=p_xstream@entry=0x55555555ba10, p_sched=0x55555555bfb0) at ../../src/stream.c:1721 #4 0x00007ffff7fa5f09 in ABT_xstream_set_main_sched (xstream=0x55555555ba10, sched=<optimized out>) at ../../src/stream.c:811 #5 0x00007ffff7fc1ee0 in margo_init_opt (addr_str=0x7fffffffc90a "na+sm://", mode=1, hg_init_info=0x0, use_progress_thread=0, rpc_thread_count=-1) at ../src/margo.c:248 #6 0x0000555555556aad in main (argc=2, argv=0x7fffffffc488) at ../examples/margo-example-server.c:39 I suspect that line number just happened to get caught in the crossfire of a memory corruption but I haven't investigated. At any rate, is there a practical (and lower than 4M) limit to how big we can set the stack size? Or is there another tunable that must be boosted as well to allow for larger stacks? thanks! -Phil On 2019-02-21 17:55:56-05:00 Iwasaki, Shintaro via discuss wrote: Hello All, Thank you for your reports! As one of the developers, I would like to summarize the current status of Argobots. As pointed out, Argobots, by default, uses 16KB for ULTs. At present, these three ways are relatively reasonable to work around, find, or solve this issue; 0. Use knowledge that, if Argobots is used alone, it should "typically" happens in ABT_finalize. The stack size is set by ABT_thread_attr_set_stacksize individually. 1. Use larger stacksize (e.g., ABT_THREAD_STACKSIZE=$((4 * 1024 * 1024)) for example) and see what happens. 2. [Uncertain] Use Valgrind with --enable-valgrind (although it is extremely slow, so not practical for large applications) For now, I think the workaround 1. (using larger stack size by default) is best among them. I haven't tried other tools, but I strongly believe that Argobots-unaware tools won't detect this problem; only Valgind can detect it. --- There are several issues: 1. Too small default stack size It might be too small to drive large system libraries (e.g., a ULT as a progress thread) I'm not sure how much it should be increased, or first of all, whether we should increase it. This does not solve the problem of the "silent stack corruption", though. In other words, if Argobots can detect stack overflow, users can change the value by increasing the default stack size or the stack size of a specific thread requiring large amount of stack. 2. Lack of stack overflow detection For example, the following two are often used; - Stack canaries (lazy but cheap) - mprotect (eager but expensive) I will create a GitHub issue for further details if detection is preferable. 3. Check if Valgrind works for this issue If --enable-valgrind is set, Argobots registers ULT's stacks to Valgrind. It should work but I haven't tested it yet. Another problem is that --enable-valgrind degrades performance of Argobots even if it is not run on Valgrind (see https://github.com/pmodels/argobots/issues/78). We would appreciate any feedback. Thank You, Shintaro Iwasaki ________________________________ From: Lombardi, Johann via discuss <[email protected]<mailto:[email protected]>> Sent: Thursday, February 21, 2019 4:30 PM To: [email protected]<mailto:[email protected]> Cc: Lombardi, Johann; Liu, Xuezhao; Wang, Di Subject: Re: [argobots-discuss] how to debug a stack overrun in Argobots Hi Phil, I think we hit the same issue recently on the DAOS side and had to bump the stack size as well. Wangdi & Xuezhao should know more. Maybe a regression in ABT? Cheers, Johann From: "Carns, Philip H. via discuss" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, 21 February 2019 at 15:50 To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Cc: "Carns, Philip H." <[email protected]<mailto:[email protected]>> Subject: Re: [argobots-discuss] how to debug a stack overrun in Argobots Just to follow up a little bit; I realized from looking at README.envvar just now that the default value of ABT_THREAD_STACKSIZE is 16K. That's almost certainly too low for us because we have ULTs that make calls into a variety of system libraries (including fairly big things like libfabric) that are beyond our control. It seems likely that we will have to run with a larger stack size, but I would still like to have a better understanding of where the problem paths are, and how much head room we really need, if anyone has suggestions. thanks! -Phil On 2019-02-21 15:31:53-05:00 Carns, Philip H. via discuss wrote: Hi all, There is a little bit of back story on https://github.com/pmodels/argobots/issues/93 , but make a long story short we have realized that we have some code that is overflowing the stack in Argobots. Many thanks to Shintaro for his help and insight or we may have never figured this out. We can work around the problem with `export ABT_THREAD_STACKSIZE=$((1024 * 1024))`. This not only fixes a Power8 test case for us, but also appears to solve a different frustrating, nonsensical segmentation fault that we've been chasing with a different code permutation on x86_64. Any suggestions on how to track down what's triggering this in our code or get a better idea of how much stack we need? We are using a considerable number of libraries, many of which are not maintained by us, so I don't even know where to start looking yet. My usual go to tool for this would be asan in gcc or clang, but I don't think that will work correctly with Argobots, and maybe there is a better solution anyway. thanks, -Phil --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ discuss mailing list [email protected]<mailto:[email protected]> https://lists.argobots.org/mailman/listinfo/discuss
Thanks Shintaro. I confirmed (with my existing argobots build) that it worked fine with large stacks as long as the stack size wasn't precisely 4 MiB. Bad luck for me that I happened to pick the one value that wouldn't work, but I'm glad the problem has been identified :) thanks, -Phil On 2019-02-22 18:26:44-05:00 Iwasaki, Shintaro wrote: Hello, Phil, Thank you for your report. We could find a bug. The problem you reported seems to happen when `ABT_THREAD_STACKSIZE` is equal to `ABT_SCHED_STACKSIZE` (by default 4MB). For now, it can be avoided by setting a different value (e.g., `ABT_THREAD_STACKSIZE=$((4 * 1024 * 1024 + 64))`). If the problem persists, please let us know. Please see the GitHub issue (https://github.com/pmodels/argobots/issues/94) and the PR (https://github.com/pmodels/argobots/pull/95) for details. I note that an error might occur if quite large thread stack size is requested (e.g., `ABT_THREAD_STACKSIZE` > `ABTD_MEM_STACK_PAGE_SIZE`, which is by default 8MB). If really needed, please increase `ABTD_MEM_STACK_PAGE_SIZE` as well, though I believe 2MB ~ 4MB should be enough in most cases. The latter issue will be investigated and fixed in the future. Best Regards, Shintaro Iwasaki On Fri, Feb 22, 2019 at 10:34 AM Iwasaki, Shintaro via discuss <[email protected]<mailto:[email protected]>> wrote: Hello, Thank you, Phil. I will create an GitHub issue about the the stack overflow detection once we understand all the problems. I'm happy to hear that margo worked with 1MB stack size. However, Argobots should work with 4MB stack size (which is the default Pthreads stack size on some machines) or larger as well; there should be no upper bound. A scheduler has its own stack size (tunable via ABT_SCHED_STACKSIZE) and I suspect it is related since the bug happens in ABT_xstream_set_main_sched. I will check this issue soon after some urgent tasks. (Note that scheduler stack size is independent of ULT stack size by design since it virtually affects the stack size of every tasklet execution) Thank You, Shintaro Iwasaki On Fri, Feb 22, 2019 at 9:57 AM Carns, Philip H. <[email protected]<mailto:[email protected]>> wrote: Thanks Shintaro. It seems like a minimum we will have to use 1. We can't control our component scope enough to get away with 0 generally. I'm interested in following up on 2, or techniques like it, later on. Maybe for regression testing. I'll have to circle back to it a little later, though. Fortunately for our use case we have one particular library that is used in most of our builds where we can centralize some of these parameters. I'm setting this (and another parameter we have been using) in this subroutine: https://xgitlab.cels.anl.gov/sds/margo/blob/master/src/margo.c#L180 This works fine (setting stack size to 1M). My first guess was to set it to 4M to be even more conservative until we understand the issue better, but that caused a different crash almost immediately when I tried that: (gdb) where #0 __GI___libc_free (mem=0x7ffff72d9000) at malloc.c:3085 #1 0x00007ffff7fb16d8 in ABTI_sched_free ( p_sched=p_sched@entry=0x55555555bbb0) at ../../src/sched/sched.c:747 #2 0x00007ffff7fa5795 in ABTI_sched_discard_and_free (p_sched=0x55555555bbb0) at ../../src/include/abti_sched.h:51 #3 ABTI_xstream_set_main_sched (p_xstream=p_xstream@entry=0x55555555ba10, p_sched=0x55555555bfb0) at ../../src/stream.c:1721 #4 0x00007ffff7fa5f09 in ABT_xstream_set_main_sched (xstream=0x55555555ba10, sched=<optimized out>) at ../../src/stream.c:811 #5 0x00007ffff7fc1ee0 in margo_init_opt (addr_str=0x7fffffffc90a "na+sm://", mode=1, hg_init_info=0x0, use_progress_thread=0, rpc_thread_count=-1) at ../src/margo.c:248 #6 0x0000555555556aad in main (argc=2, argv=0x7fffffffc488) at ../examples/margo-example-server.c:39 I suspect that line number just happened to get caught in the crossfire of a memory corruption but I haven't investigated. At any rate, is there a practical (and lower than 4M) limit to how big we can set the stack size? Or is there another tunable that must be boosted as well to allow for larger stacks? thanks! -Phil On 2019-02-21 17:55:56-05:00 Iwasaki, Shintaro via discuss wrote: Hello All, Thank you for your reports! As one of the developers, I would like to summarize the current status of Argobots. As pointed out, Argobots, by default, uses 16KB for ULTs. At present, these three ways are relatively reasonable to work around, find, or solve this issue; 0. Use knowledge that, if Argobots is used alone, it should "typically" happens in ABT_finalize. The stack size is set by ABT_thread_attr_set_stacksize individually. 1. Use larger stacksize (e.g., ABT_THREAD_STACKSIZE=$((4 * 1024 * 1024)) for example) and see what happens. 2. [Uncertain] Use Valgrind with --enable-valgrind (although it is extremely slow, so not practical for large applications) For now, I think the workaround 1. (using larger stack size by default) is best among them. I haven't tried other tools, but I strongly believe that Argobots-unaware tools won't detect this problem; only Valgind can detect it. --- There are several issues: 1. Too small default stack size It might be too small to drive large system libraries (e.g., a ULT as a progress thread) I'm not sure how much it should be increased, or first of all, whether we should increase it. This does not solve the problem of the "silent stack corruption", though. In other words, if Argobots can detect stack overflow, users can change the value by increasing the default stack size or the stack size of a specific thread requiring large amount of stack. 2. Lack of stack overflow detection For example, the following two are often used; - Stack canaries (lazy but cheap) - mprotect (eager but expensive) I will create a GitHub issue for further details if detection is preferable. 3. Check if Valgrind works for this issue If --enable-valgrind is set, Argobots registers ULT's stacks to Valgrind. It should work but I haven't tested it yet. Another problem is that --enable-valgrind degrades performance of Argobots even if it is not run on Valgrind (see https://github.com/pmodels/argobots/issues/78). We would appreciate any feedback. Thank You, Shintaro Iwasaki ________________________________ From: Lombardi, Johann via discuss <[email protected]<mailto:[email protected]>> Sent: Thursday, February 21, 2019 4:30 PM To: [email protected]<mailto:[email protected]> Cc: Lombardi, Johann; Liu, Xuezhao; Wang, Di Subject: Re: [argobots-discuss] how to debug a stack overrun in Argobots Hi Phil, I think we hit the same issue recently on the DAOS side and had to bump the stack size as well. Wangdi & Xuezhao should know more. Maybe a regression in ABT? Cheers, Johann From: "Carns, Philip H. via discuss" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, 21 February 2019 at 15:50 To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Cc: "Carns, Philip H." <[email protected]<mailto:[email protected]>> Subject: Re: [argobots-discuss] how to debug a stack overrun in Argobots Just to follow up a little bit; I realized from looking at README.envvar just now that the default value of ABT_THREAD_STACKSIZE is 16K. That's almost certainly too low for us because we have ULTs that make calls into a variety of system libraries (including fairly big things like libfabric) that are beyond our control. It seems likely that we will have to run with a larger stack size, but I would still like to have a better understanding of where the problem paths are, and how much head room we really need, if anyone has suggestions. thanks! -Phil On 2019-02-21 15:31:53-05:00 Carns, Philip H. via discuss wrote: Hi all, There is a little bit of back story on https://github.com/pmodels/argobots/issues/93 , but make a long story short we have realized that we have some code that is overflowing the stack in Argobots. Many thanks to Shintaro for his help and insight or we may have never figured this out. We can work around the problem with `export ABT_THREAD_STACKSIZE=$((1024 * 1024))`. This not only fixes a Power8 test case for us, but also appears to solve a different frustrating, nonsensical segmentation fault that we've been chasing with a different code permutation on x86_64. Any suggestions on how to track down what's triggering this in our code or get a better idea of how much stack we need? We are using a considerable number of libraries, many of which are not maintained by us, so I don't even know where to start looking yet. My usual go to tool for this would be asan in gcc or clang, but I don't think that will work correctly with Argobots, and maybe there is a better solution anyway. thanks, -Phil --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ discuss mailing list [email protected]<mailto:[email protected]> https://lists.argobots.org/mailman/listinfo/discuss
participants (3)
-
Carns, Philip H. -
Iwasaki, Shintaro -
Lombardi, Johann