Darshan crashing compute node
Hi, I'm traying to use darshan (tested all 3.1.X versions) on a HPC cluster running with Red Hat 7.6 (kernel 3.10.0-957.el7.x86_64) and using Lustre Filesystem version 2.11.0.300_cray_102_g3dbace1, but when the application starts, the compute node crashes with a kernel panic message on the console. While searching on the Darshan mailing list I came across this message: https://lists.mcs.anl.gov/mailman/htdig/darshan-users/2019-October/000542.ht... I disabled the Lustre module and recompiled darshan. The crash isn't happening anymore, but now it's unable to write the darshan log file on the lustre filesystem, with the following error: *Using OpenMPI 3.1.5 and GCC 7 Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation. Backtrace for this error: #0 0x7f4f6759f27f in ??? #1 0x7f4f687ababe in ??? #2 0x7f4f687add06 in ??? #3 0x7f4f687db6c0 in ??? #4 0x7f4f687dbddb in ??? #5 0x7f4f6879d6f1 in ??? #6 0x7f4f6871892b in ??? #7 0x7f4f691d0ae1 in MPI_File_write_at_all at lib/darshan-mpiio.c:536 #8 0x7f4f691bea7f in darshan_log_append_all at lib/darshan-core.c:1800 #9 0x7f4f691c1907 in darshan_log_write_name_record_hash at lib/darshan-core.c:1761 #10 0x7f4f691c1907 in darshan_core_shutdown at lib/darshan-core.c:546 #11 0x7f4f691be402 in MPI_Finalize at lib/darshan-core-init-finalize.c:82 #12 0x7f4f68b6a798 in ??? #13 0x4023bb in ??? #14 0x401ae6 in ??? #15 0x7f4f6758b3d4 in ??? #16 0x401b16 in ??? #17 0xffffffffffffffff in ??? -------------------------------------------------------------------------- *Using Intel PSXE 2018 with Intel MPI forrtl: severe (71): integer divide by zero Image PC Routine Line Source exec.exe 000000000045282E Unknown Unknown Unknown libpthread-2.17.s 00002B8B5A5FE5D0 Unknown Unknown Unknown libmpi_lustre.so. 00002B8B659D4FDF ADIOI_LUSTRE_Get_ Unknown Unknown libmpi_lustre.so. 00002B8B659CFFD9 ADIOI_LUSTRE_Writ Unknown Unknown libmpi.so.12.0 00002B8B59A4C15C Unknown Unknown Unknown libmpi.so.12 00002B8B59A4D1D5 PMPI_File_write_a Unknown Unknown libdarshan.so 00002B8B58F90312 MPI_File_write_at Unknown Unknown libdarshan.so 00002B8B58F7E63A Unknown Unknown Unknown libdarshan.so 00002B8B58F815B0 darshan_core_shut Unknown Unknown libdarshan.so 00002B8B58F7DFF3 MPI_Finalize Unknown Unknown libmpifort.so.12. 00002B8B592414DA pmpi_finalize__ Unknown Unknown exec.exe 00000000004490A5 Unknown Unknown Unknown exec.exe 00000000004032DE Unknown Unknown Unknown libc-2.17.so 00002B8B5AB2F3D5 __libc_start_main Unknown Unknown exec.exe 00000000004031E9 Unknown Unknown Unknown If I configure darshan to write on a different filesystem (local /tmp of the first compute node), it works out fine, but then I'm restricted to use only one compute node, since the output dir have to be shared among all nodes (MPI tasks). Is there a workaround for this? At the moment, my cluster only have the lustre filesystem as a shared filesystem among all compute nodes. Best regards. -- Abraços³, André Ramos Carneiro.
Thanks for the bug reports André! You are describing two different problems: 1) incompatibility between the Darshan Lustre instrumentation and recent Lustre releases: we'll definitely look into this. As you found in the mailing list archives there has been a known problem here, but this is a new permutation that it causes a node crash now. That's disappointing, to say the least. If anyone happens to have a pointer to how to stand up a Lustre 2.11 instance in docker or a VM they can share, that would be greatly appreciated and would help us track compatibility better. The Lustre systems we have access to are not new enough. In the mean time you are correct to disable that component at build time. 2) floating point or divide by zero when writing log to Lustre: This one is odd because at the point of the problem, Darshan is not instrumenting anything: it is simply an application writing data to Lustre. We don't generally see problems here. Darshan is doing a collective write here as the last step of aggregating instrumentation from all ranks into a single log file. The only slightly unusual things that Darshan could be doing are a) setting MPI-IO hints or b) using particular datatypes to organize the writes. Neither should be a problem, but it looks like they might have triggered a bug somewhere. If you don't mind trying another test case, could you repeat one of the experiments that crashes with a floating point or divide by zero bug down in MPI_File_write_at_all() with the following environment variable set? (please keep disabling the Lustre module at build time, we have no other fix for that ready yet) export DARSHAN_LOGHINTS= (in other words, set the DARSHAN_LOGHINTS environment variable to an empty string) I believe that this will clear the MPI-IO hints that Darshan would normally specify when opening the output log file. If that makes your example work cleanly, then we can narrow down which hint is the problem and maybe get some help outside of Darshan for the root cause. If it still fails, then we need to look elsewhere for the problem. thanks, -Phil ________________________________ From: Darshan-users <[email protected]> on behalf of André R. Carneiro <[email protected]> Sent: Wednesday, April 1, 2020 1:39 PM To: [email protected] <[email protected]> Subject: [Darshan-users] Darshan crashing compute node Hi, I'm traying to use darshan (tested all 3.1.X versions) on a HPC cluster running with Red Hat 7.6 (kernel 3.10.0-957.el7.x86_64) and using Lustre Filesystem version 2.11.0.300_cray_102_g3dbace1, but when the application starts, the compute node crashes with a kernel panic message on the console. While searching on the Darshan mailing list I came across this message: https://lists.mcs.anl.gov/mailman/htdig/darshan-users/2019-October/000542.ht... I disabled the Lustre module and recompiled darshan. The crash isn't happening anymore, but now it's unable to write the darshan log file on the lustre filesystem, with the following error: *Using OpenMPI 3.1.5 and GCC 7 Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation. Backtrace for this error: #0 0x7f4f6759f27f in ??? #1 0x7f4f687ababe in ??? #2 0x7f4f687add06 in ??? #3 0x7f4f687db6c0 in ??? #4 0x7f4f687dbddb in ??? #5 0x7f4f6879d6f1 in ??? #6 0x7f4f6871892b in ??? #7 0x7f4f691d0ae1 in MPI_File_write_at_all at lib/darshan-mpiio.c:536 #8 0x7f4f691bea7f in darshan_log_append_all at lib/darshan-core.c:1800 #9 0x7f4f691c1907 in darshan_log_write_name_record_hash at lib/darshan-core.c:1761 #10 0x7f4f691c1907 in darshan_core_shutdown at lib/darshan-core.c:546 #11 0x7f4f691be402 in MPI_Finalize at lib/darshan-core-init-finalize.c:82 #12 0x7f4f68b6a798 in ??? #13 0x4023bb in ??? #14 0x401ae6 in ??? #15 0x7f4f6758b3d4 in ??? #16 0x401b16 in ??? #17 0xffffffffffffffff in ??? -------------------------------------------------------------------------- *Using Intel PSXE 2018 with Intel MPI forrtl: severe (71): integer divide by zero Image PC Routine Line Source exec.exe 000000000045282E Unknown Unknown Unknown libpthread-2.17.s 00002B8B5A5FE5D0 Unknown Unknown Unknown libmpi_lustre.so. 00002B8B659D4FDF ADIOI_LUSTRE_Get_ Unknown Unknown libmpi_lustre.so. 00002B8B659CFFD9 ADIOI_LUSTRE_Writ Unknown Unknown libmpi.so.12.0 00002B8B59A4C15C Unknown Unknown Unknown libmpi.so.12 00002B8B59A4D1D5 PMPI_File_write_a Unknown Unknown libdarshan.so 00002B8B58F90312 MPI_File_write_at Unknown Unknown libdarshan.so 00002B8B58F7E63A Unknown Unknown Unknown libdarshan.so 00002B8B58F815B0 darshan_core_shut Unknown Unknown libdarshan.so 00002B8B58F7DFF3 MPI_Finalize Unknown Unknown libmpifort.so.12. 00002B8B592414DA pmpi_finalize__ Unknown Unknown exec.exe 00000000004490A5 Unknown Unknown Unknown exec.exe 00000000004032DE Unknown Unknown Unknown libc-2.17.so<http://libc-2.17.so> 00002B8B5AB2F3D5 __libc_start_main Unknown Unknown exec.exe 00000000004031E9 Unknown Unknown Unknown If I configure darshan to write on a different filesystem (local /tmp of the first compute node), it works out fine, but then I'm restricted to use only one compute node, since the output dir have to be shared among all nodes (MPI tasks). Is there a workaround for this? At the moment, my cluster only have the lustre filesystem as a shared filesystem among all compute nodes. Best regards. -- Abraços³, André Ramos Carneiro.
FYI, I opened a GitLab issue to track this problem: https://xgitlab.cels.anl.gov/darshan/darshan/issues/270 The crashes are a new surprise to us, so would like to have a look before our next release. I'll look to see if I can get a newer Lustre version working in a container or something to test against and will report back what I find. Thanks, --Shane ________________________________ From: Darshan-users <[email protected]> on behalf of Carns, Philip H. <[email protected]> Sent: Thursday, April 2, 2020 10:27 AM To: André R. Carneiro <[email protected]>; [email protected] <[email protected]> Subject: Re: [Darshan-users] Darshan crashing compute node Thanks for the bug reports André! You are describing two different problems: 1) incompatibility between the Darshan Lustre instrumentation and recent Lustre releases: we'll definitely look into this. As you found in the mailing list archives there has been a known problem here, but this is a new permutation that it causes a node crash now. That's disappointing, to say the least. If anyone happens to have a pointer to how to stand up a Lustre 2.11 instance in docker or a VM they can share, that would be greatly appreciated and would help us track compatibility better. The Lustre systems we have access to are not new enough. In the mean time you are correct to disable that component at build time. 2) floating point or divide by zero when writing log to Lustre: This one is odd because at the point of the problem, Darshan is not instrumenting anything: it is simply an application writing data to Lustre. We don't generally see problems here. Darshan is doing a collective write here as the last step of aggregating instrumentation from all ranks into a single log file. The only slightly unusual things that Darshan could be doing are a) setting MPI-IO hints or b) using particular datatypes to organize the writes. Neither should be a problem, but it looks like they might have triggered a bug somewhere. If you don't mind trying another test case, could you repeat one of the experiments that crashes with a floating point or divide by zero bug down in MPI_File_write_at_all() with the following environment variable set? (please keep disabling the Lustre module at build time, we have no other fix for that ready yet) export DARSHAN_LOGHINTS= (in other words, set the DARSHAN_LOGHINTS environment variable to an empty string) I believe that this will clear the MPI-IO hints that Darshan would normally specify when opening the output log file. If that makes your example work cleanly, then we can narrow down which hint is the problem and maybe get some help outside of Darshan for the root cause. If it still fails, then we need to look elsewhere for the problem. thanks, -Phil ________________________________ From: Darshan-users <[email protected]> on behalf of André R. Carneiro <[email protected]> Sent: Wednesday, April 1, 2020 1:39 PM To: [email protected] <[email protected]> Subject: [Darshan-users] Darshan crashing compute node Hi, I'm traying to use darshan (tested all 3.1.X versions) on a HPC cluster running with Red Hat 7.6 (kernel 3.10.0-957.el7.x86_64) and using Lustre Filesystem version 2.11.0.300_cray_102_g3dbace1, but when the application starts, the compute node crashes with a kernel panic message on the console. While searching on the Darshan mailing list I came across this message: https://lists.mcs.anl.gov/mailman/htdig/darshan-users/2019-October/000542.ht... I disabled the Lustre module and recompiled darshan. The crash isn't happening anymore, but now it's unable to write the darshan log file on the lustre filesystem, with the following error: *Using OpenMPI 3.1.5 and GCC 7 Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation. Backtrace for this error: #0 0x7f4f6759f27f in ??? #1 0x7f4f687ababe in ??? #2 0x7f4f687add06 in ??? #3 0x7f4f687db6c0 in ??? #4 0x7f4f687dbddb in ??? #5 0x7f4f6879d6f1 in ??? #6 0x7f4f6871892b in ??? #7 0x7f4f691d0ae1 in MPI_File_write_at_all at lib/darshan-mpiio.c:536 #8 0x7f4f691bea7f in darshan_log_append_all at lib/darshan-core.c:1800 #9 0x7f4f691c1907 in darshan_log_write_name_record_hash at lib/darshan-core.c:1761 #10 0x7f4f691c1907 in darshan_core_shutdown at lib/darshan-core.c:546 #11 0x7f4f691be402 in MPI_Finalize at lib/darshan-core-init-finalize.c:82 #12 0x7f4f68b6a798 in ??? #13 0x4023bb in ??? #14 0x401ae6 in ??? #15 0x7f4f6758b3d4 in ??? #16 0x401b16 in ??? #17 0xffffffffffffffff in ??? -------------------------------------------------------------------------- *Using Intel PSXE 2018 with Intel MPI forrtl: severe (71): integer divide by zero Image PC Routine Line Source exec.exe 000000000045282E Unknown Unknown Unknown libpthread-2.17.s 00002B8B5A5FE5D0 Unknown Unknown Unknown libmpi_lustre.so. 00002B8B659D4FDF ADIOI_LUSTRE_Get_ Unknown Unknown libmpi_lustre.so. 00002B8B659CFFD9 ADIOI_LUSTRE_Writ Unknown Unknown libmpi.so.12.0 00002B8B59A4C15C Unknown Unknown Unknown libmpi.so.12 00002B8B59A4D1D5 PMPI_File_write_a Unknown Unknown libdarshan.so 00002B8B58F90312 MPI_File_write_at Unknown Unknown libdarshan.so 00002B8B58F7E63A Unknown Unknown Unknown libdarshan.so 00002B8B58F815B0 darshan_core_shut Unknown Unknown libdarshan.so 00002B8B58F7DFF3 MPI_Finalize Unknown Unknown libmpifort.so.12. 00002B8B592414DA pmpi_finalize__ Unknown Unknown exec.exe 00000000004490A5 Unknown Unknown Unknown exec.exe 00000000004032DE Unknown Unknown Unknown libc-2.17.so<http://libc-2.17.so> 00002B8B5AB2F3D5 __libc_start_main Unknown Unknown exec.exe 00000000004031E9 Unknown Unknown Unknown If I configure darshan to write on a different filesystem (local /tmp of the first compute node), it works out fine, but then I'm restricted to use only one compute node, since the output dir have to be shared among all nodes (MPI tasks). Is there a workaround for this? At the moment, my cluster only have the lustre filesystem as a shared filesystem among all compute nodes. Best regards. -- Abraços³, André Ramos Carneiro.
Hi Phil. First of all, thank you very much for the attention and the information. The "export DARSHAN_LOGHINTS=" solved the problem! Now the darshan log file is written without problem. I would like to point out that before the "configuration" of DARSHAN_LOGHINTS, when I used the darshan-parser to extract some info from the log file, I got the following: # metadata: lib_ver = 3.1.8 # metadata: h = romio_no_indep_rw=true;cb_nodes=4 With the configuration of "DARSHAN_LOGHINTS=" there's no info about metadata or the hints used. Thank you very much for the valuable help! Best regards. On Thu, Apr 2, 2020 at 12:27 PM Carns, Philip H. <[email protected]> wrote:
Thanks for the bug reports André! You are describing two different problems:
1) incompatibility between the Darshan Lustre instrumentation and recent Lustre releases: we'll definitely look into this. As you found in the mailing list archives there has been a known problem here, but this is a new permutation that it causes a node crash now. That's disappointing, to say the least. If anyone happens to have a pointer to how to stand up a Lustre 2.11 instance in docker or a VM they can share, that would be greatly appreciated and would help us track compatibility better. The Lustre systems we have access to are not new enough. In the mean time you are correct to disable that component at build time.
2) floating point or divide by zero when writing log to Lustre: This one is odd because at the point of the problem, Darshan is not instrumenting anything: it is simply an application writing data to Lustre. We don't generally see problems here. Darshan is doing a collective write here as the last step of aggregating instrumentation from all ranks into a single log file. The only slightly unusual things that Darshan could be doing are a) setting MPI-IO hints or b) using particular datatypes to organize the writes. Neither should be a problem, but it looks like they might have triggered a bug somewhere.
If you don't mind trying another test case, could you repeat one of the experiments that crashes with a floating point or divide by zero bug down in MPI_File_write_at_all() with the following environment variable set? (please keep disabling the Lustre module at build time, we have no other fix for that ready yet)
export DARSHAN_LOGHINTS=
(in other words, set the DARSHAN_LOGHINTS environment variable to an empty string)
I believe that this will clear the MPI-IO hints that Darshan would normally specify when opening the output log file. If that makes your example work cleanly, then we can narrow down which hint is the problem and maybe get some help outside of Darshan for the root cause. If it still fails, then we need to look elsewhere for the problem.
thanks, -Phil ------------------------------ *From:* Darshan-users <[email protected]> on behalf of André R. Carneiro <[email protected]> *Sent:* Wednesday, April 1, 2020 1:39 PM *To:* [email protected] <[email protected]> *Subject:* [Darshan-users] Darshan crashing compute node
Hi,
I'm traying to use darshan (tested all 3.1.X versions) on a HPC cluster running with Red Hat 7.6 (kernel 3.10.0-957.el7.x86_64) and using Lustre Filesystem version 2.11.0.300_cray_102_g3dbace1, but when the application starts, the compute node crashes with a kernel panic message on the console.
While searching on the Darshan mailing list I came across this message: https://lists.mcs.anl.gov/mailman/htdig/darshan-users/2019-October/000542.ht...
I disabled the Lustre module and recompiled darshan. The crash isn't happening anymore, but now it's unable to write the darshan log file on the lustre filesystem, with the following error:
*Using OpenMPI 3.1.5 and GCC 7
Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation. Backtrace for this error: #0 0x7f4f6759f27f in ??? #1 0x7f4f687ababe in ??? #2 0x7f4f687add06 in ??? #3 0x7f4f687db6c0 in ??? #4 0x7f4f687dbddb in ??? #5 0x7f4f6879d6f1 in ??? #6 0x7f4f6871892b in ??? #7 0x7f4f691d0ae1 in MPI_File_write_at_all at lib/darshan-mpiio.c:536 #8 0x7f4f691bea7f in darshan_log_append_all at lib/darshan-core.c:1800 #9 0x7f4f691c1907 in darshan_log_write_name_record_hash at lib/darshan-core.c:1761 #10 0x7f4f691c1907 in darshan_core_shutdown at lib/darshan-core.c:546 #11 0x7f4f691be402 in MPI_Finalize at lib/darshan-core-init-finalize.c:82 #12 0x7f4f68b6a798 in ??? #13 0x4023bb in ??? #14 0x401ae6 in ??? #15 0x7f4f6758b3d4 in ??? #16 0x401b16 in ??? #17 0xffffffffffffffff in ??? --------------------------------------------------------------------------
*Using Intel PSXE 2018 with Intel MPI
forrtl: severe (71): integer divide by zero Image PC Routine Line Source
exec.exe 000000000045282E Unknown Unknown Unknown libpthread-2.17.s 00002B8B5A5FE5D0 Unknown Unknown Unknown libmpi_lustre.so. 00002B8B659D4FDF ADIOI_LUSTRE_Get_ Unknown Unknown libmpi_lustre.so. 00002B8B659CFFD9 ADIOI_LUSTRE_Writ Unknown Unknown libmpi.so.12.0 00002B8B59A4C15C Unknown Unknown Unknown libmpi.so.12 00002B8B59A4D1D5 PMPI_File_write_a Unknown Unknown libdarshan.so 00002B8B58F90312 MPI_File_write_at Unknown Unknown libdarshan.so 00002B8B58F7E63A Unknown Unknown Unknown libdarshan.so 00002B8B58F815B0 darshan_core_shut Unknown Unknown libdarshan.so 00002B8B58F7DFF3 MPI_Finalize Unknown Unknown libmpifort.so.12. 00002B8B592414DA pmpi_finalize__ Unknown Unknown exec.exe 00000000004490A5 Unknown Unknown Unknown exec.exe 00000000004032DE Unknown Unknown Unknown libc-2.17.so 00002B8B5AB2F3D5 __libc_start_main Unknown Unknown exec.exe 00000000004031E9 Unknown Unknown Unknown
If I configure darshan to write on a different filesystem (local /tmp of the first compute node), it works out fine, but then I'm restricted to use only one compute node, since the output dir have to be shared among all nodes (MPI tasks).
Is there a workaround for this? At the moment, my cluster only have the lustre filesystem as a shared filesystem among all compute nodes.
Best regards.
-- Abraços³, André Ramos Carneiro.
-- Abraços³, André Ramos Carneiro.
It's my day for lucky guesses 🙂 Thank you for testing that. Those MPI hints (obviously) shouldn't make the MPI library crash. Do you have a record of how OpenMPI 3.1.5 was built? If there is some permutation of build options / mpi versions / lustre versions causing a problem then we might have to detect that somehow in Darshan's build configuration. If you are willing to spend some time on it you should be able to isolate whether it's romio_no_indep_rw=true or cb_nodes=4 (or both) triggering the problem. That's probably a bug that should be opened somewhere; any user app could set those hints, not just Darshan. That combination of hints is a performance optimization from Darshan's perspective. In particular, it makes it such that regardless of how large your application is, no more than 4 of it's processes will actually open the Darshan log file and aggregate data for the other ranks. This keeps overhead down at scale. If there are many ranks and only a little bit of Darshan data to write (a typical scenario) then the cost of having all ranks call open() and close() will be far higher than the cost of relaying through a small number of aggregators. thanks, -Phil ________________________________ From: André R. Carneiro <[email protected]> Sent: Thursday, April 2, 2020 4:44 PM To: Carns, Philip H. <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [Darshan-users] Darshan crashing compute node Hi Phil. First of all, thank you very much for the attention and the information. The "export DARSHAN_LOGHINTS=" solved the problem! Now the darshan log file is written without problem. I would like to point out that before the "configuration" of DARSHAN_LOGHINTS, when I used the darshan-parser to extract some info from the log file, I got the following: # metadata: lib_ver = 3.1.8 # metadata: h = romio_no_indep_rw=true;cb_nodes=4 With the configuration of "DARSHAN_LOGHINTS=" there's no info about metadata or the hints used. Thank you very much for the valuable help! Best regards. On Thu, Apr 2, 2020 at 12:27 PM Carns, Philip H. <[email protected]<mailto:[email protected]>> wrote: Thanks for the bug reports André! You are describing two different problems: 1) incompatibility between the Darshan Lustre instrumentation and recent Lustre releases: we'll definitely look into this. As you found in the mailing list archives there has been a known problem here, but this is a new permutation that it causes a node crash now. That's disappointing, to say the least. If anyone happens to have a pointer to how to stand up a Lustre 2.11 instance in docker or a VM they can share, that would be greatly appreciated and would help us track compatibility better. The Lustre systems we have access to are not new enough. In the mean time you are correct to disable that component at build time. 2) floating point or divide by zero when writing log to Lustre: This one is odd because at the point of the problem, Darshan is not instrumenting anything: it is simply an application writing data to Lustre. We don't generally see problems here. Darshan is doing a collective write here as the last step of aggregating instrumentation from all ranks into a single log file. The only slightly unusual things that Darshan could be doing are a) setting MPI-IO hints or b) using particular datatypes to organize the writes. Neither should be a problem, but it looks like they might have triggered a bug somewhere. If you don't mind trying another test case, could you repeat one of the experiments that crashes with a floating point or divide by zero bug down in MPI_File_write_at_all() with the following environment variable set? (please keep disabling the Lustre module at build time, we have no other fix for that ready yet) export DARSHAN_LOGHINTS= (in other words, set the DARSHAN_LOGHINTS environment variable to an empty string) I believe that this will clear the MPI-IO hints that Darshan would normally specify when opening the output log file. If that makes your example work cleanly, then we can narrow down which hint is the problem and maybe get some help outside of Darshan for the root cause. If it still fails, then we need to look elsewhere for the problem. thanks, -Phil ________________________________ From: Darshan-users <[email protected]<mailto:[email protected]>> on behalf of André R. Carneiro <[email protected]<mailto:[email protected]>> Sent: Wednesday, April 1, 2020 1:39 PM To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: [Darshan-users] Darshan crashing compute node Hi, I'm traying to use darshan (tested all 3.1.X versions) on a HPC cluster running with Red Hat 7.6 (kernel 3.10.0-957.el7.x86_64) and using Lustre Filesystem version 2.11.0.300_cray_102_g3dbace1, but when the application starts, the compute node crashes with a kernel panic message on the console. While searching on the Darshan mailing list I came across this message: https://lists.mcs.anl.gov/mailman/htdig/darshan-users/2019-October/000542.ht... I disabled the Lustre module and recompiled darshan. The crash isn't happening anymore, but now it's unable to write the darshan log file on the lustre filesystem, with the following error: *Using OpenMPI 3.1.5 and GCC 7 Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation. Backtrace for this error: #0 0x7f4f6759f27f in ??? #1 0x7f4f687ababe in ??? #2 0x7f4f687add06 in ??? #3 0x7f4f687db6c0 in ??? #4 0x7f4f687dbddb in ??? #5 0x7f4f6879d6f1 in ??? #6 0x7f4f6871892b in ??? #7 0x7f4f691d0ae1 in MPI_File_write_at_all at lib/darshan-mpiio.c:536 #8 0x7f4f691bea7f in darshan_log_append_all at lib/darshan-core.c:1800 #9 0x7f4f691c1907 in darshan_log_write_name_record_hash at lib/darshan-core.c:1761 #10 0x7f4f691c1907 in darshan_core_shutdown at lib/darshan-core.c:546 #11 0x7f4f691be402 in MPI_Finalize at lib/darshan-core-init-finalize.c:82 #12 0x7f4f68b6a798 in ??? #13 0x4023bb in ??? #14 0x401ae6 in ??? #15 0x7f4f6758b3d4 in ??? #16 0x401b16 in ??? #17 0xffffffffffffffff in ??? -------------------------------------------------------------------------- *Using Intel PSXE 2018 with Intel MPI forrtl: severe (71): integer divide by zero Image PC Routine Line Source exec.exe 000000000045282E Unknown Unknown Unknown libpthread-2.17.s 00002B8B5A5FE5D0 Unknown Unknown Unknown libmpi_lustre.so. 00002B8B659D4FDF ADIOI_LUSTRE_Get_ Unknown Unknown libmpi_lustre.so. 00002B8B659CFFD9 ADIOI_LUSTRE_Writ Unknown Unknown libmpi.so.12.0 00002B8B59A4C15C Unknown Unknown Unknown libmpi.so.12 00002B8B59A4D1D5 PMPI_File_write_a Unknown Unknown libdarshan.so 00002B8B58F90312 MPI_File_write_at Unknown Unknown libdarshan.so 00002B8B58F7E63A Unknown Unknown Unknown libdarshan.so 00002B8B58F815B0 darshan_core_shut Unknown Unknown libdarshan.so 00002B8B58F7DFF3 MPI_Finalize Unknown Unknown libmpifort.so.12. 00002B8B592414DA pmpi_finalize__ Unknown Unknown exec.exe 00000000004490A5 Unknown Unknown Unknown exec.exe 00000000004032DE Unknown Unknown Unknown libc-2.17.so<http://libc-2.17.so> 00002B8B5AB2F3D5 __libc_start_main Unknown Unknown exec.exe 00000000004031E9 Unknown Unknown Unknown If I configure darshan to write on a different filesystem (local /tmp of the first compute node), it works out fine, but then I'm restricted to use only one compute node, since the output dir have to be shared among all nodes (MPI tasks). Is there a workaround for this? At the moment, my cluster only have the lustre filesystem as a shared filesystem among all compute nodes. Best regards. -- Abraços³, André Ramos Carneiro. -- Abraços³, André Ramos Carneiro.
Hi Phil, Here's how OpenMPI 3.1.5 was configured/built: '--enable-mpi-cxx' '--enable-static' '--enable-shared' '--with-verbs' '--with-libltdl' '--with-slurm' '--enable-orterun-prefix-by-default' '--enable-mpirun-prefix-by-default' '--enable-openib-rdmacm-ibaddr' '--with-pmi=/usr' '--without-cuda' '--with-knem=/opt/knem-1.1.3.90mlnx1' '--with-lustre=/usr' '--with-io-romio-flags=--with-file-system=ufs+nfs+lustre CFLAGS=-I/usr/include/lustre CPPFLAGS=-I/usr/include/lustre' '--with-ucx=/usr' '--with-ucx-libdir=/usr/lib64' '--without-mxm I repeated the experiment, using OMPIO from OpenMPI (export OMPI_MCA_io=ompio), instead of ROMIO. With this scenario I'm unable to write the darshan log file on lustre, with or without the configuration of "export DARSHAN_LOGHINTS=", with the error being slightly different: --------------------- Program received signal SIGSEGV: Segmentation fault - invalid memory reference. Backtrace for this error: error opening file /PATH/TO/FILE.darshan_partial .... #0 0x7f204bc7f27f in ??? #1 0x7f204d89f4fd in darshan_core_lookup_record_name at lib/darshan-core.c:2125 #2 0x7f204d8bb844 in darshan_stdio_lookup_record_name at lib/darshan-stdio.c:1456 #3 0x7f204d8a6f87 in fileno at lib/darshan-posix.c:590 #4 0x7f204a22a504 in ??? #5 0x7f204a22b920 in ??? #6 0x7f204ce79817 in ??? #7 0x7f204ce1bf1a in ??? #8 0x7f204cec1361 in ??? #9 0x7f204ce7b5c8 in ??? #10 0x7f204cdc7feb in ??? #11 0x7f204cdf5903 in ??? #12 0x7f204d8aef9b in MPI_File_open at lib/darshan-mpiio.c:316 #13 0x7f204d8a0b98 in darshan_log_open_all at lib/darshan-core.c:1574 #14 0x7f204d8a0b98 in darshan_core_shutdown at lib/darshan-core.c:477 #15 0x7f204d89e402 in MPI_Finalize at lib/darshan-core-init-finalize.c:82 #16 0x7f204d24a798 in ??? #17 0x4023bb in ??? #18 0x401ae6 in ??? #19 0x7f204bc6b3d4 in ??? #20 0x401b16 in ??? #21 0xffffffffffffffff in ??? -------------------------------------------------------------------------- We also have at our disposal the OpenMPI 2.4 (which came as the default MPI on the system) and I'm still unable to write the log file on Lustre filesystem, even with "export DARSHAN_LOGHINTS=". I would also like to point out that with an older version of the Lustre Client (2.4.3), both OMPIO and ROMIO worked out fine, with no problems to write the darshan log file. Once again, thank you very much the help and attention. I'll try to do some digging on this issue about the hints and the lustre filesystem. I'll give the feedback as soon as I find out anything. Best regards. On Thu, Apr 2, 2020 at 5:58 PM Carns, Philip H. <[email protected]> wrote:
It's my day for lucky guesses 🙂 Thank you for testing that.
Those MPI hints (obviously) shouldn't make the MPI library crash. Do you have a record of how OpenMPI 3.1.5 was built? If there is some permutation of build options / mpi versions / lustre versions causing a problem then we might have to detect that somehow in Darshan's build configuration.
If you are willing to spend some time on it you should be able to isolate whether it's romio_no_indep_rw=true or cb_nodes=4 (or both) triggering the problem. That's probably a bug that should be opened somewhere; any user app could set those hints, not just Darshan.
That combination of hints is a performance optimization from Darshan's perspective. In particular, it makes it such that regardless of how large your application is, no more than 4 of it's processes will actually open the Darshan log file and aggregate data for the other ranks. This keeps overhead down at scale. If there are many ranks and only a little bit of Darshan data to write (a typical scenario) then the cost of having all ranks call open() and close() will be far higher than the cost of relaying through a small number of aggregators.
thanks, -Phil ------------------------------ *From:* André R. Carneiro <[email protected]> *Sent:* Thursday, April 2, 2020 4:44 PM *To:* Carns, Philip H. <[email protected]> *Cc:* [email protected] <[email protected]> *Subject:* Re: [Darshan-users] Darshan crashing compute node
Hi Phil.
First of all, thank you very much for the attention and the information.
The "export DARSHAN_LOGHINTS=" solved the problem!
Now the darshan log file is written without problem.
I would like to point out that before the "configuration" of DARSHAN_LOGHINTS, when I used the darshan-parser to extract some info from the log file, I got the following:
# metadata: lib_ver = 3.1.8 # metadata: h = romio_no_indep_rw=true;cb_nodes=4
With the configuration of "DARSHAN_LOGHINTS=" there's no info about metadata or the hints used.
Thank you very much for the valuable help!
Best regards.
On Thu, Apr 2, 2020 at 12:27 PM Carns, Philip H. <[email protected]> wrote:
Thanks for the bug reports André! You are describing two different problems:
1) incompatibility between the Darshan Lustre instrumentation and recent Lustre releases: we'll definitely look into this. As you found in the mailing list archives there has been a known problem here, but this is a new permutation that it causes a node crash now. That's disappointing, to say the least. If anyone happens to have a pointer to how to stand up a Lustre 2.11 instance in docker or a VM they can share, that would be greatly appreciated and would help us track compatibility better. The Lustre systems we have access to are not new enough. In the mean time you are correct to disable that component at build time.
2) floating point or divide by zero when writing log to Lustre: This one is odd because at the point of the problem, Darshan is not instrumenting anything: it is simply an application writing data to Lustre. We don't generally see problems here. Darshan is doing a collective write here as the last step of aggregating instrumentation from all ranks into a single log file. The only slightly unusual things that Darshan could be doing are a) setting MPI-IO hints or b) using particular datatypes to organize the writes. Neither should be a problem, but it looks like they might have triggered a bug somewhere.
If you don't mind trying another test case, could you repeat one of the experiments that crashes with a floating point or divide by zero bug down in MPI_File_write_at_all() with the following environment variable set? (please keep disabling the Lustre module at build time, we have no other fix for that ready yet)
export DARSHAN_LOGHINTS=
(in other words, set the DARSHAN_LOGHINTS environment variable to an empty string)
I believe that this will clear the MPI-IO hints that Darshan would normally specify when opening the output log file. If that makes your example work cleanly, then we can narrow down which hint is the problem and maybe get some help outside of Darshan for the root cause. If it still fails, then we need to look elsewhere for the problem.
thanks, -Phil ------------------------------ *From:* Darshan-users <[email protected]> on behalf of André R. Carneiro <[email protected]> *Sent:* Wednesday, April 1, 2020 1:39 PM *To:* [email protected] <[email protected]> *Subject:* [Darshan-users] Darshan crashing compute node
Hi,
I'm traying to use darshan (tested all 3.1.X versions) on a HPC cluster running with Red Hat 7.6 (kernel 3.10.0-957.el7.x86_64) and using Lustre Filesystem version 2.11.0.300_cray_102_g3dbace1, but when the application starts, the compute node crashes with a kernel panic message on the console.
While searching on the Darshan mailing list I came across this message: https://lists.mcs.anl.gov/mailman/htdig/darshan-users/2019-October/000542.ht...
I disabled the Lustre module and recompiled darshan. The crash isn't happening anymore, but now it's unable to write the darshan log file on the lustre filesystem, with the following error:
*Using OpenMPI 3.1.5 and GCC 7
Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation. Backtrace for this error: #0 0x7f4f6759f27f in ??? #1 0x7f4f687ababe in ??? #2 0x7f4f687add06 in ??? #3 0x7f4f687db6c0 in ??? #4 0x7f4f687dbddb in ??? #5 0x7f4f6879d6f1 in ??? #6 0x7f4f6871892b in ??? #7 0x7f4f691d0ae1 in MPI_File_write_at_all at lib/darshan-mpiio.c:536 #8 0x7f4f691bea7f in darshan_log_append_all at lib/darshan-core.c:1800 #9 0x7f4f691c1907 in darshan_log_write_name_record_hash at lib/darshan-core.c:1761 #10 0x7f4f691c1907 in darshan_core_shutdown at lib/darshan-core.c:546 #11 0x7f4f691be402 in MPI_Finalize at lib/darshan-core-init-finalize.c:82 #12 0x7f4f68b6a798 in ??? #13 0x4023bb in ??? #14 0x401ae6 in ??? #15 0x7f4f6758b3d4 in ??? #16 0x401b16 in ??? #17 0xffffffffffffffff in ??? --------------------------------------------------------------------------
*Using Intel PSXE 2018 with Intel MPI
forrtl: severe (71): integer divide by zero Image PC Routine Line Source
exec.exe 000000000045282E Unknown Unknown Unknown libpthread-2.17.s 00002B8B5A5FE5D0 Unknown Unknown Unknown libmpi_lustre.so. 00002B8B659D4FDF ADIOI_LUSTRE_Get_ Unknown Unknown libmpi_lustre.so. 00002B8B659CFFD9 ADIOI_LUSTRE_Writ Unknown Unknown libmpi.so.12.0 00002B8B59A4C15C Unknown Unknown Unknown libmpi.so.12 00002B8B59A4D1D5 PMPI_File_write_a Unknown Unknown libdarshan.so 00002B8B58F90312 MPI_File_write_at Unknown Unknown libdarshan.so 00002B8B58F7E63A Unknown Unknown Unknown libdarshan.so 00002B8B58F815B0 darshan_core_shut Unknown Unknown libdarshan.so 00002B8B58F7DFF3 MPI_Finalize Unknown Unknown libmpifort.so.12. 00002B8B592414DA pmpi_finalize__ Unknown Unknown exec.exe 00000000004490A5 Unknown Unknown Unknown exec.exe 00000000004032DE Unknown Unknown Unknown libc-2.17.so 00002B8B5AB2F3D5 __libc_start_main Unknown Unknown exec.exe 00000000004031E9 Unknown Unknown Unknown
If I configure darshan to write on a different filesystem (local /tmp of the first compute node), it works out fine, but then I'm restricted to use only one compute node, since the output dir have to be shared among all nodes (MPI tasks).
Is there a workaround for this? At the moment, my cluster only have the lustre filesystem as a shared filesystem among all compute nodes.
Best regards.
-- Abraços³, André Ramos Carneiro.
-- Abraços³, André Ramos Carneiro.
-- Abraços³, André Ramos Carneiro.
participants (3)
-
André R. Carneiro -
Carns, Philip H. -
Snyder, Shane