December 2021 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Why stuck in MPI_Finalize?
by Thomas Jahns 09 Dec '21

09 Dec '21

Hi, On 12/1/21 1:31 PM, Joachim Protze via discuss wrote: > I think, calling MPI_Comm_free from the garbage collector in random > ordering is not valid MPI code, since this function is a collective > operation. The non-normative advice to implementers reads: > > Though collective, it is anticipated that this operation will normally > be implemented to be local, though a debugging version of an MPI library > might choose to synchronize. just an additional thought: for MPI_Win_free, ordering also matters, _and_ it must be called for each and every corresponding window creation before calling MPI_Finalize. Regards, Thomas

1 0

Re: [mpich-discuss] Error in mutex destruction at the end of MPI Program
by Protze, Joachim 08 Dec '21

08 Dec '21

OpenMP has no explicit finalization call. The runtime detects library destruction and performs cleanup code. Such cleanup code includes finalization of offloading devices. If your openmp library supports omp_pause_resource_all (https://www.openmp.org/spec-html/5.0/openmpsu153.html#x190-9040003.2.44) you should call this function before the actual mpi_finalize() -Joachim ________________________________ From: Raffenetti, Ken via discuss <discuss(a)mpich.org> Sent: Wednesday, December 8, 2021 7:02:54 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Raffenetti, Ken <raffenet(a)anl.gov> Subject: Re: [mpich-discuss] Error in mutex destruction at the end of MPI Program We do not see this issue in our regular multithreaded testing. Is it possible the other thread is using MPI while it is being finalized? I imagine that could lead to an error when destroying the mutex. It would also explain the nondeterministic nature of the error. Ken On 12/8/21, 7:26 AM, "Pedro Henrique Di Francia Rosso via discuss" <discuss(a)mpich.org> wrote: Hello there, I'm Pedro, and I work in a research group that researches the use of OpenMP in distributed systems using MPI as a communication layer. In particular, we are working with multithreaded MPICH, where there are two main "users" of MPI in our system, an Event System and a Fault Tolerance (FT) system that work together in separate threads. Those systems carry mainly MPI asynchronous messages where the requests were freed or tested until the completion. Everything works fine and correctly. Recently, sometimes we are getting an assert error in the program ending when calling MPI_Finalize(), here is the error with some callstack: (An important note: this error does not always happen. In fact, it is much more common for the application to finish correctly, instead of asserting like that) Error in system call pthread_mutex_destroy: Device or resource busy src/mpi/init/mutex.c:90 Assertion failed in file src/mpi/init/mutex.c at line 91: err == 0 /usr/local/mpi/lib/libmpi.so.12(MPL_backtrace_show+0x35) [0x7f3429505673] /usr/local/mpi/lib/libmpi.so.12(+0x3248b4) [0x7f34294a38b4] /usr/local/mpi/lib/libmpi.so.12(MPI_Finalize+0xb8) [0x7f34293a4b28] /builds/ompcluster/llvm-project/build/projects/openmp/libomptarget/libomptarget.rtl.mpi.so <http://libomptarget.rtl.mpi.so>(+0xafa5) [0x7f342b800fa5] /lib/x86_64-linux-gnu/libc.so.6(+0x43161) [0x7f342a849161] /lib/x86_64-linux-gnu/libc.so.6(+0x4325a) [0x7f342a84925a] /builds/ompcluster/llvm-project/build/projects/openmp/libomptarget/libomptarget.rtl.mpi.so <http://libomptarget.rtl.mpi.so>(+0xf3d9) [0x7f342b8053d9] /builds/ompcluster/llvm-project/build/projects/openmp/libomptarget/libomptarget.so.12(__tgt_register_lib+0xf9) [0x7f342b8673c9] ./ompcluster/main() [0x40ef3d] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x78) [0x7f342a827b88] ./ompcluster/main() [0x402a5a] Abort(1) on node 2: Internal error I've looked at the mutex.c file and saw that this is a problem when destroying the global mutex employed in the multithread MPI. I would like to ask if there are any known scenarios, or common reasons for this problem to occur, to help me find what could be happening at the end of the execution. Here is the MPICH configuration in our container: $ mpichversion MPICH Version: 3.4.2 MPICH Release date: Wed May 26 15:51:40 CDT 2021 MPICH Device: ch4:ucx MPICH configure: --prefix=/usr/local/mpi --disable-static --with-device=ch4:ucx --with-ucx=/usr/local/ucx MPICH CC: gcc -O2 MPICH CXX: g++ -O2 MPICH F77: gfortran -O2 MPICH FC: gfortran -O2 MPICH Custom Information: Regards, Pedro _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Error in mutex destruction at the end of MPI Program
by Raffenetti, Ken 08 Dec '21

08 Dec '21

We do not see this issue in our regular multithreaded testing. Is it possible the other thread is using MPI while it is being finalized? I imagine that could lead to an error when destroying the mutex. It would also explain the nondeterministic nature of the error. Ken On 12/8/21, 7:26 AM, "Pedro Henrique Di Francia Rosso via discuss" <discuss(a)mpich.org> wrote: Hello there, I'm Pedro, and I work in a research group that researches the use of OpenMP in distributed systems using MPI as a communication layer. In particular, we are working with multithreaded MPICH, where there are two main "users" of MPI in our system, an Event System and a Fault Tolerance (FT) system that work together in separate threads. Those systems carry mainly MPI asynchronous messages where the requests were freed or tested until the completion. Everything works fine and correctly. Recently, sometimes we are getting an assert error in the program ending when calling MPI_Finalize(), here is the error with some callstack: (An important note: this error does not always happen. In fact, it is much more common for the application to finish correctly, instead of asserting like that) Error in system call pthread_mutex_destroy: Device or resource busy src/mpi/init/mutex.c:90 Assertion failed in file src/mpi/init/mutex.c at line 91: err == 0 /usr/local/mpi/lib/libmpi.so.12(MPL_backtrace_show+0x35) [0x7f3429505673] /usr/local/mpi/lib/libmpi.so.12(+0x3248b4) [0x7f34294a38b4] /usr/local/mpi/lib/libmpi.so.12(MPI_Finalize+0xb8) [0x7f34293a4b28] /builds/ompcluster/llvm-project/build/projects/openmp/libomptarget/libomptarget.rtl.mpi.so <http://libomptarget.rtl.mpi.so>(+0xafa5) [0x7f342b800fa5] /lib/x86_64-linux-gnu/libc.so.6(+0x43161) [0x7f342a849161] /lib/x86_64-linux-gnu/libc.so.6(+0x4325a) [0x7f342a84925a] /builds/ompcluster/llvm-project/build/projects/openmp/libomptarget/libomptarget.rtl.mpi.so <http://libomptarget.rtl.mpi.so>(+0xf3d9) [0x7f342b8053d9] /builds/ompcluster/llvm-project/build/projects/openmp/libomptarget/libomptarget.so.12(__tgt_register_lib+0xf9) [0x7f342b8673c9] ./ompcluster/main() [0x40ef3d] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x78) [0x7f342a827b88] ./ompcluster/main() [0x402a5a] Abort(1) on node 2: Internal error I've looked at the mutex.c file and saw that this is a problem when destroying the global mutex employed in the multithread MPI. I would like to ask if there are any known scenarios, or common reasons for this problem to occur, to help me find what could be happening at the end of the execution. Here is the MPICH configuration in our container: $ mpichversion MPICH Version: 3.4.2 MPICH Release date: Wed May 26 15:51:40 CDT 2021 MPICH Device: ch4:ucx MPICH configure: --prefix=/usr/local/mpi --disable-static --with-device=ch4:ucx --with-ucx=/usr/local/ucx MPICH CC: gcc -O2 MPICH CXX: g++ -O2 MPICH F77: gfortran -O2 MPICH FC: gfortran -O2 MPICH Custom Information: Regards, Pedro

1 0

Error in mutex destruction at the end of MPI Program
by Pedro Henrique Di Francia Rosso 08 Dec '21

08 Dec '21

Hello there, I'm Pedro, and I work in a research group that researches the use of OpenMP in distributed systems using MPI as a communication layer. In particular, we are working with multithreaded MPICH, where there are two main "users" of MPI in our system, an Event System and a Fault Tolerance (FT) system that work together in separate threads. Those systems carry mainly MPI asynchronous messages where the requests were freed or tested until the completion. Everything works fine and correctly. Recently, sometimes we are getting an assert error in the program ending when calling MPI_Finalize(), here is the error with some callstack: (*An important note: this error does not always happen. In fact, it is much more common for the application to finish correctly, instead of asserting like that*) *Error in system call pthread_mutex_destroy: Device or resource busy src/mpi/init/mutex.c:90Assertion failed in file src/mpi/init/mutex.c at line 91: err == 0/usr/local/mpi/lib/libmpi.so.12(MPL_backtrace_show+0x35) [0x7f3429505673]/usr/local/mpi/lib/libmpi.so.12(+0x3248b4) [0x7f34294a38b4]/usr/local/mpi/lib/libmpi.so.12(MPI_Finalize+0xb8) [0x7f34293a4b28]/builds/ompcluster/llvm-project/build/projects/openmp/libomptarget/libomptarget.rtl.mpi.so <http://libomptarget.rtl.mpi.so>(+0xafa5) [0x7f342b800fa5]/lib/x86_64-linux-gnu/libc.so.6(+0x43161) [0x7f342a849161]/lib/x86_64-linux-gnu/libc.so.6(+0x4325a) [0x7f342a84925a]/builds/ompcluster/llvm-project/build/projects/openmp/libomptarget/libomptarget.rtl.mpi.so <http://libomptarget.rtl.mpi.so>(+0xf3d9) [0x7f342b8053d9]/builds/ompcluster/llvm-project/build/projects/openmp/libomptarget/libomptarget.so.12(__tgt_register_lib+0xf9) [0x7f342b8673c9]./ompcluster/main() [0x40ef3d]/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x78) [0x7f342a827b88]./ompcluster/main() [0x402a5a]Abort(1) on node 2: Internal error* I've looked at the mutex.c file and saw that this is a problem when destroying the global mutex employed in the multithread MPI. I would like to ask if there are any known scenarios, or common reasons for this problem to occur, to help me find what could be happening at the end of the execution. Here is the MPICH configuration in our container: *$ mpichversion* *MPICH Version: 3.4.2MPICH Release date: Wed May 26 15:51:40 CDT 2021MPICH Device: ch4:ucxMPICH configure: --prefix=/usr/local/mpi --disable-static --with-device=ch4:ucx --with-ucx=/usr/local/ucxMPICH CC: gcc -O2MPICH CXX: g++ -O2MPICH F77: gfortran -O2MPICH FC: gfortran -O2MPICH Custom Information: * Regards, Pedro

1 0

Re: [mpich-discuss] How locking on multi-VCI works
by Zhou, Hui 06 Dec '21

06 Dec '21

Guilherme, With mpich-4.0b1, you can try building mpich using --with-ch4-vci-method=implicit. In addition, try pass "mpi_assert_no_any_tag" info hint during comm creation. That will allow mpich to distribute operations with different tags even within the same communicator to different VCIs, again, in a hashing fashion. This feature is in the experimental stage, please file GitHub issues if you encounter problems. It is also possible to explicitly specify which VCI you would like to use for arbitrary individual pt2pt communication by embedding the vci information in the tag. This feature is not exactly standard-conforming, thus we are only using it for experimental purposes. You'll need to edit src/mpid/ch4/src/ch4_vci.h and remove the line "#error MPICH_VCI__TAG not implemented." to enable it. With the tag method, tags are restricted to 15bits, the highest 5 bits designate source vci, the middle 5 bits designate destination vci, and the last 5 bits for user-defined purposes. The drawback, of course, is you will have much-reduced tag space for application logic. Both the communicator, implicit, or tag vci-methods are MPI standard conformant, so you can use it with any MPI implementations without breaking your applications. But the actual performance of course will depend on the implementation. -- Hui Zhou ________________________________ From: Guilherme Valarini <guilherme.a.valarini(a)gmail.com> Sent: Monday, December 6, 2021 12:16 PM To: Zhou, Hui <zhouh(a)anl.gov> Cc: discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] How locking on multi-VCI works Hello Zhou, First of all, thanks for the help! Let me explain a little bit my use case: my team and I have a distributed and multithreaded event system implemented on top of MPI, where multiple non-blocking MPI messages are exchanged between multiple nodes. Checking our internal performance traces, we saw that there was some contention happening at the MPI layer, especially when many threads were being used. Digging a little bit deeper we found some studies explaining the current multithread support of many MPI implementations and even limitations regarding the standard itself, which might explain the problems we encountered. Since each event is mapped to a unique TAG, that would be the preferred mechanism of extracting network parallelism. But since we also want to support other MPI implementations (e.g. openmpi), we think that using multiple communicators might be a better option. I was hoping that messages sent to two different processes but from the same process through the same VCI would use two different locks at their origin. But now I see that a VCI is directly mapped to a hardware context of some sort. So it makes sense that the same lock would be shared between the two previously described messages. If you have any other general hints on how to better extract network parallelism at the MPI level, I would be grateful. 😉 Note: Sorry for the duplicate. I forgot to reply to the mailing list as well. Thaks again for the help, Guilherme Valarini Em seg., 6 de dez. de 2021 às 14:34, Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> escreveu: The total number of VCIs are configured with --with-ch4-max-vcis=#. The maximum is 64. The default used to be 1, but it is changed to 64 in 4.0b1 release. There is also an option to control the vci assignment method: --enable-ch4-vci-method={communicator,tag,implicit}. The default is communicator, with which we assign vci to communicators in a round-robin fashion. If you create communicators consecutively, they are expected to have different VCIs. The other vci-methods are at the experimental stage. If the communicator method is insufficient for your application, it may be worth a try. We'd like to understand your use case better before pointing you that way. I am not exactly understanding your question. The vci locks are local process locks, so if you have N VCIs, you will have N channels for each process. With vci-method=communicator, the vcis are one-to-one matched, i.e. rank 1 vci 1 only communicates to rank 2 (any ranks with the same communicator) vci 1. -- Hui Zhou ________________________________ From: Guilherme Valarini via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Monday, December 6, 2021 10:10 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Guilherme Valarini <guilherme.a.valarini(a)gmail.com<mailto:[email protected]>> Subject: [mpich-discuss] How locking on multi-VCI works Hello everyone, I got one question regarding the multi-VCI support and possible locking contentions of MPICH on multi-threaded environments. I understand that there is a direct mapping between a VCI and a communicator, so global locking is avoided on a multi-threaded application. But I wanted to know: how do these VCIs work? When I have N VCIs, do I have N virtual channels per rank (thus, one global lock per VCI-rank pair) or only 2 channels at all (one lock per VCI)? I was wondering if, for example, two MPI_Sends targeting different ranks on the same comm might need to be synchronized using such a global lock. Thanks for the help! Guilherme Valarini

1 0

Re: [mpich-discuss] How locking on multi-VCI works
by Guilherme Valarini 06 Dec '21

06 Dec '21

Hello Zhou, First of all, thanks for the help! Let me explain a little bit my use case: my team and I have a distributed and multithreaded event system implemented on top of MPI, where multiple non-blocking MPI messages are exchanged between multiple nodes. Checking our internal performance traces, we saw that there was some contention happening at the MPI layer, especially when many threads were being used. Digging a little bit deeper we found some studies explaining the current multithread support of many MPI implementations and even limitations regarding the standard itself, which might explain the problems we encountered. Since each event is mapped to a unique TAG, that would be the preferred mechanism of extracting network parallelism. But since we also want to support other MPI implementations (e.g. openmpi), we think that using multiple communicators might be a better option. I was hoping that messages sent to two different processes but from the same process through the same VCI would use two different locks at their origin. But now I see that a VCI is directly mapped to a hardware context of some sort. So it makes sense that the same lock would be shared between the two previously described messages. If you have any other general hints on how to better extract network parallelism at the MPI level, I would be grateful. 😉 Note: Sorry for the duplicate. I forgot to reply to the mailing list as well. Thaks again for the help, Guilherme Valarini Em seg., 6 de dez. de 2021 às 14:34, Zhou, Hui <zhouh(a)anl.gov> escreveu: > The total number of VCIs are configured with --with-ch4-max-vcis=#. The > maximum is 64. The default used to be 1, but it is changed to 64 in 4.0b1 > release. There is also an option to control the vci assignment method: > --enable-ch4-vci-method={communicator,tag,implicit}. The default is > communicator, with which we assign vci to communicators in a round-robin > fashion. If you create communicators consecutively, they are expected to > have different VCIs. The other vci-methods are at the experimental > stage. If the communicator method is insufficient for your application, it > may be worth a try. We'd like to understand your use case better before > pointing you that way. > > I am not exactly understanding your question. The vci locks are local > process locks, so if you have N VCIs, you will have N channels for each > process. With vci-method=communicator, the vcis are one-to-one matched, > i.e. rank 1 vci 1 only communicates to rank 2 (any ranks with the same > communicator) vci 1. > > -- > Hui Zhou > ------------------------------ > *From:* Guilherme Valarini via discuss <discuss(a)mpich.org> > *Sent:* Monday, December 6, 2021 10:10 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Guilherme Valarini <guilherme.a.valarini(a)gmail.com> > *Subject:* [mpich-discuss] How locking on multi-VCI works > > Hello everyone, > > I got one question regarding the multi-VCI support and possible locking > contentions of MPICH on multi-threaded environments. > > I understand that there is a direct mapping between a VCI and a > communicator, so global locking is avoided on a multi-threaded application. > But I wanted to know: how do these VCIs work? When I have N VCIs, do I have > N virtual channels per rank (thus, one global lock per VCI-rank pair) or > only 2 channels at all (one lock per VCI)? I was wondering if, for example, > two MPI_Sends targeting different ranks on the same comm might need to be > synchronized using such a global lock. > > Thanks for the help! > Guilherme Valarini >

1 0

Re: [mpich-discuss] How locking on multi-VCI works
by Zhou, Hui 06 Dec '21

06 Dec '21

The total number of VCIs are configured with --with-ch4-max-vcis=#. The maximum is 64. The default used to be 1, but it is changed to 64 in 4.0b1 release. There is also an option to control the vci assignment method: --enable-ch4-vci-method={communicator,tag,implicit}. The default is communicator, with which we assign vci to communicators in a round-robin fashion. If you create communicators consecutively, they are expected to have different VCIs. The other vci-methods are at the experimental stage. If the communicator method is insufficient for your application, it may be worth a try. We'd like to understand your use case better before pointing you that way. I am not exactly understanding your question. The vci locks are local process locks, so if you have N VCIs, you will have N channels for each process. With vci-method=communicator, the vcis are one-to-one matched, i.e. rank 1 vci 1 only communicates to rank 2 (any ranks with the same communicator) vci 1. -- Hui Zhou ________________________________ From: Guilherme Valarini via discuss <discuss(a)mpich.org> Sent: Monday, December 6, 2021 10:10 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Guilherme Valarini <guilherme.a.valarini(a)gmail.com> Subject: [mpich-discuss] How locking on multi-VCI works Hello everyone, I got one question regarding the multi-VCI support and possible locking contentions of MPICH on multi-threaded environments. I understand that there is a direct mapping between a VCI and a communicator, so global locking is avoided on a multi-threaded application. But I wanted to know: how do these VCIs work? When I have N VCIs, do I have N virtual channels per rank (thus, one global lock per VCI-rank pair) or only 2 channels at all (one lock per VCI)? I was wondering if, for example, two MPI_Sends targeting different ranks on the same comm might need to be synchronized using such a global lock. Thanks for the help! Guilherme Valarini

1 0

How locking on multi-VCI works
by Guilherme Valarini 06 Dec '21

06 Dec '21

Hello everyone, I got one question regarding the multi-VCI support and possible locking contentions of MPICH on multi-threaded environments. I understand that there is a direct mapping between a VCI and a communicator, so global locking is avoided on a multi-threaded application. But I wanted to know: how do these VCIs work? When I have N VCIs, do I have N virtual channels per rank (thus, one global lock per VCI-rank pair) or only 2 channels at all (one lock per VCI)? I was wondering if, for example, two MPI_Sends targeting different ranks on the same comm might need to be synchronized using such a global lock. Thanks for the help! Guilherme Valarini

1 0

Re: [mpich-discuss] How to configure mpich without X?
by John Cary 04 Dec '21

04 Dec '21

That worked. Thanks! Another question coming....John Cary On 12/4/21 3:39 AM, Brice Goglin wrote: > These likely come from hwloc. If only -XNVCtrl -Xext -lX11 matter > here, passing --disable-gl to the hwloc configure should work. If > hwloc is built inside mpich, try passing --disable-gl to mpich's > configure in case it passes this option back to hwloc. > > Brice > > > > > Le 03/12/2021 à 23:43, John Cary via discuss a écrit : >> This is bringing in a shared library that is causing havoc. No >> matter how I configure, I get >> >> $ grep -i wrapper_libs= config.log >> WRAPPER_LIBS='-lm -ludev -L/usr/lib64 -lpciaccess -lXNVCtrl -lXext >> -lX11 -L/usr/lib64 -lxml2 -lpthread -lrt ' >> >> I have tried >> >> --with-x=no >> --without-x >> >> Possible? >> >> Thx...John Cary >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] How to configure mpich without X?
by Brice Goglin 04 Dec '21

04 Dec '21

These likely come from hwloc. If only -XNVCtrl -Xext -lX11 matter here, passing --disable-gl to the hwloc configure should work. If hwloc is built inside mpich, try passing --disable-gl to mpich's configure in case it passes this option back to hwloc. Brice Le 03/12/2021 à 23:43, John Cary via discuss a écrit : > This is bringing in a shared library that is causing havoc. No matter > how I configure, I get > > $ grep -i wrapper_libs= config.log > WRAPPER_LIBS='-lm -ludev -L/usr/lib64 -lpciaccess -lXNVCtrl -lXext > -lX11 -L/usr/lib64 -lxml2 -lpthread -lrt ' > > I have tried > > --with-x=no > --without-x > > Possible? > > Thx...John Cary > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0