December 2021 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] How to configure mpich without X?
by John Cary 03 Dec '21

03 Dec '21

This is bringing in a shared library that is causing havoc. No matter how I configure, I get $ grep -i wrapper_libs= config.log WRAPPER_LIBS='-lm -ludev -L/usr/lib64 -lpciaccess -lXNVCtrl -lXext -lX11 -L/usr/lib64 -lxml2 -lpthread -lrt ' I have tried --with-x=no --without-x Possible? Thx...John Cary

1 0

Re: [mpich-discuss] Why stuck in MPI_Finalize?
by Erik Schnetter 01 Dec '21

01 Dec '21

On Wed, Dec 1, 2021 at 7:42 AM Joachim Protze via discuss <discuss(a)mpich.org> wrote: > > Hi Erik, > > I think, calling MPI_Comm_free from the garbage collector in random > ordering is not valid MPI code, since this function is a collective > operation. Thank you for pointing this out. This might be the culprit. I'll have to re-think the code. -erik > The non-normative advice to implementers reads: > > Though collective, it is anticipated that this operation will normally > be implemented to be local, though a debugging version of an MPI library > might choose to synchronize. > > Best > Joachim > > Am 01.12.21 um 09:33 schrieb Protze, Joachim via discuss: > > Hi Erik, > > > > You can try and run your application with MUST: > > https://itc.rwth-aachen.de/must/ <https://itc.rwth-aachen.de/must/> > > > > The report should tell you what cleanup is missing and most probably > > also why your application deadlocks in finalize. To get source line > > information, you should build your code (and Julia runtime?) with debug > > flag. Also check the section about backward in the documentation of MUST. > > > > While the most recent version was not tested on MacOS yet, it should > > work on both Linux and MacOS. It might be interesting to see whether > > analysis results differ between the different os > > > > Best > > Joachim > > ------------------------------------------------------------------------ > > *From:* Erik Schnetter via discuss <discuss(a)mpich.org> > > *Sent:* Tuesday, November 30, 2021 4:54:28 PM > > *To:* Zhou, Hui <zhouh(a)anl.gov> > > *Cc:* Erik Schnetter <schnetter(a)gmail.com>; discuss(a)mpich.org > > <discuss(a)mpich.org> > > *Subject:* Re: [mpich-discuss] Why stuck in MPI_Finalize? > > Hui > > > > A garbage collector can't reliably reclaim resources. When > > MPI_Finalize is called by the garbage collector, then there's a chance > > it won't be run (e.g. if things live in global variables), and then > > the application won't shut down completely. Therefore, MPI_Finalize is > > called explicitly, while e.g. MPI_Comm_free usually would be called > > automatically. > > > > Is there a way to see what is happening during MPI_Finalize, maybe > > some kind of `verbose` setting? > > > > -erik > > > > On Mon, Nov 29, 2021 at 11:18 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > >> > >> Hi Erik, > >> > >> Shouldn't MPI_Finalize run by the garbage collector as well by treating the entire MPI as an object? > >> > >> -- > >> Hui Zhou > >> ________________________________ > >> From: Erik Schnetter <schnetter(a)gmail.com> > >> Sent: Monday, November 29, 2021 7:23 PM > >> To: Zhou, Hui <zhouh(a)anl.gov> > >> Cc: discuss(a)mpich.org <discuss(a)mpich.org> > >> Subject: Re: [mpich-discuss] Why stuck in MPI_Finalize? > >> > >> Hui > >> > >> Julia is garbage collected, and our MPI wrappers free MPI handles > >> automatically when the wrapped handles become garbage collected. I > >> thus don't have direct control over when and which handles are freed > >> before MPI_Finalize is called at the end of the test case. Our > >> approach is to let things be as they are, ensuring that no MPI calls > >> are made any more when MPI_Finalized returns true. That means that > >> different MPI processes might (or might not ) free handles before > >> calling MPI_Finalize. I notice that this seems to work fine on Linux, > >> but not always on macOS. I don't have a simple reproducer in C. > >> > >> Pointers to setting up Julia: See e.g. > >> <https://github.com/JuliaParallel/MPI.jl/blob/master/.github/workflows/UnitT… > > <https://github.com/JuliaParallel/MPI.jl/blob/master/.github/workflows/UnitT…>> > >> which runs the unit tests for the Julia package `MPI.jl` which wraps > >> MPI implementations. There are several stanzas; the one called > >> `test-system-brew` runs MPI.jl on a system-provided (external MPI) > >> installation that has been installed via Homebrew on macOS. You would > >> replace the step "Install MPI via homebrew" by a step that downloads > >> and installs MPICH into /usr/local. I'll be happy to discuss details, > >> possibly over Zoom. > >> > >> -erik > >> > >> > >> On Mon, Nov 29, 2021 at 5:12 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > >> > > >> > Hi Erik, > >> > > >> > It is not just a barrier in MPI_Finalize. It also needs to make sure pending communications are completed. Sometimes this is the user's fault, such as mismatched communication or partially freed communicators. But sometimes this is due to lower-level network library. For example, some libfabric provider does not always flush > > send. We tried to detect and prevent all cases that result in hanging, > > but there always seem to be cases that escape our solution. If you can > > drill down some simple reproducible (even with 5% chance) cases, please > > create a github issue and we'll track it down. > >> > > >> > By the way, if you have pointers on setting up Julia CI testing, we can try to setup nightly testing on our end as well. Catching the errors when it appear always makes troubleshooting easier. > >> > > >> > -- > >> > Hui > >> > ________________________________ > >> > From: Erik Schnetter via discuss <discuss(a)mpich.org> > >> > Sent: Monday, November 29, 2021 12:30 PM > >> > To: discuss(a)mpich.org <discuss(a)mpich.org> > >> > Cc: Erik Schnetter <schnetter(a)gmail.com> > >> > Subject: [mpich-discuss] Why stuck in MPI_Finalize? > >> > > >> > I have a Julia test case on macOS where MPICH randomly gets stuck in > >> > MPI_Finalize (with about a 5% chance). See e.g. > >> > https://github.com/JuliaParallel/MPI.jl/runs/4357341818 > > <https://github.com/JuliaParallel/MPI.jl/runs/4357341818> > >> > > >> > Can you advise under what circumstances MPICH could get stuck there? > >> > The respective run uses 3 processes, and all 3 processes call into > >> > MPI_Finalize, but no process returns. > >> > > >> > I assume that MPI_Finalize contains internally the equivalent to an > >> > MPI_Barrier, but that should succeed here. Are there other actions > >> > taken in MPI_Finalize that would require some kind of consistent state > >> > across the application? For example, if a communicator was created on > >> > all processes, but freed only on some processes, could this cause such > >> > a deadlock? > >> > > >> > -erik > >> > > >> > -- > >> > Erik Schnetter <schnetter(a)gmail.com> > >> > http://www.perimeterinstitute.ca/personal/eschnetter/ > > <http://www.perimeterinstitute.ca/personal/eschnetter/> > >> > _______________________________________________ > >> > discuss mailing list discuss(a)mpich.org > >> > To manage subscription options or unsubscribe: > >> > https://lists.mpich.org/mailman/listinfo/discuss > > <https://lists.mpich.org/mailman/listinfo/discuss> > >> > >> > >> > >> -- > >> Erik Schnetter <schnetter(a)gmail.com> > >> http://www.perimeterinstitute.ca/personal/eschnetter/ > > <http://www.perimeterinstitute.ca/personal/eschnetter/> > > > > > > > > -- > > Erik Schnetter <schnetter(a)gmail.com> > > http://www.perimeterinstitute.ca/personal/eschnetter/ > > <http://www.perimeterinstitute.ca/personal/eschnetter/> > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > <https://lists.mpich.org/mailman/listinfo/discuss> > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > Dr. rer. nat. Joachim Protze > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering > RWTH Aachen University > Seffenter Weg 23 > D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > protze(a)itc.rwth-aachen.de > www.itc.rwth-aachen.de > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Erik Schnetter <schnetter(a)gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/

1 0

Re: [mpich-discuss] Why stuck in MPI_Finalize?
by Joachim Protze 01 Dec '21

01 Dec '21

Hi Erik, I think, calling MPI_Comm_free from the garbage collector in random ordering is not valid MPI code, since this function is a collective operation. The non-normative advice to implementers reads: Though collective, it is anticipated that this operation will normally be implemented to be local, though a debugging version of an MPI library might choose to synchronize. Best Joachim Am 01.12.21 um 09:33 schrieb Protze, Joachim via discuss: > Hi Erik, > > You can try and run your application with MUST: > https://itc.rwth-aachen.de/must/ <https://itc.rwth-aachen.de/must/> > > The report should tell you what cleanup is missing and most probably > also why your application deadlocks in finalize. To get source line > information, you should build your code (and Julia runtime?) with debug > flag. Also check the section about backward in the documentation of MUST. > > While the most recent version was not tested on MacOS yet, it should > work on both Linux and MacOS. It might be interesting to see whether > analysis results differ between the different os > > Best > Joachim > ------------------------------------------------------------------------ > *From:* Erik Schnetter via discuss <discuss(a)mpich.org> > *Sent:* Tuesday, November 30, 2021 4:54:28 PM > *To:* Zhou, Hui <zhouh(a)anl.gov> > *Cc:* Erik Schnetter <schnetter(a)gmail.com>; discuss(a)mpich.org > <discuss(a)mpich.org> > *Subject:* Re: [mpich-discuss] Why stuck in MPI_Finalize? > Hui > > A garbage collector can't reliably reclaim resources. When > MPI_Finalize is called by the garbage collector, then there's a chance > it won't be run (e.g. if things live in global variables), and then > the application won't shut down completely. Therefore, MPI_Finalize is > called explicitly, while e.g. MPI_Comm_free usually would be called > automatically. > > Is there a way to see what is happening during MPI_Finalize, maybe > some kind of `verbose` setting? > > -erik > > On Mon, Nov 29, 2021 at 11:18 PM Zhou, Hui <zhouh(a)anl.gov> wrote: >> >> Hi Erik, >> >> Shouldn't MPI_Finalize run by the garbage collector as well by treating the entire MPI as an object? >> >> -- >> Hui Zhou >> ________________________________ >> From: Erik Schnetter <schnetter(a)gmail.com> >> Sent: Monday, November 29, 2021 7:23 PM >> To: Zhou, Hui <zhouh(a)anl.gov> >> Cc: discuss(a)mpich.org <discuss(a)mpich.org> >> Subject: Re: [mpich-discuss] Why stuck in MPI_Finalize? >> >> Hui >> >> Julia is garbage collected, and our MPI wrappers free MPI handles >> automatically when the wrapped handles become garbage collected. I >> thus don't have direct control over when and which handles are freed >> before MPI_Finalize is called at the end of the test case. Our >> approach is to let things be as they are, ensuring that no MPI calls >> are made any more when MPI_Finalized returns true. That means that >> different MPI processes might (or might not ) free handles before >> calling MPI_Finalize. I notice that this seems to work fine on Linux, >> but not always on macOS. I don't have a simple reproducer in C. >> >> Pointers to setting up Julia: See e.g. >> <https://github.com/JuliaParallel/MPI.jl/blob/master/.github/workflows/UnitT… > <https://github.com/JuliaParallel/MPI.jl/blob/master/.github/workflows/UnitT…>> >> which runs the unit tests for the Julia package `MPI.jl` which wraps >> MPI implementations. There are several stanzas; the one called >> `test-system-brew` runs MPI.jl on a system-provided (external MPI) >> installation that has been installed via Homebrew on macOS. You would >> replace the step "Install MPI via homebrew" by a step that downloads >> and installs MPICH into /usr/local. I'll be happy to discuss details, >> possibly over Zoom. >> >> -erik >> >> >> On Mon, Nov 29, 2021 at 5:12 PM Zhou, Hui <zhouh(a)anl.gov> wrote: >> > >> > Hi Erik, >> > >> > It is not just a barrier in MPI_Finalize. It also needs to make sure pending communications are completed. Sometimes this is the user's fault, such as mismatched communication or partially freed communicators. But sometimes this is due to lower-level network library. For example, some libfabric provider does not always flush > send. We tried to detect and prevent all cases that result in hanging, > but there always seem to be cases that escape our solution. If you can > drill down some simple reproducible (even with 5% chance) cases, please > create a github issue and we'll track it down. >> > >> > By the way, if you have pointers on setting up Julia CI testing, we can try to setup nightly testing on our end as well. Catching the errors when it appear always makes troubleshooting easier. >> > >> > -- >> > Hui >> > ________________________________ >> > From: Erik Schnetter via discuss <discuss(a)mpich.org> >> > Sent: Monday, November 29, 2021 12:30 PM >> > To: discuss(a)mpich.org <discuss(a)mpich.org> >> > Cc: Erik Schnetter <schnetter(a)gmail.com> >> > Subject: [mpich-discuss] Why stuck in MPI_Finalize? >> > >> > I have a Julia test case on macOS where MPICH randomly gets stuck in >> > MPI_Finalize (with about a 5% chance). See e.g. >> > https://github.com/JuliaParallel/MPI.jl/runs/4357341818 > <https://github.com/JuliaParallel/MPI.jl/runs/4357341818> >> > >> > Can you advise under what circumstances MPICH could get stuck there? >> > The respective run uses 3 processes, and all 3 processes call into >> > MPI_Finalize, but no process returns. >> > >> > I assume that MPI_Finalize contains internally the equivalent to an >> > MPI_Barrier, but that should succeed here. Are there other actions >> > taken in MPI_Finalize that would require some kind of consistent state >> > across the application? For example, if a communicator was created on >> > all processes, but freed only on some processes, could this cause such >> > a deadlock? >> > >> > -erik >> > >> > -- >> > Erik Schnetter <schnetter(a)gmail.com> >> > http://www.perimeterinstitute.ca/personal/eschnetter/ > <http://www.perimeterinstitute.ca/personal/eschnetter/> >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> >> >> >> >> -- >> Erik Schnetter <schnetter(a)gmail.com> >> http://www.perimeterinstitute.ca/personal/eschnetter/ > <http://www.perimeterinstitute.ca/personal/eschnetter/> > > > > -- > Erik Schnetter <schnetter(a)gmail.com> > http://www.perimeterinstitute.ca/personal/eschnetter/ > <http://www.perimeterinstitute.ca/personal/eschnetter/> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dr. rer. nat. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Why stuck in MPI_Finalize?
by Protze, Joachim 01 Dec '21

01 Dec '21

Hi Erik, You can try and run your application with MUST: https://itc.rwth-aachen.de/must/ The report should tell you what cleanup is missing and most probably also why your application deadlocks in finalize. To get source line information, you should build your code (and Julia runtime?) with debug flag. Also check the section about backward in the documentation of MUST. While the most recent version was not tested on MacOS yet, it should work on both Linux and MacOS. It might be interesting to see whether analysis results differ between the different os Best Joachim ________________________________ From: Erik Schnetter via discuss <discuss(a)mpich.org> Sent: Tuesday, November 30, 2021 4:54:28 PM To: Zhou, Hui <zhouh(a)anl.gov> Cc: Erik Schnetter <schnetter(a)gmail.com>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Why stuck in MPI_Finalize? Hui A garbage collector can't reliably reclaim resources. When MPI_Finalize is called by the garbage collector, then there's a chance it won't be run (e.g. if things live in global variables), and then the application won't shut down completely. Therefore, MPI_Finalize is called explicitly, while e.g. MPI_Comm_free usually would be called automatically. Is there a way to see what is happening during MPI_Finalize, maybe some kind of `verbose` setting? -erik On Mon, Nov 29, 2021 at 11:18 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Hi Erik, > > Shouldn't MPI_Finalize run by the garbage collector as well by treating the entire MPI as an object? > > -- > Hui Zhou > ________________________________ > From: Erik Schnetter <schnetter(a)gmail.com> > Sent: Monday, November 29, 2021 7:23 PM > To: Zhou, Hui <zhouh(a)anl.gov> > Cc: discuss(a)mpich.org <discuss(a)mpich.org> > Subject: Re: [mpich-discuss] Why stuck in MPI_Finalize? > > Hui > > Julia is garbage collected, and our MPI wrappers free MPI handles > automatically when the wrapped handles become garbage collected. I > thus don't have direct control over when and which handles are freed > before MPI_Finalize is called at the end of the test case. Our > approach is to let things be as they are, ensuring that no MPI calls > are made any more when MPI_Finalized returns true. That means that > different MPI processes might (or might not ) free handles before > calling MPI_Finalize. I notice that this seems to work fine on Linux, > but not always on macOS. I don't have a simple reproducer in C. > > Pointers to setting up Julia: See e.g. > <https://github.com/JuliaParallel/MPI.jl/blob/master/.github/workflows/UnitT…> > which runs the unit tests for the Julia package `MPI.jl` which wraps > MPI implementations. There are several stanzas; the one called > `test-system-brew` runs MPI.jl on a system-provided (external MPI) > installation that has been installed via Homebrew on macOS. You would > replace the step "Install MPI via homebrew" by a step that downloads > and installs MPICH into /usr/local. I'll be happy to discuss details, > possibly over Zoom. > > -erik > > > On Mon, Nov 29, 2021 at 5:12 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > > > Hi Erik, > > > > It is not just a barrier in MPI_Finalize. It also needs to make sure pending communications are completed. Sometimes this is the user's fault, such as mismatched communication or partially freed communicators. But sometimes this is due to lower-level network library. For example, some libfabric provider does not always flush send. We tried to detect and prevent all cases that result in hanging, but there always seem to be cases that escape our solution. If you can drill down some simple reproducible (even with 5% chance) cases, please create a github issue and we'll track it down. > > > > By the way, if you have pointers on setting up Julia CI testing, we can try to setup nightly testing on our end as well. Catching the errors when it appear always makes troubleshooting easier. > > > > -- > > Hui > > ________________________________ > > From: Erik Schnetter via discuss <discuss(a)mpich.org> > > Sent: Monday, November 29, 2021 12:30 PM > > To: discuss(a)mpich.org <discuss(a)mpich.org> > > Cc: Erik Schnetter <schnetter(a)gmail.com> > > Subject: [mpich-discuss] Why stuck in MPI_Finalize? > > > > I have a Julia test case on macOS where MPICH randomly gets stuck in > > MPI_Finalize (with about a 5% chance). See e.g. > > https://github.com/JuliaParallel/MPI.jl/runs/4357341818 > > > > Can you advise under what circumstances MPICH could get stuck there? > > The respective run uses 3 processes, and all 3 processes call into > > MPI_Finalize, but no process returns. > > > > I assume that MPI_Finalize contains internally the equivalent to an > > MPI_Barrier, but that should succeed here. Are there other actions > > taken in MPI_Finalize that would require some kind of consistent state > > across the application? For example, if a communicator was created on > > all processes, but freed only on some processes, could this cause such > > a deadlock? > > > > -erik > > > > -- > > Erik Schnetter <schnetter(a)gmail.com> > > http://www.perimeterinstitute.ca/personal/eschnetter/ > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Erik Schnetter <schnetter(a)gmail.com> > http://www.perimeterinstitute.ca/personal/eschnetter/ -- Erik Schnetter <schnetter(a)gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0