prevent linking to multithreaded BLAS?
I ran into an unexpected issue -- on an NP-core machine, each MPI rank of my application was launching NP threads, such that when running with multiple ranks the machine was quickly oversubscribed and performance tanked. The root cause of this was petsc linking against the system-provided library (libopenblas0-pthread in this case) set by the update-alternatives in ubuntu. At some point this machine got updated to using the threaded blas implementation instead of serial; not sure how, and I wouldn't have noticed if I weren't running interactively. Is there any mechanism in petsc or its build system to prevent linking against an inappropriate BLAS, or do I need to be diligent about manually setting the BLAS library in the configuration stage? Thanks, Mark
We don't have configure code to detect if the BLAS is thread parallel, nor do we have code to tell it not to use a thread parallel version. Except if it is using MKL then we do force it to not use the threaded BLAS. A "cheat" would be for you to just set the environmental variable BLAS uses for number of threads to 1 always, then you would not need to worry about checking to avoid the "bad" library. Barry
On Dec 7, 2022, at 4:21 PM, Mark Lohry <[email protected]> wrote:
I ran into an unexpected issue -- on an NP-core machine, each MPI rank of my application was launching NP threads, such that when running with multiple ranks the machine was quickly oversubscribed and performance tanked.
The root cause of this was petsc linking against the system-provided library (libopenblas0-pthread in this case) set by the update-alternatives in ubuntu. At some point this machine got updated to using the threaded blas implementation instead of serial; not sure how, and I wouldn't have noticed if I weren't running interactively.
Is there any mechanism in petsc or its build system to prevent linking against an inappropriate BLAS, or do I need to be diligent about manually setting the BLAS library in the configuration stage?
Thanks, Mark
If you don't specify a blas to use - petsc configure will guess and use what it can find. So only way to force it use a particular blas is to specify one [one way is --download-fblaslapack] Wrt multi-thread openblas - you can force it run single threaded [by one of these 2 env variables] # Use single thread openblas export OPENBLAS_NUM_THREADS=1 export OMP_NUM_THREADS=1 Satish On Wed, 7 Dec 2022, Mark Lohry wrote:
I ran into an unexpected issue -- on an NP-core machine, each MPI rank of my application was launching NP threads, such that when running with multiple ranks the machine was quickly oversubscribed and performance tanked.
The root cause of this was petsc linking against the system-provided library (libopenblas0-pthread in this case) set by the update-alternatives in ubuntu. At some point this machine got updated to using the threaded blas implementation instead of serial; not sure how, and I wouldn't have noticed if I weren't running interactively.
Is there any mechanism in petsc or its build system to prevent linking against an inappropriate BLAS, or do I need to be diligent about manually setting the BLAS library in the configuration stage?
Thanks, Mark
Thanks, yes, I figured out the OMP_NUM_THREADS=1 way while triaging it, and the --download-fblaslapack way occurred to me. I was hoping for something that "just worked" (refuse to build in this case) but I don't know if it's programmatically possible for petsc to tell whether or not it's linking to a threaded BLAS? On Wed, Dec 7, 2022 at 4:35 PM Satish Balay <[email protected]> wrote:
If you don't specify a blas to use - petsc configure will guess and use what it can find.
So only way to force it use a particular blas is to specify one [one way is --download-fblaslapack]
Wrt multi-thread openblas - you can force it run single threaded [by one of these 2 env variables]
# Use single thread openblas export OPENBLAS_NUM_THREADS=1 export OMP_NUM_THREADS=1
Satish
On Wed, 7 Dec 2022, Mark Lohry wrote:
I ran into an unexpected issue -- on an NP-core machine, each MPI rank of my application was launching NP threads, such that when running with multiple ranks the machine was quickly oversubscribed and performance tanked.
The root cause of this was petsc linking against the system-provided library (libopenblas0-pthread in this case) set by the update-alternatives in ubuntu. At some point this machine got updated to using the threaded blas implementation instead of serial; not sure how, and I wouldn't have noticed if I weren't running interactively.
Is there any mechanism in petsc or its build system to prevent linking against an inappropriate BLAS, or do I need to be diligent about manually setting the BLAS library in the configuration stage?
Thanks, Mark
There would need to be, for example, some symbol in all the threaded BLAS libraries that is not in the unthreaded libraries. Of at least in some of the threaded libraries but never in the unthreaded. BlasLapack.py could check for the special symbol(s) to determine. Barry
On Dec 7, 2022, at 4:47 PM, Mark Lohry <[email protected]> wrote:
Thanks, yes, I figured out the OMP_NUM_THREADS=1 way while triaging it, and the --download-fblaslapack way occurred to me.
I was hoping for something that "just worked" (refuse to build in this case) but I don't know if it's programmatically possible for petsc to tell whether or not it's linking to a threaded BLAS?
On Wed, Dec 7, 2022 at 4:35 PM Satish Balay <[email protected] <mailto:[email protected]>> wrote:
If you don't specify a blas to use - petsc configure will guess and use what it can find.
So only way to force it use a particular blas is to specify one [one way is --download-fblaslapack]
Wrt multi-thread openblas - you can force it run single threaded [by one of these 2 env variables]
# Use single thread openblas export OPENBLAS_NUM_THREADS=1 export OMP_NUM_THREADS=1
Satish
On Wed, 7 Dec 2022, Mark Lohry wrote:
I ran into an unexpected issue -- on an NP-core machine, each MPI rank of my application was launching NP threads, such that when running with multiple ranks the machine was quickly oversubscribed and performance tanked.
The root cause of this was petsc linking against the system-provided library (libopenblas0-pthread in this case) set by the update-alternatives in ubuntu. At some point this machine got updated to using the threaded blas implementation instead of serial; not sure how, and I wouldn't have noticed if I weren't running interactively.
Is there any mechanism in petsc or its build system to prevent linking against an inappropriate BLAS, or do I need to be diligent about manually setting the BLAS library in the configuration stage?
Thanks, Mark
It isn't always wrong to link threaded BLAS. For example, a user might need to call threaded BLAS on the side (but the application can only link one) or a sparse direct solver might want threading for the supernode. We could test at runtime whether child threads exist/are created when calling BLAS and deliver a warning. Barry Smith <[email protected]> writes:
There would need to be, for example, some symbol in all the threaded BLAS libraries that is not in the unthreaded libraries. Of at least in some of the threaded libraries but never in the unthreaded.
BlasLapack.py could check for the special symbol(s) to determine.
Barry
On Dec 7, 2022, at 4:47 PM, Mark Lohry <[email protected]> wrote:
Thanks, yes, I figured out the OMP_NUM_THREADS=1 way while triaging it, and the --download-fblaslapack way occurred to me.
I was hoping for something that "just worked" (refuse to build in this case) but I don't know if it's programmatically possible for petsc to tell whether or not it's linking to a threaded BLAS?
On Wed, Dec 7, 2022 at 4:35 PM Satish Balay <[email protected] <mailto:[email protected]>> wrote:
If you don't specify a blas to use - petsc configure will guess and use what it can find.
So only way to force it use a particular blas is to specify one [one way is --download-fblaslapack]
Wrt multi-thread openblas - you can force it run single threaded [by one of these 2 env variables]
# Use single thread openblas export OPENBLAS_NUM_THREADS=1 export OMP_NUM_THREADS=1
Satish
On Wed, 7 Dec 2022, Mark Lohry wrote:
I ran into an unexpected issue -- on an NP-core machine, each MPI rank of my application was launching NP threads, such that when running with multiple ranks the machine was quickly oversubscribed and performance tanked.
The root cause of this was petsc linking against the system-provided library (libopenblas0-pthread in this case) set by the update-alternatives in ubuntu. At some point this machine got updated to using the threaded blas implementation instead of serial; not sure how, and I wouldn't have noticed if I weren't running interactively.
Is there any mechanism in petsc or its build system to prevent linking against an inappropriate BLAS, or do I need to be diligent about manually setting the BLAS library in the configuration stage?
Thanks, Mark
On Dec 7, 2022, at 11:56 PM, Jed Brown <[email protected]> wrote:
It isn't always wrong to link threaded BLAS. For example, a user might need to call threaded BLAS on the side (but the application can only link one) or a sparse direct solver might want threading for the supernode.
Indeed, the user asked specifically for their work flow if configure could, based on additional configure argument, ensure that they did not get a threaded BLAS; they did not ask that configure never give a threaded BLAS or even that it give a non-threaded BLAS by default.
We could test at runtime whether child threads exist/are created when calling BLAS and deliver a warning.
How does one test for this? Some standard Unix API for checking this?
Barry Smith <[email protected]> writes:
There would need to be, for example, some symbol in all the threaded BLAS libraries that is not in the unthreaded libraries. Of at least in some of the threaded libraries but never in the unthreaded.
BlasLapack.py could check for the special symbol(s) to determine.
Barry
On Dec 7, 2022, at 4:47 PM, Mark Lohry <[email protected]> wrote:
Thanks, yes, I figured out the OMP_NUM_THREADS=1 way while triaging it, and the --download-fblaslapack way occurred to me.
I was hoping for something that "just worked" (refuse to build in this case) but I don't know if it's programmatically possible for petsc to tell whether or not it's linking to a threaded BLAS?
On Wed, Dec 7, 2022 at 4:35 PM Satish Balay <[email protected] <mailto:[email protected]>> wrote:
If you don't specify a blas to use - petsc configure will guess and use what it can find.
So only way to force it use a particular blas is to specify one [one way is --download-fblaslapack]
Wrt multi-thread openblas - you can force it run single threaded [by one of these 2 env variables]
# Use single thread openblas export OPENBLAS_NUM_THREADS=1 export OMP_NUM_THREADS=1
Satish
On Wed, 7 Dec 2022, Mark Lohry wrote:
I ran into an unexpected issue -- on an NP-core machine, each MPI rank of my application was launching NP threads, such that when running with multiple ranks the machine was quickly oversubscribed and performance tanked.
The root cause of this was petsc linking against the system-provided library (libopenblas0-pthread in this case) set by the update-alternatives in ubuntu. At some point this machine got updated to using the threaded blas implementation instead of serial; not sure how, and I wouldn't have noticed if I weren't running interactively.
Is there any mechanism in petsc or its build system to prevent linking against an inappropriate BLAS, or do I need to be diligent about manually setting the BLAS library in the configuration stage?
Thanks, Mark
Barry Smith <[email protected]> writes:
We could test at runtime whether child threads exist/are created when calling BLAS and deliver a warning.
How does one test for this? Some standard Unix API for checking this?
I'm not sure, the ids of child threads are in /proc/$pid/task/ and (when opened by a process) /proc/self/task/. See man procfs for details.
participants (4)
-
Barry Smith -
Jed Brown -
Mark Lohry -
Satish Balay