- discuss - mailman3.cels.anl.gov

error building cuda from yaksa
by Allen McIntosh 02 May '22

02 May '22

Mpich version: 4.0.2 OS: Ubuntu 20.04 (but likely irrelevant) Line 45 (approximately) of mpich-4.0.2/modules/yaksa/src/backend/cuda/cudalt.sh invokes libtool without providing its own version. This results in an error if there is no system version of libtool. I got around this by changing the invocation to "./libtool", but that is pretty ugly. I have asked to have libtool-bin installed on the build system. I can post with any results. [I see that I really should have posted this to yaksa-users. I will subscribe and post there.]

1 0

Re: [mpich-discuss] CUDA support with clang instead of nvcc
by Sajid Ali 15 Apr '22

15 Apr '22

Hi Hui, I was referring to the possibility of avoiding nvcc altogether rather than setting the --ccbin (essentially eschewing nvcc to do this instead : ` https://llvm.org/docs/CompileCudaWithLLVM.html`) As Ken noted previously, it looks like building yaksa (with CUDA support) requires the use of nvcc. Thank You, Sajid Ali (he/him) | Research Associate Scientific Computing Division Fermi National Accelerator Laboratory s-sajid-ali.github.io

1 0

Re: [mpich-discuss] CUDA support with clang instead of nvcc
by Zhou, Hui 15 Apr '22

15 Apr '22

The configure in yaksa will set correct -ccbin option for nvcc to use the compiler set by CC. Reference https://github.com/pmodels/yaksa/blob/e4063c04b6ceeda7d863449a4e92124feb5b7… Ideally, it should use CXX rather than CC. But it currently works to compile the cu files in yaksa. -- Hui [https://opengraph.githubassets.com/23e05374cb64ce76ccf37f70c107678c6cd0fc7b…]<https://github.com/pmodels/yaksa/blob/e4063c04b6ceeda7d863449a4e92124feb5b7…> yaksa/subconfigure.m4 at e4063c04b6ceeda7d863449a4e92124feb5b739a · pmodels/yaksa<https://github.com/pmodels/yaksa/blob/e4063c04b6ceeda7d863449a4e92124feb5b7…> Yaksa: High-performance Noncontiguous Data Management - yaksa/subconfigure.m4 at e4063c04b6ceeda7d863449a4e92124feb5b739a · pmodels/yaksa github.com ________________________________ From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, April 15, 2022 3:02 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Zhou, Hui <zhouh(a)anl.gov> Subject: Re: [mpich-discuss] CUDA support with clang instead of nvcc The Yaksa configure is currently hardcoded to use `nvcc` for .cu files. We'll have to patch it to support alternatives, right? Ken On 4/15/22, 2:57 PM, "Zhou, Hui via discuss" <discuss(a)mpich.org> wrote: Yes, just set CC=clang and CXX=clang++. -- Hui Zhou ________________________________________ From: Sajid Ali via discuss <discuss(a)mpich.org> Sent: Friday, April 15, 2022 11:49 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Sajid Ali <sajidsyed2021(a)u.northwestern.edu> Subject: [mpich-discuss] CUDA support with clang instead of nvcc Hi MPICH-developers, Is it possible to compile MPICH with CUDA awareness, but with clang as the CUDA compiler? (i.e. use clang capable of targeting the nvptx backend to compiler cuda files instead of nvcc). Thank You, Sajid Ali (he/him) | Research Associate Scientific Computing Division Fermi National Accelerator Laboratory s-sajid-ali.github.io <http://s-sajid-ali.github.io>

1 0

Re: [mpich-discuss] CUDA support with clang instead of nvcc
by Raffenetti, Ken 15 Apr '22

15 Apr '22

The Yaksa configure is currently hardcoded to use `nvcc` for .cu files. We'll have to patch it to support alternatives, right? Ken On 4/15/22, 2:57 PM, "Zhou, Hui via discuss" <discuss(a)mpich.org> wrote: Yes, just set CC=clang and CXX=clang++. -- Hui Zhou ________________________________________ From: Sajid Ali via discuss <discuss(a)mpich.org> Sent: Friday, April 15, 2022 11:49 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Sajid Ali <sajidsyed2021(a)u.northwestern.edu> Subject: [mpich-discuss] CUDA support with clang instead of nvcc Hi MPICH-developers, Is it possible to compile MPICH with CUDA awareness, but with clang as the CUDA compiler? (i.e. use clang capable of targeting the nvptx backend to compiler cuda files instead of nvcc). Thank You, Sajid Ali (he/him) | Research Associate Scientific Computing Division Fermi National Accelerator Laboratory s-sajid-ali.github.io <http://s-sajid-ali.github.io>

1 0

Re: [mpich-discuss] CUDA support with clang instead of nvcc
by Zhou, Hui 15 Apr '22

15 Apr '22

Yes, just set CC=clang and CXX=clang++. -- Hui Zhou ________________________________ From: Sajid Ali via discuss <discuss(a)mpich.org> Sent: Friday, April 15, 2022 11:49 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Sajid Ali <sajidsyed2021(a)u.northwestern.edu> Subject: [mpich-discuss] CUDA support with clang instead of nvcc Hi MPICH-developers, Is it possible to compile MPICH with CUDA awareness, but with clang as the CUDA compiler? (i.e. use clang capable of targeting the nvptx backend to compiler cuda files instead of nvcc). Thank You, Sajid Ali (he/him) | Research Associate Scientific Computing Division Fermi National Accelerator Laboratory s-sajid-ali.github.io<http://s-sajid-ali.github.io>

1 0

CUDA support with clang instead of nvcc
by Sajid Ali 15 Apr '22

15 Apr '22

Hi MPICH-developers, Is it possible to compile MPICH with CUDA awareness, but with clang as the CUDA compiler? (i.e. use clang capable of targeting the nvptx backend to compiler cuda files instead of nvcc). Thank You, Sajid Ali (he/him) | Research Associate Scientific Computing Division Fermi National Accelerator Laboratory s-sajid-ali.github.io

1 0

Re: [mpich-discuss] MPI_Init and MPI_COMM_WORLD
by Raffenetti, Ken 13 Apr '22

13 Apr '22

Bruce, According to the standard, MPI_COMM_WORLD is a named constant handle which does not change value between MPI_INIT and MPI_FINALIZE. Therefore, I believe your example is OK from the standard perspective. In MPICH, MPI_COMM_WORLD is an integer constant defined at compile-time. I believe in Open MPI the value is defined at link-time, which should still meet the requirements of your example. Ken On 4/13/22, 12:29 PM, "Palmer, Bruce J via discuss" <discuss(a)mpich.org> wrote: Hi, Does MPI_COMM_WORLD exist in any sense before MPI_Init is called? In particular, does this code violate the standard? InitializeMyApp(MPI_Comm comm, int argc, char **argv) { int initialized; MPI_Initialized(&initialized); if (!initialized) MPI_Init(&argc, &argv); /* call some function that uses comm */ } int main(int argc, char **argv) { MPI_Comm comm = MPI_COMM_WORLD; InitializeMyApp(comm, argc, argv); : } My instinct is that this is wrong, but I would like to be sure before I complain about it. It appears to work with some existing implementations, but it looks sketchy to me. Bruce

1 0

MPI_Init and MPI_COMM_WORLD
by Palmer, Bruce J 13 Apr '22

13 Apr '22

Hi, Does MPI_COMM_WORLD exist in any sense before MPI_Init is called? In particular, does this code violate the standard? InitializeMyApp(MPI_Comm comm, int argc, char **argv) { int initialized; MPI_Initialized(&initialized); if (!initialized) MPI_Init(&argc, &argv); /* call some function that uses comm */ } int main(int argc, char **argv) { MPI_Comm comm = MPI_COMM_WORLD; InitializeMyApp(comm, argc, argv); : } My instinct is that this is wrong, but I would like to be sure before I complain about it. It appears to work with some existing implementations, but it looks sketchy to me. Bruce

1 0

Re: [mpich-discuss] Hydra WARNING: too many ssh connections
by Zhou, Hui 01 Apr '22

01 Apr '22

The spawned process launches the same way the first MPI_COMM_WORLD is launched, i.e. in a round-robin fashion through the list of nodes. -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Sent: Friday, April 1, 2022 4:22 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: Hydra WARNING: too many ssh connections Thanks Hui, is the spawned process on the local host, or the remote host or both? Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Friday, April 1, 2022 4:20 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: Hydra WARNING: too many ssh connections Every time you call MPI_Comm_spawn, hydra will launch a ssh (for each host) to create a proxy. It is certainly not ideal for applications relying on spawning many processes. ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Friday, April 1, 2022 4:08 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Hydra WARNING: too many ssh connections Hi, you provided the following information about the warning “too many ssh connections”: The particular warning is issued by hydra, MPICH’s process manager. Following excerpt is the comment in that source code: /* ssh has many types of security controls that do not allow a * user to ssh to the same node multiple times very * quickly. If this happens, the ssh daemons disables ssh * connections causing the job to fail. This is basically a * hack to slow down ssh connections to the same node. We * check for offset == 0 before applying this hack, so we only * slow down the cases where ssh is being used, and not the * cases where we fall back to fork. */ Is this just during an initial ssh connection attempt? I’m trying to figure out where my code is triggering this warning. Could it be from 1. MPI_Intercomm_create 2. MPI_Comm_spawn 3. others? I’m calling mpiexec with the “—launcher ssh” option, MPICH 4.0.1. Thanks, Kurt

1 0

Re: [mpich-discuss] Hydra WARNING: too many ssh connections
by Mccall, Kurt E. (MSFC-EV41) 01 Apr '22

01 Apr '22

Thanks Hui, is the spawned process on the local host, or the remote host or both? Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Friday, April 1, 2022 4:20 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: Hydra WARNING: too many ssh connections Every time you call MPI_Comm_spawn, hydra will launch a ssh (for each host) to create a proxy. It is certainly not ideal for applications relying on spawning many processes. ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Friday, April 1, 2022 4:08 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Hydra WARNING: too many ssh connections Hi, you provided the following information about the warning "too many ssh connections": The particular warning is issued by hydra, MPICH's process manager. Following excerpt is the comment in that source code: /* ssh has many types of security controls that do not allow a * user to ssh to the same node multiple times very * quickly. If this happens, the ssh daemons disables ssh * connections causing the job to fail. This is basically a * hack to slow down ssh connections to the same node. We * check for offset == 0 before applying this hack, so we only * slow down the cases where ssh is being used, and not the * cases where we fall back to fork. */ Is this just during an initial ssh connection attempt? I'm trying to figure out where my code is triggering this warning. Could it be from 1. MPI_Intercomm_create 2. MPI_Comm_spawn 3. others? I'm calling mpiexec with the "-launcher ssh" option, MPICH 4.0.1. Thanks, Kurt

1 0