- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Custom rank for processes
by Niyaz Murshed 01 Jul '24

01 Jul '24

Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:37 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Yes, you can use rankmap to simply list out the node assignment for each rank. "-rankmap (vector,1,1,0,0,0)" is a list of 5 node ids, one for each rank. So rank 0 gets node 1, rank 1 gets node 1, rank 2 gets node 0, and so on. The -hosts options is convenient if you have somewhat uniform assignment. If you want arbitrary assignment, just use -rankmap option. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 4:30 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192. 168. 2. 100: 2,192. 168. 2. 200: 3 : this will give 2 process in ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192.168.2.100:2,192.168.2.200:3 : this will give 2 process in 100 and 3 in 200 .. Still won’t be able to specify which ranks goes to which node. Rank0 will be on node 100 .. what If I want Rank0 on node 200? From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:09 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes > root@ampere-altra-2-1:/# mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors Hello world from processor dpr740, rank 0 out of 5 processors Hello world from processor dpr740, rank 4 out of 5 processors Hello world from processor dpr740, rank 2 out of 5 processors > -bind-to user:10,11,12,13 > This would mean on host 192.168.2.100 > P0=>10 , P2=>11 > This would mean on host 192.168.2.200 > P0=>10 , P2=>11, P3=12 > Is this correct understanding ? Is it also possible to say which rank process will be pinned to which core ? Yes, that is correct. The ranks are assigned to hosts as shown in the hello world output. > About the rankmap:, trying to understand if I can select where a particular rank would be from > list of hosts. Currently, the first host in the list always get rank0. > Can I specify the below ranks? > mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world > > Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors => rank0 > Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors. => rank1 > Hello world from processor dpr740, rank 0 out of 5 processors =>rank2 > Hello world from processor dpr740, rank 4 out of 5 processors =>rank3 > Hello world from processor dpr740, rank 2 out of 5 processors =>rank4 Yes. You can use "-rankmap (vector,1,1,0,0,0)". Alternatively, you can use "-hosts 192.168.2.100:2,192.168.2.200:3", the colon syntax specifies how many processes you want to assign to each host. -- Hui

1 0

Re: [mpich-discuss] Custom rank for processes
by Zhou, Hui 01 Jul '24

01 Jul '24

Yes, you can use rankmap to simply list out the node assignment for each rank. "-rankmap (vector,1,1,0,0,0)" is a list of 5 node ids, one for each rank. So rank 0 gets node 1, rank 1 gets node 1, rank 2 gets node 0, and so on. The -hosts options is convenient if you have somewhat uniform assignment. If you want arbitrary assignment, just use -rankmap option. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 4:30 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192. 168. 2. 100: 2,192. 168. 2. 200: 3 : this will give 2 process in ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192.168.2.100:2,192.168.2.200:3 : this will give 2 process in 100 and 3 in 200 .. Still won’t be able to specify which ranks goes to which node. Rank0 will be on node 100 .. what If I want Rank0 on node 200? From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:09 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes > root@ampere-altra-2-1:/# mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors Hello world from processor dpr740, rank 0 out of 5 processors Hello world from processor dpr740, rank 4 out of 5 processors Hello world from processor dpr740, rank 2 out of 5 processors > -bind-to user:10,11,12,13 > This would mean on host 192.168.2.100 > P0=>10 , P2=>11 > This would mean on host 192.168.2.200 > P0=>10 , P2=>11, P3=12 > Is this correct understanding ? Is it also possible to say which rank process will be pinned to which core ? Yes, that is correct. The ranks are assigned to hosts as shown in the hello world output. > About the rankmap:, trying to understand if I can select where a particular rank would be from > list of hosts. Currently, the first host in the list always get rank0. > Can I specify the below ranks? > mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world > > Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors => rank0 > Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors. => rank1 > Hello world from processor dpr740, rank 0 out of 5 processors =>rank2 > Hello world from processor dpr740, rank 4 out of 5 processors =>rank3 > Hello world from processor dpr740, rank 2 out of 5 processors =>rank4 Yes. You can use "-rankmap (vector,1,1,0,0,0)". Alternatively, you can use "-hosts 192.168.2.100:2,192.168.2.200:3", the colon syntax specifies how many processes you want to assign to each host. -- Hui

1 0

Re: [mpich-discuss] Custom rank for processes
by Niyaz Murshed 01 Jul '24

01 Jul '24

Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192.168.2.100:2,192.168.2.200:3 : this will give 2 process in 100 and 3 in 200 .. Still won’t be able to specify which ranks goes to which node. Rank0 will be on node 100 .. what If I want Rank0 on node 200? From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:09 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes > root@ampere-altra-2-1:/# mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors Hello world from processor dpr740, rank 0 out of 5 processors Hello world from processor dpr740, rank 4 out of 5 processors Hello world from processor dpr740, rank 2 out of 5 processors > -bind-to user:10,11,12,13 > This would mean on host 192.168.2.100 > P0=>10 , P2=>11 > This would mean on host 192.168.2.200 > P0=>10 , P2=>11, P3=12 > Is this correct understanding ? Is it also possible to say which rank process will be pinned to which core ? Yes, that is correct. The ranks are assigned to hosts as shown in the hello world output. > About the rankmap:, trying to understand if I can select where a particular rank would be from > list of hosts. Currently, the first host in the list always get rank0. > Can I specify the below ranks? > mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world > > Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors => rank0 > Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors. => rank1 > Hello world from processor dpr740, rank 0 out of 5 processors =>rank2 > Hello world from processor dpr740, rank 4 out of 5 processors =>rank3 > Hello world from processor dpr740, rank 2 out of 5 processors =>rank4 Yes. You can use "-rankmap (vector,1,1,0,0,0)". Alternatively, you can use "-hosts 192.168.2.100:2,192.168.2.200:3", the colon syntax specifies how many processes you want to assign to each host. -- Hui

1 0

Re: [mpich-discuss] Custom rank for processes
by Zhou, Hui 01 Jul '24

01 Jul '24

> root@ampere-altra-2-1:/# mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors Hello world from processor dpr740, rank 0 out of 5 processors Hello world from processor dpr740, rank 4 out of 5 processors Hello world from processor dpr740, rank 2 out of 5 processors > -bind-to user:10,11,12,13 > This would mean on host 192.168.2.100 > P0=>10 , P2=>11 > This would mean on host 192.168.2.200 > P0=>10 , P2=>11, P3=12 > Is this correct understanding ? Is it also possible to say which rank process will be pinned to which core ? Yes, that is correct. The ranks are assigned to hosts as shown in the hello world output. > About the rankmap:, trying to understand if I can select where a particular rank would be from > list of hosts. Currently, the first host in the list always get rank0. > Can I specify the below ranks? > mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world > > Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors => rank0 > Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors. => rank1 > Hello world from processor dpr740, rank 0 out of 5 processors =>rank2 > Hello world from processor dpr740, rank 4 out of 5 processors =>rank3 > Hello world from processor dpr740, rank 2 out of 5 processors =>rank4 Yes. You can use "-rankmap (vector,1,1,0,0,0)". Alternatively, you can use "-hosts 192.168.2.100:2,192.168.2.200:3", the colon syntax specifies how many processes you want to assign to each host. -- Hui

1 0

Re: [mpich-discuss] Custom rank for processes
by Niyaz Murshed 01 Jul '24

01 Jul '24

Thank you for the explanation, Hui. root@ampere-altra-2-1:/# mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors Hello world from processor dpr740, rank 0 out of 5 processors Hello world from processor dpr740, rank 4 out of 5 processors Hello world from processor dpr740, rank 2 out of 5 processors -bind-to user:10,11,12,13 This would mean on host 192.168.2.100 P0=>10 , P2=>11 This would mean on host 192.168.2.200 P0=>10 , P2=>11, P3=12 Is this correct understanding ? Is it also possible to say which rank process will be pinned to which core ? About the rankmap:, trying to understand if I can select where a particular rank would be from list of hosts. Currently, the first host in the list always get rank0. Can I specify the below ranks? mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors => rank0 Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors. => rank1 Hello world from processor dpr740, rank 0 out of 5 processors =>rank2 Hello world from processor dpr740, rank 4 out of 5 processors =>rank3 Hello world from processor dpr740, rank 2 out of 5 processors =>rank4 From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 3:43 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Hi Niyaz, With 5 processes and two hosts, by default it is equivalent to "-rankmap (vector,(0,2,5))" or "-rankmap (vector,0,1,0,1,0)", which is the same with option "-ppn 1"-- a round-robin assignment of one process per host. The result is, rank 0,2,4 on the fist host and rank 1,3 on the second host. Let me know what is your desired rank ordering, I'll help you on the options. You can use "-bind-to" to specify binding. Unfortunately, you only can specify the same binding policy for all hosts. The option "-bind-to user:0+2,1+4,3,2" binds the first process to core 0 and 2, the 2nd process to core 1 and 4, the 3rd process to core 3, the 4th process to core 2. If there are only 3 or 2 processes on the node, the later binding specifications are ignored. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 3:17 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Thank you Joachim and Hui. -rankmap {rank map} comma separated rank to node id list How is the format if I have 4 processes ? mpirun -prepend-rank -n 5 -hosts 192. 168. 2. 100,192. 168. 2. 200 hello-world How do I add rankmap to the above? Just add ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you Joachim and Hui. -rankmap {rank map} comma separated rank to node id list How is the format if I have 4 processes ? mpirun -prepend-rank -n 5 -hosts 192.168.2.100,192.168.2.200 hello-world How do I add rankmap to the above? Just add ranks 0,1,2,3 after –rankmap? Secondly, can I pin each of these 5 processes to a particular core? Could you give an example of the “user” option? user:0+2,1+4,3,2 -- user specified binding does the above mean rank 0 will run on 2? From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 3:10 PM To: discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: Custom rank for processes >What Joachim points out is that you can set a rank of 2, 3, 20, 30 if your total number of processes is 4. Of course, I meant for "can't". Here are some additional pointers: * The PR adding the rankmap option: https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/6213__;!!G… <https://urldefense.us/v3/__https:/github.com/pmodels/mpich/pull/6213__;!!G_…> * Other options for specifying rank ordering and binding:https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/ma… <https://urldefense.us/v3/__https:/github.com/pmodels/mpich/blob/main/doc/wi…> Hui ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, July 1, 2024 3:04 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: Custom rank for processes Hi Niyaz, Yes, you can use the -rankmap option to specify a full rank ordering. The syntax for rankmap string is: ``` /* rankmap string format (used in PMI_process_mapping) in ABNF: * * mapping = '(' format ',' (block / block_repeats) *[',' (block / block_repeats)] ')' * format = 'vector' * block = num / '(' num ',' num ',' num ')' * block_repeats = '[' block *[',' block] ']x' num * num = 1*DIGIT * * If a block is in 3-tuple of (X,Y,Z), the meaning is: * X - node id start value * Y - number of nodes with size Z * Z - number of processes assigned to each node * * If a block is a single number id, it denotes the node id, which is equivallen to * (id, 1, 1). * * A block_repeat repeats the bracketed block list by num of times. */ ``` What Joachim points out is that you can set a rank of 2, 3, 20, 30 if your total number of processes is 4. You can only order the processes. Hope that helps. -- Hui ________________________________ From: Niyaz Murshed via discuss <discuss(a)mpich.org> Sent: Monday, July 1, 2024 2:43 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes Am actually trying to understand how/if I can pin a rank to a process .. also pin a core to a process. From: Niyaz Murshed via discuss <discuss@ mpich. org> Date: Monday, July 1, 2024 at 2: 34 PM To: Jenke, Joachim <jenke@ itc. rwth-aachen. de>, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Am actually trying to understand how/if I can pin a rank to a process .. also pin a core to a process. From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Monday, July 1, 2024 at 2:34 PM To: Jenke, Joachim <jenke(a)itc.rwth-aachen.de>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes What is the rankmap used for ? From: Jenke, Joachim <jenke@ itc. rwth-aachen. de> Sent: Monday, July 1, 2024 2: 21: 53 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Niyaz Murshed <Niyaz. Murshed@ arm. com>; nd <nd@ arm. com> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd What is the rankmap used for ? ________________________________ From: Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Sent: Monday, July 1, 2024 2:21:53 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: Custom rank for processes MPI defines rank numbers to be consecutive from 0 to size-1. Your suggested numbering will not work. -Joachim ________________________________ From: Niyaz Murshed via discuss <discuss(a)mpich.org> Sent: Monday, July 1, 2024 9:09:41 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: [mpich-discuss] Custom rank for processes Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank 20,30 Thanks.

1 0

Re: [mpich-discuss] Custom rank for processes
by Zhou, Hui 01 Jul '24

01 Jul '24

Hi Niyaz, With 5 processes and two hosts, by default it is equivalent to "-rankmap (vector,(0,2,5))" or "-rankmap (vector,0,1,0,1,0)", which is the same with option "-ppn 1"-- a round-robin assignment of one process per host. The result is, rank 0,2,4 on the fist host and rank 1,3 on the second host. Let me know what is your desired rank ordering, I'll help you on the options. You can use "-bind-to" to specify binding. Unfortunately, you only can specify the same binding policy for all hosts. The option "-bind-to user:0+2,1+4,3,2" binds the first process to core 0 and 2, the 2nd process to core 1 and 4, the 3rd process to core 3, the 4th process to core 2. If there are only 3 or 2 processes on the node, the later binding specifications are ignored. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 3:17 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Thank you Joachim and Hui. -rankmap {rank map} comma separated rank to node id list How is the format if I have 4 processes ? mpirun -prepend-rank -n 5 -hosts 192. 168. 2. 100,192. 168. 2. 200 hello-world How do I add rankmap to the above? Just add ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you Joachim and Hui. -rankmap {rank map} comma separated rank to node id list How is the format if I have 4 processes ? mpirun -prepend-rank -n 5 -hosts 192.168.2.100,192.168.2.200 hello-world How do I add rankmap to the above? Just add ranks 0,1,2,3 after –rankmap? Secondly, can I pin each of these 5 processes to a particular core? Could you give an example of the “user” option? user:0+2,1+4,3,2 -- user specified binding does the above mean rank 0 will run on 2? From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 3:10 PM To: discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: Custom rank for processes >What Joachim points out is that you can set a rank of 2, 3, 20, 30 if your total number of processes is 4. Of course, I meant for "can't". Here are some additional pointers: * The PR adding the rankmap option: https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/6213__;!!G… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/6213__;!!G…> * Other options for specifying rank ordering and binding:https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/ma… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/main/doc/w…> Hui ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, July 1, 2024 3:04 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: Custom rank for processes Hi Niyaz, Yes, you can use the -rankmap option to specify a full rank ordering. The syntax for rankmap string is: ``` /* rankmap string format (used in PMI_process_mapping) in ABNF: * * mapping = '(' format ',' (block / block_repeats) *[',' (block / block_repeats)] ')' * format = 'vector' * block = num / '(' num ',' num ',' num ')' * block_repeats = '[' block *[',' block] ']x' num * num = 1*DIGIT * * If a block is in 3-tuple of (X,Y,Z), the meaning is: * X - node id start value * Y - number of nodes with size Z * Z - number of processes assigned to each node * * If a block is a single number id, it denotes the node id, which is equivallen to * (id, 1, 1). * * A block_repeat repeats the bracketed block list by num of times. */ ``` What Joachim points out is that you can set a rank of 2, 3, 20, 30 if your total number of processes is 4. You can only order the processes. Hope that helps. -- Hui ________________________________ From: Niyaz Murshed via discuss <discuss(a)mpich.org> Sent: Monday, July 1, 2024 2:43 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes Am actually trying to understand how/if I can pin a rank to a process .. also pin a core to a process. From: Niyaz Murshed via discuss <discuss@ mpich. org> Date: Monday, July 1, 2024 at 2: 34 PM To: Jenke, Joachim <jenke@ itc. rwth-aachen. de>, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Am actually trying to understand how/if I can pin a rank to a process .. also pin a core to a process. From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Monday, July 1, 2024 at 2:34 PM To: Jenke, Joachim <jenke(a)itc.rwth-aachen.de>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes What is the rankmap used for ? From: Jenke, Joachim <jenke@ itc. rwth-aachen. de> Sent: Monday, July 1, 2024 2: 21: 53 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Niyaz Murshed <Niyaz. Murshed@ arm. com>; nd <nd@ arm. com> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd What is the rankmap used for ? ________________________________ From: Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Sent: Monday, July 1, 2024 2:21:53 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: Custom rank for processes MPI defines rank numbers to be consecutive from 0 to size-1. Your suggested numbering will not work. -Joachim ________________________________ From: Niyaz Murshed via discuss <discuss(a)mpich.org> Sent: Monday, July 1, 2024 9:09:41 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: [mpich-discuss] Custom rank for processes Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank 20,30 Thanks.

1 0

Re: [mpich-discuss] Custom rank for processes
by Niyaz Murshed 01 Jul '24

01 Jul '24

Thank you Joachim and Hui. -rankmap {rank map} comma separated rank to node id list How is the format if I have 4 processes ? mpirun -prepend-rank -n 5 -hosts 192.168.2.100,192.168.2.200 hello-world How do I add rankmap to the above? Just add ranks 0,1,2,3 after –rankmap? Secondly, can I pin each of these 5 processes to a particular core? Could you give an example of the “user” option? user:0+2,1+4,3,2 -- user specified binding does the above mean rank 0 will run on 2? From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 3:10 PM To: discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: Custom rank for processes >What Joachim points out is that you can set a rank of 2, 3, 20, 30 if your total number of processes is 4. Of course, I meant for "can't". Here are some additional pointers: * The PR adding the rankmap option: https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/6213__;!!G… * Other options for specifying rank ordering and binding:https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/ma… Hui ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, July 1, 2024 3:04 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: Custom rank for processes Hi Niyaz, Yes, you can use the -rankmap option to specify a full rank ordering. The syntax for rankmap string is: ``` /* rankmap string format (used in PMI_process_mapping) in ABNF: * * mapping = '(' format ',' (block / block_repeats) *[',' (block / block_repeats)] ')' * format = 'vector' * block = num / '(' num ',' num ',' num ')' * block_repeats = '[' block *[',' block] ']x' num * num = 1*DIGIT * * If a block is in 3-tuple of (X,Y,Z), the meaning is: * X - node id start value * Y - number of nodes with size Z * Z - number of processes assigned to each node * * If a block is a single number id, it denotes the node id, which is equivallen to * (id, 1, 1). * * A block_repeat repeats the bracketed block list by num of times. */ ``` What Joachim points out is that you can set a rank of 2, 3, 20, 30 if your total number of processes is 4. You can only order the processes. Hope that helps. -- Hui ________________________________ From: Niyaz Murshed via discuss <discuss(a)mpich.org> Sent: Monday, July 1, 2024 2:43 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes Am actually trying to understand how/if I can pin a rank to a process .. also pin a core to a process. From: Niyaz Murshed via discuss <discuss@ mpich. org> Date: Monday, July 1, 2024 at 2: 34 PM To: Jenke, Joachim <jenke@ itc. rwth-aachen. de>, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Am actually trying to understand how/if I can pin a rank to a process .. also pin a core to a process. From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Monday, July 1, 2024 at 2:34 PM To: Jenke, Joachim <jenke(a)itc.rwth-aachen.de>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes What is the rankmap used for ? From: Jenke, Joachim <jenke@ itc. rwth-aachen. de> Sent: Monday, July 1, 2024 2: 21: 53 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Niyaz Murshed <Niyaz. Murshed@ arm. com>; nd <nd@ arm. com> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd What is the rankmap used for ? ________________________________ From: Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Sent: Monday, July 1, 2024 2:21:53 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: Custom rank for processes MPI defines rank numbers to be consecutive from 0 to size-1. Your suggested numbering will not work. -Joachim ________________________________ From: Niyaz Murshed via discuss <discuss(a)mpich.org> Sent: Monday, July 1, 2024 9:09:41 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: [mpich-discuss] Custom rank for processes Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank 20,30 Thanks.

1 0

Re: [mpich-discuss] Custom rank for processes
by Zhou, Hui 01 Jul '24

01 Jul '24

>What Joachim points out is that you can set a rank of 2, 3, 20, 30 if your total number of processes is 4. Of course, I meant for "can't". Here are some additional pointers: * The PR adding the rankmap option: https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/6213__;!!G… * Other options for specifying rank ordering and binding:https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/ma… Hui ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, July 1, 2024 3:04 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: Custom rank for processes Hi Niyaz, Yes, you can use the -rankmap option to specify a full rank ordering. The syntax for rankmap string is: ``` /* rankmap string format (used in PMI_process_mapping) in ABNF: * * mapping = '(' format ',' (block / block_repeats) *[',' (block / block_repeats)] ')' * format = 'vector' * block = num / '(' num ',' num ',' num ')' * block_repeats = '[' block *[',' block] ']x' num * num = 1*DIGIT * * If a block is in 3-tuple of (X,Y,Z), the meaning is: * X - node id start value * Y - number of nodes with size Z * Z - number of processes assigned to each node * * If a block is a single number id, it denotes the node id, which is equivallen to * (id, 1, 1). * * A block_repeat repeats the bracketed block list by num of times. */ ``` What Joachim points out is that you can set a rank of 2, 3, 20, 30 if your total number of processes is 4. You can only order the processes. Hope that helps. -- Hui ________________________________ From: Niyaz Murshed via discuss <discuss(a)mpich.org> Sent: Monday, July 1, 2024 2:43 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes Am actually trying to understand how/if I can pin a rank to a process .. also pin a core to a process. From: Niyaz Murshed via discuss <discuss@ mpich. org> Date: Monday, July 1, 2024 at 2: 34 PM To: Jenke, Joachim <jenke@ itc. rwth-aachen. de>, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Am actually trying to understand how/if I can pin a rank to a process .. also pin a core to a process. From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Monday, July 1, 2024 at 2:34 PM To: Jenke, Joachim <jenke(a)itc.rwth-aachen.de>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes What is the rankmap used for ? From: Jenke, Joachim <jenke@ itc. rwth-aachen. de> Sent: Monday, July 1, 2024 2: 21: 53 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Niyaz Murshed <Niyaz. Murshed@ arm. com>; nd <nd@ arm. com> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd What is the rankmap used for ? ________________________________ From: Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Sent: Monday, July 1, 2024 2:21:53 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: Custom rank for processes MPI defines rank numbers to be consecutive from 0 to size-1. Your suggested numbering will not work. -Joachim ________________________________ From: Niyaz Murshed via discuss <discuss(a)mpich.org> Sent: Monday, July 1, 2024 9:09:41 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: [mpich-discuss] Custom rank for processes Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank 20,30 Thanks.

1 0

Re: [mpich-discuss] Custom rank for processes
by Zhou, Hui 01 Jul '24

01 Jul '24

Hi Niyaz, Yes, you can use the -rankmap option to specify a full rank ordering. The syntax for rankmap string is: ``` /* rankmap string format (used in PMI_process_mapping) in ABNF: * * mapping = '(' format ',' (block / block_repeats) *[',' (block / block_repeats)] ')' * format = 'vector' * block = num / '(' num ',' num ',' num ')' * block_repeats = '[' block *[',' block] ']x' num * num = 1*DIGIT * * If a block is in 3-tuple of (X,Y,Z), the meaning is: * X - node id start value * Y - number of nodes with size Z * Z - number of processes assigned to each node * * If a block is a single number id, it denotes the node id, which is equivallen to * (id, 1, 1). * * A block_repeat repeats the bracketed block list by num of times. */ ``` What Joachim points out is that you can set a rank of 2, 3, 20, 30 if your total number of processes is 4. You can only order the processes. Hope that helps. -- Hui ________________________________ From: Niyaz Murshed via discuss <discuss(a)mpich.org> Sent: Monday, July 1, 2024 2:43 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes Am actually trying to understand how/if I can pin a rank to a process .. also pin a core to a process. From: Niyaz Murshed via discuss <discuss@ mpich. org> Date: Monday, July 1, 2024 at 2: 34 PM To: Jenke, Joachim <jenke@ itc. rwth-aachen. de>, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Am actually trying to understand how/if I can pin a rank to a process .. also pin a core to a process. From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Monday, July 1, 2024 at 2:34 PM To: Jenke, Joachim <jenke(a)itc.rwth-aachen.de>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes What is the rankmap used for ? From: Jenke, Joachim <jenke@ itc. rwth-aachen. de> Sent: Monday, July 1, 2024 2: 21: 53 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Niyaz Murshed <Niyaz. Murshed@ arm. com>; nd <nd@ arm. com> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd What is the rankmap used for ? ________________________________ From: Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Sent: Monday, July 1, 2024 2:21:53 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: Custom rank for processes MPI defines rank numbers to be consecutive from 0 to size-1. Your suggested numbering will not work. -Joachim ________________________________ From: Niyaz Murshed via discuss <discuss(a)mpich.org> Sent: Monday, July 1, 2024 9:09:41 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: [mpich-discuss] Custom rank for processes Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank 20,30 Thanks.

1 0

Re: [mpich-discuss] Custom rank for processes
by Niyaz Murshed 01 Jul '24

01 Jul '24

Am actually trying to understand how/if I can pin a rank to a process .. also pin a core to a process. From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Monday, July 1, 2024 at 2:34 PM To: Jenke, Joachim <jenke(a)itc.rwth-aachen.de>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes What is the rankmap used for ? From: Jenke, Joachim <jenke@ itc. rwth-aachen. de> Sent: Monday, July 1, 2024 2: 21: 53 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Niyaz Murshed <Niyaz. Murshed@ arm. com>; nd <nd@ arm. com> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd What is the rankmap used for ? ________________________________ From: Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Sent: Monday, July 1, 2024 2:21:53 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: Custom rank for processes MPI defines rank numbers to be consecutive from 0 to size-1. Your suggested numbering will not work. -Joachim ________________________________ From: Niyaz Murshed via discuss <discuss(a)mpich.org> Sent: Monday, July 1, 2024 9:09:41 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: [mpich-discuss] Custom rank for processes Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank 20,30 Thanks.

1 0