June 2019 - discuss - mailman3.cels.anl.gov

Fwd: Issue in submitting jobs in round robin fashion
by Rajeshwar Gande 06 Jun '19

06 Jun '19

Hi all, I am building a cluster for GPU intensive application called Relion. The requirement is that the mpi occupies processes in round robin fashion across the nodes. I have tried using -map-by node parameter to achieve the same. It failed to implement round robin algorithm. I am running mpirun.mpich as the executable since my mpirun corresponds to open mpi which is working in round robin fashion but is not capable for multi threading. Hence using MPICH2. Can someone help me in figuring out how to submit a mpi job to schedule in rr fashion. Compute nodes: 64 vCPU, 8 GPU. Parameters: 24MPI, 8 Threads I am looking forward to schedule 8mpi processes on each node instead mpich2 is scheduling 24 on one node. Hope I made my question clear. Regards, Rajeshwar G.

1 0

RMA and multithreading (MPI_THREAD_MULTIPLE)
by Alexey Paznikov 06 Jun '19

06 Jun '19

Hi, I have multithreaded MPI program (MPI_THREAD_MULTIPLE mode) with RMA calls.This is a simplified example: #include <stdio.h> #include <mpi.h> #include <pthread.h> MPI_Win win; pthread_mutex_t lock; void *thread(void *arg) { MPI_Win_lock_all(0, win); MPI_Win_unlock_all(win); return NULL; } int main(int argc, char *argv[]) { MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, NULL); pthread_mutex_init(&lock, NULL); int *buf = NULL; const int bufsize = 1; MPI_Alloc_mem(bufsize, MPI_INFO_NULL, &buf); MPI_Win_create(buf, 1, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win); pthread_t tid; pthread_create(&tid, NULL, thread, NULL); MPI_Win_lock_all(0, win); MPI_Win_unlock_all(win); pthread_join(tid, NULL); MPI_Win_free(&win); MPI_Finalize(); return 0; } If I run such program, it crashes with an error message: Fatal error in PMPI_Win_lock_all: Wrong synchronization of RMA calls , error stack: PMPI_Win_lock_all(149).: MPI_Win_lock_all(assert=0, win=0xa0000000) failed MPID_Win_lock_all(1522): Wrong synchronization of RMA calls If I replace MPI_Win_lock_all with MPI_Win_lock the problem remains: Fatal error in PMPI_Win_lock: Wrong synchronization of RMA calls , error stack: PMPI_Win_lock(157).: MPI_Win_lock(lock_type=234, rank=0, assert=0, win=0xa0000000) failed MPID_Win_lock(1163): Wrong synchronization of RMA calls (this message is repeated many times) If I protect RMA operations with a mutex, the problem disappears. MPICH version 3.3. Similar problem is also in MVAPICH2 2.3.1. Are the RMA operations not thread safe at the moment? Could you tell me how to deal with this problem?

1 0

Re: [mpich-discuss] A possible bug in HYD_pmcd_pmi_allocate_kvs
by Raffenetti, Kenneth J. 06 Jun '19

06 Jun '19

On 6/6/19 12:52 AM, Xiaopeng Duan wrote: > Thank you, Ken. > > We were having another problem with 3.3, and will try it once we fixed > our issue. > > Just my couriosity, why a random number was chosen for the fix instead > of hostname or address? Looks to me the random number still has some > possibility to repeat (although very rare), but hostnames and addresses > should be unique in a system. I had the same thought when looking back at this patch. Maybe Giuseppe can share why that was added. I'm fairly sure it can be safely removed. Ken > > Regards, > Xiaopeng > > On Wed, Jun 5, 2019, 8:29 AM Raffenetti, Kenneth J. > <raffenet(a)mcs.anl.gov <mailto:[email protected]>> wrote: > > We added a similar fix in https://github.com/pmodels/mpich/pull/2788. > This was included in the MPICH 3.3 release. > > Ken > > On 6/4/19 11:27 PM, Xiaopeng Duan via discuss wrote: > > Hi, MPICH experts, > > > > We are working on a dynamic master-worker flow using > > mpi_comm_connect/mpi_com_accept. In some cases when the total > number of > > worker process is large, they may get the same kvs_name and > confuse the > > internal group identifiers. This was traced to the naming > convention in > > HYD_pmcd_pmi_allocate_kvs() that considers only process id, while > two > > processes on different machines may have the same pid. I tried to > add > > host name (from unistd.h>gethostname) to the name, i.e. > > 'kvs_HOSTNAME_PID_pgid', then everything is working fine in our > testing. > > > > So I'm wondering if this change is safe (we may need it for our > release) > > and if it would go into the official MPICH release some time. > > > > Thank you very much. > > > > Xiaopeng > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > >

1 0

Re: [mpich-discuss] A possible bug in HYD_pmcd_pmi_allocate_kvs
by Xiaopeng Duan 06 Jun '19

06 Jun '19

Thank you, Ken. We were having another problem with 3.3, and will try it once we fixed our issue. Just my couriosity, why a random number was chosen for the fix instead of hostname or address? Looks to me the random number still has some possibility to repeat (although very rare), but hostnames and addresses should be unique in a system. Regards, Xiaopeng On Wed, Jun 5, 2019, 8:29 AM Raffenetti, Kenneth J. <raffenet(a)mcs.anl.gov> wrote: > We added a similar fix in https://github.com/pmodels/mpich/pull/2788. > This was included in the MPICH 3.3 release. > > Ken > > On 6/4/19 11:27 PM, Xiaopeng Duan via discuss wrote: > > Hi, MPICH experts, > > > > We are working on a dynamic master-worker flow using > > mpi_comm_connect/mpi_com_accept. In some cases when the total number of > > worker process is large, they may get the same kvs_name and confuse the > > internal group identifiers. This was traced to the naming convention in > > HYD_pmcd_pmi_allocate_kvs() that considers only process id, while two > > processes on different machines may have the same pid. I tried to add > > host name (from unistd.h>gethostname) to the name, i.e. > > 'kvs_HOSTNAME_PID_pgid', then everything is working fine in our testing. > > > > So I'm wondering if this change is safe (we may need it for our release) > > and if it would go into the official MPICH release some time. > > > > Thank you very much. > > > > Xiaopeng > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > >

1 0

Re: [mpich-discuss] How to know total memory allocated by MPICH at a certain time?
by Raffenetti, Kenneth J. 05 Jun '19

05 Jun '19

We don't currently have any support for this feature. MPICH internally tracks memory allocations (with --enable-g[=mem] configuration option), and has the option to dump a report at MPI_Finalize (export MPIR_CVAR_MEMDUMP=1) that summarizes types of allocations, # of allocations, and a high water mark. I don't think it would take much to hack an on-demand usage report, however I can't give any ETA on when it might be added. It might be faster if you do a draft first and post a pull request on Github :). Ken On 6/3/19 3:56 PM, Zhang, Junchao via discuss wrote: > Hello, > Is there a way for users to know total memory allocated by MPICH at a > certain time? I am asking that since I want to know the difference of > this value before/after a user function. The function is quite complex > and calls many other functions. > The background is that a PETSc user claimed a PETSc routine increased > RSS (resident set size) periodically when running with MPICH, but not > with OpenMPI. The application itself is valgrind clean and has no > memory leaks. > Thank you. > --Junchao Zhang > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] A possible bug in HYD_pmcd_pmi_allocate_kvs
by Raffenetti, Kenneth J. 05 Jun '19

05 Jun '19

We added a similar fix in https://github.com/pmodels/mpich/pull/2788. This was included in the MPICH 3.3 release. Ken On 6/4/19 11:27 PM, Xiaopeng Duan via discuss wrote: > Hi, MPICH experts, > > We are working on a dynamic master-worker flow using > mpi_comm_connect/mpi_com_accept. In some cases when the total number of > worker process is large, they may get the same kvs_name and confuse the > internal group identifiers. This was traced to the naming convention in > HYD_pmcd_pmi_allocate_kvs() that considers only process id, while two > processes on different machines may have the same pid. I tried to add > host name (from unistd.h>gethostname) to the name, i.e. > 'kvs_HOSTNAME_PID_pgid', then everything is working fine in our testing. > > So I'm wondering if this change is safe (we may need it for our release) > and if it would go into the official MPICH release some time. > > Thank you very much. > > Xiaopeng > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

mpich/master: Something got wrong since may 24
by Eric Chamberland 05 Jun '19

05 Jun '19

Hi, I am in the early debugging stages of something that broke on may 24 on our test database with 17 tests against mpich/master branch. It seems related to MPI I/O, since we have observed the bugs just after MPI I/O reads. Everything is fine with other MPI libs, even older mpich. This is only a mpich/master problem. My early query: am I alone to have observed this? Thanks, Eric

1 0

A possible bug in HYD_pmcd_pmi_allocate_kvs
by Xiaopeng Duan 04 Jun '19

04 Jun '19

Hi, MPICH experts, We are working on a dynamic master-worker flow using mpi_comm_connect/mpi_com_accept. In some cases when the total number of worker process is large, they may get the same kvs_name and confuse the internal group identifiers. This was traced to the naming convention in HYD_pmcd_pmi_allocate_kvs() that considers only process id, while two processes on different machines may have the same pid. I tried to add host name (from unistd.h>gethostname) to the name, i.e. 'kvs_HOSTNAME_PID_pgid', then everything is working fine in our testing. So I'm wondering if this change is safe (we may need it for our release) and if it would go into the official MPICH release some time. Thank you very much. Xiaopeng

1 0

How to know total memory allocated by MPICH at a certain time?
by Zhang, Junchao 03 Jun '19

03 Jun '19

Hello, Is there a way for users to know total memory allocated by MPICH at a certain time? I am asking that since I want to know the difference of this value before/after a user function. The function is quite complex and calls many other functions. The background is that a PETSc user claimed a PETSc routine increased RSS (resident set size) periodically when running with MPICH, but not with OpenMPI. The application itself is valgrind clean and has no memory leaks. Thank you. --Junchao Zhang

1 0