August 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI 3.0 RMA question
by Rajeev Thakur 20 Aug '14

20 Aug '14

If you are using passive target RMA (lock-unlock), you have to use some out-of-band method for the target to know that the data transfer has happened. If you use active target RMA (fence or post/start/complete/wait), the target knows the data has been transferred when its call to fence or wait returns. Rajeev On Aug 20, 2014, at 8:34 PM, "Boisvert, Sebastien" <boisvert(a)anl.gov> wrote: > Hi, > > Let's say I have 20 ranks. They all call MPI_Win_create_dynamic to create a window. > Then, every rank attach a buffer with MPI_Win_attach to their window. > > Then, rank 3 calls MPI_Put to put, say, some data on the window of rank 99. > Rank 3 can then rely on MPI_Win_test to verify the completion of the operation > via the flag (or it can use MPI_Rput + MPI_Test). > > Is there a way for rank 99 to know that its window (its buffer) contains new data ? > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI point-to-point question
by Rajeev Thakur 20 Aug '14

20 Aug '14

Just use MPI_ANY_SOURCE and MPI_ANY_TAG in the MPI_Iprobe, so it will return the first available message. Rajeev On Aug 20, 2014, at 8:37 PM, "Boisvert, Sebastien" <boisvert(a)anl.gov> wrote: > Hi, > > When using MPI_Isend + MPI_Test + MPI_Iprobe + MPI_Recv (MPI point-to-point), > is there a way to give a hint to MPICH to tell it that the order of messages between each > pair of MPI ranks does not need to be enforced ? > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI point-to-point question
by Boisvert, Sebastien 20 Aug '14

20 Aug '14

Hi, When using MPI_Isend + MPI_Test + MPI_Iprobe + MPI_Recv (MPI point-to-point), is there a way to give a hint to MPICH to tell it that the order of messages between each pair of MPI ranks does not need to be enforced ?

1 0

MPI 3.0 RMA question
by Boisvert, Sebastien 20 Aug '14

20 Aug '14

Hi, Let's say I have 20 ranks. They all call MPI_Win_create_dynamic to create a window. Then, every rank attach a buffer with MPI_Win_attach to their window. Then, rank 3 calls MPI_Put to put, say, some data on the window of rank 99. Rank 3 can then rely on MPI_Win_test to verify the completion of the operation via the flag (or it can use MPI_Rput + MPI_Test). Is there a way for rank 99 to know that its window (its buffer) contains new data ?

1 0

Re: [mpich-discuss] BLCR restart makes processes hang
by myself 18 Aug '14

18 Aug '14

Well. Thanks! Look forward to the version which is supported with the fault tolerance. At 2014-08-18 09:58:59, "Bland, Wesley B." <wbland(a)anl.gov> wrote: Unfortunately, you’re correct that there isn’t currently a solution. Until we fix that ticket, checkpointing is currently not functioning in MPICH. It’s on the roadmap to be fixed along with some new fault tolerance features in the future, but it’s not there yet. Thanks, Wesley On Aug 18, 2014, at 8:51 AM, myself <chcdlf(a)126.com> wrote: I tried to use BLCR with MPICH3. However, it seems not to work. I compile the blcr in CentOS and `make test` show not fail tests. Then, I compile mpich with BLCR. The information is shown as follows, $ mpichversion MPICH Version: 3.1.2 MPICH Release date:Mon Jul 21 16:00:21 CDT 2014 MPICH Device: ch3:nemesis MPICH configure: --prefix=/home/test/develop/mpich3-blcr --with-device=ch3:nemesis CFLAGS=-fPIC --enable-checkpointing --with-blcr=/home/test/develop/blcr-0.8.5 --with-hydra-ckpointlib=blcr MPICH CC: gcc -fPIC -O2 MPICH CXX: g++ -O2 MPICH F77: gfortran -O2 MPICH FC: gfortran -O2 After that, I compile my application like this $ mpicc mpiblcr.c -o mpiblcr -lcr When I firstly run the application, it seems ok to make the checkpoint files, such as context-num0-0-0. $ mpiexec -ckpointlib blcr -ckpoint-prefix `pwd` -ckpoint-interval 2 -n 2 ./mpiblcr 5411) Step 0 5410) Step 0 5410) Step 1 5411) Step 1 [proxy:0:0@node1] requesting checkpoint [proxy:0:0@node1] checkpoint completed 5410) Step 2 5411) Step 2 However, when I try to restart the process with checkpoint, it hangs and thereis no information printed. $ mpiexec -ckpointlib blcr -ckpoint-prefix `pwd` -n 2 -ckpoint-num 1 The pstree shows the pmi start application process ├─sshd─┬─3*[sshd───sshd───bash] │ ├─sshd───sshd───bash───mpiexec───hydra_pmi_proxy───mpiblcr │ └─sshd───sshd───bash───pstree and `ps aux` shows the process is defunct $ ps aux | grep osu_bw test 15290 0.0 0.0 0 0 ? Z 21:44 0:00 [mpiblcr] <defunct> I don't know how to identify this problem. I also see someone had the same problem like me several years ago #1144. But, there are no solutions. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] BLCR restart makes processes hang
by Bland, Wesley B. 18 Aug '14

18 Aug '14

Unfortunately, you’re correct that there isn’t currently a solution. Until we fix that ticket, checkpointing is currently not functioning in MPICH. It’s on the roadmap to be fixed along with some new fault tolerance features in the future, but it’s not there yet. Thanks, Wesley On Aug 18, 2014, at 8:51 AM, myself <chcdlf(a)126.com<mailto:[email protected]>> wrote: I tried to use BLCR with MPICH3. However, it seems not to work. I compile the blcr in CentOS and `make test` show not fail tests. Then, I compile mpich with BLCR. The information is shown as follows, $ mpichversion MPICH Version: 3.1.2 MPICH Release date: Mon Jul 21 16:00:21 CDT 2014 MPICH Device: ch3:nemesis MPICH configure: --prefix=/home/test/develop/mpich3-blcr --with-device=ch3:nemesis CFLAGS=-fPIC --enable-checkpointing --with-blcr=/home/test/develop/blcr-0.8.5 --with-hydra-ckpointlib=blcr MPICH CC: gcc -fPIC -O2 MPICH CXX: g++ -O2 MPICH F77: gfortran -O2 MPICH FC: gfortran -O2 After that, I compile my application like this $ mpicc mpiblcr.c -o mpiblcr -lcr When I firstly run the application, it seems ok to make the checkpoint files, such as context-num0-0-0. $ mpiexec -ckpointlib blcr -ckpoint-prefix `pwd` -ckpoint-interval 2 -n 2 ./mpiblcr 5411) Step 0 5410) Step 0 5410) Step 1 5411) Step 1 [proxy:0:0@node1] requesting checkpoint [proxy:0:0@node1] checkpoint completed 5410) Step 2 5411) Step 2 However, when I try to restart the process with checkpoint, it hangs and thereis no information printed. $ mpiexec -ckpointlib blcr -ckpoint-prefix `pwd` -n 2 -ckpoint-num 1 The pstree shows the pmi start application process ├─sshd─┬─3*[sshd───sshd───bash] │ ├─sshd───sshd───bash───mpiexec───hydra_pmi_proxy───mpiblcr │ └─sshd───sshd───bash───pstree and `ps aux` shows the process is defunct $ ps aux | grep osu_bw test 15290 0.0 0.0 0 0 ? Z 21:44 0:00 [mpiblcr] <defunct> I don't know how to identify this problem. I also see someone had the same problem like me several years ago #1144<http://trac.mpich.org/projects/mpich/ticket/1144>. But, there are no solutions. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

BLCR restart makes processes hang
by myself 18 Aug '14

18 Aug '14

I tried to use BLCR with MPICH3. However, it seems not to work. I compile the blcr in CentOS and `make test` show not fail tests. Then, I compile mpich with BLCR. The information is shown as follows, $ mpichversion MPICH Version: 3.1.2 MPICH Release date:Mon Jul 21 16:00:21 CDT 2014 MPICH Device: ch3:nemesis MPICH configure: --prefix=/home/test/develop/mpich3-blcr --with-device=ch3:nemesis CFLAGS=-fPIC --enable-checkpointing --with-blcr=/home/test/develop/blcr-0.8.5 --with-hydra-ckpointlib=blcr MPICH CC: gcc -fPIC -O2 MPICH CXX: g++ -O2 MPICH F77: gfortran -O2 MPICH FC: gfortran -O2 After that, I compile my application like this $ mpicc mpiblcr.c -o mpiblcr -lcr When I firstly run the application, it seems ok to make the checkpoint files, such as context-num0-0-0. $ mpiexec -ckpointlib blcr -ckpoint-prefix `pwd` -ckpoint-interval 2 -n 2 ./mpiblcr 5411) Step 0 5410) Step 0 5410) Step 1 5411) Step 1 [proxy:0:0@node1] requesting checkpoint [proxy:0:0@node1] checkpoint completed 5410) Step 2 5411) Step 2 However, when I try to restart the process with checkpoint, it hangs and thereis no information printed. $ mpiexec -ckpointlib blcr -ckpoint-prefix `pwd` -n 2 -ckpoint-num 1 The pstree shows the pmi start application process ├─sshd─┬─3*[sshd───sshd───bash] │ ├─sshd───sshd───bash───mpiexec───hydra_pmi_proxy───mpiblcr │ └─sshd───sshd───bash───pstree and `ps aux` shows the process is defunct $ ps aux | grep osu_bw test 15290 0.0 0.0 0 0 ? Z 21:44 0:00 [mpiblcr] <defunct> I don't know how to identify this problem. I also see someone had the same problem like me several years ago #1144. But, there are no solutions.

1 0

Re: [mpich-discuss] jumpshot memory requirements
by Rob Latham 13 Aug '14

13 Aug '14

On 08/13/2014 01:30 PM, Bland, Wesley B. wrote: > I think you might have sent this message to the wrong list. This list is > for discussing MPICH. For Tau/Jumpshot questions, you should probably > contact them. Wes, there's no "jumpshot" list. Never has been. Used to be, Anthony Chan would chime in on this list... He's not around but I'll try to fill in best I can. The original poster is running into a limitation not of system memory, but of Java itself. In order to bump up the Java heap, there are some flags you can pass: -Xms<size> set initial Java heap size -Xmx<size> set maximum Java heap size -Xss<size> set java thread stack size for example -- $ JVMFLAGS="-Xms512m -Xmx2048m" ~/work/soft/mpich2/bin/jumpshot coll_perf.slog2 Though you might want to make these values even larger. It's true, Jumpshot is a memory hog. Good luck. ==rob > >> On Aug 13, 2014, at 1:18 PM, Martin, Eileen Rose <martin243(a)llnl.gov >> <mailto:[email protected]>> wrote: >> >> Hi, >> I am trying to profile some code with Tau on a single node with 12 >> threads on an X86 machine. I am able to view the PAPI counter results >> with paraprof without any problems. However, when I follow these >> directions >> (https://www.cs.uoregon.edu/research/tau/docs/old/bk01ch04s03.html) , >> then try to open the tau.slog2 file (which is 814MB) with Jumpshot to >> view the timeline, I am only able to view the main and legend windows, >> then I get a bunch of errors saying itï¿½s out of memory in the Java >> heap (see errors below). When I run Jumpshot, I am grabbing a whole >> node that has 24 GB of memory available, which I think should be >> enough. About how much memory should I expect Jumpshot to require on a >> slog2 file that is just under 1 GB? >> Thank you, >> Eileen Martin >> >> >> Hereï¿½s an example of the errors I am getting, in case you recognize >> these (I get the same errors for other threads): >> >> slog_ins.tZoomFtr = 2.0 >> Exception in thread "Thread-1" java.lang.OutOfMemoryError: Java heap space >> at base.drawable.InfoBox.resolveCategory(InfoBox.java:103) >> at logformat.slog2.BufForDrawables.readObject(BufForDrawables.java:249) >> at logformat.slog2.input.TreeNode.readObject(TreeNode.java:137) >> at logformat.slog2.input.TreeNode.<init>(TreeNode.java:130) >> at logformat.slog2.input.InputLog.readTreeNode(InputLog.java:327) >> at logformat.slog2.input.TreeTrunk.growChildren(TreeTrunk.java:138) >> at logformat.slog2.input.TreeTrunk.growChildren(TreeTrunk.java:147) >> at logformat.slog2.input.TreeTrunk.growChildren(TreeTrunk.java:147) >> at logformat.slog2.input.TreeTrunk.growInTreeWindow(TreeTrunk.java:110) >> at viewer.timelines.CanvasTimeline.<init>(CanvasTimeline.java:113) >> at viewer.timelines.TimelinePanel.<init>(TimelinePanel.java:167) >> at viewer.timelines.TimelineFrame.<init>(TimelineFrame.java:37) >> at viewer.first.LogFileOperations$1.construct(LogFileOperations.java:238) >> at viewer.common.SwingWorker$2.run(SwingWorker.java:109) >> at java.lang.Thread.run(Thread.java:682) >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-discuss] jumpshot memory requirements
by Bland, Wesley B. 13 Aug '14

13 Aug '14

I think you might have sent this message to the wrong list. This list is for discussing MPICH. For Tau/Jumpshot questions, you should probably contact them. On Aug 13, 2014, at 1:18 PM, Martin, Eileen Rose <martin243(a)llnl.gov<mailto:[email protected]>> wrote: Hi, I am trying to profile some code with Tau on a single node with 12 threads on an X86 machine. I am able to view the PAPI counter results with paraprof without any problems. However, when I follow these directions (https://www.cs.uoregon.edu/research/tau/docs/old/bk01ch04s03.html) , then try to open the tau.slog2 file (which is 814MB) with Jumpshot to view the timeline, I am only able to view the main and legend windows, then I get a bunch of errors saying it’s out of memory in the Java heap (see errors below). When I run Jumpshot, I am grabbing a whole node that has 24 GB of memory available, which I think should be enough. About how much memory should I expect Jumpshot to require on a slog2 file that is just under 1 GB? Thank you, Eileen Martin Here’s an example of the errors I am getting, in case you recognize these (I get the same errors for other threads): slog_ins.tZoomFtr = 2.0 Exception in thread "Thread-1" java.lang.OutOfMemoryError: Java heap space at base.drawable.InfoBox.resolveCategory(InfoBox.java:103) at logformat.slog2.BufForDrawables.readObject(BufForDrawables.java:249) at logformat.slog2.input.TreeNode.readObject(TreeNode.java:137) at logformat.slog2.input.TreeNode.<init>(TreeNode.java:130) at logformat.slog2.input.InputLog.readTreeNode(InputLog.java:327) at logformat.slog2.input.TreeTrunk.growChildren(TreeTrunk.java:138) at logformat.slog2.input.TreeTrunk.growChildren(TreeTrunk.java:147) at logformat.slog2.input.TreeTrunk.growChildren(TreeTrunk.java:147) at logformat.slog2.input.TreeTrunk.growInTreeWindow(TreeTrunk.java:110) at viewer.timelines.CanvasTimeline.<init>(CanvasTimeline.java:113) at viewer.timelines.TimelinePanel.<init>(TimelinePanel.java:167) at viewer.timelines.TimelineFrame.<init>(TimelineFrame.java:37) at viewer.first.LogFileOperations$1.construct(LogFileOperations.java:238) at viewer.common.SwingWorker$2.run(SwingWorker.java:109) at java.lang.Thread.run(Thread.java:682) _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

jumpshot memory requirements
by Martin, Eileen Rose 13 Aug '14

13 Aug '14

Hi, I am trying to profile some code with Tau on a single node with 12 threads on an X86 machine. I am able to view the PAPI counter results with paraprof without any problems. However, when I follow these directions (https://www.cs.uoregon.edu/research/tau/docs/old/bk01ch04s03.html) , then try to open the tau.slog2 file (which is 814MB) with Jumpshot to view the timeline, I am only able to view the main and legend windows, then I get a bunch of errors saying it’s out of memory in the Java heap (see errors below). When I run Jumpshot, I am grabbing a whole node that has 24 GB of memory available, which I think should be enough. About how much memory should I expect Jumpshot to require on a slog2 file that is just under 1 GB? Thank you, Eileen Martin Here’s an example of the errors I am getting, in case you recognize these (I get the same errors for other threads): slog_ins.tZoomFtr = 2.0 Exception in thread "Thread-1" java.lang.OutOfMemoryError: Java heap space at base.drawable.InfoBox.resolveCategory(InfoBox.java:103) at logformat.slog2.BufForDrawables.readObject(BufForDrawables.java:249) at logformat.slog2.input.TreeNode.readObject(TreeNode.java:137) at logformat.slog2.input.TreeNode.<init>(TreeNode.java:130) at logformat.slog2.input.InputLog.readTreeNode(InputLog.java:327) at logformat.slog2.input.TreeTrunk.growChildren(TreeTrunk.java:138) at logformat.slog2.input.TreeTrunk.growChildren(TreeTrunk.java:147) at logformat.slog2.input.TreeTrunk.growChildren(TreeTrunk.java:147) at logformat.slog2.input.TreeTrunk.growInTreeWindow(TreeTrunk.java:110) at viewer.timelines.CanvasTimeline.<init>(CanvasTimeline.java:113) at viewer.timelines.TimelinePanel.<init>(TimelinePanel.java:167) at viewer.timelines.TimelineFrame.<init>(TimelineFrame.java:37) at viewer.first.LogFileOperations$1.construct(LogFileOperations.java:238) at viewer.common.SwingWorker$2.run(SwingWorker.java:109) at java.lang.Thread.run(Thread.java:682)

1 0