May 2015 - Darshan-users - mailman3.cels.anl.gov

Re: [Darshan-users] LD_PRELOAD system() call
by Phil Carns 01 Jun '15

01 Jun '15

Hi Cristian, I just tried to narrow this down further again, but Edison is now running craype version 2.3.1 and the behavior seems to have changed. There is no longer a hang, and the system() call executes correctly. I get the following error messages in stderr, though (in addition to the wrapper output): Sun May 31 06:30:30 2015: [unset]:_pmi_alps_sync:alps response not OKAY Sun May 31 06:30:30 2015: [unset]:_pmiu_daemon:_pmi_alps_sync failed Sun May 31 06:30:30 2015: [PE_0]:_pmi_daemon_barrier:PE pipe read failed from daemon errno = Success Sun May 31 06:30:30 2015: [PE_0]:_pmi_init:_pmi_daemon_barrier returned -1 -Phil On 05/06/2015 04:47 PM, Carns, Philip H. wrote: > Hi Cristian, > > Here are the results from an execution on Edison, with that fprintf > added just before the existing one. I think line 260 is probably the > last valid line before the hang (the rest are messages printed after the > scheduler starts trying to kill the job). > > thanks, > -Phil > > On 05/06/2015 02:33 PM, Phil Carns wrote: >> On 05/05/2015 07:09 AM, Cristian Simarro wrote: >>> Hi Phill, >>> >>> I have tried your test in our Cray but I can not reproduce the >>> behaviour. Actually the test program is finishing properly. >> Interesting! I assume it is printing output showing that the wrappers >> were triggered, though, right? >> >> I'm attaching my job script just in case there is any difference there >> in how we are executing the test case. I think I probably compiled >> the example program with Intel compilers, while the read-wrapper >> library was compiled with GNU, though I wouldn't think that part would >> matter here. The system is running craype 2.2.1. >> >> >> >>> Can you add traceback information about the call? >>> Dl_info dli; >>> >>> original_read = dlsym(RTLD_NEXT, "read"); >>> dladdr(original_read,&dli); >>> fprintf(stderr, "debug trace [%d]: %s " >>> "called by %p [ %s(%p) %s(%p) ].\n", >>> getpid(), __func__, >>> __builtin_return_address(0), >>> strrchr(dli.dli_fname, '/') ? >>> strrchr(dli.dli_fname, '/')+1 : >>> dli.dli_fname, >>> dli.dli_fbase, dli.dli_sname, dli.dli_saddr); >> Sure, I'll give that a try and report back. Thanks for the example; I >> was thinking that a backtrace might be very helpful but I wasn't sure >> how to best go about collecting it :) >> >> thanks, >> -Phil >> >>> Best regards, >>> Cristian >>> >>> ------------------------------------------------------------------ >>> Cristian Simarro >>> Analyst, User Support Section >>> European Centre for Medium-Range Weather Forecasts (ECMWF) >>> Shinfield Park, Reading, RG2 9AX, United Kingdom >>> Tel: (+44 118) 9499315 Fax: (+44 118) 9869450 >>> E-mail: Cristian.Simarro(a)ecmwf.int http://www.ecmwf.int >>> ------------------------------------------------------------------ >>> >>> ----- Original Message ----- >>> From: "Phil Carns" <carns(a)mcs.anl.gov> >>> To: darshan-users(a)lists.mcs.anl.gov >>> Sent: Sunday, 3 May, 2015 3:35:46 PM >>> Subject: Re: [Darshan-users] LD_PRELOAD system() call >>> >>> Hi Cristian, >>> >>> This is definitely the same problem that Kalyana and I were looking at >>> earlier with fork(). I built a very small example reproducer library to >>> try to simplify the problem (see Makefile and read-wrapper.c). The >>> read-wrapper.c isn't doing anything except intercepting the read() >>> function, printing some information, then calling the real read() >>> function. I've been building this library with PrgEnv-gnu. >>> >>> The test.c is your example program, and the >>> test-preload-read-wrapper.pbs.e* is an example stderr file from trying >>> to run it with the example read wrapper library preloaded. >>> >>> There isn't any Darshan code involved here, but the example still >>> hangs. It looks like you could trigger it with *any* wrapper on the >>> read() function in the Cray environment in conjunction with a fork() or >>> system() call. Maybe there is some sort of recursion here? >>> >>> I'll keep thinking about this some, but I thought I would share what I'm >>> seeing with the list in case anyone else has an idea. >>> >>> thanks, >>> -Phil >>> >>> On 05/01/2015 12:06 PM, Carns, Philip H. wrote: >>>> Hi Cristian, >>>> >>>> I was testing on Edison, an XC30 system at NERSC. I compiled with >>>> cray-mpich 7.1.1, and I think it is using Torque as the batch system. >>>> FYI, to run this example program I have to launch the executable using >>>> aprun (otherwise MPI won't initialize properly). I think this will be >>>> reproducible with non-MPI programs as well, though. >>>> >>>> thanks, >>>> -Phil >>>> >>>> On 05/01/2015 04:53 AM, Cristian Simarro wrote: >>>>> Hi Phil, >>>>> >>>>> Could you please tell me the batch system that you are using in >>>>> your Cray machine? Is the MPI implementation cray-mpich? >>>>> >>>>> Thanks, >>>>> Cristian >>>>> >>>>> ----- Original Message ----- >>>>> From: "Phil Carns" <carns(a)mcs.anl.gov> >>>>> To: "Cristian Simarro" <cristian.simarro(a)ecmwf.int> >>>>> Cc: darshan-users(a)lists.mcs.anl.gov >>>>> Sent: Thursday, 30 April, 2015 8:45:11 PM >>>>> Subject: Re: [Darshan-users] LD_PRELOAD system() call >>>>> >>>>> Thanks for the test program, Cristian. I can confirm that it hangs >>>>> with >>>>> LD_PRELOAD on a Cray, but not on a Linux workstation. I'm not exactly >>>>> sure what the underlying difference is in this case, but it is >>>>> definitely 100% reproducible in the Cray environment. >>>>> >>>>> Kalyana Chadalavada has actually observed something very similar when >>>>> using fork() directly; I imagine that it is the underlying fork() >>>>> within >>>>> the system() call that is causing the problem. >>>>> >>>>> thanks, >>>>> -Phil >>>>> >>>>> On 04/30/2015 03:07 AM, Cristian Simarro wrote: >>>>>> Hi Phill, >>>>>> >>>>>> Actually any command under system() call is triggering the >>>>>> problem. The spawned process do not finish and then the task that >>>>>> has issued the call is hung on the waitpid. >>>>>> >>>>>> This example hangs if we are using LD_PRELOAD mechanism: >>>>>> >>>>>> #include <stdio.h> >>>>>> #include <mpi.h> >>>>>> #include <stdlib.h> >>>>>> >>>>>> int main (int argc, char *argv[]) >>>>>> { >>>>>> int rank, size; >>>>>> int ret; >>>>>> >>>>>> MPI_Init (&argc, &argv); >>>>>> MPI_Comm_rank (MPI_COMM_WORLD, &rank); >>>>>> MPI_Comm_size (MPI_COMM_WORLD, &size); >>>>>> if(rank == 0) { >>>>>> ret = system("echo calling system"); >>>>>> } >>>>>> printf( "Hello world from process %d of %d\n", rank, size ); >>>>>> MPI_Finalize(); >>>>>> return 0; >>>>>> } >>>>>> >>>>>> Thanks, >>>>>> Cristian >>>>>> >>>>>> ------------------------------------------------------------------ >>>>>> Cristian Simarro >>>>>> Analyst, User Support Section >>>>>> European Centre for Medium-Range Weather Forecasts (ECMWF) >>>>>> Shinfield Park, Reading, RG2 9AX, United Kingdom >>>>>> Tel: (+44 118) 9499315 Fax: (+44 118) 9869450 >>>>>> E-mail: Cristian.Simarro(a)ecmwf.int http://www.ecmwf.int >>>>>> ------------------------------------------------------------------ >>>>>> >>>>>> ----- Original Message ----- >>>>>> From: "Phil Carns" <carns(a)mcs.anl.gov> >>>>>> To: darshan-users(a)lists.mcs.anl.gov >>>>>> Sent: Wednesday, 29 April, 2015 10:13:54 PM >>>>>> Subject: Re: [Darshan-users] LD_PRELOAD system() call >>>>>> >>>>>> On 04/29/2015 02:54 PM, Phil Carns wrote: >>>>>>> On 04/29/2015 12:17 PM, Cristian Simarro wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> We have been facing some problems with system() call inside some >>>>>>>> C/Fortran codes in our Cray machine. >>>>>>>> >>>>>>>> The method used here is compile dynamically and then use >>>>>>>> LD_PRELOAD. >>>>>>>> When the code calls system(command), it hangs the execution if >>>>>>>> preloaded with Darshan because it is trying to instrument an >>>>>>>> internal >>>>>>>> system read() with no initialization. >>>>>>>> >>>>>>>> The solution we have designed is to unset LD_PRELOAD (if set >>>>>>>> before) >>>>>>>> in the darshan_mpi_initialize function. >>>>>>>> >>>>>>>> Has anybody found a similar problem with LD_PRELOAD + system() >>>>>>>> calls? >>>>>>> Hi Cristian, >>>>>>> >>>>>>> I don't think I've seen this exact combination before, but it seems >>>>>>> like something we should be able to reproduce and isolate. >>>>>>> >>>>>>> If I understand correctly, it sounds like the underlying process >>>>>>> spawned by system() is inheriting the LD_PRELOAD environment >>>>>>> variable >>>>>>> from the parent program, and it is the underlying process that is >>>>>>> getting hung? If so, does it matter what you run in the system() >>>>>>> call >>>>>>> or does it seem like pretty anything triggers it? >>>>>>> >>>>>>> thanks, >>>>>>> -Phil >>>>>> The solution you have suggested (unsetting LD_PRELOAD grammatically >>>>>> during Darshan initialization) might not be a bad long term solution, >>>>>> maybe with some extra safety logic to make sure we don't accidentally >>>>>> unset unrelated LD_PRELOAD entries. I imagine that once the >>>>>> application >>>>>> has gotten to darshan initialization, then the loader has already >>>>>> processed the LD_PRELOAD environment variable and we don't need to >>>>>> keep >>>>>> it set any longer. That would help keep it from interfering with >>>>>> child >>>>>> processes. >>>>>> >>>>>> We would definitely need to do some testing to confirm, though. >>>>>> >>>>>> thanks, >>>>>> -Phil >>>>>> _______________________________________________ >>>>>> Darshan-users mailing list >>>>>> Darshan-users(a)lists.mcs.anl.gov >>>>>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users >>>> _______________________________________________ >>>> Darshan-users mailing list >>>> Darshan-users(a)lists.mcs.anl.gov >>>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users >>> _______________________________________________ >>> Darshan-users mailing list >>> Darshan-users(a)lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users

2 1

Re: [Darshan-users] Darshan error with HDF5
by Phil Carns 28 May '15

28 May '15

Hi Adnan! Is everything in the application statically linked (maybe check with ldd <executable>)? That error message is supposed to only be generated by the dynamic (ld-preloaded) darshan library, because it has to locate underlying symbols at run time. The symbol name mapping should be set at link time for a static executable. thanks, -Phil On 05/28/2015 04:20 PM, Adnan Haider wrote: > Hi, > I am hoping to use Darshan for analyzing our application's > performance. We are using HDF5 and netcdf 4. I installed Darshan > version 2.3.1 and mvapich2 for our mpi implementation. Darshan works > fine for a standard mpi pingpong application. When we run our > application, however, it gives the following error: > Darshan failed to map symbol: H5Fopen > Darshan failed to map symbol: H5Fopen > Darshan failed to map symbol: H5Fopen > Darshan failed to map symbol: H5Fopen > > The four instances of the error probably have to do with the 4 mpi > ranks specified at runtime. Nothing is written into the log directory. > This is the only output, unless there is some error file I am not > aware of. I used the static instrumentation approach for enabling > Darshan. Thanks for the help! > > Thanks, Adnan Haider > B.S Candidate, Computer Science > Illinois Institute of Technology

1 0

Darshan error with HDF5
by Adnan Haider 28 May '15

28 May '15

Hi, I am hoping to use Darshan for analyzing our application's performance. We are using HDF5 and netcdf 4. I installed Darshan version 2.3.1 and mvapich2 for our mpi implementation. Darshan works fine for a standard mpi pingpong application. When we run our application, however, it gives the following error: Darshan failed to map symbol: H5Fopen Darshan failed to map symbol: H5Fopen Darshan failed to map symbol: H5Fopen Darshan failed to map symbol: H5Fopen The four instances of the error probably have to do with the 4 mpi ranks specified at runtime. Nothing is written into the log directory. This is the only output, unless there is some error file I am not aware of. I used the static instrumentation approach for enabling Darshan. Thanks for the help! Thanks, Adnan Haider B.S Candidate, Computer Science Illinois Institute of Technology

1 0

Re: [Darshan-users] LD_PRELOAD system() call
by Phil Carns 06 May '15

06 May '15

On 05/05/2015 07:09 AM, Cristian Simarro wrote: > Hi Phill, > > I have tried your test in our Cray but I can not reproduce the behaviour. Actually the test program is finishing properly. Interesting! I assume it is printing output showing that the wrappers were triggered, though, right? I'm attaching my job script just in case there is any difference there in how we are executing the test case. I think I probably compiled the example program with Intel compilers, while the read-wrapper library was compiled with GNU, though I wouldn't think that part would matter here. The system is running craype 2.2.1. > Can you add traceback information about the call? > > Dl_info dli; > > original_read = dlsym(RTLD_NEXT, "read"); > dladdr(original_read,&dli); > fprintf(stderr, "debug trace [%d]: %s " > "called by %p [ %s(%p) %s(%p) ].\n", > getpid(), __func__, > __builtin_return_address(0), > strrchr(dli.dli_fname, '/') ? > strrchr(dli.dli_fname, '/')+1 : dli.dli_fname, > dli.dli_fbase, dli.dli_sname, dli.dli_saddr); Sure, I'll give that a try and report back. Thanks for the example; I was thinking that a backtrace might be very helpful but I wasn't sure how to best go about collecting it :) thanks, -Phil > Best regards, > Cristian > > ------------------------------------------------------------------ > Cristian Simarro > Analyst, User Support Section > European Centre for Medium-Range Weather Forecasts (ECMWF) > Shinfield Park, Reading, RG2 9AX, United Kingdom > Tel: (+44 118) 9499315 Fax: (+44 118) 9869450 > E-mail: Cristian.Simarro(a)ecmwf.int http://www.ecmwf.int > ------------------------------------------------------------------ > > ----- Original Message ----- > From: "Phil Carns" <carns(a)mcs.anl.gov> > To: darshan-users(a)lists.mcs.anl.gov > Sent: Sunday, 3 May, 2015 3:35:46 PM > Subject: Re: [Darshan-users] LD_PRELOAD system() call > > Hi Cristian, > > This is definitely the same problem that Kalyana and I were looking at > earlier with fork(). I built a very small example reproducer library to > try to simplify the problem (see Makefile and read-wrapper.c). The > read-wrapper.c isn't doing anything except intercepting the read() > function, printing some information, then calling the real read() > function. I've been building this library with PrgEnv-gnu. > > The test.c is your example program, and the > test-preload-read-wrapper.pbs.e* is an example stderr file from trying > to run it with the example read wrapper library preloaded. > > There isn't any Darshan code involved here, but the example still > hangs. It looks like you could trigger it with *any* wrapper on the > read() function in the Cray environment in conjunction with a fork() or > system() call. Maybe there is some sort of recursion here? > > I'll keep thinking about this some, but I thought I would share what I'm > seeing with the list in case anyone else has an idea. > > thanks, > -Phil > > On 05/01/2015 12:06 PM, Carns, Philip H. wrote: >> Hi Cristian, >> >> I was testing on Edison, an XC30 system at NERSC. I compiled with >> cray-mpich 7.1.1, and I think it is using Torque as the batch system. >> FYI, to run this example program I have to launch the executable using >> aprun (otherwise MPI won't initialize properly). I think this will be >> reproducible with non-MPI programs as well, though. >> >> thanks, >> -Phil >> >> On 05/01/2015 04:53 AM, Cristian Simarro wrote: >>> Hi Phil, >>> >>> Could you please tell me the batch system that you are using in your Cray machine? Is the MPI implementation cray-mpich? >>> >>> Thanks, >>> Cristian >>> >>> ----- Original Message ----- >>> From: "Phil Carns" <carns(a)mcs.anl.gov> >>> To: "Cristian Simarro" <cristian.simarro(a)ecmwf.int> >>> Cc: darshan-users(a)lists.mcs.anl.gov >>> Sent: Thursday, 30 April, 2015 8:45:11 PM >>> Subject: Re: [Darshan-users] LD_PRELOAD system() call >>> >>> Thanks for the test program, Cristian. I can confirm that it hangs with >>> LD_PRELOAD on a Cray, but not on a Linux workstation. I'm not exactly >>> sure what the underlying difference is in this case, but it is >>> definitely 100% reproducible in the Cray environment. >>> >>> Kalyana Chadalavada has actually observed something very similar when >>> using fork() directly; I imagine that it is the underlying fork() within >>> the system() call that is causing the problem. >>> >>> thanks, >>> -Phil >>> >>> On 04/30/2015 03:07 AM, Cristian Simarro wrote: >>>> Hi Phill, >>>> >>>> Actually any command under system() call is triggering the problem. The spawned process do not finish and then the task that has issued the call is hung on the waitpid. >>>> >>>> This example hangs if we are using LD_PRELOAD mechanism: >>>> >>>> #include <stdio.h> >>>> #include <mpi.h> >>>> #include <stdlib.h> >>>> >>>> int main (int argc, char *argv[]) >>>> { >>>> int rank, size; >>>> int ret; >>>> >>>> MPI_Init (&argc, &argv); >>>> MPI_Comm_rank (MPI_COMM_WORLD, &rank); >>>> MPI_Comm_size (MPI_COMM_WORLD, &size); >>>> if(rank == 0) { >>>> ret = system("echo calling system"); >>>> } >>>> printf( "Hello world from process %d of %d\n", rank, size ); >>>> MPI_Finalize(); >>>> return 0; >>>> } >>>> >>>> Thanks, >>>> Cristian >>>> >>>> ------------------------------------------------------------------ >>>> Cristian Simarro >>>> Analyst, User Support Section >>>> European Centre for Medium-Range Weather Forecasts (ECMWF) >>>> Shinfield Park, Reading, RG2 9AX, United Kingdom >>>> Tel: (+44 118) 9499315 Fax: (+44 118) 9869450 >>>> E-mail: Cristian.Simarro(a)ecmwf.int http://www.ecmwf.int >>>> ------------------------------------------------------------------ >>>> >>>> ----- Original Message ----- >>>> From: "Phil Carns" <carns(a)mcs.anl.gov> >>>> To: darshan-users(a)lists.mcs.anl.gov >>>> Sent: Wednesday, 29 April, 2015 10:13:54 PM >>>> Subject: Re: [Darshan-users] LD_PRELOAD system() call >>>> >>>> On 04/29/2015 02:54 PM, Phil Carns wrote: >>>>> On 04/29/2015 12:17 PM, Cristian Simarro wrote: >>>>>> Hello, >>>>>> >>>>>> We have been facing some problems with system() call inside some >>>>>> C/Fortran codes in our Cray machine. >>>>>> >>>>>> The method used here is compile dynamically and then use LD_PRELOAD. >>>>>> When the code calls system(command), it hangs the execution if >>>>>> preloaded with Darshan because it is trying to instrument an internal >>>>>> system read() with no initialization. >>>>>> >>>>>> The solution we have designed is to unset LD_PRELOAD (if set before) >>>>>> in the darshan_mpi_initialize function. >>>>>> >>>>>> Has anybody found a similar problem with LD_PRELOAD + system() calls? >>>>> Hi Cristian, >>>>> >>>>> I don't think I've seen this exact combination before, but it seems >>>>> like something we should be able to reproduce and isolate. >>>>> >>>>> If I understand correctly, it sounds like the underlying process >>>>> spawned by system() is inheriting the LD_PRELOAD environment variable >>>>> from the parent program, and it is the underlying process that is >>>>> getting hung? If so, does it matter what you run in the system() call >>>>> or does it seem like pretty anything triggers it? >>>>> >>>>> thanks, >>>>> -Phil >>>> The solution you have suggested (unsetting LD_PRELOAD grammatically >>>> during Darshan initialization) might not be a bad long term solution, >>>> maybe with some extra safety logic to make sure we don't accidentally >>>> unset unrelated LD_PRELOAD entries. I imagine that once the application >>>> has gotten to darshan initialization, then the loader has already >>>> processed the LD_PRELOAD environment variable and we don't need to keep >>>> it set any longer. That would help keep it from interfering with child >>>> processes. >>>> >>>> We would definitely need to do some testing to confirm, though. >>>> >>>> thanks, >>>> -Phil >>>> _______________________________________________ >>>> Darshan-users mailing list >>>> Darshan-users(a)lists.mcs.anl.gov >>>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users >> _______________________________________________ >> Darshan-users mailing list >> Darshan-users(a)lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users > > _______________________________________________ > Darshan-users mailing list > Darshan-users(a)lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users

1 1

Re: [Darshan-users] LD_PRELOAD system() call
by Phil Carns 05 May '15

05 May '15

Hi Cristian, This is definitely the same problem that Kalyana and I were looking at earlier with fork(). I built a very small example reproducer library to try to simplify the problem (see Makefile and read-wrapper.c). The read-wrapper.c isn't doing anything except intercepting the read() function, printing some information, then calling the real read() function. I've been building this library with PrgEnv-gnu. The test.c is your example program, and the test-preload-read-wrapper.pbs.e* is an example stderr file from trying to run it with the example read wrapper library preloaded. There isn't any Darshan code involved here, but the example still hangs. It looks like you could trigger it with *any* wrapper on the read() function in the Cray environment in conjunction with a fork() or system() call. Maybe there is some sort of recursion here? I'll keep thinking about this some, but I thought I would share what I'm seeing with the list in case anyone else has an idea. thanks, -Phil On 05/01/2015 12:06 PM, Carns, Philip H. wrote: > Hi Cristian, > > I was testing on Edison, an XC30 system at NERSC. I compiled with > cray-mpich 7.1.1, and I think it is using Torque as the batch system. > FYI, to run this example program I have to launch the executable using > aprun (otherwise MPI won't initialize properly). I think this will be > reproducible with non-MPI programs as well, though. > > thanks, > -Phil > > On 05/01/2015 04:53 AM, Cristian Simarro wrote: >> Hi Phil, >> >> Could you please tell me the batch system that you are using in your Cray machine? Is the MPI implementation cray-mpich? >> >> Thanks, >> Cristian >> >> ----- Original Message ----- >> From: "Phil Carns" <carns(a)mcs.anl.gov> >> To: "Cristian Simarro" <cristian.simarro(a)ecmwf.int> >> Cc: darshan-users(a)lists.mcs.anl.gov >> Sent: Thursday, 30 April, 2015 8:45:11 PM >> Subject: Re: [Darshan-users] LD_PRELOAD system() call >> >> Thanks for the test program, Cristian. I can confirm that it hangs with >> LD_PRELOAD on a Cray, but not on a Linux workstation. I'm not exactly >> sure what the underlying difference is in this case, but it is >> definitely 100% reproducible in the Cray environment. >> >> Kalyana Chadalavada has actually observed something very similar when >> using fork() directly; I imagine that it is the underlying fork() within >> the system() call that is causing the problem. >> >> thanks, >> -Phil >> >> On 04/30/2015 03:07 AM, Cristian Simarro wrote: >>> Hi Phill, >>> >>> Actually any command under system() call is triggering the problem. The spawned process do not finish and then the task that has issued the call is hung on the waitpid. >>> >>> This example hangs if we are using LD_PRELOAD mechanism: >>> >>> #include <stdio.h> >>> #include <mpi.h> >>> #include <stdlib.h> >>> >>> int main (int argc, char *argv[]) >>> { >>> int rank, size; >>> int ret; >>> >>> MPI_Init (&argc, &argv); >>> MPI_Comm_rank (MPI_COMM_WORLD, &rank); >>> MPI_Comm_size (MPI_COMM_WORLD, &size); >>> if(rank == 0) { >>> ret = system("echo calling system"); >>> } >>> printf( "Hello world from process %d of %d\n", rank, size ); >>> MPI_Finalize(); >>> return 0; >>> } >>> >>> Thanks, >>> Cristian >>> >>> ------------------------------------------------------------------ >>> Cristian Simarro >>> Analyst, User Support Section >>> European Centre for Medium-Range Weather Forecasts (ECMWF) >>> Shinfield Park, Reading, RG2 9AX, United Kingdom >>> Tel: (+44 118) 9499315 Fax: (+44 118) 9869450 >>> E-mail: Cristian.Simarro(a)ecmwf.int http://www.ecmwf.int >>> ------------------------------------------------------------------ >>> >>> ----- Original Message ----- >>> From: "Phil Carns" <carns(a)mcs.anl.gov> >>> To: darshan-users(a)lists.mcs.anl.gov >>> Sent: Wednesday, 29 April, 2015 10:13:54 PM >>> Subject: Re: [Darshan-users] LD_PRELOAD system() call >>> >>> On 04/29/2015 02:54 PM, Phil Carns wrote: >>>> On 04/29/2015 12:17 PM, Cristian Simarro wrote: >>>>> Hello, >>>>> >>>>> We have been facing some problems with system() call inside some >>>>> C/Fortran codes in our Cray machine. >>>>> >>>>> The method used here is compile dynamically and then use LD_PRELOAD. >>>>> When the code calls system(command), it hangs the execution if >>>>> preloaded with Darshan because it is trying to instrument an internal >>>>> system read() with no initialization. >>>>> >>>>> The solution we have designed is to unset LD_PRELOAD (if set before) >>>>> in the darshan_mpi_initialize function. >>>>> >>>>> Has anybody found a similar problem with LD_PRELOAD + system() calls? >>>> Hi Cristian, >>>> >>>> I don't think I've seen this exact combination before, but it seems >>>> like something we should be able to reproduce and isolate. >>>> >>>> If I understand correctly, it sounds like the underlying process >>>> spawned by system() is inheriting the LD_PRELOAD environment variable >>>> from the parent program, and it is the underlying process that is >>>> getting hung? If so, does it matter what you run in the system() call >>>> or does it seem like pretty anything triggers it? >>>> >>>> thanks, >>>> -Phil >>> The solution you have suggested (unsetting LD_PRELOAD grammatically >>> during Darshan initialization) might not be a bad long term solution, >>> maybe with some extra safety logic to make sure we don't accidentally >>> unset unrelated LD_PRELOAD entries. I imagine that once the application >>> has gotten to darshan initialization, then the loader has already >>> processed the LD_PRELOAD environment variable and we don't need to keep >>> it set any longer. That would help keep it from interfering with child >>> processes. >>> >>> We would definitely need to do some testing to confirm, though. >>> >>> thanks, >>> -Phil >>> _______________________________________________ >>> Darshan-users mailing list >>> Darshan-users(a)lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users > _______________________________________________ > Darshan-users mailing list > Darshan-users(a)lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users

2 1

Opening at Livermore National Laboratory for I/O SPecialist
by Hedges, Richard M. 04 May '15

04 May '15

Hello Darshan Users community, We have a posting for an I/O spevialist at LLNL (see link or description below). I think that this is intended as an early career type position, and the person would work closely with me and with our local file system development team. Also, please feel free to forward to any of your contacts. I would be glad to answer questions should there be a need for that. Thanks, - Richard ==================================================== Richard Hedges Customer Support and Test - File Systems Project Development Environment Group - Livermore Computing Lawrence Livermore National Laboratory 7000 East Avenue, MS L-557 Livermore, CA 94551 v: (925) 423-2699 f: (925) 423-6961 E: richard-hedges(a)llnl.gov<mailto:[email protected]> https://careers-prd.llnl.gov/psc/careers/EMPLOYEE/HRMS/s/WEBLIB_LL.VIEW_JOB… Job Description Job Title: System Software Developer Science & Engineering MTS 2 (SES.2) / Science & Engineering MTS 3 (SES.3) Job Code(s): Job ID: 12979 ________________________________ Return to Previous Page ________________________________ NOTE: This is a Career Indefinite position. For important information about LLNL appointments refer to the FAQs above. Lab employees and external candidates may be considered for this position. NATURE AND SCOPE OF JOB As a member of the Livermore Computing (LC) High Performance Computer Center, the successful candidate will work as a Software Developer supporting LC and application teams that use LC’s clusters. The candidate will interface directly with application teams in order to improve application I/O performance using existing I/O characterization tools and creating new tools as needed. The primary duty of the candidate will be to understand and improve application access patterns to LLNL file system and storage resources. Increasingly, the performance of applications and services running at LC depends on the efficiency of application I/O access patterns. Poor I/O access patterns can lead to inefficient application execution and can impact the productivity of unrelated applications in the Center. The successful candidate will analyze cluster and file system utilization to target those applications which would best benefit from an investment in I/O improvement. The candidate will identify target application development teams, analyze their I/O patterns, and work with those teams to improve I/O efficiency. The successful candidate will become a focal point of application I/O analysis, improvement, and expertise in the LC. Livermore Computing is one of the largest supercomputer Centers in the world. Its resources include the world's largest Linux clusters as well as Sequoia, a 20 PetaFLOP/s IBM system. The working environment is challenging and team-oriented. The successful candidate will interact with LC researchers, system administrators, system programmers, application developers and external colleagues. The position will involve occasional travel for meetings, training, and conferences. Candidate will report to the Development Environment Group Leader. Further information about LC can be found on the web at: https://computing.llnl.gov IN ADDITION AT THE SES.3 LEVEL Will provide I/O and software expertise to achieve novel improvements and enhancements to existing applications and software. Candidate will apply advanced knowledge to independently contribute to technical projects using creativity and imagination. The work may not be not clearly defined; the candidate is expected to be self-directed and show initiative in scoping out work and proposing solutions. ESSENTIAL DUTIES - Provide I/O analysis and code development expertise to application teams spanning LC’s user community. - Work closely and effectively with application teams to improve overall code performance. - Identify high Return On Investment (ROI) target applications through analysis of Center I/O patterns. - Provide and support I/O related code development tools and libraries for HPC from open source, vendors, academia, and other labs on all LC production platforms, including BlueGene/Q and Linux Clusters, and to prepare for future architectures. - Perform research and development activities on new parallel I/O tools, and enhancements to existing I/O tools with other DOE laboratories, academia, and industry. - Identify user I/O requirements and evaluate I/O tool effectiveness in collaboration with the application development user community. - Advise and give presentations to the users and customers of LC on I/O performance and best practices. - Perform all assignments in accordance with ES&H, Security, and business practice requirements and policies. IN ADDITION AT THE SES.3 LEVEL - Troubleshoot and resolve complex software issues which may also involve interfacing with various LC technical staff with different areas of expertise. - Independently research, investigate and recommend new techniques, tools, and technologies for system- and application-level software. - Design and develop software to solve complex systems-level problems, and provide expanded feature sets. - Develop funding proposals to improve existing I/O HPC development environment tools and technologies, or to research and develop new I/O HPC technologies. ESSENTIAL SKILLS, KNOWLEDGE, AND ABILITIES - BS in computer science, or related field, or equivalent level of demonstrated knowledge. - Comprehensive knowledge of programming and the I/O aspects of programming. - Experience with UNIX/Linux and standard compiled programming languages. - Basic knowledge of parallel programming models and techniques. - Demonstrated ability to work effectively under limited direction in a dynamic environment. - Demonstrated effective analytical and problem-solving skills to contribute to creative solutions to complex problems. - Proficient verbal and written communication skills and interpersonal skills necessary to work effectively with system administrators, system programmers, application developers and to represent LLNL in public forums (open source projects and technical conferences). - Ability to travel for meetings, training, and conferences. IN ADDITION AT THE SES.3 LEVEL - Significant experience writing I/O packages or routines. - Demonstrated advanced code development skills applied to I/O, HPC tools, and libraries. - Advanced knowledge of HPC code development tools and infrastructure components. - Significant experience developing robust and scalable software infrastructure targeting debugging and performance analysis. - Demonstrated ability to work effectively with independence and minimal direction in a dynamic environment with competing priorities. - Significant experience providing innovative solutions to broadly defined tasks and problems. - Demonstrated effective analytical and problem-solving skills for solving systems software problems. - Advanced verbal and written communication and interpersonal skills necessary to work effectively in a technical team environment. DESIRED SKILLS, KNOWLEDGE, AND ABILITIES - MS or PhD in Computer Science or related field. - Experience designing and implementing complex system software for production use. - Experience analyzing and improving application I/O. - Significant knowledge and experience with distributed file systems. - Experience with support of scientific applications programming. - Experience in a high performance computing environments. - Experience with HPC performance measurement tools. SPECIAL REQUIREMENTS Pre-Placement Medical Exam: None required. Pre-Employment Drug Test: External applicant(s) selected for this position will be required to pass a post-offer, pre-employment drug test. Anticipated Clearance Level: Q (Position will be cleared to this level). Applicants selected will be subject to a Federal background investigation and must meet eligibility requirements for access to classified information or matter. In addition, all L or Q cleared employees are subject to random drug testing. If you hold multiple citizenships (U.S. and another country), you may be required to renounce your non-U.S. citizenship before a DOE L or Q clearance will be processed/granted. LLNL is an affirmative action/ equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, marital status, national origin, ancestry, sex, sexual orientation, disability, medical condition, protected veteran status, age, citizenship, or any other characteristic protected by law.

1 0

Re: [Darshan-users] LD_PRELOAD system() call
by Phil Carns 01 May '15

01 May '15

Hi Cristian, I was testing on Edison, an XC30 system at NERSC. I compiled with cray-mpich 7.1.1, and I think it is using Torque as the batch system. FYI, to run this example program I have to launch the executable using aprun (otherwise MPI won't initialize properly). I think this will be reproducible with non-MPI programs as well, though. thanks, -Phil On 05/01/2015 04:53 AM, Cristian Simarro wrote: > Hi Phil, > > Could you please tell me the batch system that you are using in your Cray machine? Is the MPI implementation cray-mpich? > > Thanks, > Cristian > > ----- Original Message ----- > From: "Phil Carns" <carns(a)mcs.anl.gov> > To: "Cristian Simarro" <cristian.simarro(a)ecmwf.int> > Cc: darshan-users(a)lists.mcs.anl.gov > Sent: Thursday, 30 April, 2015 8:45:11 PM > Subject: Re: [Darshan-users] LD_PRELOAD system() call > > Thanks for the test program, Cristian. I can confirm that it hangs with > LD_PRELOAD on a Cray, but not on a Linux workstation. I'm not exactly > sure what the underlying difference is in this case, but it is > definitely 100% reproducible in the Cray environment. > > Kalyana Chadalavada has actually observed something very similar when > using fork() directly; I imagine that it is the underlying fork() within > the system() call that is causing the problem. > > thanks, > -Phil > > On 04/30/2015 03:07 AM, Cristian Simarro wrote: >> Hi Phill, >> >> Actually any command under system() call is triggering the problem. The spawned process do not finish and then the task that has issued the call is hung on the waitpid. >> >> This example hangs if we are using LD_PRELOAD mechanism: >> >> #include <stdio.h> >> #include <mpi.h> >> #include <stdlib.h> >> >> int main (int argc, char *argv[]) >> { >> int rank, size; >> int ret; >> >> MPI_Init (&argc, &argv); >> MPI_Comm_rank (MPI_COMM_WORLD, &rank); >> MPI_Comm_size (MPI_COMM_WORLD, &size); >> if(rank == 0) { >> ret = system("echo calling system"); >> } >> printf( "Hello world from process %d of %d\n", rank, size ); >> MPI_Finalize(); >> return 0; >> } >> >> Thanks, >> Cristian >> >> ------------------------------------------------------------------ >> Cristian Simarro >> Analyst, User Support Section >> European Centre for Medium-Range Weather Forecasts (ECMWF) >> Shinfield Park, Reading, RG2 9AX, United Kingdom >> Tel: (+44 118) 9499315 Fax: (+44 118) 9869450 >> E-mail: Cristian.Simarro(a)ecmwf.int http://www.ecmwf.int >> ------------------------------------------------------------------ >> >> ----- Original Message ----- >> From: "Phil Carns" <carns(a)mcs.anl.gov> >> To: darshan-users(a)lists.mcs.anl.gov >> Sent: Wednesday, 29 April, 2015 10:13:54 PM >> Subject: Re: [Darshan-users] LD_PRELOAD system() call >> >> On 04/29/2015 02:54 PM, Phil Carns wrote: >>> On 04/29/2015 12:17 PM, Cristian Simarro wrote: >>>> Hello, >>>> >>>> We have been facing some problems with system() call inside some >>>> C/Fortran codes in our Cray machine. >>>> >>>> The method used here is compile dynamically and then use LD_PRELOAD. >>>> When the code calls system(command), it hangs the execution if >>>> preloaded with Darshan because it is trying to instrument an internal >>>> system read() with no initialization. >>>> >>>> The solution we have designed is to unset LD_PRELOAD (if set before) >>>> in the darshan_mpi_initialize function. >>>> >>>> Has anybody found a similar problem with LD_PRELOAD + system() calls? >>> Hi Cristian, >>> >>> I don't think I've seen this exact combination before, but it seems >>> like something we should be able to reproduce and isolate. >>> >>> If I understand correctly, it sounds like the underlying process >>> spawned by system() is inheriting the LD_PRELOAD environment variable >>> from the parent program, and it is the underlying process that is >>> getting hung? If so, does it matter what you run in the system() call >>> or does it seem like pretty anything triggers it? >>> >>> thanks, >>> -Phil >> The solution you have suggested (unsetting LD_PRELOAD grammatically >> during Darshan initialization) might not be a bad long term solution, >> maybe with some extra safety logic to make sure we don't accidentally >> unset unrelated LD_PRELOAD entries. I imagine that once the application >> has gotten to darshan initialization, then the loader has already >> processed the LD_PRELOAD environment variable and we don't need to keep >> it set any longer. That would help keep it from interfering with child >> processes. >> >> We would definitely need to do some testing to confirm, though. >> >> thanks, >> -Phil >> _______________________________________________ >> Darshan-users mailing list >> Darshan-users(a)lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users

1 0

Re: [Darshan-users] LD_PRELOAD system() call
by Phil Carns 01 May '15

01 May '15

Thanks for the test program, Cristian. I can confirm that it hangs with LD_PRELOAD on a Cray, but not on a Linux workstation. I'm not exactly sure what the underlying difference is in this case, but it is definitely 100% reproducible in the Cray environment. Kalyana Chadalavada has actually observed something very similar when using fork() directly; I imagine that it is the underlying fork() within the system() call that is causing the problem. thanks, -Phil On 04/30/2015 03:07 AM, Cristian Simarro wrote: > Hi Phill, > > Actually any command under system() call is triggering the problem. The spawned process do not finish and then the task that has issued the call is hung on the waitpid. > > This example hangs if we are using LD_PRELOAD mechanism: > > #include <stdio.h> > #include <mpi.h> > #include <stdlib.h> > > int main (int argc, char *argv[]) > { > int rank, size; > int ret; > > MPI_Init (&argc, &argv); > MPI_Comm_rank (MPI_COMM_WORLD, &rank); > MPI_Comm_size (MPI_COMM_WORLD, &size); > if(rank == 0) { > ret = system("echo calling system"); > } > printf( "Hello world from process %d of %d\n", rank, size ); > MPI_Finalize(); > return 0; > } > > Thanks, > Cristian > > ------------------------------------------------------------------ > Cristian Simarro > Analyst, User Support Section > European Centre for Medium-Range Weather Forecasts (ECMWF) > Shinfield Park, Reading, RG2 9AX, United Kingdom > Tel: (+44 118) 9499315 Fax: (+44 118) 9869450 > E-mail: Cristian.Simarro(a)ecmwf.int http://www.ecmwf.int > ------------------------------------------------------------------ > > ----- Original Message ----- > From: "Phil Carns" <carns(a)mcs.anl.gov> > To: darshan-users(a)lists.mcs.anl.gov > Sent: Wednesday, 29 April, 2015 10:13:54 PM > Subject: Re: [Darshan-users] LD_PRELOAD system() call > > On 04/29/2015 02:54 PM, Phil Carns wrote: >> On 04/29/2015 12:17 PM, Cristian Simarro wrote: >>> Hello, >>> >>> We have been facing some problems with system() call inside some >>> C/Fortran codes in our Cray machine. >>> >>> The method used here is compile dynamically and then use LD_PRELOAD. >>> When the code calls system(command), it hangs the execution if >>> preloaded with Darshan because it is trying to instrument an internal >>> system read() with no initialization. >>> >>> The solution we have designed is to unset LD_PRELOAD (if set before) >>> in the darshan_mpi_initialize function. >>> >>> Has anybody found a similar problem with LD_PRELOAD + system() calls? >> Hi Cristian, >> >> I don't think I've seen this exact combination before, but it seems >> like something we should be able to reproduce and isolate. >> >> If I understand correctly, it sounds like the underlying process >> spawned by system() is inheriting the LD_PRELOAD environment variable >> from the parent program, and it is the underlying process that is >> getting hung? If so, does it matter what you run in the system() call >> or does it seem like pretty anything triggers it? >> >> thanks, >> -Phil > The solution you have suggested (unsetting LD_PRELOAD grammatically > during Darshan initialization) might not be a bad long term solution, > maybe with some extra safety logic to make sure we don't accidentally > unset unrelated LD_PRELOAD entries. I imagine that once the application > has gotten to darshan initialization, then the loader has already > processed the LD_PRELOAD environment variable and we don't need to keep > it set any longer. That would help keep it from interfering with child > processes. > > We would definitely need to do some testing to confirm, though. > > thanks, > -Phil > _______________________________________________ > Darshan-users mailing list > Darshan-users(a)lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users

2 1