July 2007 - Cobalt - mailman3.cels.anl.gov

limiting the number of jobs per user in a queue
by Andrew Pochinsky 27 Jul '07

27 Jul '07

Dear All, I'm trying to setup a Cobalt queue to allow serialization of jobs on BG/L. The idea is to let one jobs per user at the same time in a queue and let users use this queue if they need to run jobs in a given order. Here's what I did: - setup a queue, serial00 with one jobs per user limit. - make it running with cqadm.py --startq serial00 --- here's what cqstat thinks about this queue bglsn:bin # cqstat -fq serial00 Name Users MinTime MaxTime MaxRunning MaxQueued MaxUserNodes TotalNodes State ======================================================================== ===================== serial00 * * * 1 * * * running The setup, however, does not seem to work; queue serial00 is not scheduled for some reason, e.g., user1 has four jobs sitting there and there are resources to run them, but Cobalt does not start a job from the queue. Here's what cqstat says (job 21607 should run, but does not): bglsn:bin # cqstat -f JobID JobName User WallTime QueuedTime RunTime Nodes State Location Mode Procs Queue StartTime Index ======================================================================== ======================================================================== ==== 21480 - user1 21:43:00 142:19:28 04:29:32 512 running 0512-01 vn 1024 default 07/26/07 12:12:47 - 21485 - user2 05:40:00 145:45:04 N/ A 512 queued N/A vn 1024 default N/ A - 21486 - user2 05:40:00 140:44:59 N/ A 512 queued N/A vn 1024 default N/ A - 21487 - user2 05:40:00 140:44:58 N/ A 512 queued N/A vn 1024 default N/ A - 21488 - user1 20:01:00 124:08:49 N/ A 512 queued N/A vn 1024 default N/ A - 21493 - user2 05:40:00 115:44:41 N/ A 512 queued N/A vn 1024 default N/ A - 21494 - user2 05:40:00 110:44:36 N/ A 512 queued N/A vn 1024 default N/ A - 21495 - user2 05:40:00 110:44:35 N/ A 512 queued N/A vn 1024 default N/ A - 21496 - user2 05:40:00 105:44:30 N/ A 512 queued N/A vn 1024 default N/ A - 21497 - user1 15:02:00 90:12:27 N/ A 512 queued N/A vn 1024 default N/ A - 21506 - user2 05:40:00 85:44:16 N/ A 512 queued N/A vn 1024 default N/ A - 21507 - user2 05:40:00 80:44:12 N/ A 512 queued N/A vn 1024 default N/ A - 21516 - user2 05:40:00 75:44:07 N/ A 512 queued N/A vn 1024 default N/ A - 21521 - user1 10:00:00 71:21:01 N/ A 512 queued N/A vn 1024 default N/ A - 21591 - user1 15:03:00 02:52:28 13:24:07 128 running 0128-03 vn 256 default 07/26/07 03:18:12 - 21592 xxxxxxxxxxxxxxxxxxxx user3 04:00:00 14:40:43 N/ A 512 queued N/A vn 1024 R.user3 N/ A - 21607 - user1 01:00:00 04:47:12 N/ A 32 queued N/A vn 64 serial00 N/ A - 21608 - user1 01:00:00 04:47:08 N/ A 32 queued N/A vn 64 serial00 N/ A - 21609 - user1 01:00:00 04:47:07 N/ A 32 queued N/A vn 64 serial00 N/ A - 21610 - user1 01:00:00 04:47:06 N/ A 32 queued N/A vn 64 serial00 N/ A - 21614 - user4 00:10:00 01:13:14 N/ A 512 queued N/A vn 1024 default N/ A - 21616 - user1 15:01:00 00:00:13 00:56:22 128 running 0128-01 vn 256 default 07/26/07 15:45:57 - 21617 - user1 15:01:00 00:00:05 00:46:48 128 running 0128-02 vn 256 default 07/26/07 15:55:31 - Am I doing something wrong? Thanks, --andrew

4 14

Creating a new queue as a duplicate of an existing one, and job run behavior?
by Michael Oberg 23 Jul '07

23 Jul '07

NCAR currently has a 'default' queue, which we have been using as the main queue for all users. With our Blue Gene/L machine joining the TeraGrid on the 31st, I need to create a queue for the TeraGrid users. If I duplicate the partition membership of the default queue to this new TeraGrid queue, will cobalt still run the jobs solely based on submit time and available partitions? Thanks, Michael Oberg Research Systems Evaluation Team National Center for Atmospheric Research (NCAR) Office: 303.497.1268, Cell: 720.938.6585 oberg(a)ucar.edu

2 2

cobalt 0.97 and new features
by Narayan Desai 18 Jul '07

18 Jul '07

In the 0.97 release, a number of new resource management features were added, including new queue policies, queue restrictions, reservations queues, and so forth. Now that several sites have upgraded to 0.97, I am hoping to start a discussion about uses of these features. We (at ANL) are using them for some things, and NCAR is using them differently. Theron has written up a description of how NCAR has its policies setup, and the features they use to get these results, and we will be posting something similar soon. So, what sorts of scheduling policies are people using, and how are they accomplishing them? Also, what other policies would sites like to implement but are unable to do so? Theron's summary is up at: http://trac.mcs.anl.gov/projects/cobalt/wiki/NeatStuff That is also where our summary will be posted, once it is done. For sites that haven't upgraded yet, have you run into any issues, or is it just a time issue? -nld

1 0

Re: 0.96.2 to 0.97 upgrade
by Joy Khoriaty 09 Jul '07

09 Jul '07

Hi, Thanks Theron, I've dumped the backup successfully - I installed from the rpms, (with -Uvh), and started the new cobalt server. When I do a cqstat, I get: $ cqstat Traceback (most recent call last): File "/usr/bin/cqstat.py", line 170, in ? j['queuedtime'] = get_elapsed_time(float(j.get('submittime')), time.time()) TypeError: float() argument must be a string or a number Also, trying to load the old config gives: $ python cdump.py --load cobalt_config.xml Setting queues: No handlers could be found for logger "queue-manager" Traceback (most recent call last): File "cdump.py", line 208, in ? [response] = cqm.SetQueues(query, eq) File "<string>", line 1, in <lambda> File "/var/tmp/cobalt-0.97.0-1-root/usr/lib/python2.3/site-packages/Cobalt/Proxy.py", line 70, in run_method xmlrpclib.Fault: <Fault 20: 'Server Failure'> Any clues? Cheers, On Tue, 3 Jul 2007, Theron R Voran wrote: > It doesn't look like the rpms ever were built. I'll see what I can do > about building them today. > -Theron > > > Joy Khoriaty said the following on 07/03/2007 02:26 PM CDT: > > Hi Theron, > > > > No I haven't installed the 0.97 library yet - > > > > I just grabbed the cdump.py file alone. > > > > In that case, I'll comment it out and add the next jobid as you > > mentioned. > > > > Are the 0.97 rpms online? > > > > Cheers, > > > > On Tue, 3 Jul 2007, Theron R Voran wrote: > > > > > >> Have you already installed the 0.97 python libraries? The cdump.py > >> script checks the revisions of a couple of the Cobalt python libraries > >> to see whether it can run GetJobID or not, so my guess is that it found > >> the 0.97 libraries somehow. > >> > >> In any case, you may also just comment out the section that calls > >> GetJobID, lines 125-133. You will then of course need to set the next > >> jobid with cqadm.py -j after installing and starting 0.97.0. > >> -Theron > >> > >> > >> > >> Joy Khoriaty said the following on 07/03/2007 01:35 PM CDT: > >> > >>> Hi, > >>> > >>> I get the following when trying to dump the old cobalt configuration: > >>> > >>> $python cdump.py --dump > cobalt_config.xml > >>> Traceback (most recent call last): > >>> File "cdump.py", line 126, in ? > >>> next_jobid = cqm.GetJobID() > >>> AttributeError: queue_manager instance has no attribute 'GetJobID' > >>> > >>> Any clues? > >>> > >>> Cheers, > >>> > >>> > >>> On Sun, 1 Jul 2007, Joy Khoriaty wrote: > >>> > >>> > >>> > >>>> Hi, > >>>> > >>>> I'd like to know if the upgrade to the latest Cobalt will > >>>> also preserve our current Cobalt Job ID count - otherwise is there > >>>> a way to do that? > >>>> > >>>> Cheers, > >>>> > >>>> _______________________________________________________________ > >>>> > >>>> Joy Khoriaty tel: (617) 253 8865 > >>>> MIT fax: (617) 253 8674 > >>>> Center for Theoretical Physics email: khoriaty(a)lns.mit.edu > >>>> 77 Massachusetts Ave. web: > >>>> Building NE25-4080 > >>>> Cambridge MA 02139 > >>>> _______________________________________________________________ > >>>> > >>>> > >>>> > >>> _______________________________________________________________ > >>> > >>> Joy Khoriaty tel: (617) 253 8865 > >>> MIT fax: (617) 253 8674 > >>> Center for Theoretical Physics email: khoriaty(a)lns.mit.edu > >>> 77 Massachusetts Ave. web: > >>> Building NE25-4080 > >>> Cambridge MA 02139 > >>> _______________________________________________________________ > >>> > >>> > >>> > > > > _______________________________________________________________ > > > > Joy Khoriaty tel: (617) 253 8865 > > MIT fax: (617) 253 8674 > > Center for Theoretical Physics email: khoriaty(a)lns.mit.edu > > 77 Massachusetts Ave. web: > > Building NE25-4080 > > Cambridge MA 02139 > > _______________________________________________________________ > > > _______________________________________________________________ Joy Khoriaty tel: (617) 253 8865 MIT fax: (617) 253 8674 Center for Theoretical Physics email: khoriaty(a)lns.mit.edu 77 Massachusetts Ave. web: Building NE25-4080 Cambridge MA 02139 _______________________________________________________________

4 15

0.97 rpms online?
by Joy Khoriaty 03 Jul '07

03 Jul '07

Hi, The BGL instructions suggest downloading: # rpm -ihv ftp://ftp.mcs.anl.gov/pub/cobalt/rpms/sles9-ppc64/cobalt-0.97-1.ppc64.rpm The 0.97 rpms don't seem to be online - are they available? or should the sources be used instead. Cheers, On Tue, 3 Jul 2007, Joy Khoriaty wrote: > Hi, > > I get the following when trying to dump the old cobalt configuration: > > $python cdump.py --dump > cobalt_config.xml > Traceback (most recent call last): > File "cdump.py", line 126, in ? > next_jobid = cqm.GetJobID() > AttributeError: queue_manager instance has no attribute 'GetJobID' > > Any clues? > > Cheers, > > > On Sun, 1 Jul 2007, Joy Khoriaty wrote: > > > Hi, > > > > I'd like to know if the upgrade to the latest Cobalt will > > also preserve our current Cobalt Job ID count - otherwise is there > > a way to do that? > > > > Cheers, > > > > _______________________________________________________________ > > > > Joy Khoriaty tel: (617) 253 8865 > > MIT fax: (617) 253 8674 > > Center for Theoretical Physics email: khoriaty(a)lns.mit.edu > > 77 Massachusetts Ave. web: > > Building NE25-4080 > > Cambridge MA 02139 > > _______________________________________________________________ > > > > _______________________________________________________________ > > Joy Khoriaty tel: (617) 253 8865 > MIT fax: (617) 253 8674 > Center for Theoretical Physics email: khoriaty(a)lns.mit.edu > 77 Massachusetts Ave. web: > Building NE25-4080 > Cambridge MA 02139 > _______________________________________________________________ > _______________________________________________________________ Joy Khoriaty tel: (617) 253 8865 MIT fax: (617) 253 8674 Center for Theoretical Physics email: khoriaty(a)lns.mit.edu 77 Massachusetts Ave. web: Building NE25-4080 Cambridge MA 02139 _______________________________________________________________

2 1

upgrade to latest Cobalt
by Joy Khoriaty 03 Jul '07

03 Jul '07

Hi, I'd like to know if the upgrade to the latest Cobalt will also preserve our current Cobalt Job ID count - otherwise is there a way to do that? Cheers, _______________________________________________________________ Joy Khoriaty tel: (617) 253 8865 MIT fax: (617) 253 8674 Center for Theoretical Physics email: khoriaty(a)lns.mit.edu 77 Massachusetts Ave. web: Building NE25-4080 Cambridge MA 02139 _______________________________________________________________

3 3