Hello,
I see this has been mentioned on the list previously, but I am having
the same problem.
I have the following partition setup:
<Partition admin="online"
deps="R000_J108-32:R000_J106-32:R000_J104-32:R000_J102-32:R000_J111-128:
R000_J203-128:R000_J210-128" name="R000_J102-512" queue="m512"
size="512" state="idle" />
<Partition admin="online" deps="" name="R000_J108-32"
queue="m32" size="32" state="idle" />
<Partition admin="online" deps="" name="R000_J106-32"
queue="m32" size="32" state="idle" />
<Partition admin="online" deps="" name="R000_J104-32"
queue="m32" size="32" state="idle" />
<Partition admin="online" deps="" name="R000_J102-32"
queue="m32" size="32" state="idle" />
<Partition admin="online" deps="" name="R000_J111-128"
queue="m128" size="128" state="idle" />
<Partition admin="online" deps="" name="R000_J203-128"
queue="m128" size="128" state="idle" />
<Partition admin="online" deps="" name="R000_J210-128"
queue="m128" size="128" state="idle" />
However, the dependency of the 512-node queue is not respected. If I
do this:
$cqsub -q m128 -n 128 -t 5 cpi; cqsub -q m512 -n 512 -t 5 cpi; cqsub -
q m128 -n 128 -t 5 cpi
It attempts to run all of them at once, with the 512 node partition
winning, and the others complaining that the partition is already
allocated.
JobID User WallTime Nodes State Location
==========================================================
70 jmesterh 00:05:00 128 running R000_J111-128
71 jmesterh 00:05:00 512 running R000_J102-512
72 jmesterh 00:05:00 128 running R000_J203-128
I restarted cobalt, sss, and bglmaster to no effect.
Any suggestions?
Joe
--
Joe Mesterhazy
ECpE UNIX Administrator
2101 Coover Hall, Iowa State University
Ames, IA 50011. (515) 294-7359
http://www.mesterhazy.net
Susan,
We don't have 64, 256 node partitions like Joseph's site.
Is it possible to create these node set for ourselves? or should we ask IBM
to define it?
It seems that we have to modify some xml files and database,
if there are some tips, please let me know.
Best Regards,
Junko Ikeda
NIWS Co., Ltd.
Kayabacho-First Bldg., 1-17-21 Shinkawa, Chuo-ku, Tokyo 104-0033 Japan
TEL : 81-3-5117-7833 / FAX : 81-3-5117-7913
mailto:[email protected]
-----Original Message-----
From: owner-cobalt(a)mcs.anl.gov [mailto:[email protected]]On
Behalf Of Susan Coghlan
Sent: Thursday, February 09, 2006 2:47 AM
To: Joseph Mesterhazy
Cc: Narayan Desai; cobalt(a)mcs.anl.gov
Subject: Re: Scheduler question
Joseph,
Narayan is out of the office right now, so I'll answer a few of your
questions. Users can specify arbitrary numbers of cpus. Our users do it
all the time. Basically, what is happening is binning. Jobs are run on
the smallest partition they will fit. This way, we don't have a 32 node
job using a 256 node partition.
Yes, you can create 256 node and 64 node partitions. We have 32, 64, 128,
256, 512, and 1024 node partitions that are used all the time. You won't
necessarily get the best performance, but you can certainly use them.
Susan.
On Wed, 8 Feb 2006, Joseph Mesterhazy wrote:
> So does this mean that users cannot specify arbitrary numbers of
> CPUs? This is something our users do a lot to test algorithms.
>
> Also, IBM told us you cannot create 256 node and 64 node partitions,
> and the solution was to allocate a 512 node or 128 node partition and
> specify less CPU's to mpirun in that case (which is what I was trying
> to do by allocating the 128 node queue and use only 64 CPUs).
>
> Joe
>
> On Feb 8, 2006, at 11:03 AM, Narayan Desai wrote:
>
> >>>>>> "Joe" == Joseph Mesterhazy <jmesterh(a)iastate.edu> writes:
> >
> > Joe> Hello, I have setup two partitions of 128 nodes, and put both
> > Joe> of them in a queue called p128, like so:
> >
> > Joe> <Partition admin="online" name="R001_J210-128" queue="p128"
> > Joe> state="idle" /> <Partition admin="online" name="R001_J203-128"
> > Joe> queue="p128" state="idle" />
> >
> > Joe> If I submit a job using cqsub (eg. cqsub -q p128 -n 128 -t 5
> > Joe> cpi) it starts fine. But if I request less than 128 nodes, eg
> > Joe> -n 64, the job remains in the queued state. I am using the
> > Joe> default scheduling.
> >
> > Joe> Is this the normal behavior, or have I mis-configured
> > Joe> something?
> >
> > This is normal behavior. The way to get the behavior you are looking
> > for is to define several smaller overlapping partitions. you can
> > define 64 and 32 way partitions that overlap with the 128 way ones you
> > have already defined. You can setup partition dependencies which will
> > cause the scheduler to be able to use peer partitions
> > simultaneously. In the case you described, the 64 node job would run
> > on one of the 64 way partitions and the other 64 nodes (of the 128
> > node partition) would be usable for other jobs. Take a look for
> > partition dependencies in the docs; that is the keyword to look for.
> > -nld
>
> --
> Joe Mesterhazy
> ECpE UNIX Administrator
> 2101 Coover Hall, Iowa State University
> Ames, IA 50011. (515) 294-7359
> http://www.mesterhazy.net
>
>
I'm wondering, has anyone on this list attempted to restrict access
to mpirun on the frontend, to prevent users from bypassing the
scheduler?
Joe
--
Joe Mesterhazy
ECpE UNIX Administrator
2101 Coover Hall, Iowa State University
Ames, IA 50011. (515) 294-7359
http://www.mesterhazy.net
