March 2010 - Hpc-admins - mailman3.cels.anl.gov

[CELS Sysadmins] Geek Beer Movie Night
by Craig Stacey 01 Apr '10

01 Apr '10

Tonight, my house. Free beer, chip in $5 for pizza. Office Space. We'll start rolling around 7. Please let me know if you're coming so (A) we know how much pizza to order and (B) we know to wait for you. Also voice any pizza topping preferences. == Craig _______________________________________________ This is the CELS Sysadmins mailing list. It is sent to memebers of: core-admins(a)mcs.anl.gov, hpc-admins(a)mcs.anl.gov, systems(a)alcf.anl.gov, aig(a)alcf.anl.gov. Do not subscribe directly to this list. Thanks.

4 3

[CELS Sysadmins] Geekbeer
by Craig Stacey 26 Mar '10

26 Mar '10

I have to make an Ikea run at some point this weekend. If we beer at Gordon Biersch or Auld Dubliner, I can do that with ease. == Craig _______________________________________________ This is the CELS Sysadmins mailing list. It is sent to memebers of: core-admins(a)mcs.anl.gov, hpc-admins(a)mcs.anl.gov, systems(a)alcf.anl.gov, aig(a)alcf.anl.gov. Do not subscribe directly to this list. Thanks.

3 4

GPFS Backup script
by Ti Leggett 26 Mar '10

26 Mar '10

One of the problems we've faced with the PADS GPFS filesystem is how to reliably backup all the projects and do so with little impact to the performance of the filesystem and the storage nodes, and do all this as efficiently as possible. Our first pass was me manually balancing the projects across all of our storage nodes (we have 10). This was poor because my basis was on their total capacity and number of files not by their rate of change. It also meant that we scanned projects even when those projects had no change that day. So I came up with the attached script. The code is heavily commented but I'll explain what it does and what it requires. First it requires TSM. It could probably use some other backup service, but TSM has a nice method of querying occupancy. Second we use GPFS filesets - each project is its own fileset. This makes it very nice to get per project capacity and file usage regardless of the ownership of those files in the project. We have a process that nightly gathers each projects capacity and file usage (mmlsfileset) and dumps the data in a database. We use this database data to determine which projects have changed from the day before. Third, each project is backed up as its own TSM virtual mountpoint and domain. This makes it simple to query each project's usage on a daily basis and query if any given project has been backed up before. With those requirements in place here's what the script does in somewhat fine detail. If you don't care about any of this or the details you can ignore the remainder of this lengthy email. The first thing to do is get a list of all the projects on the filesystem. This is easy and amounts to an 'ls' on the proejcts directory. We then use that list to see if there are any projects that have never been backed up to TSM before. If so, we definitely need to add them and we use their current capacity and file usage combined as the project's 'weight'. Next we get the list of projects that may still be backing up from the day before. This also amounts to an 'ls' on our backup run directory. We verify the project is indeed still backing up and not just a stale run file by ssh'ing to the host reportedly still backing up the project and querying the PID that's in the run file. If that PID exists and is a dsmc process, we remove that project from the candidate projects to backup today. For all the projects not currently backing up and that have never been backed up, we get their usage reported from today and yesterday and take the difference in capacity used and file counts. If one of these values has changed, positively or negatively, the project needs to be backed up. We take the difference of the capacity used and add it to it's total file count today and that is the project's weight. If this number is negative - i.e., it has removed a lot of files - the weight is assigned as the total number of files it contains. We use the project's weight as a rough estimate of how much effort will be required to backup the project - if there has been a lot of positive change in data or if the project has a large amount of files, it will take longer to backup. Once we have all the projects we want to backup, we put them in a heap with the weightiest project at the top. Next we got through our available nodes to perform the backups and assign them a 'depth'. If a nodes is still performing a backup from the day before we assign it a very high 'depth' otherwise each node gets its node number as its node number as its depth - stor1 -> depth of 1, stor2 -> depth of 2, etc. We then put these nodes in a heap with the node with the shallowest depth at the top. Now that we have our project heap and node heap, we go through and extract a project from the heap - maintaing heap order - so that whatever project we get is guaranteed to be the weightiest. We extract a node from the heap so that the node we get is guaranteed to have the shallowest depth. We assign that node to backup that project, add the project's weight to the node's depth and add the node back to the heap - maintaining heap order. When we're done with this process we guaranteed to have an indexed list (indexed on node number) such that node's are as balanced as can be to backup all the projects. Because each node runs this script independently, it's very important that this indexed list be exactly the same on each node so that no project gets backed up by two separate nodes at the same time and because of the two heaps - that's the case. Then it's a matter of each node getting the list of projects it needs to backup and doing so, serially, keeping track of which project it's currently backing and the PID of that process in our run directory. That's it. Feel free to drop by and grill me on this if you think this may be useful to you in your setup. There are some TSM configurations that make this possible, so if you're using TSM ask me what I do to make sure TSM doesn't run itself out of memory.

1 0

[CELS Sysadmins] eConnectix postponed
by Tisha Stacey 24 Mar '10

24 Mar '10

The eConnectix visit that was scheduled for tomorrow (that involved installing a demo unit and presenting their "Flash School" for us) has to be postponed. When I hear of the new date, I'll let you know. Thanks. - Tisha _______________________________________________ This is the CELS Sysadmins mailing list. It is sent to memebers of: core-admins(a)mcs.anl.gov, hpc-admins(a)mcs.anl.gov, systems(a)alcf.anl.gov, aig(a)alcf.anl.gov. Do not subscribe directly to this list. Thanks.

1 0

[CELS Sysadmins] google voice invite?
by David Ressman 22 Mar '10

22 Mar '10

Probably a long shot, but does anyone here have a google voice invite you could spare? Thanks! _______________________________________________ This is the CELS Sysadmins mailing list. It is sent to memebers of: core-admins(a)mcs.anl.gov, hpc-admins(a)mcs.anl.gov, systems(a)alcf.anl.gov, aig(a)alcf.anl.gov. Do not subscribe directly to this list. Thanks.

1 1

[CELS Sysadmins] Late start for movie night
by Tisha Stacey 19 Mar '10

19 Mar '10

Because things are all bustificated here at Computer Central, we're going to have to push the start time for movie night back a bit. We'll get started at 8:00 instead of 7:00. Hopefully anti-bustification measures will have kicked in by then. - Tisha _______________________________________________ This is the CELS Sysadmins mailing list. It is sent to memebers of: core-admins(a)mcs.anl.gov, hpc-admins(a)mcs.anl.gov, systems(a)alcf.anl.gov, aig(a)alcf.anl.gov. Do not subscribe directly to this list. Thanks.

1 0

[CELS Sysadmins] Interest in eConnectix "flash school"?
by Tisha Stacey 19 Mar '10

19 Mar '10

(Apologies to ALCF people who have already heard this.) Some eConnectix representatives will be here, probably sometime next week (maybe Thursday), and can present their "Flash School" presentation to those who are interested. It's a maybe 90-minute-ish (probably more if there are really a lot of questions) presentation on flash storage, its limitations, what people do to work with (or around) those limitations, and how eConnectix's RAM products compare. For those not already familiar with eConnectix: http://www.econnectix.com/ Please let me know so I can gauge potential attendance. Thanks. - Tisha _______________________________________________ This is the CELS Sysadmins mailing list. It is sent to memebers of: core-admins(a)mcs.anl.gov, hpc-admins(a)mcs.anl.gov, systems(a)alcf.anl.gov, aig(a)alcf.anl.gov. Do not subscribe directly to this list. Thanks.

1 0

HPC Admin Meeting
by Ti Leggett 19 Mar '10

19 Mar '10

The following meeting has been modified: Subject: HPC Admin Meeting Organizer: "Ti Leggett" <tleggett(a)zimbra.mcs.anl.gov> Resources: Projector NEC / 333506 (Site: MCS, Building: 240, Floor: 2, Room: 2E15) [MODIFIED] Time: Friday, March 19, 2010, 9:30:00 AM - 12:00:00 PM GMT -06:00 US/Canada Central Invitees: "Narayan Desai" <b52935(a)zimbra.mcs.anl.gov>; "hpc-admins(a)lists.mcs.anl.gov" <hpc-admins(a)lists.mcs.anl.gov>; "Ti Leggett" <leggett(a)zimbra.mcs.anl.gov>; b53880(a)zimbra.mcs.anl.gov *~*~*~*~*~*~*~*~*~*

1 0

[CELS Sysadmins] Geek Beer Movie Night
by Craig Stacey 17 Mar '10

17 Mar '10

This Friday we're hosting another Geek Beer Movie Night. Cory & Lis are going to try to come, too, I think. Any votes on what to watch? Immediately available are: Galaxy Quest Sneakers Blade Runner The Princess Bride Office Space Tron Any suggestions? == Craig _______________________________________________ This is the CELS Sysadmins mailing list. It is sent to memebers of: core-admins(a)mcs.anl.gov, hpc-admins(a)mcs.anl.gov, systems(a)alcf.anl.gov, aig(a)alcf.anl.gov. Do not subscribe directly to this list. Thanks.

7 6

Meeting Request: HPC Admin Meeting
by Ti Leggett 16 Mar '10

16 Mar '10

1 0