One of the problems we've faced with the PADS GPFS filesystem is how to reliably backup all the projects and do so with little impact to the performance of the filesystem and the storage nodes, and do all this as efficiently as possible. Our first pass was me manually balancing the projects across all of our storage nodes (we have 10). This was poor because my basis was on their total capacity and number of files not by their rate of change. It also meant that we scanned projects even when those projects had no change that day. So I came up with the attached script. The code is heavily commented but I'll explain what it does and what it requires. First it requires TSM. It could probably use some other backup service, but TSM has a nice method of querying occupancy. Second we use GPFS filesets - each project is its own fileset. This makes it very nice to get per project capacity and file usage regardless of the ownership of those files in the project. We have a process that nightly gathers each projects capacity and file usage (mmlsfileset) and dumps the data in a database. We use this database data to determine which projects have changed from the day before. Third, each project is backed up as its own TSM virtual mountpoint and domain. This makes it simple to query each project's usage on a daily basis and query if any given project has been backed up before. With those requirements in place here's what the script does in somewhat fine detail. If you don't care about any of this or the details you can ignore the remainder of this lengthy email. The first thing to do is get a list of all the projects on the filesystem. This is easy and amounts to an 'ls' on the proejcts directory. We then use that list to see if there are any projects that have never been backed up to TSM before. If so, we definitely need to add them and we use their current capacity and file usage combined as the project's 'weight'. Next we get the list of projects that may still be backing up from the day before. This also amounts to an 'ls' on our backup run directory. We verify the project is indeed still backing up and not just a stale run file by ssh'ing to the host reportedly still backing up the project and querying the PID that's in the run file. If that PID exists and is a dsmc process, we remove that project from the candidate projects to backup today. For all the projects not currently backing up and that have never been backed up, we get their usage reported from today and yesterday and take the difference in capacity used and file counts. If one of these values has changed, positively or negatively, the project needs to be backed up. We take the difference of the capacity used and add it to it's total file count today and that is the project's weight. If this number is negative - i.e., it has removed a lot of files - the weight is assigned as the total number of files it contains. We use the project's weight as a rough estimate of how much effort will be required to backup the project - if there has been a lot of positive change in data or if the project has a large amount of files, it will take longer to backup. Once we have all the projects we want to backup, we put them in a heap with the weightiest project at the top. Next we got through our available nodes to perform the backups and assign them a 'depth'. If a nodes is still performing a backup from the day before we assign it a very high 'depth' otherwise each node gets its node number as its node number as its depth - stor1 -> depth of 1, stor2 -> depth of 2, etc. We then put these nodes in a heap with the node with the shallowest depth at the top. Now that we have our project heap and node heap, we go through and extract a project from the heap - maintaing heap order - so that whatever project we get is guaranteed to be the weightiest. We extract a node from the heap so that the node we get is guaranteed to have the shallowest depth. We assign that node to backup that project, add the project's weight to the node's depth and add the node back to the heap - maintaining heap order. When we're done with this process we guaranteed to have an indexed list (indexed on node number) such that node's are as balanced as can be to backup all the projects. Because each node runs this script independently, it's very important that this indexed list be exactly the same on each node so that no project gets backed up by two separate nodes at the same time and because of the two heaps - that's the case. Then it's a matter of each node getting the list of projects it needs to backup and doing so, serially, keeping track of which project it's currently backing and the PID of that process in our run directory. That's it. Feel free to drop by and grill me on this if you think this may be useful to you in your setup. There are some TSM configurations that make this possible, so if you're using TSM ask me what I do to make sure TSM doesn't run itself out of memory.