Zeptoos-users

Download

[email protected]

October 2005

2 participants
2 discussions

maximum CIOD_RDWR_BUFFER_SIZE
by Kazutomo Yoshii 31 Oct '05

31 Oct '05

Hi, Probably, we may be able to use the following formula to calculate the maximum number of CIOD_RDWR_BUFFER_SIZE. CIOD_RDWR_BUFFER_SIZE_max = (MemTotal-MemSysUsed)/(2*N) where, MemTotal : total installed memory MemSysUsed : memory used by system software N : the number of compute nodes per one I/O node. (PSET) The reason why we need (2*N) is that we have to think about the virtual mode. MemTotal is 512MByte per default( no exception?). ZeptoOS1.1 uses approx. 60MB which includes pvfs2, sshd, etc memory usage. On the most of BGL systems, PSET is 32 and CIOD_RDWR_BUFFER_SIZE_max is going to be approx. 7 MBytes. PSET is available from /proc/personality.sh, so we could automatically calculate the max size during ION boot-time. If something wrong, please correct me. Thanks, Kaz

1 0

Notes from BGW tests
by Pete Beckman 28 Oct '05

28 Oct '05

Hello ZeptoOS users (or those thinking about it for their BG/L). Yesterday we ran on 16 BG/L racks at Watson. The results were great. PVFS and ZeptoOS did well, and we learned a lot. A couple quick notes: 1) Thanks to Susan, we now have a "switcher" script that can be run from the command line. It will switch a partition to ZeptoOS and then back to the default with a simple command from the shell. This is perfect for installations that want to flipflop a partition between ZeptoOS and the default IBM setup for the purposes of testing performance or getting IBM support. Naturally, when you report a bug to IBM, you should do it against their default configuration, and "switcher" lets you easily flop back and forth (without Cobalt). The new script and documentation will be released in the next version of ZeptoOS, which bundled up in the next couple weeks. If you want a copy of the program now, let me know. 2) While running at BGW (20 rack machine at Watson), we realized that the default CIOD_RDWR_BUFFER_SIZE set to 4M by ZeptoOS is perfect for configurations that have 32 compute nodes per I/O node, but is too big for machines with 64 compute nodes per I/O node (16 per rack). On such I/O-poor machines, the CIOD will run out of memory trying to allocate extra buffer space for 64 compute nodes. Of course this is easily fixed with the ZeptoOS configuration tool. We suggest 3M for I/O poor machines. Similarly, machines that are I/O-rich, with 8 or 16 compute nodes per I/O node can probably increase the variable and therefore increase I/O performance. The next version of ZeptoOS will automatically default to the best number for your configuration. We are suggesting that IBM allow some of their 1 Megabyte node cards (twice the current memory) be made available as I/O nodes, even for standard 512M systems. Our Linux would be happy with the extra buffering :-) 3) On the BGW, The PVFS2 team tested PVFS2 on ZeptoOS up through 16K nodes writing to 33 PVFS2 servers. I'm betting that for those of you signed up for the SC05 tutorial, "S01: Parallel I/O in Practice" (http://sc05.supercomputing.org/schedule/event_detail.php? evid=5124) you may get a sneak peak at their results, or at least you can raise your hand and ask :-). In any case, they warmed the ethernet and hard drive with aggregate I/O speeds in the gigabytes/ sec range with various PVFS2 configurations. We also had a chance to test ZeptoOS with Jumbo frames. Worked perfectly. So if everything on your switch can be Jumbo, we recommend it. 4) The current version of ZeptoOS was also run with "Version 1" of the IBM BG/L software without changes. So no worries there. 5) Kamil and Kaz recently made some improvements to the ZeptoOS installation options. The improvements let you keep all the ZeptoOS files, even etc/rc scripts in a separate directory from your existing BG/L configuration. So if you want to keep all of your Zepto mods clean, and away from the IBM setup, that's now quite easy. In fact, we tested this on the BGW machine, since obviously IBM was not willing to give us root to install files :-). Instead, all we needed was a directory on /bgl, and our switcher script, and we were able to do it all without root :-) The installation improvements will be packaged up for release in the next couple weeks, or sooner if folks need them. 6) The ZeptoOS club is growing :-) Current users or exploratory testers include ANL, ASTRON, MIT, NIWS, EPCC, and of course BGW :-) Please let us know additional features you need (or just want). You can post them at bugzilla.mcs.anl.gov/zeptoos. Simply mark them "enhancements" and we will see what we can do. We have a couple things currently under way. The PVFS team has some sneaky plans for even faster I/O with clever hacks on the ION. We are also exploring the idea of running gateway daemons on the I/O nodes that could let you couple apps running on your BG/L nodes with external machines, either via GridMPI-style or WAN-style links. Thanks. -Pete

1 0