September 2008 - Topg-ssf - mailman3.cels.anl.gov

Re: [topg-ssf] Meeting notes
by Corby Schmitz 03 Oct '08

03 Oct '08

This is a difficult situation. It would be great to have a single solution, but that's not going to happen. There are very few 60a PDUs available which meet the needs of the average rack. 2x 3-phase 30a is sufficient to run a rack of systems for the average instance. We know that a single circuit will not work of system's resources as we need a one UPS and one COM connection. The nice part is that even with two 30a COM circuits, a pair (fused management interface) of PDUs provides more than enough power. We are talking about ~100a @ 208vac. The same solution with s single 60a does provide the same power, but minimizes our capabilities for getting PDUs to front-end the system. There is an alternative to trying to pick the right receptacle to be in place. We could have power fee troughs under the floor. This would allow us to drop in multiple conductors of various sizes and get them where they need to be. We have such things in the BMR, but they have been pretty much abandoned as they only cover a small portion of the underfloor space. Using a more modular approach would allow us to have one distribution system (cable-centric), with custom accessibility where it needs to be customized. corby ----- Original Message ----- From: "Craig Stacey" <stace(a)mcs.anl.gov> To: "Schmitz Corby" <schmitz(a)anl.gov> Cc: topg-ssf(a)lists.mcs.anl.gov Sent: Monday, September 29, 2008 2:03:20 PM GMT -06:00 US/Canada Central Subject: Re: [topg-ssf] Meeting notes -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm rethinking our assumptions from earlier this year, and we need to settle on this soon. We need to make sure we're choosing the *most useful* configuration for power in the SSF. Nothing we choose will be universally applicable, but as long as we come up with something that's going to be the trend, we can buy the right PDUs to make it useful to whatever going into the cabinets. What concerns me is that the trend in the computing clusters I see coming in is definitely skewing to the 60A range and not in the 30A range -- this is precisely what prompted our discussion in the meeting on Friday about converting to 60A. Should we be looking at going with a 60A 3 Phase run to every cabinet, alternating UPS and non-UPS for each receptacle? Should we further adopt a hybrid strategy, wherein we deploy 4 x single phase 208 to ALCF's area, 2 x 30A 3 phase to a portion of the room, and 60A 3 phase to another portion? It's starting to feel like the latter is the right choice. Incidentally, the proposed (by Teng) SSF PDU schematic and data sheet is now the wiki: http://wiki.mcs.anl.gov/SSF/index.php/Image:PDM_Mk_II.pdf http://wiki.mcs.anl.gov/SSF/index.php/Image:PDM-MK-II-Schematic.pdf == Craig On Sep 29, 2008, at 1:40 PM, Schmitz Corby wrote: > The wiring would need to change. #10 stranded will be too small and, > we should check with Paul, but I believe that using parallel > conductors is not allowed with smaller than a #2. We would have to > consult the tables, but this would be a #4 I believe. There is > another consideration here, the standard conduit size for a 5 wire 3- > phase circuit is 3/4" (squeezing it into a 1/2" is not advisable). > We would need at least a 1-1/2" conduit for 5x #4 conductors to > support the 60a circuits. > I am not sure what we are planning as far as electrical > distribution, but it is something that we should hash out as it > drives the maximum size of circuits going out to our cabinets. > > -corby -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkjhJngACgkQbVNGA3HOzQssQACfXPNyKErlWQnIMY4PeV+QVRiR QJwAn3ntnBZ2NqX0X0PSfF3RCofr5Lix =icSa -----END PGP SIGNATURE-----

7 10

Re: [topg-ssf] Meeting notes
by Craig Stacey 29 Sep '08

29 Sep '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm rethinking our assumptions from earlier this year, and we need to settle on this soon. We need to make sure we're choosing the *most useful* configuration for power in the SSF. Nothing we choose will be universally applicable, but as long as we come up with something that's going to be the trend, we can buy the right PDUs to make it useful to whatever going into the cabinets. What concerns me is that the trend in the computing clusters I see coming in is definitely skewing to the 60A range and not in the 30A range -- this is precisely what prompted our discussion in the meeting on Friday about converting to 60A. Should we be looking at going with a 60A 3 Phase run to every cabinet, alternating UPS and non-UPS for each receptacle? Should we further adopt a hybrid strategy, wherein we deploy 4 x single phase 208 to ALCF's area, 2 x 30A 3 phase to a portion of the room, and 60A 3 phase to another portion? It's starting to feel like the latter is the right choice. Incidentally, the proposed (by Teng) SSF PDU schematic and data sheet is now the wiki: http://wiki.mcs.anl.gov/SSF/index.php/Image:PDM_Mk_II.pdf http://wiki.mcs.anl.gov/SSF/index.php/Image:PDM-MK-II-Schematic.pdf == Craig On Sep 29, 2008, at 1:40 PM, Schmitz Corby wrote: > The wiring would need to change. #10 stranded will be too small and, > we should check with Paul, but I believe that using parallel > conductors is not allowed with smaller than a #2. We would have to > consult the tables, but this would be a #4 I believe. There is > another consideration here, the standard conduit size for a 5 wire 3- > phase circuit is 3/4" (squeezing it into a 1/2" is not advisable). > We would need at least a 1-1/2" conduit for 5x #4 conductors to > support the 60a circuits. > I am not sure what we are planning as far as electrical > distribution, but it is something that we should hash out as it > drives the maximum size of circuits going out to our cabinets. > > -corby -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkjhJngACgkQbVNGA3HOzQssQACfXPNyKErlWQnIMY4PeV+QVRiR QJwAn3ntnBZ2NqX0X0PSfF3RCofr5Lix =icSa -----END PGP SIGNATURE-----

1 0

Re: [topg-ssf] Meeting notes
by Craig Stacey 29 Sep '08

29 Sep '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 We were thinking replace the two 30A breakers at the panel with a single 60A, and the two L21-30R with a single L21-60R, replacing the wiring through the conduit if required. == Craig On Sep 26, 2008, at 5:31 PM, Schmitz Corby wrote: > can you detail how you convert two 30a to a single 60a? I am not > sure what the plan is, but it sounds fishy to me. > > - - -corby > > > > On Sep 26, 2008, at 4:56 PM, Craig Stacey wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> http://wiki.mcs.anl.gov/SSF/index.php/2008-09-26 >> >> Agenda >> ∙ Get everyone up to speed on where we are >> ∙ Look at the drawings for the SSF mechanical and electrical >> support areas >> ∙ Hash out the WBS items before submitting to Ron Richardson for >> inclusion into the WBS >> ∙ Review the preliminary power layout to ensure it meets our needs. >> >> Items >> Power Layout >> Since the time for the power layout is upon us, it's time to >> finalize the design. >> >> ∙ We looked at the existing SSF layout. General consensus was that >> the proposed plan of 2 x NEMA L21-30R 3 Phase 208 per non-ALCF >> cabinet (with ALCF using 4 x NEMA L6-30R 208 for their cabinets) >> was good. >> ∙ Ti will check if the iDataPlex racks can make use of a 2 x 30A 3 >> Phase, or if they require a single 60A 3 Phase. We discussed in the >> meeting that it should not be a complex operation to convert a dual >> 30A to a single 60A, so this should not be a show-stopper. >> ∙ Craig will produce a straw-man layout of power connectors and >> upload to the wiki by COB Monday, 9/29. >> >> WBS Items >> ∙ We went over the initial WBS layout. Bobby will provide some >> numbers for the EVS rack move, and the others will have their >> numbers fleshed out over the next two weeks. >> ∙ Action item is to have initial numbers submitted to Ron >> Richardson in FMS by next TOPG-SSF meeting on 10/10/08. >> ∙ A reminder to people in estimating labor time for their moves. >> We're right now focussing on the layers in the spread sheet. These >> items will have sub-tasks such as removing old cables and such. >> Make sure you consider those tasks. We'll break out those finer >> tasks once we've settled on what the higher level tasks are. >> >> Next Meeting: 10/10/2009 >> >> == >> Craig >> >> == >> Craig >> >> >> >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v1.4.8 (Darwin) >> >> iEYEARECAAYFAkjdWpQACgkQbVNGA3HOzQsBswCgpT1z+Efn5Nex17aSNxHdWrSw >> IAUAoJzmQ9klyNCqgXlyOHpxy5BKRFKq >> =sUp2 >> -----END PGP SIGNATURE----- >> _______________________________________________ >> topg-ssf mailing list >> topg-ssf(a)lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/topg-ssf > > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkjhHK8ACgkQbVNGA3HOzQtclACfdPg5ykZISvlP7ox9SM3lwjMx f04Ani11LY9tWyPCC8Sg4QaZ9/M+/yo5 =4iwe -----END PGP SIGNATURE-----

2 1

RFI: Mechanical Room Cross Section answered
by Craig Stacey 29 Sep '08

29 Sep '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The wiki's been updated with a PDF of the Mechanical Room Cross Section for those who are interested. http://wiki.mcs.anl.gov/SSF/index.php/ 2008-09-15#RFI:_Mechanical_Room_Cross_Section == Craig -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkjg/1QACgkQbVNGA3HOzQtnegCffFTGavcrSlMbRuC5XQ+Ko3HI XHcAn14aYdQkNWUYBXj28h5viAuGC5Ti =uei+ -----END PGP SIGNATURE-----

1 0

Meeting notes
by Craig Stacey 26 Sep '08

26 Sep '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 http://wiki.mcs.anl.gov/SSF/index.php/2008-09-26 Agenda ∙ Get everyone up to speed on where we are ∙ Look at the drawings for the SSF mechanical and electrical support areas ∙ Hash out the WBS items before submitting to Ron Richardson for inclusion into the WBS ∙ Review the preliminary power layout to ensure it meets our needs. Items Power Layout Since the time for the power layout is upon us, it's time to finalize the design. ∙ We looked at the existing SSF layout. General consensus was that the proposed plan of 2 x NEMA L21-30R 3 Phase 208 per non-ALCF cabinet (with ALCF using 4 x NEMA L6-30R 208 for their cabinets) was good. ∙ Ti will check if the iDataPlex racks can make use of a 2 x 30A 3 Phase, or if they require a single 60A 3 Phase. We discussed in the meeting that it should not be a complex operation to convert a dual 30A to a single 60A, so this should not be a show-stopper. ∙ Craig will produce a straw-man layout of power connectors and upload to the wiki by COB Monday, 9/29. WBS Items ∙ We went over the initial WBS layout. Bobby will provide some numbers for the EVS rack move, and the others will have their numbers fleshed out over the next two weeks. ∙ Action item is to have initial numbers submitted to Ron Richardson in FMS by next TOPG-SSF meeting on 10/10/08. ∙ A reminder to people in estimating labor time for their moves. We're right now focussing on the layers in the spread sheet. These items will have sub-tasks such as removing old cables and such. Make sure you consider those tasks. We'll break out those finer tasks once we've settled on what the higher level tasks are. Next Meeting: 10/10/2009 == Craig == Craig -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkjdWpQACgkQbVNGA3HOzQsBswCgpT1z+Efn5Nex17aSNxHdWrSw IAUAoJzmQ9klyNCqgXlyOHpxy5BKRFKq =sUp2 -----END PGP SIGNATURE-----

2 1

Topg-SSF
by stace＠mcs.anl.gov 26 Sep '08

26 Sep '08

A single instance of the following meeting has been modified: Subject: Topg-SSF Organizer: stace(a)zimbra.mcs.anl.gov Resources: 221-a261(a)zimbra.mcs.anl.gov Time: Friday, September 26, 2008, 1:00:00 PM - 2:00:00 PM GMT -06:00 US/Canada Central Invitees: topg-ssf(a)lists.mcs.anl.gov *~*~*~*~*~*~*~*~*~*

1 0

Wiki update and meeting change
by Craig Stacey 24 Sep '08

24 Sep '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I think I have general buy-in for the Friday meeting schedule, though I'd like to move it to 1:00 PM instead of 1:30 to allow another attendee to make it. Please let me know if this poses a problem. I populated the wiki with a lot of data on the previous Requests for Information (RFI) both from the trust to us and from us to the trust. This isn't an exhaustive list of RFI, just the ones I've been a part of that pertain to the SSF. More downloadable files are available on the site as well. http://wiki.mcs.anl.gov/SSF Thanks. == Craig -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkja2EIACgkQbVNGA3HOzQvDPwCfR2zONTLbWN6PDG+7e5cjTtRt zw4AoJBX7zbs/EgwlTCjJIUUAfcRk7mQ =OJ3f -----END PGP SIGNATURE-----

1 0

RFI from EYP on 13.2 kv electrical supply
by Sims, Jeffrey A. 24 Sep '08

24 Sep '08

Craig: Here is a request for information from EYP: Are there spares, spaces or provisions (for draw-out CB's or fuse/switches) in the 15kV switchgear to feed the BG/Q system racks and mechanical equipment? What are the sizes of the spares and/or cubical spaces? Jeff

2 3

Time for regular meetings
by Craig Stacey 23 Sep '08

23 Sep '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 (I've moved this list to its new home at lists.mcs.anl.gov.) We're at the point in this process where we need to ramp up activity. I've set up a new wiki (for which you all should have received login information) since the intial Trac/Wiki site I set up was not well suited to this project. We can use this wiki to trade files easily and maintain revision control. Please check out the wiki for notes. Here's where we are so far: 1) Preliminary numbers on WBS 1.3 in place, need to flesh out more. Spreadsheet is on the wiki. 2) Power layout discussion needs to happen. Current plan is to put two 3 phase 30A circuits per cabinet, except in ALCF space which requires 4 x 208V single phase circuits. I'd like to start having regular bi-weekly meetings on this in the hopes of keeping everyone in sync, and keeping meeting lengths to something reasonable. I know there's not a time when everyone on this list is going to be able to make it, but as long as we have a reasonable quorum, we'll be fine. I propose every other Friday at 1:30PM in 221 beginning 9/26 (room TBD). If you generally can't make that schedule (and you *want* to be able to attend), let me know and we'll find something else. The first meeting's agenda will be * Get everyone up to speed on where we are * Look at the drawings for the SSF mechanical and electrical support areas * Hash out the WBS items before submitting to Ron Richardson for inclusion into the WBS * Review the preliminary power layout to ensure it meets our needs. Thanks! == Craig -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkjZfGgACgkQbVNGA3HOzQuXbQCfd5v2mv5lukzSOz/zz+5BRG0l zBgAn0SMbAwi/Zgf6/dhwvPs1dJS28ww =76q7 -----END PGP SIGNATURE-----

1 0