May 2012 - Core-other - mailman3.cels.anl.gov

New Student is here
by Craig Stacey 29 May '12

29 May '12

Jeff Rossiter is our new student, and today's his first day. He's sitting in front of the help desk. Drop by and say hi when you get a chance. We'll get him spun up on help desk, added to the right mailing lists, unix groups, etc. and get him working on the self-service SVN repo machine over the coming week. He's at orientation again tomorrow. -- Craig

1 0

Thought this would interest you guys...
by Tina Stanton-Piersanti 29 May '12

29 May '12

http://triblocal.com/joliet/calendar/2012/06/01/craft-import-beer-tasting-a…

1 0

Need new badge
by John Roberts 29 May '12

29 May '12

Have to stop first for a badge. Mine is expired Sent from my iPhone

1 0

Today's user termination list
by stace＠mcs.anl.gov 29 May '12

29 May '12

Full list of people who no longer appear on the active employee list =========================================================================== 57016 MCS DeJongh, Matthew 216406 PSC Zhang, Huan

1 0

Where's Tina Tuesday Morning
by Tina Stanton-Piersanti 28 May '12

28 May '12

I need to stop at the vet tomorrow morning on my way in to pick up some more pain killers for my dog. They open at 9:00. T. Sent from my iPad

1 0

Re: 221 vserver issues
by Dan Olson 28 May '12

28 May '12

I just looked at this, and the filesystem is in a very bad state. It was the systems@alcf list that was bad. Also we were missing mbox files from the p2s2-chairs list that look like they were on disk for some time. I'm reverting to the owney that is on vserver3-221, it is much better than stumbling over filesystem problems for the next few months. live migration=bad! ---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055 ----- Original Message ----- From: "Craig Stacey" <stace(a)mcs.anl.gov> To: "Core Admins" <core-admins(a)mcs.anl.gov> Sent: Monday, May 28, 2012 12:00:58 AM Subject: Re: 221 vserver issues I got mailman working for lists.mcs.anl.gov working again by recreating the master mailman list. This trick did not work for lists.alcf.anl.gov. So, as a quick hack that needs to be undone, I took out the integrity check to make it just start the qrunner. The following commented lines in /etc/init.d/lists.alcf.anl.gov need to be uncommented once we find the list that's a problem. > case "$1" in > start) > # if [ "$(/www/lists.alcf.anl.gov/bin/list_lists -b | grep ^mailman$ )" = "" ]; then > # echo "Site list for mailman (usually named mailman) missing" > # echo "Please create it; until then, mailman will refuse to start" > # exit 0 > # fi > $DAEMON -s start > ;; > stop) > $DAEMON stop > ;; > restart|force-reload) On May 27, 2012, at 11:30 PM, Craig Stacey wrote: > Some lists and domains look like they didn't come back cleanly. We may need to restore from tape. It also looks like mail is not getting delivered (I tried sending this to core-admins(a)mcs.anl.gov first). > >> Starting Mailman's master qrunner. >> Traceback (most recent call last): >> File "/www/lists.alcf.anl.gov/bin/list_lists", line 122, in <module> >> main() >> File "/www/lists.alcf.anl.gov/bin/list_lists", line 94, in main >> mlist = MailList.MailList(n, lock=0) >> File "/www/lists.alcf.anl.gov/Mailman/MailList.py", line 130, in __init__ >> self.Load() >> File "/www/lists.alcf.anl.gov/Mailman/MailList.py", line 676, in Load >> self.CheckVersion(dict) >> File "/www/lists.alcf.anl.gov/Mailman/MailList.py", line 715, in CheckVersion >> if self.data_version >= mm_cfg.DATA_FILE_VERSION: >> File "/www/lists.alcf.anl.gov/Mailman/MailList.py", line 146, in __getattr__ >> raise AttributeError, name >> AttributeError: data_version >> Site list for mailman (usually named mailman) missing >> Please create it; until then, mailman will refuse to start >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Traceback (most recent call last): >> File "/www/lists.mcs.anl.gov/bin/list_lists", line 122, in <module> >> main() >> File "/www/lists.mcs.anl.gov/bin/list_lists", line 94, in main >> mlist = MailList.MailList(n, lock=0) >> File "/www/lists.mcs.anl.gov/Mailman/MailList.py", line 130, in __init__ >> self.Load() >> File "/www/lists.mcs.anl.gov/Mailman/MailList.py", line 655, in Load >> raise Errors.MMCorruptListDatabaseError, e >> Mailman.Errors.MMCorruptListDatabaseError: [Errno 2] No such file or directory: '/www/lists.mcs.anl.gov/lists/p2s2-chairs/config.db.last' >> Site list for mailman (usually named mailman) missing >> Please create it; until then, mailman will refuse to start >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. >> Starting Mailman's master qrunner. > > > > On May 27, 2012, at 11:13 PM, Craig Stacey wrote: > >> fsck on /dev/vda1 came up clean, rebooting... >> >> On May 27, 2012, at 11:06 PM, Craig Stacey wrote: >> >>> From the logs on owney: >>> >>> May 27 22:38:41 owney kernel: [41501.728056] EXT4-fs error (device vda1): ext4_lookup: deleted inode referenced: 285537 >>> May 27 22:38:41 owney kernel: [41501.956729] Aborting journal on device vda1-8. >>> May 27 22:38:41 owney kernel: [41502.169946] EXT4-fs error (device vda1): ext4_journal_start_sb: Detected aborted journal >>> May 27 22:38:41 owney kernel: [41502.393211] EXT4-fs (vda1): Remounting filesystem read-only >>> >>> Then lots of complaints of read-only fs. I'm going to reboot owney unless I hear otherwise in the next 10 minutes. >>> >>> On May 27, 2012, at 6:13 PM, Dan Olson wrote: >>> >>>> franklin's hot migration finished a while ago. It looks like its working OK. >>>> I think were done at this point. We can look at the rest on Tuesday. >>>> >>>> ---- >>>> Daniel Murphy-Olson >>>> Systems Administrator >>>> Mathematics & Computer Science Division >>>> Argonne National Laboratory >>>> 630-252-0055 >>>> >>>> ----- Original Message ----- >>>> From: "Craig Stacey" <stace(a)mcs.anl.gov> >>>> To: "Dan Olson" <dolson(a)mcs.anl.gov> >>>> Cc: core-admins(a)mcs.anl.gov >>>> Sent: Sunday, May 27, 2012 2:10:10 PM >>>> Subject: Re: 221 vserver issues >>>> >>>> Agreed. >>>> >>>> And thanks again. >>>> >>>> -- >>>> Craig >>>> >>>> On May 27, 2012, at 1:58 PM, Dan Olson <dolson(a)mcs.anl.gov> wrote: >>>> >>>>> franklin is currently at 42% transferred to vserver4-221. It should complete around 5:30 or 6pm. Hopefully that goes well, if >>>>> it doesn't vserver3-221 will leave the source VM paused, we can unpause it if necessary. >>>>> >>>>> I verified rt, squall, and wind (wikis) are using coredb, not coredb-2. I think its a good time to walk away for a bit. >>>>> >>>>> ---- >>>>> Daniel Murphy-Olson >>>>> Systems Administrator >>>>> Mathematics & Computer Science Division >>>>> Argonne National Laboratory >>>>> 630-252-0055 >>>>> >>>>> ----- Original Message ----- >>>>> From: "Dan Olson" <dolson(a)mcs.anl.gov> >>>>> To: "Craig Stacey" <stace(a)mcs.anl.gov> >>>>> Cc: core-admins(a)mcs.anl.gov >>>>> Sent: Sunday, May 27, 2012 1:41:09 PM >>>>> Subject: Re: 221 vserver issues >>>>> >>>>> Yes >>>>> >>>>> ---- >>>>> Daniel Murphy-Olson >>>>> Systems Administrator >>>>> Mathematics & Computer Science Division >>>>> Argonne National Laboratory >>>>> 630-252-0055 >>>>> >>>>> ----- Original Message ----- >>>>> From: "Craig Stacey" <stace(a)mcs.anl.gov> >>>>> To: "Ken Raffenetti" <raffenet(a)mcs.anl.gov> >>>>> Cc: "Dan Olson" <dolson(a)mcs.anl.gov>, core-admins(a)mcs.anl.gov >>>>> Sent: Sunday, May 27, 2012 1:38:00 PM >>>>> Subject: Re: 221 vserver issues >>>>> >>>>> Thanks a lot. So the status of things as I understand it is: >>>>> >>>>> ALCF website moved to new vserver. >>>>> franklin/owney move is in progress, happening live >>>>> >>>>> status.alcf, blizzard, and dust are all staying on the server with the memory error for now. >>>>> >>>>> Yes? >>>>> >>>>> On May 27, 2012, at 1:36 PM, Ken Raffenetti wrote: >>>>> >>>>>> I think I've got the grant issues for wind figured out. All those extra >>>>>> gateway entries in /etc/network/interfaces were affecting how the mysql >>>>>> server saw connections. >>>>>> >>>>>> Ken Raffenetti >>>>>> Systems Administration Associate >>>>>> MCS Division - Argonne National Laboratory >>>>>> >>>>>> On 05/27/2012 01:17 PM, Dan Olson wrote: >>>>>>> squall (www.alcf) move is complete to vserver1-221. >>>>>>> >>>>>>> ---- >>>>>>> Daniel Murphy-Olson >>>>>>> Systems Administrator >>>>>>> Mathematics & Computer Science Division >>>>>>> Argonne National Laboratory >>>>>>> 630-252-0055 >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>> From: "Dan Olson" <dolson(a)mcs.anl.gov> >>>>>>> To: "Ken Raffenetti" <raffenet(a)mcs.anl.gov> >>>>>>> Cc: core-admins(a)mcs.anl.gov >>>>>>> Sent: Sunday, May 27, 2012 1:05:19 PM >>>>>>> Subject: Re: 221 vserver issues >>>>>>> >>>>>>> It should be up. The root volume spec in grub is probably wrong. >>>>>>> >>>>>>> ---- >>>>>>> Daniel Murphy-Olson >>>>>>> Systems Administrator >>>>>>> Mathematics & Computer Science Division >>>>>>> Argonne National Laboratory >>>>>>> 630-252-0055 >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>> From: "Ken Raffenetti" <raffenet(a)mcs.anl.gov> >>>>>>> To: core-admins(a)mcs.anl.gov >>>>>>> Sent: Sunday, May 27, 2012 12:59:37 PM >>>>>>> Subject: Re: 221 vserver issues >>>>>>> >>>>>>> I'm kicking it. The console showed it in busybox. If I shouldn't be >>>>>>> doing that, please let me know. >>>>>>> >>>>>>> Ken Raffenetti >>>>>>> Systems Administration Associate >>>>>>> MCS Division - Argonne National Laboratory >>>>>>> >>>>>>> On 05/27/2012 12:49 PM, Ken Raffenetti wrote: >>>>>>>> Should wind be up? It's not pingable right now. >>>>>>>> >>>>>>>> Ken Raffenetti >>>>>>>> Systems Administration Associate >>>>>>>> MCS Division - Argonne National Laboratory >>>>>>>> >>>>>>>> On 05/27/2012 12:43 PM, Craig Stacey wrote: >>>>>>>>> Thanks. I'm not sure how much help I can be but I can take a look at >>>>>>>>> coredb in a little bit. >>>>>>>>> -- Craig (from my mobile) >>>>>>>>> >>>>>>>>> Dan Olson <dolson(a)mcs.anl.gov> wrote: >>>>>>>>> >>>>>>>>> I've spent the last couple hours trying to find new homes for the critical vms that are on vserver3-221. Memory mirroring seems to have made the problem worse, instead of just panicing the box, it would reboot after a bit with no log info. So I turned that off. >>>>>>>>> >>>>>>>>> Moving the VMs live also failed, I attempted to move status.alcf to vserver1-221, which panicd that machine. >>>>>>>>> >>>>>>>>> I'm currently trying to move franklin to vserver4-221, and its progressing, but slowly. >>>>>>>>> >>>>>>>>> As far as I can tell the machines that are critical are >>>>>>>>> squall (www.alcf) - move in progress, currently offline >>>>>>>>> franklin (lists) - hot migration still progressing to vserver4-221, we will see how this pans out. If we need to migrate it offline it will take 5.5 hrs. >>>>>>>>> >>>>>>>>> Needs work - if anyone wants to assist - >>>>>>>>> coredb-2, something is wrong with the grants for the wikis on coredb, when I >>>>>>>>> point >>>>>>>>> wind at coredb the wikis don't work. >>>>>>>>> Other than this we should be able to point everything at coredb. >>>>>>>>> >>>>>>>>> May stay on the troubled vserver, we will try to get them off from the top to the bottom >>>>>>>>> blizzard (not sure how important confluence is at this point) >>>>>>>>> status.alcf >>>>>>>>> dust >>>>>>>>> >>>>>>>>> We do have an additional chassis that is not being used by alcf that matches vserver3-221 labeled vserver2-alcf in building 240, >>>>>>>>> however swapping the disks out of vserver3-221 into that machine seems more risky than the VM migration. >>>>>>>>> It would be nice to have that machine moved into place in 221 just in case it comes to that. >>>>>>>>> >>>>>>>>> >>>>>>>>> ---- >>>>>>>>> Daniel Murphy-Olson >>>>>>>>> Systems Administrator >>>>>>>>> Mathematics & Computer Science Division >>>>>>>>> Argonne National Laboratory >>>>>>>>> 630-252-0055 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Craig >>>>> >>>>> >>>>> >>>>> >>>>> >>> >>> -- >>> Craig >>> >>> >>> >> >> -- >> Craig >> >> >> > > -- > Craig > > > -- Craig

3 4

221 vserver issues
by Dan Olson 27 May '12

27 May '12

I've spent the last couple hours trying to find new homes for the critical vms that are on vserver3-221. Memory mirroring seems to have made the problem worse, instead of just panicing the box, it would reboot after a bit with no log info. So I turned that off. Moving the VMs live also failed, I attempted to move status.alcf to vserver1-221, which panicd that machine. I'm currently trying to move franklin to vserver4-221, and its progressing, but slowly. As far as I can tell the machines that are critical are squall (www.alcf) - move in progress, currently offline franklin (lists) - hot migration still progressing to vserver4-221, we will see how this pans out. If we need to migrate it offline it will take 5.5 hrs. Needs work - if anyone wants to assist - coredb-2, something is wrong with the grants for the wikis on coredb, when I point wind at coredb the wikis don't work. Other than this we should be able to point everything at coredb. May stay on the troubled vserver, we will try to get them off from the top to the bottom blizzard (not sure how important confluence is at this point) status.alcf dust We do have an additional chassis that is not being used by alcf that matches vserver3-221 labeled vserver2-alcf in building 240, however swapping the disks out of vserver3-221 into that machine seems more risky than the VM migration. It would be nice to have that machine moved into place in 221 just in case it comes to that. ---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055

3 10

Today's user termination list
by stace＠mcs.anl.gov 27 May '12

27 May '12

Full list of people who no longer appear on the active employee list =========================================================================== 59963 CLS Difley, Seth W. 67749 PSC Tu, Chao 68025 PSC Kim, Hyekyung Clarisse 83018 PSC Lee, Seok-Yong 83170 PSC Liedel, Clemens C. 83496 PSC Pemble, Charles W. 87368 PSC Yoo, Choong-Shik 213501 PSC Neubauer, Julie L. 215427 PSC Johnson, Zachary L. 216699 PSC Chung, Chih-Pin 217218 PSC Masoudi, Ali 217219 PSC Emptage, Ryan P. 218036 PSC Rein, Francisca N. 218478 PSC Niemeyer, Juergen 219508 PSC Liu, Nian 219589 PSC Fitzsimmons, Shawn R. 219592 PSC Tockstein, Michael A. 219593 PSC Hall, Victoria J. 225335 PSC Sun, Haohua

1 0

Today's user termination list
by stace＠mcs.anl.gov 26 May '12

26 May '12

Full list of people who no longer appear on the active employee list =========================================================================== 34013 NE Douglas, Barbara J. 63207 CSE Luo, Guangming 66135 PSC Lomakin, Ivan B. 67688 PSC Chiu, Wilson K. S. 67919 PSC Eliason, William K. 83807 PSC Luzin, Vladimir 83809 PSC Gretarsson, Hlynur 84378 PSC Buckley, David P. 84465 PSC DeVreugd, Christopher P. 84516 PSC Ahr, Brian C. 85605 PSC Eres, Gyula 86133 PSC Hemley, Russell J. 205189 PSC Steitz, Tom 206236 PSC Nelson, George J. 210793 PSC Lombardo, Jeffrey J. 216491 PSC Mangala Prasad, Vidya 217118 PSC Harris, Willaim M. 217133 PSC Kiss, Andrew M. 217171 PSC Konigsberg, William H. 217475 PSC Gagnon, Matthieu 217827 ES Torrel, Sol 217880 CIS Pattermann, Elizabeth M. 219554 PSC McDonough, Thomas J. 219558 PSC Crothamel, Matthew T. 219559 OTD DePaolo, Donald 219560 OTD Aselage, Terrence Lee 219561 OTD Wolinsky, Viviana 219562 OTD Bartlett, Bart 219563 OTD Davis, Jon C. 219564 PSC Beatty, Sarah M. B. 219565 PSC Chao, Emmanuel 219567 PSC Nguyen, Trongtuong B. 219569 PSC Aruda, Kenneth D. 219573 PSC Podkaminer, Jacob P. 219578 PSC Tariq, Faryal 219579 PSC Mahat, Meg B.

1 0

[CELS Sysadmins] Beer
by Tisha Stacey 25 May '12

25 May '12

For those of us who haven't already left -- I suggest 3 Corners in Lemont. - Tisha _______________________________________________ This is the CELS Sysadmins mailing list. It is sent to memebers of: core-admins(a)mcs.anl.gov, hpc-admins(a)mcs.anl.gov, systems(a)alcf.anl.gov, aig(a)alcf.anl.gov. Do not subscribe directly to this list. Thanks.

5 7