October 2007 - Mg-rast - mailman3.cels.anl.gov

mg-rast Pipeline added diskspace check
by Robert Olson 31 Oct '07

31 Oct '07

I have added a check for available disk space (arbitrarily set a 10G minimum for the pipeline's purposes) in the RAST and MG-RAST pipeline scripts. I have installed the updated script in the RAST dev, test, production backend serers. --bob

1 0

Re: mg-rast job 304 bad again
by Robert Olson 31 Oct '07

31 Oct '07

yes, all_jobs returns all jobs (inactive jobs are still valid). we should probably defer the creation of the meta object in the job48 init func until it is used. better, we'd defer any nontrivial operation until after the job obj is used. --bob On Oct 31, 2007, at 11:41 AM, Daniel Paarmann wrote: > mg-rast is responsive again. > > Marking a job as inactive apparently does not change the fact that > it is completely read/parsed by Job48->all_jobs. > > Daniel > > On Oct 31, 2007, at 10:20 AM, Robert Olson wrote: > >> The job that was bad because of the ^M line endings got restarted >> and still generated a bad meta.xml. The fix I made to invoke >> reformat_contigs should get installed ASAP onto the server. (as I >> am mostly out of the loop on the MGRAST I would prefer not to be >> updating the code there). >> >> I have marked that job inactive as it has bogus fasta data. >> >> Also, the manually changed job 305 meta.xml had bad perms. >> >> olson@bio-ppc-head-2:~$ ls -l /vol/metagenome-48-hour/Jobs.prod/ >> 305/meta.xml >> -rw-r--r-- 1 redwards metag48 2258876 Oct 22 20:05 /vol/ >> metagenome-48-hour/Jobs.prod/305/meta.xml >> >> I have fixed that as well. >> >

1 0

Re: mg-rast job 304 bad again
by Daniel Paarmann 31 Oct '07

31 Oct '07

mg-rast is responsive again. Marking a job as inactive apparently does not change the fact that it is completely read/parsed by Job48->all_jobs. Daniel On Oct 31, 2007, at 10:20 AM, Robert Olson wrote: > The job that was bad because of the ^M line endings got restarted > and still generated a bad meta.xml. The fix I made to invoke > reformat_contigs should get installed ASAP onto the server. (as I > am mostly out of the loop on the MGRAST I would prefer not to be > updating the code there). > > I have marked that job inactive as it has bogus fasta data. > > Also, the manually changed job 305 meta.xml had bad perms. > > olson@bio-ppc-head-2:~$ ls -l /vol/metagenome-48-hour/Jobs.prod/305/ > meta.xml > -rw-r--r-- 1 redwards metag48 2258876 Oct 22 20:05 /vol/ > metagenome-48-hour/Jobs.prod/305/meta.xml > > I have fixed that as well. >

1 0

Re: mg-rast Application heartbeat monitoring
by Tobias Paczian 31 Oct '07

31 Oct '07

I suppose this is a writeup of what has been decided last week where we already assigned people to these tasks? The reason why the monitoring can not yet be effective is that it will require (at least for RAST and MG-RAST) a backdoor to access the page to be monitored for the WGET. This backdoor will have to be implemented, tested and put into production. We need to assign who is supposed to do this. Cheers Tobi On Oct 31, 2007, at 11:14 AM, Folker Meyer wrote: > Hi all, > > caused not only by the latest series of Application failures > (NMPDR, RAST and today MG-RAST), I think it is time we did > something about this. > > My suggestion is to monitor the applications and have a chain of > events set into motion if certain conditions are not met. > > The applications to monitor and the email address to contact as I > see them today. > > NMPDR -> email to Bruce > RAST -> rast(a)mcs.anl.gov > MG-RAST -> mg-rast(a)mcs.anl.gov > > (It seems clear that email is not the right medium in a case of a > complete application failure!) > > --------------------- > MONITORING MECHANISM > ******************** > > I suggest a very simple mechanism to implement this: > > - compare a html page obtained by wget with a static page stored on > a monitoring server > > Explanation: > 0) define a set of URLs to monitor > 1) at the time of a system update, store a copy of the monitoring > results > 2) every 5-10 minutes re-run the wget procedure and compare to the > stored results > a) if no differences -> OK > b) if there are differences -> trigger event chain > > [The monitoring needs to include a notion of response time!] > > --------------------- > CHAIN OF EVENTS > *************** > In an optimal world, we would have on pager per application and the > responsible person would be notified if they need to intervene. > > I think this is not required at this stage, if we can assume that > emails will be read in a timely manner by the people that can > intervene. Maybe the production team to be established can discuss > the need for a pager. > > ---- > > SUGGESTED SET OF MILESTONES > *************************** > > To achieve this I suggest the following steps: > > - server teams define a set of URLs into the apps to be monitored > [TODAY, I will send them as I define them] > (any problems aka user/pw requirements should be resolved!] > - initial monitoring framework is established [THIS WEEK] > - first test for monitoring [NEXT WEEK] > - we will artificially introduce problems to the development > servers (that will also be monitored for this test only] > (Bob and Folker) > - the production team will react to the error messages sent by > the monitoring (if monitoring works!) > > > I hope we can have a very brief discussion about this and then > proceed to implement it. > > > Folker > > > > > > > > > > > > > > > >

1 0

mg-rast Application heartbeat monitoring
by Folker Meyer 31 Oct '07

31 Oct '07

Hi all, caused not only by the latest series of Application failures (NMPDR, RAST and today MG-RAST), I think it is time we did something about this. My suggestion is to monitor the applications and have a chain of events set into motion if certain conditions are not met. The applications to monitor and the email address to contact as I see them today. NMPDR -> email to Bruce RAST -> rast(a)mcs.anl.gov MG-RAST -> mg-rast(a)mcs.anl.gov (It seems clear that email is not the right medium in a case of a complete application failure!) --------------------- MONITORING MECHANISM ******************** I suggest a very simple mechanism to implement this: - compare a html page obtained by wget with a static page stored on a monitoring server Explanation: 0) define a set of URLs to monitor 1) at the time of a system update, store a copy of the monitoring results 2) every 5-10 minutes re-run the wget procedure and compare to the stored results a) if no differences -> OK b) if there are differences -> trigger event chain [The monitoring needs to include a notion of response time!] --------------------- CHAIN OF EVENTS *************** In an optimal world, we would have on pager per application and the responsible person would be notified if they need to intervene. I think this is not required at this stage, if we can assume that emails will be read in a timely manner by the people that can intervene. Maybe the production team to be established can discuss the need for a pager. ---- SUGGESTED SET OF MILESTONES *************************** To achieve this I suggest the following steps: - server teams define a set of URLs into the apps to be monitored [TODAY, I will send them as I define them] (any problems aka user/pw requirements should be resolved!] - initial monitoring framework is established [THIS WEEK] - first test for monitoring [NEXT WEEK] - we will artificially introduce problems to the development servers (that will also be monitored for this test only] (Bob and Folker) - the production team will react to the error messages sent by the monitoring (if monitoring works!) I hope we can have a very brief discussion about this and then proceed to implement it. Folker

1 0

Re: mg-rast job 304 bad again
by Daniel Paarmann 31 Oct '07

31 Oct '07

I've updated mg_preprocess on the production mg-rast. - Daniel On Oct 31, 2007, at 10:20 AM, Robert Olson wrote: > The job that was bad because of the ^M line endings got restarted > and still generated a bad meta.xml. The fix I made to invoke > reformat_contigs should get installed ASAP onto the server. (as I > am mostly out of the loop on the MGRAST I would prefer not to be > updating the code there). > > I have marked that job inactive as it has bogus fasta data. > > Also, the manually changed job 305 meta.xml had bad perms. > > olson@bio-ppc-head-2:~$ ls -l /vol/metagenome-48-hour/Jobs.prod/305/ > meta.xml > -rw-r--r-- 1 redwards metag48 2258876 Oct 22 20:05 /vol/ > metagenome-48-hour/Jobs.prod/305/meta.xml > > I have fixed that as well. >

1 0

mg-rast job 304 bad again
by Robert Olson 31 Oct '07

31 Oct '07

The job that was bad because of the ^M line endings got restarted and still generated a bad meta.xml. The fix I made to invoke reformat_contigs should get installed ASAP onto the server. (as I am mostly out of the loop on the MGRAST I would prefer not to be updating the code there). I have marked that job inactive as it has bogus fasta data. Also, the manually changed job 305 meta.xml had bad perms. olson@bio-ppc-head-2:~$ ls -l /vol/metagenome-48-hour/Jobs.prod/305/ meta.xml -rw-r--r-- 1 redwards metag48 2258876 Oct 22 20:05 /vol/ metagenome-48-hour/Jobs.prod/305/meta.xml I have fixed that as well.

1 0

Re: mg-rast down
by Folker Meyer 31 Oct '07

31 Oct '07

Update: ------------snip------------- Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, olson(a)mcs.anl.gov and inform them of the time the error occurred, and anything you might have done that may have caused the error. More information about this error may be available in the server error log. ------------snip------------- after about 15 minutes. Did we start to recycle old job IDs? I have noticed job numbers 1-5 being touched on October 30th? I assume at this time that the database on bioseed is the culprit... Folker On Oct 31, 2007, at 7:25 AM, Folker Meyer wrote: > Hi all, > > when I go to the production MG-Rast I do not get this message! > > However I do not get any of our web-pages instead I get the spinned > wheel of death for ever. > > I am now looking at the internals of treponema. > > Folker > > > > On Oct 30, 2007, at 11:59 PM, Rob Edwards wrote: > >> mg-rast is down at the moment, returning an internal server error. >> >> This is from the log on bioseed: >> >> [Tue Oct 30 23:28:10 2007] [error] [client 130.202.241.242] Useless >> use of division (/) in void context at /vol/48-hour/FIGdisk.dev/ >> dist/releases/dev/cee/lib/FortyEight/WebPage/ControlCenter.pm line >> 331., referer: http://bioseed.mcs.anl.gov/48-hour-dev/FIG/rast.cgi >> [Tue Oct 30 23:28:10 2007] [error] [client 130.202.241.242] >> Bareword "g" not allowed while "strict subs" in use at /vol/48-hour/ >> FIGdisk.dev/dist/releases/dev/cee/lib/FortyEight/WebPage/ >> ControlCenter.pm line 331., referer: http://bioseed.mcs.anl.gov/48-hour-dev/FIG/rast.cgi >> [Tue Oct 30 23:28:10 2007] [error] [client 130.202.241.242] >> Compilation failed in require at (eval 27) line 3., referer: http://bioseed.mcs.anl.gov/48-hour-dev/FIG/rast.cgi >> >> >> Rob >> >> >

1 0

Re: mg-rast down
by Folker Meyer 31 Oct '07

31 Oct '07

Hi all, when I go to the production MG-Rast I do not get this message! However I do not get any of our web-pages instead I get the spinned wheel of death for ever. I am now looking at the internals of treponema. Folker On Oct 30, 2007, at 11:59 PM, Rob Edwards wrote: > mg-rast is down at the moment, returning an internal server error. > > This is from the log on bioseed: > > [Tue Oct 30 23:28:10 2007] [error] [client 130.202.241.242] Useless > use of division (/) in void context at /vol/48-hour/FIGdisk.dev/dist/ > releases/dev/cee/lib/FortyEight/WebPage/ControlCenter.pm line 331., > referer: http://bioseed.mcs.anl.gov/48-hour-dev/FIG/rast.cgi > [Tue Oct 30 23:28:10 2007] [error] [client 130.202.241.242] Bareword > "g" not allowed while "strict subs" in use at /vol/48-hour/ > FIGdisk.dev/dist/releases/dev/cee/lib/FortyEight/WebPage/ > ControlCenter.pm line 331., referer: http://bioseed.mcs.anl.gov/48-hour-dev/FIG/rast.cgi > [Tue Oct 30 23:28:10 2007] [error] [client 130.202.241.242] > Compilation failed in require at (eval 27) line 3., referer: http://bioseed.mcs.anl.gov/48-hour-dev/FIG/rast.cgi > > > Rob > >

1 0

mg-rast down
by Rob Edwards 30 Oct '07

30 Oct '07

mg-rast is down at the moment, returning an internal server error. This is from the log on bioseed: [Tue Oct 30 23:28:10 2007] [error] [client 130.202.241.242] Useless use of division (/) in void context at /vol/48-hour/FIGdisk.dev/dist/ releases/dev/cee/lib/FortyEight/WebPage/ControlCenter.pm line 331., referer: http://bioseed.mcs.anl.gov/48-hour-dev/FIG/rast.cgi [Tue Oct 30 23:28:10 2007] [error] [client 130.202.241.242] Bareword "g" not allowed while "strict subs" in use at /vol/48-hour/ FIGdisk.dev/dist/releases/dev/cee/lib/FortyEight/WebPage/ ControlCenter.pm line 331., referer: http://bioseed.mcs.anl.gov/48- hour-dev/FIG/rast.cgi [Tue Oct 30 23:28:10 2007] [error] [client 130.202.241.242] Compilation failed in require at (eval 27) line 3., referer: http:// bioseed.mcs.anl.gov/48-hour-dev/FIG/rast.cgi Rob

1 0