December 2007 - Mg-rast - mailman3.cels.anl.gov

things missing from the mg-rast
by Rob Edwards 15 Dec '07

15 Dec '07

These are somethings that are missing from the mg-rast that I have thought of while thinking about the NIH HMP RFA. I think that some of these could be trivial for a first-order implementation and would help us while writing the grant. 1. Cross references It would be invaluable if we have a way to link our metagenomes to other sites, because that is one of the immediate requirements of the RFA. I think we can easily add something to the meta.xml that houses linkouts, and ask for them when people upload the data. Obviously it would be nice for a way to represent them in the output, but I am not sure that is essential at the moment. I suggest we ask submittors whether they have accession numbers for NCBI, EBI, DDBJ, CAMERA, and IMG, stuff them in the XML and call it good! 2. Downloading raw data Can we easily provide a link so others can download the raw, unprocessed data for public metagenomes. 3. Will public metagenomes be available without logging in? I'm undecided on this, but overall I'd say probably. We could just set up a guest account that people can use to see public metagenomes 4. MEGAN format data Can we generate a file in megan format? I think someone (Folker?) was playing with that, and it should be relatively trivial to generate the data and provide a link so users can download it, right? Rob p.s. We should have about 100 public metagenomes by the end of the week ...

1 0

mg-rast Re: Server downtime
by Robert Olson 13 Dec '07

13 Dec '07

We are back, upgraded to SGE 6.1U3. You shouldn't see any changes except that the qstat command now only shows your jobs by default. You can see them all by adding the argument -u "*". For example: $ qstat $ qstat -u '*' job-ID prior name user state submit/start at queue slots ja-task-ID ------------------------------------------------------------------------ ----------------------------------------- 87814 0.10500 sleep rastcode r 12/13/2007 17:31:31 all.q(a)bio-ppc-3.mcs.anl.gov 1 I have restarted the servers' backends. --bob On Dec 13, 2007, at 10:50 AM, Robert Olson wrote: > I would like to idle the rast and mg-rast servers today so we can > drain the scheduler queues so I can upgrade to the new version of > SGE; this will let us put the policies in place that let MGRAST not > be starved for cycles when RAST is busy. > > Do we have an easy mechanism for putting a message of the day on > the frontpage of the servers? > > I'm thinking we can leave the frontend active and just stop the > pipeline (that's what I mean by idling the server, btw). That way > uploads and browsing of completed jobs can still go on; job > processing would just not happen). > > --bob >

1 0

mg-rast Jobs
by Robert Olson 13 Dec '07

13 Dec '07

Looking thru the running sims jobs .. there were two stuck on node 4, which is down (more busted hardware I think). I rescheduled them to run. The job on node 13 has a humongous single query sequence: olson@bio-ppc-13:~$ ls -l /vol/metagenome-48-hour/Jobs.prod/214/proc/ sims.seed/sims.in/317.fa total 3520 -rw-rw-rw- 1 olson metag48 3598632 Nov 29 10:06 in.1 this seems to be a bug ... the blast has been running since nov 29 and has written zero bytes of output: olson@bio-ppc-13:~$ ps uww14844 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND olson 14844 99.8 28.8 1736452 1164836 ? R Nov29 20243:40 /vol/ metagenome-48-hour/FIGdisk.backend.prod/env/cee/bin/blastall -m 8 -e 0.1 -p blastx -i /vol/metagenome-48-hour/Jobs.prod/214/proc/sims.seed/ sims.in/317.fa/in.1 -d /scratch/metagenome-48-hour/Data/ SEED_2006_07_01 -o /vol/metagenome-48-hour/Jobs.prod/214/proc/ sims.seed/sims.raw/317.fa/out.1 olson@bio-ppc-13:~$ ls -l /vol/metagenome-48-hour/Jobs.prod/214/proc/ sims.seed/sims.raw/317.fa/out.1 -rw-r--r-- 1 olson metag48 0 Nov 29 10:06 /vol/metagenome-48-hour/ Jobs.prod/214/proc/sims.seed/sims.raw/317.fa/out.1 I have killed that process. It appears that the blast for job 215 is similarly operating on a huge single input file: olson@bio-ppc-26:~$ ps wwuw14054 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND olson 14054 99.7 9.9 1483480 401176 ? R Dec11 2747:32 /vol/ metagenome-48-hour/FIGdisk.backend.prod/env/cee/bin/blastall -m 8 -e 0.1 -p blastx -i /vol/metagenome-48-hour/Jobs.prod/215/proc/sims.seed/ sims.in/334.fa/in.1 -d /scratch/metagenome-48-hour/Data/ SEED_2006_07_01 -o /vol/metagenome-48-hour/Jobs.prod/215/proc/ sims.seed/sims.raw/334.fa/out.1 olson@bio-ppc-26:~$ ls -l /vol/metagenome-48-hour/Jobs.prod/215/proc/ sims.seed/sims.in/334.fa/in.1 -rw-rw-rw- 1 olson metag48 2847768 Dec 6 11:42 /vol/metagenome-48- hour/Jobs.prod/215/proc/sims.seed/sims.in/334.fa/in.1 olson@bio-ppc-26:~$ ls -l /vol/metagenome-48-hour/Jobs.prod/215/proc/ sims.seed/sims.raw/334.fa/out.1 -rw-r--r-- 1 olson metag48 0 Dec 11 14:04 /vol/metagenome-48-hour/ Jobs.prod/215/proc/sims.seed/sims.raw/334.fa/out.1 and should probably be killed and the job fixed to split the input more appropriately.

1 0

mg-rast Server downtime
by Robert Olson 13 Dec '07

13 Dec '07

I would like to idle the rast and mg-rast servers today so we can drain the scheduler queues so I can upgrade to the new version of SGE; this will let us put the policies in place that let MGRAST not be starved for cycles when RAST is busy. Do we have an easy mechanism for putting a message of the day on the frontpage of the servers? I'm thinking we can leave the frontend active and just stop the pipeline (that's what I mean by idling the server, btw). That way uploads and browsing of completed jobs can still go on; job processing would just not happen). --bob

1 0

mg-rast Metagenomics Jobs
by Rob Edwards 11 Dec '07

11 Dec '07

Andrew, The jobs that you submitted to the metagenomics RAST service back in July have finally finished. I am truly sorry that they took so long, they should not have done so. Typically jobs take just a day or two to be analyzed. Unfortunately we didn't notice that yours was stuck in an unresolved state until late last week. Once the error was cleared everything processed as normal. We're working through a backlog of analyses and things should be back to normal next week. If you have questions about the service or processing, please contact mg-rast(a)mcs.anl.gov My apologies, Rob Edwards for the mg-rast team.

1 0

Re: mg-rast Node Undefined Error
by Rob Edwards 07 Dec '07

07 Dec '07

the last job that I submitted that had the error was a while back, but I only just figured out the fix. I'm not sure whether it is still occurring or if all the ones that i fixed yesterday were the result of old submissions. I'll let you know if I see it crop up again Rob On Dec 7, 2007, at 4:44 AM, Robert Olson wrote: > > there may be an update needed to GenomeMeta.pm if you are seeing that > error; I thought I had fixed that. > > On Thu, 6 Dec 2007, Rob Edwards wrote: > >> Hey all >> >> I tracked down the problem that was randomly giving this error: >> >> Can't call method "nodeValue" on an undefined value at /vol/ >> metagenome-48-hour/FIGdisk.prod/dist/releases/dev/cee/lib/ >> FigKernelPackages/GenomeMeta.pm line 398. >> >> The offending line in the meta.xml file is: >> >> <entry name="optional_info.time"> >> <scalar/> >> >> >> Note the malformed XML <scalar/> there -- it should at least be >> <scalar /> although that doesn't appear to fix the problem. >> >> Replacing that line with >> >> <entry name="optional_info.time"> >> <scalar> >> </scalar> >> >> resolves the bug. >> >> In jobs that have stalled, I then have to set the stage that they are >> complete on as complete, and then the job seems to pick up fine from >> there. >> >> Rob >> >> >> >>

1 0

Re: mg-rast Node Undefined Error
by Robert Olson 07 Dec '07

07 Dec '07

there may be an update needed to GenomeMeta.pm if you are seeing that error; I thought I had fixed that. On Thu, 6 Dec 2007, Rob Edwards wrote: > Hey all > > I tracked down the problem that was randomly giving this error: > > Can't call method "nodeValue" on an undefined value at /vol/ > metagenome-48-hour/FIGdisk.prod/dist/releases/dev/cee/lib/ > FigKernelPackages/GenomeMeta.pm line 398. > > The offending line in the meta.xml file is: > > <entry name="optional_info.time"> > <scalar/> > > > Note the malformed XML <scalar/> there -- it should at least be > <scalar /> although that doesn't appear to fix the problem. > > Replacing that line with > > <entry name="optional_info.time"> > <scalar> > </scalar> > > resolves the bug. > > In jobs that have stalled, I then have to set the stage that they are > complete on as complete, and then the job seems to pick up fine from > there. > > Rob > > > >

1 0

Re: mg-rast 6.fasta
by Jeremy Koenig 06 Dec '07

06 Dec '07

That's great thanks Rob! -JK On Dec 6, 2007 1:07 PM, Rob Edwards <redwards(a)mcs.anl.gov> wrote: > Jeremy > > I have been trying to debug mg-rast jobs, and your job 6.fasta failed > because the fasta file was not correctly formated. First, it had X's > in the sequence - this is a problem with some software, X is not a > valid character in a nucleotide fasta file (N means unknown in > nucleotide, X in protein), and second there were some comments at the > end of the file. I have fixed this problem, and hopefully this job > will complete soon. > > Rob > > >

1 0

Re: mg-rast error
by Blaby, Ian K 06 Dec '07

06 Dec '07

Hi Rob, Thanks for getting back to me. I've resubmitted the file to the other server. I generated the FASTA file by copy+pasting into notepad and saving as a *.fa file (save as type=all files), so presumably this should be saving as the correct file type. Thanks for your help Ian -- I.K. Blaby PhD Dept Microbiology & Cell Sciences University of Florida Rob Edwards wrote: > Dear Ian > > I have been catching errors in the metagenomics rast server, and > noticed that you submitted job #215 (Hfx.volc) that had an error. The > error was that the fasta file was in html format and not text format. > > I have fixed the error, and your job should complete. However, in the > process I also noticed that you have a single sequence that is > 2,847,757 bp. The metagenomics rast is not designed to handle such > sequences efficiently, but we also provide a service for annotating > larger fragments. This has a much better algorithm for finding the > genes (which in metagenomes is particularly tricky), and I recommend > that you run your sequence through that service. > > The URL is > > http://rast.nmpdr.org/ > > The login and account information are the same. > > Please note that you should remove the HTML tags from your fasta file > or else your sequence won't work on that service either! > > Thanks, and please let us know if you have questions > > Rob Edwards > On behalf of the RAST and MG-RAST team > >

1 0

mg-rast 6.fasta
by Rob Edwards 06 Dec '07

06 Dec '07

Jeremy I have been trying to debug mg-rast jobs, and your job 6.fasta failed because the fasta file was not correctly formated. First, it had X's in the sequence - this is a problem with some software, X is not a valid character in a nucleotide fasta file (N means unknown in nucleotide, X in protein), and second there were some comments at the end of the file. I have fixed this problem, and hopefully this job will complete soon. Rob

1 0