April 2009 - Mg-rast - mailman3.cels.anl.gov

Re: [mg-rast] Your MG-RAST job (3300)
by Mark D'Souza 30 Apr '09

30 Apr '09

Hi Ramiro, 2828 is proceeding normally. There are over 3 Gbp of sequence data in the queue so some slowdown is inevitable, especially for the large jobs. I am checking into the other two. Regards, Mark ----- "Ramiro Logares" <ramiro.logares(a)gmail.com> wrote: > From: "Ramiro Logares" <ramiro.logares(a)gmail.com> > To: "Mark D'Souza" <dsouza(a)mcs.anl.gov> > Sent: Thursday, April 30, 2009 10:25:52 AM GMT -06:00 US/Canada Central > Subject: Re: Your MG-RAST job (3300) > > Hi, > Thanks a lot for checking and fixing this. BTW, I have two jobs > running (3160 and 1112) that seem to be stopped due to a failure. > Then > the job 2828 seems to be OK, but it has been several days in the same > status, maybe it just stopped? should I resubmit these jobs? > > Thanks for having this great portal running! > Ramiro > > On Thu, Apr 30, 2009 at 3:27 AM, Mark D'Souza <dsouza(a)mcs.anl.gov> > wrote: > > Hi, > > > > The job you submitted to MG-RAST (3300) has failed as the sequence > file has incorrectly formatted fasta -- the sequence ID Gruv_6 should > be on a new line. > > > >>Gruv_5 > > > CGTAATCAACGCGAGCTAGTGACTCGCGCTTACTGGGAATTCCACGTTCACGGAGAAAAATTTCAAGCCCCGATCCCAATCACGAAGGGGTTTCAGCGGGTTACCCACTCCTTTCGGAGCAGGAAGAACACGCTTATCCCTTCAGTGTAGCGCGCGTGCGGCCCCGAACATCTAAGGGCATCACAGACCTGTTATTGCTCAATTTC > > > GTGTAGCTAACCGCTACTTGTCCCTCTAAGAAGTACGCGCCGACAAAAGATCGGCGTACTATTTAGCAGGCTAGAGTCTCGTTCGTTATCGGAATTAACCAGACAAATCGCTCCACCAACTAAGAACGGCCATGCACCACCACCCACCGAATCAAGAAAGAGCTCTCAATCTGTCAATCCTTACAGTGTCCGGGCCGGGTGAGTTT > > > TCCCGTGTTGAGTCAAATTAAGCCGCAGGCTCCACTCCTGGTGGTGCCCTTCCGTCAATTCCTTTAAGTTTCAGCTCTGCAACCATACTTCCCCCGGAACCCAAAGACTTTGG>Gruv_6 > > > CGTAGATATTCGGAAGAACACCAGTGGCGAAGGCGGCTTACTGGCCCATTACTGACGCTGAGGCTCGAAAGCGTGGGGAGCAAACAGGATTAGATACCCTGGTAGTCCACGCTGTAAACGATGAGTGCTAGTTGTCGGCATGCATGCATGTCGGTGACGCAGCTAACGCCTTAAGCACTCCGCCTGGGGAGTACGGTCGCAAGATT > > > AAAACTCAAAGGAATTGACGGGGGCCCGCACAAGCGGTGGAGCATGTGGTTTAATTCGAAGCAACGCGCAGAACCTTACCACCTTTTGACATGCCTGGACCGCCACAGAGATGTGGTTTTCCCTTCGGGGACTAGGACACAGGTGCTGCATGGCTGTCGTCAGCTCGTGTCGTGAGATGTTGGGTTAAGTCCCGCAACGAGCGCAA > > CCCTCGCTATTAGTTGCCATCAGGTTTGGCTGGGCACTCTAATAGGACCGCCGGTGGTAAGCCGGA > > > > I made this change and restarted the job, please check your sequence > data to make sure that nothing was lost while preparing the files for > resubmission. You may also choose to delete this job and resubmit a > corrected fasta file. > > > > Regards, > > Mark D'Souza > > -- for the MG-RAST team > > > > mg-rast(a)mcs.anl.gov > > http://metagenomics.nmpdr.org/dsouza > > -- Regards, Mark D'Souza -- for the MG-RAST team mg-rast(a)mcs.anl.gov http://metagenomics.nmpdr.org/

1 0

Re: [mg-rast] Your MG-RAST jobs (3283, 3284, 3285)
by Mark D'Souza 30 Apr '09

30 Apr '09

Hi Isaac, This seems to be a rare case as we have never had this problem before. I suspect that there are at least two independent issues at play -- the inclusion of the './' in the file names and the inclusion of the '._' files in the tarball. The first could be the result of invoking tar with './' before each file to be included, and the second is a Mac thing. I tried doing a tar on my Mac and untar on Linux, and the following seems to work: tar czvf bc.tgz bc.fasta bc.qual bc.fasta bc.qual where bc.fasta and bc.qual are plain text uncompressed files. Listing the files should then give: tar tvzf bc.tgz -rw-r--r-- dsouza/ 1330 2009-04-30 13:36:35 bc.fasta -rw-r--r-- dsouza/ 2175 2009-04-30 13:36:45 bc.qual without the '._' files. Try this and see if it works, get in touch if it doesn't. Regards, Mark ----- "Isaac Hilburn" <ihilburn(a)caltech.edu> wrote: > From: "Isaac Hilburn" <ihilburn(a)caltech.edu> > To: "Mark D'Souza" <dsouza(a)mcs.anl.gov> > Cc: "Rachel Poretsky" <poretsky(a)caltech.edu> > Sent: Wednesday, April 29, 2009 7:09:59 PM GMT -06:00 US/Canada Central > Subject: Re: Your MG-RAST jobs (3283, 3284, 3285) > > Mark, > > Has there been a problem in the past uploading files that are > compressed using compression programs on a Mac computer? The first > set of submitted files I compressed using a sharewire tar / gzip > compressor (iArchive). I tried to submit the data again after > compressing the .qual and .fasta files into a tar.gz archive using the > > GNU tar command line application in Mac OS X, and I got the same > initial error as the first submission (i.e. the upload system sees two > > sequence files ._BC2ANME2_454-reads.fasta and BC2ANME2_454- > reads.fasta) while there is only actually one sequence file in the > tar.gz archive. > > Do you have any recommendations for how I should compress the files > > to get rid of this problem? > > Thank you, > ---Isaac > > On Apr 29, 2009, at 11:57 AM, Mark D'Souza wrote: > > > Hi, > > > > I'm afraid the MG-RAST jobs you recently submitted (3283, 3284, > > 3285) have failed. It looks like the sequence and quality files have > > > been tarred separately and then tarred (and gzipped) together a > > second time and the system was unable to figure out how to handle > > these files. Please resubmit each job as a tarred gzipped file > > containing the plain text fasta and quality files. > > > > Thanks, > > Mark D'Souza > > -- for the MG-RAST team > > > > mg-rast(a)mcs.anl.gov > > http://metagenomics.nmpdr.org/dsouza

1 0

Your MG-RAST job (3300)
by Mark D'Souza 29 Apr '09

29 Apr '09

Hi, The job you submitted to MG-RAST (3300) has failed as the sequence file has incorrectly formatted fasta -- the sequence ID Gruv_6 should be on a new line. >Gruv_5 CGTAATCAACGCGAGCTAGTGACTCGCGCTTACTGGGAATTCCACGTTCACGGAGAAAAATTTCAAGCCCCGATCCCAATCACGAAGGGGTTTCAGCGGGTTACCCACTCCTTTCGGAGCAGGAAGAACACGCTTATCCCTTCAGTGTAGCGCGCGTGCGGCCCCGAACATCTAAGGGCATCACAGACCTGTTATTGCTCAATTTC GTGTAGCTAACCGCTACTTGTCCCTCTAAGAAGTACGCGCCGACAAAAGATCGGCGTACTATTTAGCAGGCTAGAGTCTCGTTCGTTATCGGAATTAACCAGACAAATCGCTCCACCAACTAAGAACGGCCATGCACCACCACCCACCGAATCAAGAAAGAGCTCTCAATCTGTCAATCCTTACAGTGTCCGGGCCGGGTGAGTTT TCCCGTGTTGAGTCAAATTAAGCCGCAGGCTCCACTCCTGGTGGTGCCCTTCCGTCAATTCCTTTAAGTTTCAGCTCTGCAACCATACTTCCCCCGGAACCCAAAGACTTTGG>Gruv_6 CGTAGATATTCGGAAGAACACCAGTGGCGAAGGCGGCTTACTGGCCCATTACTGACGCTGAGGCTCGAAAGCGTGGGGAGCAAACAGGATTAGATACCCTGGTAGTCCACGCTGTAAACGATGAGTGCTAGTTGTCGGCATGCATGCATGTCGGTGACGCAGCTAACGCCTTAAGCACTCCGCCTGGGGAGTACGGTCGCAAGATT AAAACTCAAAGGAATTGACGGGGGCCCGCACAAGCGGTGGAGCATGTGGTTTAATTCGAAGCAACGCGCAGAACCTTACCACCTTTTGACATGCCTGGACCGCCACAGAGATGTGGTTTTCCCTTCGGGGACTAGGACACAGGTGCTGCATGGCTGTCGTCAGCTCGTGTCGTGAGATGTTGGGTTAAGTCCCGCAACGAGCGCAA CCCTCGCTATTAGTTGCCATCAGGTTTGGCTGGGCACTCTAATAGGACCGCCGGTGGTAAGCCGGA I made this change and restarted the job, please check your sequence data to make sure that nothing was lost while preparing the files for resubmission. You may also choose to delete this job and resubmit a corrected fasta file. Regards, Mark D'Souza -- for the MG-RAST team mg-rast(a)mcs.anl.gov http://metagenomics.nmpdr.org/dsouza

1 0

Your MG-RAST jobs (3283, 3284, 3285)
by Mark D'Souza 29 Apr '09

29 Apr '09

Hi, I'm afraid the MG-RAST jobs you recently submitted (3283, 3284, 3285) have failed. It looks like the sequence and quality files have been tarred separately and then tarred (and gzipped) together a second time and the system was unable to figure out how to handle these files. Please resubmit each job as a tarred gzipped file containing the plain text fasta and quality files. Thanks, Mark D'Souza -- for the MG-RAST team mg-rast(a)mcs.anl.gov http://metagenomics.nmpdr.org/dsouza

1 0

Fwd: connecting fasta names with phylogeny
by Folker Meyer 29 Apr '09

29 Apr '09

did we answer her? Begin forwarded message: > From: "Katrine Whiteson" <katrine.whiteson(a)genomic.ch> > Date: April 22, 2009 8:26:09 AM CDT > To: "Folker Meyer" <folker.meyer(a)googlemail.com> > Subject: Re: [mg-rast] connecting fasta names with phylogeny > > Hi Folker, > > Last month I asked you about whether MG_RAST can supply a file that > contains both phylogenetic classification info and the fasta header > from the sequence that was submitted. We have been submitting only > unique sequences, with the info about how many times a sequence > appears in the fasta header. A file with the classification + fasta > header would make it easy for us to look at how the bacterial > populations are distributed. > > Do you know when this featuer will be available? (We can also > submit all of the sequences, not just the unique ones...). > > I have another question about RAST genome comparison. How can I find > out how the identity values are calculated for the compare genomes > feature. These values are described as "percent protein sequence > identity", and it is great that they can be exported in a table. > Does this identity value literally mean that the amino acid > sequences are compared, and if 1 out of 100 amino acids are > different it would be an identity of 99%? > > thank you! > > Katrine > _____________ > post-doc > University of Geneva Hospitals > www.genomic.ch > > > ----- Original Message ----- > From: Folker Meyer > To: Katrine Whiteson > Sent: Wednesday, March 04, 2009 4:40 PM > Subject: Re: [mg-rast] connecting fasta names with phylogeny > > KAtherine: > > the Algorithm is independent of the RDP classifier. > > While we support the use of 16s fragments for classification > purposes, using fragments of protein coding genes is what MG-RAST is > mostly about. We attempt to place fragments phylogenetically into > the NCBI taxonomy. > > The algorithm is simply a BLAST search. > > We will in the near future replace this by a protein family (FIGfam) > based decision procedure that places the metagenomic fragments in to > an alignment of all members of a protein family and derives the > phylogenetic placement from that. > > > Hope this helps, > Folker > > > > On Mar 4, 2009, at 7:28 AM, Katrine Whiteson wrote: > >> Folker, >> I have one more question about how phylogenetic comparisons are >> made on MG-RAST. Is the algorithm used similar to the RDP >> classifier? I have read the recent paper in BMC Bioinformations but >> I couldn't figure out what kind of algorithm is being used. >> >> It would be very useful if the algorithm is independent from the >> RDP classifier, as an independent check. >> >> thank you! >> Katrine >> ----- Original Message ----- >> From: Folker Meyer >> To: Katrine Whiteson >> Cc: mg-rast(a)mcs.anl.gov >> Sent: Tuesday, March 03, 2009 8:27 PM >> Subject: Re: [mg-rast] connecting fasta names with phylogeny >> >> Katrine: >> >> we are currently working on an updated version of our genbank >> export file creation process. The new version of the genbank files >> will include a suggested phylogenetic placement. >> >> Are your sequences 100% identical when you "bin" them into "unique" >> sequence sets? >> >> Best, >> Folker >> On Mar 3, 2009, at 3:53 AM, Katrine Whiteson wrote: >> >>> Hi, >>> I would like to know if it is possible to get a single file that >>> contains the name of each of my sequences as I submitted them in >>> my fasta file, and the phylogenetic classifications. >>> >>> I see that if I click on a particular organism, I can see the >>> fasta file names for the subset of sequences that are classified >>> there, but is it possible to download a file containing all the >>> classified sequences, that has the sequence names as well as the >>> taxonomy? The table you can export with all classified sequences >>> appears to only contain the taxonomy. >>> >>> (My fasta sequence files include the number of times that a >>> particular sequence is present in my data, as I uploaded only >>> unique sequences, and I would like to reconstruct the actual >>> phylogenetic composition, taking into account the number of times >>> each sequence is present.) >>> >>> thank you! >>> >>> Katrine >>> post-doc >>> www.genomic.ch >>> >> >> > >

1 0

mg-rast sims jobs failed
by Mark D'Souza 28 Apr '09

28 Apr '09

Hi Bob, There are a number of sim jobs for 3204 and 3205 which have 0 file size in the sims.seed/sims.raw/.../ directory. This is suspicious as they are all dated around the time the disk got full. The files in sge_output are zero size too. Maybe the sim jobs did not get redone when the queue got restarted? Is there any way to know if these chunks will be recomputed? Is there any way to flag these chunks for recalculation? These are the only two jobs I checked, there may be others. Mark

1 0

Re: [mg-rast] 800bp contigs
by Mark D'Souza 27 Apr '09

27 Apr '09

Helen, I believe you wandered onto the RAST site and tried the upload there -- there are traces of this happening, and RAST does impose a contig length limit. The sites look very similar below the banner. No harm done, the job did get loaded to MG-RAST and is now being processed. Regards, Mark On Mon, 27 Apr 2009, Helen White wrote: > Mark > Yes, that is the file of mine in the queue. When I tried to upload it at one > point I received the below error, which led me to think there was a length > problem. Can you explain the error as I clearly am missing something here. > Thanks so much. > Helen > > Format: fasta > > Found 492 contigs. > > Found 388127 characters sequence data. > > Found 0 ambiguous characters. > > Found GC content to be 56.7 > > min length = 500, median length = 671, mean length = 788.9, max length = 2695 > > *Failed:* > > The mean sequence length is below 2k chars. > > > > > Mark D'Souza wrote: >> Hi Helen, >> >> I see a new MG-RAST job under your name which is queued for the first >> processing step. Until this step is completed the job will not be visible >> to you, are you interpreting this as a failed upload? >> >> There should be no contig length limit to 2Kbp for submission to MG-RAST, >> in fact the current version works best with small fragments. >> >> Regards, >> Mark D'Souza >> -- for the MG-RAST team >> >> mg-rast(a)mcs.anl.gov >> http://metagenomics.nmpdr.org/ >> >> On Mon, 27 Apr 2009, Helen White wrote: >> >>> Hi >>> I am trying to upload a fasta file to MG-RAST of a metagenome, but my >>> contigs are only ~800bp. >>> Is it possible to make an exception to the 2Kbp contig length rule for >>> this sample? >>> Thanks for any help you can offer >>> Best, >>> Helen >>> >

1 0

about the meta server
by Robert Olson 27 Apr '09

27 Apr '09

On mg-rast in /etc/httpd.mg-rast-test/conf.d/meta.conf you will find: Alias /meta /vol/mg-rast-test/meta <Location /meta> AddHandler perl-script .cgi PerlResponseHandler ModPerl::Registry PerlOptions +ParseHeaders Options +ExecCGI Order allow,deny Allow from all </Location> Thus the meta server is /vol/mg-rast-test/meta/genome-meta-server.cgi it uses postgres database genome_meta on bio-big. Table meta_stash defines the storage for a meta file: genome_meta=# select * from meta_stash where meta_path = '/vol/mg-rast- prod/jobs/901/meta.xml'; id | creation_date | meta_path | server | job_id ----+----------------------------+------------------------------------- +--------+-------- 10 | 2008-06-23 15:59:30.657418 | /vol/mg-rast-prod/jobs/901/ meta.xml | | genome_meta=# select ts, key, value from meta_data where stash_id = 10 and key like 'status.%'; ts | key | value ----------------------------+------------------------ +--------------------------------------- 2008-06-23 16:48:01.993116 | status.check_sims | <scalar><! [CDATA[complete]]></scalar> 2008-06-23 16:55:21.990019 | status.create_seed_org | <scalar><! [CDATA[complete]]></scalar> 2008-06-23 16:57:37.264052 | status.export | <scalar><! [CDATA[complete]]></scalar> 2008-06-23 17:02:26.958227 | status.final | <scalar><! [CDATA[complete]]></scalar> 2008-06-23 16:16:32.94411 | status.preprocess | <scalar><! [CDATA[complete]]></scalar> 2008-06-23 16:17:08.785478 | status.sims | <scalar><! [CDATA[complete]]></scalar> 2008-06-23 15:59:30.744101 | status.uploaded | <scalar><! [CDATA[complete]]></scalar> You can solve the copying-jobs problem by either changing the meta_path in the meta_stash table, or I suspect by inserting another row (nope, you can't; meta_stash.id is a primary key). So what your copy-job script could do is just update the meta_stash table. --bob

1 0

Re: [mg-rast] 800bp contigs
by Mark D'Souza 27 Apr '09

27 Apr '09

Hi Helen, I see a new MG-RAST job under your name which is queued for the first processing step. Until this step is completed the job will not be visible to you, are you interpreting this as a failed upload? There should be no contig length limit to 2Kbp for submission to MG-RAST, in fact the current version works best with small fragments. Regards, Mark D'Souza -- for the MG-RAST team mg-rast(a)mcs.anl.gov http://metagenomics.nmpdr.org/ On Mon, 27 Apr 2009, Helen White wrote: > Hi > I am trying to upload a fasta file to MG-RAST of a metagenome, but my contigs > are only ~800bp. > Is it possible to make an exception to the 2Kbp contig length rule for this > sample? > Thanks for any help you can offer > Best, > Helen >

1 0

800bp contigs
by Helen White 27 Apr '09

27 Apr '09

Hi I am trying to upload a fasta file to MG-RAST of a metagenome, but my contigs are only ~800bp. Is it possible to make an exception to the 2Kbp contig length rule for this sample? Thanks for any help you can offer Best, Helen

1 0