October 2008 - Mg-rast - mailman3.cels.anl.gov

Thanks Re: Uploading a large file
by moor0282＠flinders.edu.au 31 Oct '08

31 Oct '08

Thanks everyone. Great work! Quoting Mark D'Souza <dsouza(a)mcs.anl.gov>: > Hi Paul, > > You're welcome. I slipped the ftp'd sequence file into job 1580 and it > should run through the pipeline normally. > > The file size matches (121,057,323 bytes), the md5sum of the .bz2 file is > 8310567f42392ee5d0d779bdd04e3322, and it is deleted on the ftp site. > > Let us know if there are any problems with this job. > > Regards, > Mark > > On Fri, 31 Oct 2008, Paul Gardner-Stephen wrote: > > > Hi Mark, > > > > thanks for that. I have submitted job 1580, and the uploaded file is > > org1_longreads.fsa.bz2 and should be 121,057,323 bytes. It was still > > uploading when I sent this, but expect it to be there soon. > > > > Thanks again, > > > > Paul. > > > > Mark D'Souza wrote: > >> Hi Paul, > >> > >> I had set up a private area for you, but then realized the best way to do > > >> this is through the ftp drop-box set up by the MCS division here at > >> ftp.mcs.anl.gov > >> > >> You can anonymous ftp to this site, cd to 'incoming' and drop off your > >> sequence data file. Compressing the sequence file is fine with us, gzip or > > >> bzip2 will work nicely. > >> > >> The incoming directory does not allow anonymous ftp users to list the > >> contents, so no one will be able to see your file. Files deposited by > >> Anonymous can not be read (get) by Anonymous, so no one can access it > >> through ftp either. Files are automatically deleted 10 days after they are > > >> created. > >> > >> Let me know if this sounds OK. > >> > >> Regards, > >> Mark > >> > >> 1. Check http://metagenomics.nmpdr.org/rast.cgi?page=Upload for required > >> upload sequence data format > >> 2. You can deposit a single fasta file per job. > >> 3. Create a dummy job on the MG-RAST site (see URL above) with dummy > >> sequence data in a fasta file. > >> Enter the metagenome information (name, latitude etc.) that you would like > > >> used with the ftp'd date. > >> Make the first line of the fasta file '> ftp submission', i.e. no > >> sequence id, this will cause the job processing to fail, but the job > number > >> and information will be retained. > >> 4. Deposit your sequence data at the ftp site, > >> 5. Email mg-rast(a)mcs.anl.gov and let us know the job number assigned to > >> your dummy job and the name of the file you deposited at the ftp site. > >> > >> > >> On Wed, 29 Oct 2008, Paul Gardner-Stephen wrote: > >> > >>> Hi Folker, > >>> > >>> Just wondering how you were going getting the private hidden upload area > > >>> for us, as we are keen to do this. > >>> > >>> Thanks, > >>> Paul. > >>> > >>> Folker Meyer wrote: > >>>> Dear Bob: > >>>> > >>>> just a quick answer (more later): > >>>> > >>>> we are organizing a workshop in early december and are beginning to > >>>> consider a more technical workshop that is linked to that event. Please > > >>>> contact Liz Glass (a co-worker) who is organizing the workshop if there > > >>>> is any interest. > >>>> > >>>> BEst, > >>>> Folker > >>>> PS: No problem with keeping it private, I was just curious. We will > >>>> provide a "private" hidden upload area. > >>>> > >>>> > >>>> > >>>> On Oct 22, 2008, at 6:13 PM, moor0282(a)flinders.edu.au wrote: > >>>> > >>>>> > >>>>> > >>>>> Dear Folker, I'll just reply partially for Paul. Our data is > >>>>> semi-commercial, in > >>>>> the sense that its academia doing the work, but the data will belong to > > >>>>> the > >>>>> commercial partner at least until published. > >>>>> > >>>>> As such theres no problem with us giving your staff access to handling, > > >>>>> as long > >>>>> as the data isnt yet made available to other RAST users or to the > >>>>> public. Our > >>>>> understanding is thata your association with the proiject would be > >>>>> equivalent > >>>>> to AGRF who did the sequencing, i.e. you handle the data but in > >>>>> confidence. > >>>>> > >>>>> Paul can you supply the etceteras that you think Folker may need? > >>>>> > >>>>> Folker, Paul is the systems manager at our School of Engineering and > >>>>> Computer > >>>>> Science so he will be able to forward all the neccessary details. Our > >>>>> good > >>>>> luck, he is also a bioinformaticist! > >>>>> > >>>>> Thanks for the use of RAST, we anticipate it will make sifting through > > >>>>> our raw > >>>>> reads much more manageable. When the paper is published and the gene > >>>>> functions > >>>>> established (by functional testing at the bench) and patents submitted, > > >>>>> we may > >>>>> well be able to release much of the data to the public, just not the > >>>>> genes for > >>>>> pathways that we want to patent: genes that code for hypothetical > >>>>> proteins with > >>>>> functions unique to this lineage. Those kind of proteins we cant > publish > >>>>> even > >>>>> while their functions arent known, just in case their function turns > out > >>>>> to be > >>>>> comercially crucial later. Lots of membrane transporters would fall > into > >>>>> this > >>>>> category for instance. But any gene that we feel is not of commercial > >>>>> use, we > >>>>> can publish those ones when we have some kind of public outreach > >>>>> underway, > >>>>> which may be a year or two from now. In the meantime our project needs > > >>>>> to be > >>>>> kept secret from the large US sequencing centres. > >>>>> > >>>>> As well as being involved in this commercial sequencing of algal > >>>>> genomes, I am > >>>>> also undertaking ecological meta-genome work on corals and their > >>>>> symbionts, > >>>>> which is non-commercial. I can offer to put that data through RAST as a > > >>>>> public > >>>>> project to offset the use we are getting with the in-confidence stuff. > >>>>> > >>>>> Also by potentially funding two FLX sequencing machines in Adelaide, we > > >>>>> are > >>>>> possibly becoming a pipeline for other users in Australia, some of whom > > >>>>> will > >>>>> have public projects. So we could steer people your way and raise your > > >>>>> profile > >>>>> hopefully. Its such a great thing that you opened up your computing > >>>>> power to > >>>>> researchers. Kudos! > >>>>> > >>>>> Maybe we could eventually send a bioinformaticist over to learn how you > > >>>>> do > >>>>> things? We'll be clearer about our funding and longterm prospects by > >>>>> December. > >>>>> For now its a tight budget and we just have a 1.5x coverage of a 200MB > > >>>>> genome to > >>>>> analyse, as FLX reads. Some bacterial sequences that are in the > dataset, > >>>>> can be > >>>>> filtered out using your groovy platform. Wed love to cite that we used > > >>>>> RAST, > >>>>> when we publish our work as paper/s. > >>>>> > >>>>> So many thanks from Bob and the algae team. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Quoting Folker Meyer <folker.meyer(a)googlemail.com>: > >>>>> > >>>>>> Hi, > >>>>>> > >>>>>> we could create an upload area for you in a hidden directory on our > >>>>>> ftp server. > >>>>>> > >>>>>> We'd then have to upload for you. You would need to provide all > >>>>>> required info to us for the upload. GPS coords etc. > >>>>>> Is this going to be a public data set at some point? > >>>>>> > >>>>>> would that work for you? > >>>>>> > >>>>>> best, > >>>>>> Folker > >>>>>> On Oct 22, 2008, at 1:18 AM, Paul Gardner-Stephen wrote: > >>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> Our proxy server won't let me upload a large file (136MB tgz). Is > >>>>>>> there an FTP or other alternative method to upload the file to MG- > >>>>>>> RAST? > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Paul. > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >> > > > > >

1 0

Re: [mg-rast] Jettisoning tax_sim tables
by Robert Olson 31 Oct '08

31 Oct '08

heh. After truncating those tables there is 1.2TB free. --bob On Oct 31, 2008, at 2:14 PM, Jared Wilkening wrote: > Our central access module for the postgres db is > MetagenomeAnalysis.pm. Doing a search through that all of our access > methods use tax_sim_best_psc so there shouldn't be a problem with > abandoning the full hit tables. > > Jared > > On Fri, Oct 31, 2008 at 2:07 PM, Robert Olson <olson(a)mcs.anl.gov> > wrote: > I have truncated the tax_sim_XXX tables in jobs 45 and 1441. It > appears things still work properly, at least the initial overview, > the kegg browser, and metabolic overview. If someone can verify this > I can clear out those tables in the server and keep them from being > loaded for now in the new jobs that are running. > > --bob >

1 0

Re: [mg-rast] Jettisoning tax_sim tables
by Jared Wilkening 31 Oct '08

31 Oct '08

Our central access module for the postgres db is MetagenomeAnalysis.pm. Doing a search through that all of our access methods use tax_sim_best_psc so there shouldn't be a problem with abandoning the full hit tables. Jared On Fri, Oct 31, 2008 at 2:07 PM, Robert Olson <olson(a)mcs.anl.gov> wrote: > I have truncated the tax_sim_XXX tables in jobs 45 and 1441. It appears > things still work properly, at least the initial overview, the kegg browser, > and metabolic overview. If someone can verify this I can clear out those > tables in the server and keep them from being loaded for now in the new jobs > that are running. > > --bob >

1 0

Jettisoning tax_sim tables
by Robert Olson 31 Oct '08

31 Oct '08

I have truncated the tax_sim_XXX tables in jobs 45 and 1441. It appears things still work properly, at least the initial overview, the kegg browser, and metabolic overview. If someone can verify this I can clear out those tables in the server and keep them from being loaded for now in the new jobs that are running. --bob

1 0

Re: [mg-rast] Metagenomes ftp
by Mark D'Souza 31 Oct '08

31 Oct '08

Hi Vedran, This has now been fixed, thanks for alerting us to the problem. Regards, Mark D'Souza On Mon, 20 Oct 2008, Vedran Lucic wrote: > Dear support, > > we are experiencing some issues with > http://www.theseed.org/DinsdaleSupplementalMaterial/ site. We are tring to > obtain files: 4440437.3.fa.gz and 4440416.3.fa.gz > but none of the ftp links seem to be working. > Is there an alternative way to get these files? > > Thank you in advance, > > Kind regards, > > Vedran Lucic > >

1 0

Re: [mg-rast] Uploading a large file
by Mark D'Souza 31 Oct '08

31 Oct '08

Hi Paul, You're welcome. I slipped the ftp'd sequence file into job 1580 and it should run through the pipeline normally. The file size matches (121,057,323 bytes), the md5sum of the .bz2 file is 8310567f42392ee5d0d779bdd04e3322, and it is deleted on the ftp site. Let us know if there are any problems with this job. Regards, Mark On Fri, 31 Oct 2008, Paul Gardner-Stephen wrote: > Hi Mark, > > thanks for that. I have submitted job 1580, and the uploaded file is > org1_longreads.fsa.bz2 and should be 121,057,323 bytes. It was still > uploading when I sent this, but expect it to be there soon. > > Thanks again, > > Paul. > > Mark D'Souza wrote: >> Hi Paul, >> >> I had set up a private area for you, but then realized the best way to do >> this is through the ftp drop-box set up by the MCS division here at >> ftp.mcs.anl.gov >> >> You can anonymous ftp to this site, cd to 'incoming' and drop off your >> sequence data file. Compressing the sequence file is fine with us, gzip or >> bzip2 will work nicely. >> >> The incoming directory does not allow anonymous ftp users to list the >> contents, so no one will be able to see your file. Files deposited by >> Anonymous can not be read (get) by Anonymous, so no one can access it >> through ftp either. Files are automatically deleted 10 days after they are >> created. >> >> Let me know if this sounds OK. >> >> Regards, >> Mark >> >> 1. Check http://metagenomics.nmpdr.org/rast.cgi?page=Upload for required >> upload sequence data format >> 2. You can deposit a single fasta file per job. >> 3. Create a dummy job on the MG-RAST site (see URL above) with dummy >> sequence data in a fasta file. >> Enter the metagenome information (name, latitude etc.) that you would like >> used with the ftp'd date. >> Make the first line of the fasta file '> ftp submission', i.e. no >> sequence id, this will cause the job processing to fail, but the job number >> and information will be retained. >> 4. Deposit your sequence data at the ftp site, >> 5. Email mg-rast(a)mcs.anl.gov and let us know the job number assigned to >> your dummy job and the name of the file you deposited at the ftp site. >> >> >> On Wed, 29 Oct 2008, Paul Gardner-Stephen wrote: >> >>> Hi Folker, >>> >>> Just wondering how you were going getting the private hidden upload area >>> for us, as we are keen to do this. >>> >>> Thanks, >>> Paul. >>> >>> Folker Meyer wrote: >>>> Dear Bob: >>>> >>>> just a quick answer (more later): >>>> >>>> we are organizing a workshop in early december and are beginning to >>>> consider a more technical workshop that is linked to that event. Please >>>> contact Liz Glass (a co-worker) who is organizing the workshop if there >>>> is any interest. >>>> >>>> BEst, >>>> Folker >>>> PS: No problem with keeping it private, I was just curious. We will >>>> provide a "private" hidden upload area. >>>> >>>> >>>> >>>> On Oct 22, 2008, at 6:13 PM, moor0282(a)flinders.edu.au wrote: >>>> >>>>> >>>>> >>>>> Dear Folker, I'll just reply partially for Paul. Our data is >>>>> semi-commercial, in >>>>> the sense that its academia doing the work, but the data will belong to >>>>> the >>>>> commercial partner at least until published. >>>>> >>>>> As such theres no problem with us giving your staff access to handling, >>>>> as long >>>>> as the data isnt yet made available to other RAST users or to the >>>>> public. Our >>>>> understanding is thata your association with the proiject would be >>>>> equivalent >>>>> to AGRF who did the sequencing, i.e. you handle the data but in >>>>> confidence. >>>>> >>>>> Paul can you supply the etceteras that you think Folker may need? >>>>> >>>>> Folker, Paul is the systems manager at our School of Engineering and >>>>> Computer >>>>> Science so he will be able to forward all the neccessary details. Our >>>>> good >>>>> luck, he is also a bioinformaticist! >>>>> >>>>> Thanks for the use of RAST, we anticipate it will make sifting through >>>>> our raw >>>>> reads much more manageable. When the paper is published and the gene >>>>> functions >>>>> established (by functional testing at the bench) and patents submitted, >>>>> we may >>>>> well be able to release much of the data to the public, just not the >>>>> genes for >>>>> pathways that we want to patent: genes that code for hypothetical >>>>> proteins with >>>>> functions unique to this lineage. Those kind of proteins we cant publish >>>>> even >>>>> while their functions arent known, just in case their function turns out >>>>> to be >>>>> comercially crucial later. Lots of membrane transporters would fall into >>>>> this >>>>> category for instance. But any gene that we feel is not of commercial >>>>> use, we >>>>> can publish those ones when we have some kind of public outreach >>>>> underway, >>>>> which may be a year or two from now. In the meantime our project needs >>>>> to be >>>>> kept secret from the large US sequencing centres. >>>>> >>>>> As well as being involved in this commercial sequencing of algal >>>>> genomes, I am >>>>> also undertaking ecological meta-genome work on corals and their >>>>> symbionts, >>>>> which is non-commercial. I can offer to put that data through RAST as a >>>>> public >>>>> project to offset the use we are getting with the in-confidence stuff. >>>>> >>>>> Also by potentially funding two FLX sequencing machines in Adelaide, we >>>>> are >>>>> possibly becoming a pipeline for other users in Australia, some of whom >>>>> will >>>>> have public projects. So we could steer people your way and raise your >>>>> profile >>>>> hopefully. Its such a great thing that you opened up your computing >>>>> power to >>>>> researchers. Kudos! >>>>> >>>>> Maybe we could eventually send a bioinformaticist over to learn how you >>>>> do >>>>> things? We'll be clearer about our funding and longterm prospects by >>>>> December. >>>>> For now its a tight budget and we just have a 1.5x coverage of a 200MB >>>>> genome to >>>>> analyse, as FLX reads. Some bacterial sequences that are in the dataset, >>>>> can be >>>>> filtered out using your groovy platform. Wed love to cite that we used >>>>> RAST, >>>>> when we publish our work as paper/s. >>>>> >>>>> So many thanks from Bob and the algae team. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Quoting Folker Meyer <folker.meyer(a)googlemail.com>: >>>>> >>>>>> Hi, >>>>>> >>>>>> we could create an upload area for you in a hidden directory on our >>>>>> ftp server. >>>>>> >>>>>> We'd then have to upload for you. You would need to provide all >>>>>> required info to us for the upload. GPS coords etc. >>>>>> Is this going to be a public data set at some point? >>>>>> >>>>>> would that work for you? >>>>>> >>>>>> best, >>>>>> Folker >>>>>> On Oct 22, 2008, at 1:18 AM, Paul Gardner-Stephen wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Our proxy server won't let me upload a large file (136MB tgz). Is >>>>>>> there an FTP or other alternative method to upload the file to MG- >>>>>>> RAST? >>>>>>> >>>>>>> Thanks, >>>>>>> Paul. >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> > >

1 0

Re: [mg-rast] error on jobs page
by Folker Meyer 31 Oct '08

31 Oct '08

likely we will in the near future Folker On Oct 31, 2008, at 1:32 PM, Jared Wilkening wrote: > Space could be trimmed if we removed the best hits by alignment > tables that > we are not using. They are never used in frontend. > > Jared > > On Fri, Oct 31, 2008 at 1:08 PM, Robert Olson <olson(a)mcs.anl.gov> > wrote: > >> Well, I worked around the problem. The bigger problem is that the >> disk on >> bio-big holding the postgres database is full; we've now got 1.2 TB >> in that >> database. >> >> I deleted some stuff from the machine and we have around 44GB free, >> and >> shrinking. >> >> we'll need to either determine how to keep less stuff in the >> database, or >> add disk. >> >> >> On Oct 31, 2008, at 12:11 PM, Mark D'Souza wrote: >> >> Hi, >>> >>> The meta server was behaving badly. Bob fixed it. >>> >>> Mark >>> >>> >>> On Fri, 31 Oct 2008, Mark D'Souza wrote: >>> >>> Hi, >>>> >>>> I am getting an internal server error on the jobs page. tried as >>>> dsouza >>>> (admin) as well as regular user -- same thing. Alex tried, and he >>>> does NOT >>>> get an error. I did clear out my cookies, restarted my browser etc. >>>> >>>> OK, possibly users like Alex who do NOT have any (or access to) >>>> private >>>> jobs do not get an error. Maybe the rights are messed up. >>>> >>>> If anyone knows what the problem could be please fix it ASAP or >>>> let me >>>> know what the problem could be. >>>> >>>> Mark >>>> >>>> >>

1 0

Re: [mg-rast] error on jobs page
by Jared Wilkening 31 Oct '08

31 Oct '08

Space could be trimmed if we removed the best hits by alignment tables that we are not using. They are never used in frontend. Jared On Fri, Oct 31, 2008 at 1:08 PM, Robert Olson <olson(a)mcs.anl.gov> wrote: > Well, I worked around the problem. The bigger problem is that the disk on > bio-big holding the postgres database is full; we've now got 1.2 TB in that > database. > > I deleted some stuff from the machine and we have around 44GB free, and > shrinking. > > we'll need to either determine how to keep less stuff in the database, or > add disk. > > > On Oct 31, 2008, at 12:11 PM, Mark D'Souza wrote: > > Hi, >> >> The meta server was behaving badly. Bob fixed it. >> >> Mark >> >> >> On Fri, 31 Oct 2008, Mark D'Souza wrote: >> >> Hi, >>> >>> I am getting an internal server error on the jobs page. tried as dsouza >>> (admin) as well as regular user -- same thing. Alex tried, and he does NOT >>> get an error. I did clear out my cookies, restarted my browser etc. >>> >>> OK, possibly users like Alex who do NOT have any (or access to) private >>> jobs do not get an error. Maybe the rights are messed up. >>> >>> If anyone knows what the problem could be please fix it ASAP or let me >>> know what the problem could be. >>> >>> Mark >>> >>> >

1 0

Re: [mg-rast] error on jobs page
by Robert Olson 31 Oct '08

31 Oct '08

Well, I worked around the problem. The bigger problem is that the disk on bio-big holding the postgres database is full; we've now got 1.2 TB in that database. I deleted some stuff from the machine and we have around 44GB free, and shrinking. we'll need to either determine how to keep less stuff in the database, or add disk. On Oct 31, 2008, at 12:11 PM, Mark D'Souza wrote: > Hi, > > The meta server was behaving badly. Bob fixed it. > > Mark > > > On Fri, 31 Oct 2008, Mark D'Souza wrote: > >> Hi, >> >> I am getting an internal server error on the jobs page. tried as >> dsouza (admin) as well as regular user -- same thing. Alex tried, >> and he does NOT get an error. I did clear out my cookies, restarted >> my browser etc. >> >> OK, possibly users like Alex who do NOT have any (or access to) >> private jobs do not get an error. Maybe the rights are messed up. >> >> If anyone knows what the problem could be please fix it ASAP or let >> me know what the problem could be. >> >> Mark >>

1 0

Re: [mg-rast] error on jobs page
by Mark D'Souza 31 Oct '08

31 Oct '08

Hi, The meta server was behaving badly. Bob fixed it. Mark On Fri, 31 Oct 2008, Mark D'Souza wrote: > Hi, > > I am getting an internal server error on the jobs page. tried as dsouza > (admin) as well as regular user -- same thing. Alex tried, and he does NOT > get an error. I did clear out my cookies, restarted my browser etc. > > OK, possibly users like Alex who do NOT have any (or access to) private jobs > do not get an error. Maybe the rights are messed up. > > If anyone knows what the problem could be please fix it ASAP or let me know > what the problem could be. > > Mark >

1 0