April 2008 - Mg-rast - mailman3.cels.anl.gov

Re: mg-rast concurrent access
by Robert Olson 24 Apr '08

24 Apr '08

On Apr 24, 2008, at 3:14 PM, Robert Olson wrote: > I think we may have to put any analysis in the mgrast into short- > term batch mode. with what I'm guessing is just a couple of us > using the system I'm seeing this: Just realized I didn't really explain what I meant here. We may get better overall performance if we serialize accesses to the data in order to keep the machine from thrashing as it pulls multiple job data off disk into cache. Big memory might help here, for a while, up until the multiple users have a working set larger than the memory size, at which point we thrash again. Getting a system with a bunch of 15K rpm disks might make a difference here.

1 0

Re: mg-rast concurrent access
by Folker Meyer 24 Apr '08

24 Apr '08

might be me, I have been clicking wild on purpose. Folker On Apr 24, 2008, at 3:14 PM, Robert Olson wrote: > I think we may have to put any analysis in the mgrast into short- > term batch mode. with what I'm guessing is just a couple of us using > the system I'm seeing this: > > 14723 olson 25 0 8848m 2.8g 2.8g R 100 18.1 13:58.04 postgres > 15164 olson 25 0 8828m 1.2g 1.2g R 100 7.9 1:18.13 postgres > 14509 olson 25 0 8856m 3.0g 2.9g R 100 19.0 24:34.44 postgres > 14510 olson 18 0 8841m 3.0g 3.0g R 89 19.0 24:36.10 postgres > 15163 olson 25 0 8843m 1.3g 1.3g R 86 8.2 1:46.44 postgres > 14921 olson 25 0 8933m 6.2g 6.1g R 67 39.9 7:51.13 postgres > 15058 olson 25 0 8833m 829m 813m R 65 5.2 4:07.23 postgres > 14724 olson 25 0 8856m 2.8g 2.8g R 50 18.1 13:28.68 postgres > 15060 olson 21 0 8833m 834m 818m R 50 5.2 4:14.50 postgres > 15080 olson 25 0 8835m 1.4g 1.3g R 46 8.6 4:03.44 postgres > 14943 olson 25 0 8924m 5.8g 5.8g R 35 37.3 7:00.00 postgres > > Given the cost of these queries I'd also like to see the ajax stuff > not kick off automatically when you go to an analysis page; rather, > the user has to explicitly choose parameters (even the defaults) and > request the analysis. > > I'd also like to reiterate here in this context that we shouldn't > run multiple analyses in parallel via ajax from one of these pages. > > --bob >

1 0

mg-rast concurrent access
by Robert Olson 24 Apr '08

24 Apr '08

I think we may have to put any analysis in the mgrast into short-term batch mode. with what I'm guessing is just a couple of us using the system I'm seeing this: 14723 olson 25 0 8848m 2.8g 2.8g R 100 18.1 13:58.04 postgres 15164 olson 25 0 8828m 1.2g 1.2g R 100 7.9 1:18.13 postgres 14509 olson 25 0 8856m 3.0g 2.9g R 100 19.0 24:34.44 postgres 14510 olson 18 0 8841m 3.0g 3.0g R 89 19.0 24:36.10 postgres 15163 olson 25 0 8843m 1.3g 1.3g R 86 8.2 1:46.44 postgres 14921 olson 25 0 8933m 6.2g 6.1g R 67 39.9 7:51.13 postgres 15058 olson 25 0 8833m 829m 813m R 65 5.2 4:07.23 postgres 14724 olson 25 0 8856m 2.8g 2.8g R 50 18.1 13:28.68 postgres 15060 olson 21 0 8833m 834m 818m R 50 5.2 4:14.50 postgres 15080 olson 25 0 8835m 1.4g 1.3g R 46 8.6 4:03.44 postgres 14943 olson 25 0 8924m 5.8g 5.8g R 35 37.3 7:00.00 postgres Given the cost of these queries I'd also like to see the ajax stuff not kick off automatically when you go to an analysis page; rather, the user has to explicitly choose parameters (even the defaults) and request the analysis. I'd also like to reiterate here in this context that we shouldn't run multiple analyses in parallel via ajax from one of these pages. --bob

1 0

mg-rast failing query
by Robert Olson 24 Apr '08

24 Apr '08

postgres log reports: ERROR: column "s.tax_str" must appear in the GROUP BY clause or be used in an aggregate functionSTATEMENT: select s.tax_group_1, s.tax_group_2, s.tax_group_3, s.tax_str, count(*) as num from ( select id1, min(rank_psc) as rank from tax_sim_570 where dbid=6 and logpsc<=-100 group by id1) as b inner join tax_sim_570 as s on b.id1=s.id1 and b.rank=s.rank_psc where dbid=6 group by s.tax_group_1, s.tax_group_2, s.tax_group_3

1 0

mg-rast can't jump directly to comparison page
by Robert Olson 24 Apr '08

24 Apr '08

I wanted to compare metagenomes, but find that you can't go directly there without paying the price of an individual metagenome overview page.

1 0

mg-rast Ajax & default mgrast pages
by Robert Olson 24 Apr '08

24 Apr '08

The queries to populate the MGRAST pages are fairly expensive; I am seeing that when one goes to say the main tax overview page the ajax logic kicks off two of them at once. I'd like to see those either get serialized or the one not viewed initially be loaded on demand. better for general system load. --bob

1 0

Re: mg-rast try this out
by Robert Olson 24 Apr '08

24 Apr '08

This appears to be mostly behaving today, access time seems reasonable. There are some glitches due to postgres storing trailing padding spaces in CHAR columns; I hacked some of the queries to fix, but not all is working properly. I am going to load up the job 540 data that was causing other problems this week with a change in the schema that should make things work properly. --bob On Apr 23, 2008, at 5:08 PM, Robert Olson wrote: > mg-rast driven by the new postgres db, with job 565 > (MG_Prarie_Reminant_Sample (4440690.3)) loaded is at > > http://mg-rast.mcs.anl.gov/olson/FIG/metagenomics.cgi > > according to the seed hits that's a really boring data set, all > vibrio cholerae nrt36s, all the time. > > --bob >

1 0

Re: mg-rast Re: MG-RAST server
by Folker Meyer 24 Apr '08

24 Apr '08

ditto. depends on the data. we can not know this in advance, but 1 peg per fragment is not unreasonable. peg to frag ration is an interesting way to look at metagenomes. wonder if low ration means diverse biome or just means we have not seen a lot of this before. FOlker On Apr 24, 2008, at 9:00 AM, Robert Olson wrote: > I was talking about the fraction that got called as pegs. > > On Apr 24, 2008, at 8:56 AM, Folker Meyer wrote: > >> yes. it is reasonable. people can simply combine to mid-sized 454 >> runs and get to 600k reads... >> >> Folker >> On Apr 24, 2008, at 8:43 AM, Robert Olson wrote: >> >>> This job has 445585 reads, from which it created 282K pegs. Does >>> this seem remarkable to anyone? Wonder what's going on. Rob - does >>> it seem reasonable? >>> >>> --bob >>> >>> On Apr 24, 2008, at 8:36 AM, Robert Olson wrote: >>> >>>> This is the actual problem. >>>> >>>> root@treponema:/vol/mg-rast-prod/jobs/540# wc -l rp/4440668.3/ >>>> Features/peg/tbl >>>> 282879 rp/4440668.3/Features/peg/tbl >>>> >>>> This job produced 282K peg features. FIGV is just not up for >>>> that. we will have to put a fix in soon for this. >>>> >>>> --bob >>>> >>>> On Apr 24, 2008, at 8:22 AM, Folker Meyer wrote: >>>> >>>>> Sean: >>>>> >>>>> in the process of getting the new version of the server up and >>>>> running on the production hardware we ran into resource issues. >>>>> This killed both the new and the old version. >>>>> >>>>> THe old version will be up and running shortly. >>>>> >>>>> Sorry for the inconvienience, >>>>> FOlker >>>>> On Apr 24, 2008, at 7:56 AM, Sean Norman wrote: >>>>> >>>>>> Hi Folker, >>>>>> >>>>>> Sorry to take more of your time but I am trying to view the >>>>>> features of one of my metagenomics jobs analyzed by the MG-RAST >>>>>> using the SEED viewer and keep getting the error message >>>>>> below. I've emailed the contact listed in the error message a >>>>>> few times but have not yet received a reply. The user is >>>>>> snorman and the job# is 540 (4440668.3). Please let me know if >>>>>> there is anything I can do to remedy this problem. Thanks and >>>>>> have a good day. >>>>>> >>>>>> Cheers, >>>>>> >>>>>> Sean >>>>>> >>>>>> >>>>>> "Internal Server Error >>>>>> >>>>>> The server encountered an internal error or misconfiguration >>>>>> and was unable to complete your request. >>>>>> >>>>>> Please contact the server administrator, olson(a)mcs.anl.gov and >>>>>> inform them of the time the error occurred, and anything you >>>>>> might have done that may have caused the error. >>>>>> >>>>>> More information about this error may be available in the >>>>>> server error log." >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> R. Sean Norman, Ph.D. >>>>>> Assistant Professor >>>>>> Dept. of Environmental Health Sciences >>>>>> Arnold School of Public Health >>>>>> University of South Carolina >>>>>> 921 Assembly St. >>>>>> Columbia, SC 29208 >>>>>> Phone: 803.777.0940 >>>>>> Email: rsnorman(a)gwm.sc.edu >>>>>> >>>>> >>>> >>> >> >

1 0

Re: mg-rast Re: MG-RAST server
by Robert Olson 24 Apr '08

24 Apr '08

I was talking about the fraction that got called as pegs. On Apr 24, 2008, at 8:56 AM, Folker Meyer wrote: > yes. it is reasonable. people can simply combine to mid-sized 454 > runs and get to 600k reads... > > Folker > On Apr 24, 2008, at 8:43 AM, Robert Olson wrote: > >> This job has 445585 reads, from which it created 282K pegs. Does >> this seem remarkable to anyone? Wonder what's going on. Rob - does >> it seem reasonable? >> >> --bob >> >> On Apr 24, 2008, at 8:36 AM, Robert Olson wrote: >> >>> This is the actual problem. >>> >>> root@treponema:/vol/mg-rast-prod/jobs/540# wc -l rp/4440668.3/ >>> Features/peg/tbl >>> 282879 rp/4440668.3/Features/peg/tbl >>> >>> This job produced 282K peg features. FIGV is just not up for >>> that. we will have to put a fix in soon for this. >>> >>> --bob >>> >>> On Apr 24, 2008, at 8:22 AM, Folker Meyer wrote: >>> >>>> Sean: >>>> >>>> in the process of getting the new version of the server up and >>>> running on the production hardware we ran into resource issues. >>>> This killed both the new and the old version. >>>> >>>> THe old version will be up and running shortly. >>>> >>>> Sorry for the inconvienience, >>>> FOlker >>>> On Apr 24, 2008, at 7:56 AM, Sean Norman wrote: >>>> >>>>> Hi Folker, >>>>> >>>>> Sorry to take more of your time but I am trying to view the >>>>> features of one of my metagenomics jobs analyzed by the MG-RAST >>>>> using the SEED viewer and keep getting the error message >>>>> below. I've emailed the contact listed in the error message a >>>>> few times but have not yet received a reply. The user is >>>>> snorman and the job# is 540 (4440668.3). Please let me know if >>>>> there is anything I can do to remedy this problem. Thanks and >>>>> have a good day. >>>>> >>>>> Cheers, >>>>> >>>>> Sean >>>>> >>>>> >>>>> "Internal Server Error >>>>> >>>>> The server encountered an internal error or misconfiguration >>>>> and was unable to complete your request. >>>>> >>>>> Please contact the server administrator, olson(a)mcs.anl.gov and >>>>> inform them of the time the error occurred, and anything you >>>>> might have done that may have caused the error. >>>>> >>>>> More information about this error may be available in the >>>>> server error log." >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> R. Sean Norman, Ph.D. >>>>> Assistant Professor >>>>> Dept. of Environmental Health Sciences >>>>> Arnold School of Public Health >>>>> University of South Carolina >>>>> 921 Assembly St. >>>>> Columbia, SC 29208 >>>>> Phone: 803.777.0940 >>>>> Email: rsnorman(a)gwm.sc.edu >>>>> >>>> >>> >> >

1 0

Re: mg-rast Re: MG-RAST server
by Folker Meyer 24 Apr '08

24 Apr '08

yes. it is reasonable. people can simply combine to mid-sized 454 runs and get to 600k reads... Folker On Apr 24, 2008, at 8:43 AM, Robert Olson wrote: > This job has 445585 reads, from which it created 282K pegs. Does > this seem remarkable to anyone? Wonder what's going on. Rob - does > it seem reasonable? > > --bob > > On Apr 24, 2008, at 8:36 AM, Robert Olson wrote: > >> This is the actual problem. >> >> root@treponema:/vol/mg-rast-prod/jobs/540# wc -l rp/4440668.3/ >> Features/peg/tbl >> 282879 rp/4440668.3/Features/peg/tbl >> >> This job produced 282K peg features. FIGV is just not up for that. >> we will have to put a fix in soon for this. >> >> --bob >> >> On Apr 24, 2008, at 8:22 AM, Folker Meyer wrote: >> >>> Sean: >>> >>> in the process of getting the new version of the server up and >>> running on the production hardware we ran into resource issues. >>> This killed both the new and the old version. >>> >>> THe old version will be up and running shortly. >>> >>> Sorry for the inconvienience, >>> FOlker >>> On Apr 24, 2008, at 7:56 AM, Sean Norman wrote: >>> >>>> Hi Folker, >>>> >>>> Sorry to take more of your time but I am trying to view the >>>> features of one of my metagenomics jobs analyzed by the MG-RAST >>>> using the SEED viewer and keep getting the error message below. >>>> I've emailed the contact listed in the error message a few times >>>> but have not yet received a reply. The user is snorman and the >>>> job# is 540 (4440668.3). Please let me know if there is anything >>>> I can do to remedy this problem. Thanks and have a good day. >>>> >>>> Cheers, >>>> >>>> Sean >>>> >>>> >>>> "Internal Server Error >>>> >>>> The server encountered an internal error or misconfiguration and >>>> was unable to complete your request. >>>> >>>> Please contact the server administrator, olson(a)mcs.anl.gov and >>>> inform them of the time the error occurred, and anything you >>>> might have done that may have caused the error. >>>> >>>> More information about this error may be available in the server >>>> error log." >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> R. Sean Norman, Ph.D. >>>> Assistant Professor >>>> Dept. of Environmental Health Sciences >>>> Arnold School of Public Health >>>> University of South Carolina >>>> 921 Assembly St. >>>> Columbia, SC 29208 >>>> Phone: 803.777.0940 >>>> Email: rsnorman(a)gwm.sc.edu >>>> >>> >> >

1 0