May 2008 - Mg-rast - mailman3.cels.anl.gov

Re: [mg-rast] Re: blastall mem use
by Gordon D. Pusch 31 May '08

31 May '08

On 2008-May-31, at 3:13 PM, Folker Meyer wrote: > from the NCBI pags: BLAST 2.2.18 now available date: 2008-04-01 > > My suspicion is that now that 64 bit is more common they might have > fixed some things... Even so, we are still going to be forced to revisit this question, ~1.5--2 doubling-times from now, since available memory size does not seem to be increasing nearly as fast as the NR size is... :-( -- Gordon > On May 31, 2008, at 3:10 PM, Robert Olson wrote: > >> 2.2.14 >> >> >> ----- Original Message ----- >> From: "Folker Meyer" <folker.meyer(a)googlemail.com> >> To: "Robert Olson" <olson(a)mcs.anl.gov> >> Cc: "Folker Meyer" <folker.meyer(a)googlemail.com>, mg- >> rast(a)mcs.anl.gov, "Gordon D. Pusch" <gdpusch(a)xnet.com> >> Sent: Saturday, May 31, 2008 3:00:55 PM GMT -06:00 US/Canada Central >> Subject: Re: [mg-rast] Re: blastall mem use >> >> here is my standard question: what version? Are we using the latest >> version. >> >> Folker >> >> On May 31, 2008, at 2:52 PM, Robert Olson wrote: >> >>> we'd have to go there if the db grew enough that a single node's RAM >>> could hold it. >>> >>> here are some numbers from jobs running on one of the ppc nodes now: >>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>> 28766 olson 25 0 1222m 494m 475m R 99.3 12.5 16102:53 >>> blastall >>> 19651 mgrastte 18 0 1969m 1.5g 198m R 97.3 37.7 338:01.66 >>> blastall >>> >>> The first is an original mgrast run against /scratch/metagenome-48- >>> hour/Data/SEED_2006_07_01 at 1.6GB. >>> >>> The second is the new mgrast against /vol/mg-rast-test/Data/db/seed/ >>> 018/nr at 2.6GB. >>> >>> NR size increased by a factor of 1.7, and virtual size by 1.6, so >>> that is following. The resident size tho is much larger for the new >>> one. >>> >>> On the 64-bit machine (mg-rast) we see this tho >>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>> 12370 mgrastte 25 0 3657m 3.1g 780m R 100 20.0 85:46.74 >>> blastall >>> 4989 mgrastte 25 0 3401m 3.0g 899m R 98 19.1 451:13.21 blastall >>> 12013 mgrastte 25 0 3727m 3.2g 807m R 98 20.6 103:35.79 >>> blastall >>> >>> a near doubling in virtual size. Leads me to wonder if blast is >>> doing something dumb on 64-bit machines. though it does mean that we >>> are still good with running two copies on the PPC nodes, for now >>> anyway. >>> >>> ----- Original Message ----- >>> From: "Folker Meyer" <folker.meyer(a)googlemail.com> >>> To: "Gordon D. Pusch" <gdpusch(a)xnet.com> >>> Cc: "Robert Olson" <olson(a)mcs.anl.gov>, mg-rast(a)mcs.anl.gov >>> Sent: Saturday, May 31, 2008 10:11:05 AM GMT -06:00 US/Canada >>> Central >>> Subject: Re: [mg-rast] Re: blastall mem use >>> >>> >>> On May 31, 2008, at 10:08 AM, Gordon D. Pusch wrote: >>> >>>> On 2008-May-31, at 8:27 AM, Robert Olson wrote: >>>> >>>>> does blastx memory use increase nonlinearly with the size of >>>>> the NR? >>>> >>>> I wouldn't think so. The size of the dynamic programming matrix >>>> should at worst scale as the product of the query and comparison- >>>> sequence lengths, both of which should be uncorrelated with >>>> database >>>> size. >>>> >>>> I suppose the overhead of representing the formatdb hash in memory >>>> might very well scale nonlinearly with NR size, but I've never >>>> heard >>>> it >>>> mentioned, before... >>>> >>>> I'm not sure which of the formatdb files do what (although I would >>>> think >>>> that it must be documented _somewhere_), but you might look at >>>> whether >>>> one of them is growing superlinearly with NR size... >>>> >>>> >>>>> The mgrast blastx runs I'm doing now against the version 18 NR >>>>> (2.5G vs the former one at 1.9G) are consuming on the order of 3G >>>>> of RAM; this means among other things that we have cut the size >>>>> of the cluster in half as we don't have the RAM on the nodes >>>>> to run two copies (though we may be able to get some back >>>>> using the 2-thread option in blastall). >>>> >>>> Ugh. That's unpleasant news... :-( >>>> >>>> There are variants of BLAST that allow splitting an NR into chunks >>>> for distribution across a cluster, and then recombining the output; >>>> perhaps we will be forced to look into them... :-( >>> >>> >>> Why? If we use more threads per job we might not loose any overall >>> throughput >>> >>>> >>>> >>>> -- Gordon >>>> >>>> >>> >>> >> >> >

1 0

Re: [mg-rast] Re: blastall mem use
by Folker Meyer 31 May '08

31 May '08

from the NCBI pags: BLAST 2.2.18 now available date: 2008-04-01 My suspicion is that now that 64 bit is more common they might have fixed some things... On May 31, 2008, at 3:10 PM, Robert Olson wrote: > 2.2.14 > > > ----- Original Message ----- > From: "Folker Meyer" <folker.meyer(a)googlemail.com> > To: "Robert Olson" <olson(a)mcs.anl.gov> > Cc: "Folker Meyer" <folker.meyer(a)googlemail.com>, mg- > rast(a)mcs.anl.gov, "Gordon D. Pusch" <gdpusch(a)xnet.com> > Sent: Saturday, May 31, 2008 3:00:55 PM GMT -06:00 US/Canada Central > Subject: Re: [mg-rast] Re: blastall mem use > > here is my standard question: what version? Are we using the latest > version. > > Folker > > On May 31, 2008, at 2:52 PM, Robert Olson wrote: > >> we'd have to go there if the db grew enough that a single node's RAM >> could hold it. >> >> here are some numbers from jobs running on one of the ppc nodes now: >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 28766 olson 25 0 1222m 494m 475m R 99.3 12.5 16102:53 blastall >> 19651 mgrastte 18 0 1969m 1.5g 198m R 97.3 37.7 338:01.66 blastall >> >> The first is an original mgrast run against /scratch/metagenome-48- >> hour/Data/SEED_2006_07_01 at 1.6GB. >> >> The second is the new mgrast against /vol/mg-rast-test/Data/db/seed/ >> 018/nr at 2.6GB. >> >> NR size increased by a factor of 1.7, and virtual size by 1.6, so >> that is following. The resident size tho is much larger for the new >> one. >> >> On the 64-bit machine (mg-rast) we see this tho >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 12370 mgrastte 25 0 3657m 3.1g 780m R 100 20.0 85:46.74 blastall >> 4989 mgrastte 25 0 3401m 3.0g 899m R 98 19.1 451:13.21 blastall >> 12013 mgrastte 25 0 3727m 3.2g 807m R 98 20.6 103:35.79 blastall >> >> a near doubling in virtual size. Leads me to wonder if blast is >> doing something dumb on 64-bit machines. though it does mean that we >> are still good with running two copies on the PPC nodes, for now >> anyway. >> >> ----- Original Message ----- >> From: "Folker Meyer" <folker.meyer(a)googlemail.com> >> To: "Gordon D. Pusch" <gdpusch(a)xnet.com> >> Cc: "Robert Olson" <olson(a)mcs.anl.gov>, mg-rast(a)mcs.anl.gov >> Sent: Saturday, May 31, 2008 10:11:05 AM GMT -06:00 US/Canada Central >> Subject: Re: [mg-rast] Re: blastall mem use >> >> >> On May 31, 2008, at 10:08 AM, Gordon D. Pusch wrote: >> >>> On 2008-May-31, at 8:27 AM, Robert Olson wrote: >>> >>>> does blastx memory use increase nonlinearly with the size of the >>>> NR? >>> >>> I wouldn't think so. The size of the dynamic programming matrix >>> should at worst scale as the product of the query and comparison- >>> sequence lengths, both of which should be uncorrelated with database >>> size. >>> >>> I suppose the overhead of representing the formatdb hash in memory >>> might very well scale nonlinearly with NR size, but I've never heard >>> it >>> mentioned, before... >>> >>> I'm not sure which of the formatdb files do what (although I would >>> think >>> that it must be documented _somewhere_), but you might look at >>> whether >>> one of them is growing superlinearly with NR size... >>> >>> >>>> The mgrast blastx runs I'm doing now against the version 18 NR >>>> (2.5G vs the former one at 1.9G) are consuming on the order of 3G >>>> of RAM; this means among other things that we have cut the size >>>> of the cluster in half as we don't have the RAM on the nodes >>>> to run two copies (though we may be able to get some back >>>> using the 2-thread option in blastall). >>> >>> Ugh. That's unpleasant news... :-( >>> >>> There are variants of BLAST that allow splitting an NR into chunks >>> for distribution across a cluster, and then recombining the output; >>> perhaps we will be forced to look into them... :-( >> >> >> Why? If we use more threads per job we might not loose any overall >> throughput >> >>> >>> >>> -- Gordon >>> >>> >> >> > >

1 0

Re: [mg-rast] Re: blastall mem use
by Robert Olson 31 May '08

31 May '08

2.2.14 ----- Original Message ----- From: "Folker Meyer" <folker.meyer(a)googlemail.com> To: "Robert Olson" <olson(a)mcs.anl.gov> Cc: "Folker Meyer" <folker.meyer(a)googlemail.com>, mg-rast(a)mcs.anl.gov, "Gordon D. Pusch" <gdpusch(a)xnet.com> Sent: Saturday, May 31, 2008 3:00:55 PM GMT -06:00 US/Canada Central Subject: Re: [mg-rast] Re: blastall mem use here is my standard question: what version? Are we using the latest version. Folker On May 31, 2008, at 2:52 PM, Robert Olson wrote: > we'd have to go there if the db grew enough that a single node's RAM > could hold it. > > here are some numbers from jobs running on one of the ppc nodes now: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 28766 olson 25 0 1222m 494m 475m R 99.3 12.5 16102:53 blastall > 19651 mgrastte 18 0 1969m 1.5g 198m R 97.3 37.7 338:01.66 blastall > > The first is an original mgrast run against /scratch/metagenome-48- > hour/Data/SEED_2006_07_01 at 1.6GB. > > The second is the new mgrast against /vol/mg-rast-test/Data/db/seed/ > 018/nr at 2.6GB. > > NR size increased by a factor of 1.7, and virtual size by 1.6, so > that is following. The resident size tho is much larger for the new > one. > > On the 64-bit machine (mg-rast) we see this tho > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 12370 mgrastte 25 0 3657m 3.1g 780m R 100 20.0 85:46.74 blastall > 4989 mgrastte 25 0 3401m 3.0g 899m R 98 19.1 451:13.21 blastall > 12013 mgrastte 25 0 3727m 3.2g 807m R 98 20.6 103:35.79 blastall > > a near doubling in virtual size. Leads me to wonder if blast is > doing something dumb on 64-bit machines. though it does mean that we > are still good with running two copies on the PPC nodes, for now > anyway. > > ----- Original Message ----- > From: "Folker Meyer" <folker.meyer(a)googlemail.com> > To: "Gordon D. Pusch" <gdpusch(a)xnet.com> > Cc: "Robert Olson" <olson(a)mcs.anl.gov>, mg-rast(a)mcs.anl.gov > Sent: Saturday, May 31, 2008 10:11:05 AM GMT -06:00 US/Canada Central > Subject: Re: [mg-rast] Re: blastall mem use > > > On May 31, 2008, at 10:08 AM, Gordon D. Pusch wrote: > >> On 2008-May-31, at 8:27 AM, Robert Olson wrote: >> >>> does blastx memory use increase nonlinearly with the size of the NR? >> >> I wouldn't think so. The size of the dynamic programming matrix >> should at worst scale as the product of the query and comparison- >> sequence lengths, both of which should be uncorrelated with database >> size. >> >> I suppose the overhead of representing the formatdb hash in memory >> might very well scale nonlinearly with NR size, but I've never heard >> it >> mentioned, before... >> >> I'm not sure which of the formatdb files do what (although I would >> think >> that it must be documented _somewhere_), but you might look at >> whether >> one of them is growing superlinearly with NR size... >> >> >>> The mgrast blastx runs I'm doing now against the version 18 NR >>> (2.5G vs the former one at 1.9G) are consuming on the order of 3G >>> of RAM; this means among other things that we have cut the size >>> of the cluster in half as we don't have the RAM on the nodes >>> to run two copies (though we may be able to get some back >>> using the 2-thread option in blastall). >> >> Ugh. That's unpleasant news... :-( >> >> There are variants of BLAST that allow splitting an NR into chunks >> for distribution across a cluster, and then recombining the output; >> perhaps we will be forced to look into them... :-( > > > Why? If we use more threads per job we might not loose any overall > throughput > >> >> >> -- Gordon >> >> > >

1 0

Re: [mg-rast] Re: blastall mem use
by Folker Meyer 31 May '08

31 May '08

here is my standard question: what version? Are we using the latest version. Folker On May 31, 2008, at 2:52 PM, Robert Olson wrote: > we'd have to go there if the db grew enough that a single node's RAM > could hold it. > > here are some numbers from jobs running on one of the ppc nodes now: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 28766 olson 25 0 1222m 494m 475m R 99.3 12.5 16102:53 blastall > 19651 mgrastte 18 0 1969m 1.5g 198m R 97.3 37.7 338:01.66 blastall > > The first is an original mgrast run against /scratch/metagenome-48- > hour/Data/SEED_2006_07_01 at 1.6GB. > > The second is the new mgrast against /vol/mg-rast-test/Data/db/seed/ > 018/nr at 2.6GB. > > NR size increased by a factor of 1.7, and virtual size by 1.6, so > that is following. The resident size tho is much larger for the new > one. > > On the 64-bit machine (mg-rast) we see this tho > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 12370 mgrastte 25 0 3657m 3.1g 780m R 100 20.0 85:46.74 blastall > 4989 mgrastte 25 0 3401m 3.0g 899m R 98 19.1 451:13.21 blastall > 12013 mgrastte 25 0 3727m 3.2g 807m R 98 20.6 103:35.79 blastall > > a near doubling in virtual size. Leads me to wonder if blast is > doing something dumb on 64-bit machines. though it does mean that we > are still good with running two copies on the PPC nodes, for now > anyway. > > ----- Original Message ----- > From: "Folker Meyer" <folker.meyer(a)googlemail.com> > To: "Gordon D. Pusch" <gdpusch(a)xnet.com> > Cc: "Robert Olson" <olson(a)mcs.anl.gov>, mg-rast(a)mcs.anl.gov > Sent: Saturday, May 31, 2008 10:11:05 AM GMT -06:00 US/Canada Central > Subject: Re: [mg-rast] Re: blastall mem use > > > On May 31, 2008, at 10:08 AM, Gordon D. Pusch wrote: > >> On 2008-May-31, at 8:27 AM, Robert Olson wrote: >> >>> does blastx memory use increase nonlinearly with the size of the NR? >> >> I wouldn't think so. The size of the dynamic programming matrix >> should at worst scale as the product of the query and comparison- >> sequence lengths, both of which should be uncorrelated with database >> size. >> >> I suppose the overhead of representing the formatdb hash in memory >> might very well scale nonlinearly with NR size, but I've never heard >> it >> mentioned, before... >> >> I'm not sure which of the formatdb files do what (although I would >> think >> that it must be documented _somewhere_), but you might look at >> whether >> one of them is growing superlinearly with NR size... >> >> >>> The mgrast blastx runs I'm doing now against the version 18 NR >>> (2.5G vs the former one at 1.9G) are consuming on the order of 3G >>> of RAM; this means among other things that we have cut the size >>> of the cluster in half as we don't have the RAM on the nodes >>> to run two copies (though we may be able to get some back >>> using the 2-thread option in blastall). >> >> Ugh. That's unpleasant news... :-( >> >> There are variants of BLAST that allow splitting an NR into chunks >> for distribution across a cluster, and then recombining the output; >> perhaps we will be forced to look into them... :-( > > > Why? If we use more threads per job we might not loose any overall > throughput > >> >> >> -- Gordon >> >> > >

1 0

Re: [mg-rast] Re: blastall mem use
by Robert Olson 31 May '08

31 May '08

we'd have to go there if the db grew enough that a single node's RAM could hold it. here are some numbers from jobs running on one of the ppc nodes now: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28766 olson 25 0 1222m 494m 475m R 99.3 12.5 16102:53 blastall 19651 mgrastte 18 0 1969m 1.5g 198m R 97.3 37.7 338:01.66 blastall The first is an original mgrast run against /scratch/metagenome-48-hour/Data/SEED_2006_07_01 at 1.6GB. The second is the new mgrast against /vol/mg-rast-test/Data/db/seed/018/nr at 2.6GB. NR size increased by a factor of 1.7, and virtual size by 1.6, so that is following. The resident size tho is much larger for the new one. On the 64-bit machine (mg-rast) we see this tho PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12370 mgrastte 25 0 3657m 3.1g 780m R 100 20.0 85:46.74 blastall 4989 mgrastte 25 0 3401m 3.0g 899m R 98 19.1 451:13.21 blastall 12013 mgrastte 25 0 3727m 3.2g 807m R 98 20.6 103:35.79 blastall a near doubling in virtual size. Leads me to wonder if blast is doing something dumb on 64-bit machines. though it does mean that we are still good with running two copies on the PPC nodes, for now anyway. ----- Original Message ----- From: "Folker Meyer" <folker.meyer(a)googlemail.com> To: "Gordon D. Pusch" <gdpusch(a)xnet.com> Cc: "Robert Olson" <olson(a)mcs.anl.gov>, mg-rast(a)mcs.anl.gov Sent: Saturday, May 31, 2008 10:11:05 AM GMT -06:00 US/Canada Central Subject: Re: [mg-rast] Re: blastall mem use On May 31, 2008, at 10:08 AM, Gordon D. Pusch wrote: > On 2008-May-31, at 8:27 AM, Robert Olson wrote: > >> does blastx memory use increase nonlinearly with the size of the NR? > > I wouldn't think so. The size of the dynamic programming matrix > should at worst scale as the product of the query and comparison- > sequence lengths, both of which should be uncorrelated with database > size. > > I suppose the overhead of representing the formatdb hash in memory > might very well scale nonlinearly with NR size, but I've never heard > it > mentioned, before... > > I'm not sure which of the formatdb files do what (although I would > think > that it must be documented _somewhere_), but you might look at whether > one of them is growing superlinearly with NR size... > > >> The mgrast blastx runs I'm doing now against the version 18 NR >> (2.5G vs the former one at 1.9G) are consuming on the order of 3G >> of RAM; this means among other things that we have cut the size >> of the cluster in half as we don't have the RAM on the nodes >> to run two copies (though we may be able to get some back >> using the 2-thread option in blastall). > > Ugh. That's unpleasant news... :-( > > There are variants of BLAST that allow splitting an NR into chunks > for distribution across a cluster, and then recombining the output; > perhaps we will be forced to look into them... :-( Why? If we use more threads per job we might not loose any overall throughput > > > -- Gordon > >

1 0

Re: [mg-rast] Re: blastall mem use
by Gordon D. Pusch 31 May '08

31 May '08

On 2008-May-31, at 10:11 AM, Folker Meyer wrote: > On May 31, 2008, at 10:08 AM, Gordon D. Pusch wrote: >> On 2008-May-31, at 8:27 AM, Robert Olson wrote: >> >>> The mgrast blastx runs I'm doing now against the version 18 NR >>> (2.5G vs the former one at 1.9G) are consuming on the order of 3G >>> of RAM; this means among other things that we have cut the size >>> of the cluster in half as we don't have the RAM on the nodes >>> to run two copies (though we may be able to get some back >>> using the 2-thread option in blastall). >> >> Ugh. That's unpleasant news... :-( >> >> There are variants of BLAST that allow splitting an NR into chunks >> for distribution across a cluster, and then recombining the output; >> perhaps we will be forced to look into them... :-( > > Why? If we use more threads per job we might not loose any overall > throughput Because from what Bob is telling us, in less than 18 months or so, the NR will be so big that we won't even be able to run _one_ thread on one of the "large memory" machines... :-( -- Gordon

1 0

Re: [mg-rast] Re: blastall mem use
by Folker Meyer 31 May '08

31 May '08

On May 31, 2008, at 10:08 AM, Gordon D. Pusch wrote: > On 2008-May-31, at 8:27 AM, Robert Olson wrote: > >> does blastx memory use increase nonlinearly with the size of the NR? > > I wouldn't think so. The size of the dynamic programming matrix > should at worst scale as the product of the query and comparison- > sequence lengths, both of which should be uncorrelated with database > size. > > I suppose the overhead of representing the formatdb hash in memory > might very well scale nonlinearly with NR size, but I've never heard > it > mentioned, before... > > I'm not sure which of the formatdb files do what (although I would > think > that it must be documented _somewhere_), but you might look at whether > one of them is growing superlinearly with NR size... > > >> The mgrast blastx runs I'm doing now against the version 18 NR >> (2.5G vs the former one at 1.9G) are consuming on the order of 3G >> of RAM; this means among other things that we have cut the size >> of the cluster in half as we don't have the RAM on the nodes >> to run two copies (though we may be able to get some back >> using the 2-thread option in blastall). > > Ugh. That's unpleasant news... :-( > > There are variants of BLAST that allow splitting an NR into chunks > for distribution across a cluster, and then recombining the output; > perhaps we will be forced to look into them... :-( Why? If we use more threads per job we might not loose any overall throughput > > > -- Gordon > >

1 0

Re: blastall mem use
by Gordon D. Pusch 31 May '08

31 May '08

On 2008-May-31, at 8:27 AM, Robert Olson wrote: > does blastx memory use increase nonlinearly with the size of the NR? I wouldn't think so. The size of the dynamic programming matrix should at worst scale as the product of the query and comparison- sequence lengths, both of which should be uncorrelated with database size. I suppose the overhead of representing the formatdb hash in memory might very well scale nonlinearly with NR size, but I've never heard it mentioned, before... I'm not sure which of the formatdb files do what (although I would think that it must be documented _somewhere_), but you might look at whether one of them is growing superlinearly with NR size... > The mgrast blastx runs I'm doing now against the version 18 NR > (2.5G vs the former one at 1.9G) are consuming on the order of 3G > of RAM; this means among other things that we have cut the size > of the cluster in half as we don't have the RAM on the nodes > to run two copies (though we may be able to get some back > using the 2-thread option in blastall). Ugh. That's unpleasant news... :-( There are variants of BLAST that allow splitting an NR into chunks for distribution across a cluster, and then recombining the output; perhaps we will be forced to look into them... :-( -- Gordon

1 0

blastall mem use
by Robert Olson 31 May '08

31 May '08

does blastx memory use increase nonlinearly with the size of the NR? The mgrast blastx runs I'm doing now against the version 18 NR (2.5G vs the former one at 1.9G) are consuming on the order of 3G of RAM; this means among other things that we have cut the size of the cluster in half as we don't have the RAM on the nodes to run two copies (though we may be able to get some back using the 2-thread option in blastall). --bob

1 0

update
by Robert Olson 30 May '08

30 May '08

Quick status update ... first fullsize job is running through the newly bolted-together pipeline as a shake-out run. New features of this version include Flexible configuration of blast databases, including maintenance scripts for doing the necessary database loads from the databases (required before runs may be processed against the database). Ability to load multiple different SEED NR databases and the associated subsystem data. Incremental loading of the sims databases, as the sims runs finish. Conceivably one could get an early view of their data before it all finishes. etc. --bob

1 0