On May 31, 2008, at 10:08 AM, Gordon D. Pusch wrote:

On 2008-May-31, at 8:27 AM, Robert Olson wrote:

...

does blastx memory use increase nonlinearly with the size of the NR?

I wouldn't think so. The size of the dynamic programming matrix should at worst scale as the product of the query and comparison- sequence lengths, both of which should be uncorrelated with database size.

I suppose the overhead of representing the formatdb hash in memory might very well scale nonlinearly with NR size, but I've never heard it mentioned, before...

I'm not sure which of the formatdb files do what (although I would think that it must be documented _somewhere_), but you might look at whether one of them is growing superlinearly with NR size...

...

The mgrast blastx runs I'm doing now against the version 18 NR (2.5G vs the former one at 1.9G) are consuming on the order of 3G of RAM; this means among other things that we have cut the size of the cluster in half as we don't have the RAM on the nodes to run two copies (though we may be able to get some back using the 2-thread option in blastall).

Ugh. That's unpleasant news... :-(

There are variants of BLAST that allow splitting an NR into chunks for distribution across a cluster, and then recombining the output; perhaps we will be forced to look into them... :-(

Why? If we use more threads per job we might not loose any overall throughput

-- Gordon