Media repositories and memory usage

Media repositories and memory usage

[Nguyen Thai Ngoc Duy]

Hi,

There is another aspect I did not see Tim mention in his slides:
memory usage with large blobs. Git has tradition of loading the whole
blob in memory for easy manipulation. It could consume a lot of memory
in large blob case (and particularly worse in mingw port because it
does not support mmap).

I see these operations that need access to blobs:
 1. checkout blobs
 2. checkin blobs
 3. diff/delta blobs

Diffing blobs should be avoided any way for large blobs. Checking in
blobs does not require lots of memory. I'm working on checkout case to
reduce memory footprint. Is there any other case that will need full
blob in memory?

PS. For checkout/checkin case, if you do any conversion, full blob in
memory is still needed. But I guess that is rare for large blobs.
-- 
Duy