Re: How do I get the latest sparse?

[Christopher Li <sparse@chrisli.org>]

Yes, sparse is a memory hog if you keep adding files to it.
It is actually very hard to free the memory used in middle
stage, because they have dependences.

Here is my plan on this:

Save the linearized/AST result as byte code to file. Because
my checking is linearization based. Currently I only include the
information I cared. Most of the expression and statement
is not stored. They turn into linearize code any way.

Process all the byte code result, build a global symbol dictionary
into ident hash table. The implementation part of the symbol, namely
the function body, is dynamically loaded.

When we need some global symbol, the corresponding byte code file can
be load into memory. I don't deal with individual symbol loading because
they have dependence to other symbol in the same file any way. A byte
code file can be removed from the memory as well.

My current hacking shows that the byte code size is about 10x the size
of the .o file. Given that each linearized instruction is 32 bytes, it
is not as bad as it sounds. The total result of a linux kernel will
be a few hundred Mb. It might be possible to fit that all in the machine
I have, keep my finger crossed.

Chris


On Wed, Jan 17, 2007 at 08:05:20PM +0100, Sam Ravnborg wrote:
> 
> I experimented with a patch some time ago that allowed us to run
> sparse on all files in a directory for the kernel.
> But I experienced OOM while checking the XFS sources.
> Reply on the matter from Linus was at that time something in the line
> of that sparse did not do any atemp to free up memory after the early
> parse stage so it is consuming much more memory than needed.
> 
> This was almost one year ago so things may have changed - but at least
> something to keep an eye on.
> 
> XFS was just triggering this - it surely was due to codesize alone so
> any other big directory could have been the trigger too. It is not XFS specific.
> 
> 	Sam