How do I get the latest sparse?

How do I get the latest sparse?

[Mitesh Shah]

Hi,

How do I get the latest version of sparse? The git repository from the
sparse home page
(http://kernel.org/pub/linux/kernel/people/josh/sparse/) does not show
any update since Dec 05, 2006, while I see few patched submitted after
that.

Also the latest tarball URL gives me a version which is not updated
since Nov 06
(http://www.codemonkey.org.uk/projects/git-snapshots/sparse/sparse-lates
t.tar.gz)

Is there something I am missing?

Thanks,

-Mitesh

Re: How do I get the latest sparse?

[Christopher Li]

That is the same git repository I am pulling from. There is not
much update recently. I assume it is the Christmas/New Year holidays.

Don't worry, Josh will get back to us.

Chris

On Mon, Jan 15, 2007 at 03:48:26PM -0800, Mitesh Shah wrote:
> Hi,
> 
> How do I get the latest version of sparse? The git repository from the
> sparse home page
> (http://kernel.org/pub/linux/kernel/people/josh/sparse/) does not show
> any update since Dec 05, 2006, while I see few patched submitted after
> that.
> 
> Also the latest tarball URL gives me a version which is not updated
> since Nov 06
> (http://www.codemonkey.org.uk/projects/git-snapshots/sparse/sparse-lates
> t.tar.gz)
> 
> Is there something I am missing?
> 
> Thanks,
> 
> -Mitesh
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-sparse" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How do I get the latest sparse?

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 1670 bytes --]

Christopher Li wrote:
> That is the same git repository I am pulling from. There is not
> much update recently. I assume it is the Christmas/New Year holidays.

That combined with the back-to-school crunch and work on a research grant.
The grant helps further my work on Sparse, though. :)

I've followed all the patches on linux-sparse, and I have tested several of
them in local git; I just haven't pushed out new bits.  I particularly look
forward to integrating your NULL pointer checking code, and I think I can
adapt some of the dataflow analysis code for lock context checking as well.  I
also know at least one likely customer for the smarter Sparse-based ctags, and
I plan to look into generating cscope information as well (sparscope).  Expect
bits soon.

Also, some of the patches came with disclaimers along the lines of "don't use
this, I plan to rewrite". :) For those, I wanted to wait a bit and see where
they go; however, given the usefulness of some of those patches, I may go
ahead and merge them anyway.

- Josh Triplett

> Chris
> 
> On Mon, Jan 15, 2007 at 03:48:26PM -0800, Mitesh Shah wrote:
>> Hi,
>>
>> How do I get the latest version of sparse? The git repository from the
>> sparse home page
>> (http://kernel.org/pub/linux/kernel/people/josh/sparse/) does not show
>> any update since Dec 05, 2006, while I see few patched submitted after
>> that.
>>
>> Also the latest tarball URL gives me a version which is not updated
>> since Nov 06
>> (http://www.codemonkey.org.uk/projects/git-snapshots/sparse/sparse-lates
>> t.tar.gz)
>>
>> Is there something I am missing?
>>
>> Thanks,
>>
>> -Mitesh



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: How do I get the latest sparse?

[Christopher Li]

On Tue, Jan 16, 2007 at 04:23:57AM -0800, Josh Triplett wrote:
> I've followed all the patches on linux-sparse, and I have tested several of
> them in local git; I just haven't pushed out new bits.  I particularly look

Great to know you are back. I can resend my stack of patches now :-)

> forward to integrating your NULL pointer checking code, and I think I can

I haven't get much feed back of the new checking code yet. I will refresh my
patch as well. Without cross function/file checking, the interrupt checking
generates way too many false positives. I am working on the cross file checking
right now.

> adapt some of the dataflow analysis code for lock context checking as well.  I
> also know at least one likely customer for the smarter Sparse-based ctags, and
> I plan to look into generating cscope information as well (sparscope).  Expect
> bits soon.

Sparscope will be interesting.

Chris

Re: How do I get the latest sparse?

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 1264 bytes --]

Christopher Li wrote:
> On Tue, Jan 16, 2007 at 04:23:57AM -0800, Josh Triplett wrote:
>> I've followed all the patches on linux-sparse, and I have tested several of
>> them in local git; I just haven't pushed out new bits.  I particularly look
> 
> Great to know you are back. I can resend my stack of patches now :-)

In general, you only need to resend if you have changes.  Otherwise, I can
just use the previous versions you mailed. :)

Currently working my way through your resent patches.

>> forward to integrating your NULL pointer checking code, and I think I can
> 
> I haven't get much feed back of the new checking code yet.

Expect some comments in response to your new patch shortly.

> I will refresh my patch as well.

It probably still applied. :)

> Without cross function/file checking, the interrupt checking generates way
> too many false positives.

Sounds a lot like the context checking. :)

The use of strcmp on asm strings for x86 instruction mnemonics seems like a
problem, though.

> I am working on the cross file checking right now.

Excellent.  I would welcome any attempts at cross-function and cross-file
checking, rather than continuing to add annotations for that purpose.

- Josh Triplett


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: How do I get the latest sparse?

[Christopher Li]

On Tue, Jan 16, 2007 at 09:41:49PM -0800, Josh Triplett wrote:
> > Without cross function/file checking, the interrupt checking generates way
> > too many false positives.
> 
> Sounds a lot like the context checking. :)

That is because the rules of the checking is too restrictive. 

> The use of strcmp on asm strings for x86 instruction mnemonics seems like a
> problem, though.

Yes, I agree. The interrupt checking is an example of how the checker
can be written. It is not very usable as it is. I have a few idea to address
it, but it will not help the usability issue. Cross functions checking is the
way to go.

Chris

Re: How do I get the latest sparse?

[Sam Ravnborg]

> 
> Excellent.  I would welcome any attempts at cross-function and cross-file
> checking, rather than continuing to add annotations for that purpose.

I experimented with a patch some time ago that allowed us to run
sparse on all files in a directory for the kernel.
But I experienced OOM while checking the XFS sources.
Reply on the matter from Linus was at that time something in the line
of that sparse did not do any atemp to free up memory after the early
parse stage so it is consuming much more memory than needed.

This was almost one year ago so things may have changed - but at least
something to keep an eye on.

XFS was just triggering this - it surely was due to codesize alone so
any other big directory could have been the trigger too. It is not XFS specific.

	Sam

Re: How do I get the latest sparse?

[Christopher Li]

Yes, sparse is a memory hog if you keep adding files to it.
It is actually very hard to free the memory used in middle
stage, because they have dependences.

Here is my plan on this:

Save the linearized/AST result as byte code to file. Because
my checking is linearization based. Currently I only include the
information I cared. Most of the expression and statement
is not stored. They turn into linearize code any way.

Process all the byte code result, build a global symbol dictionary
into ident hash table. The implementation part of the symbol, namely
the function body, is dynamically loaded.

When we need some global symbol, the corresponding byte code file can
be load into memory. I don't deal with individual symbol loading because
they have dependence to other symbol in the same file any way. A byte
code file can be removed from the memory as well.

My current hacking shows that the byte code size is about 10x the size
of the .o file. Given that each linearized instruction is 32 bytes, it
is not as bad as it sounds. The total result of a linux kernel will
be a few hundred Mb. It might be possible to fit that all in the machine
I have, keep my finger crossed.

Chris


On Wed, Jan 17, 2007 at 08:05:20PM +0100, Sam Ravnborg wrote:
> 
> I experimented with a patch some time ago that allowed us to run
> sparse on all files in a directory for the kernel.
> But I experienced OOM while checking the XFS sources.
> Reply on the matter from Linus was at that time something in the line
> of that sparse did not do any atemp to free up memory after the early
> parse stage so it is consuming much more memory than needed.
> 
> This was almost one year ago so things may have changed - but at least
> something to keep an eye on.
> 
> XFS was just triggering this - it surely was due to codesize alone so
> any other big directory could have been the trigger too. It is not XFS specific.
> 
> 	Sam

Re: How do I get the latest sparse?

[Randy Dunlap]

On Mon, 15 Jan 2007 15:48:26 -0800 Mitesh Shah wrote:

> Hi,
> 
> How do I get the latest version of sparse? The git repository from the
> sparse home page
> (http://kernel.org/pub/linux/kernel/people/josh/sparse/) does not show
> any update since Dec 05, 2006, while I see few patched submitted after
> that.
> 
> Also the latest tarball URL gives me a version which is not updated
> since Nov 06
> (http://www.codemonkey.org.uk/projects/git-snapshots/sparse/sparse-lates
> t.tar.gz)
> 
> Is there something I am missing?

http://www.codemonkey.org.uk/projects/git-snapshots/sparse/ says:

[ ]	sparse-2007-01-10.tar.gz	10-Jan-2007 23:01 	169K
[ ]	sparse-2007-01-11.tar.gz	11-Jan-2007 23:01 	169K
[ ]	sparse-2007-01-12.tar.gz	12-Jan-2007 23:01 	169K
[ ]	sparse-2007-01-13.tar.gz	13-Jan-2007 23:00 	169K
[ ]	sparse-2007-01-14.tar.gz	14-Jan-2007 23:00 	169K
[ ]	sparse-2007-01-15.tar.gz	15-Jan-2007 23:01 	169K
[ ]	sparse-2007-01-16.tar.gz	16-Jan-2007 00:01 	169K
[ ]	sparse-latest.tar.gz		16-Jan-2007 00:01 	169K
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

---
~Randy

RE: How do I get the latest sparse?

[Mitesh Shah]

> http://www.codemonkey.org.uk/projects/git-snapshots/sparse/ says:
>
> [ ]	sparse-2007-01-10.tar.gz	10-Jan-2007 23:01 	169K
> [ ]	sparse-2007-01-11.tar.gz	11-Jan-2007 23:01 	169K
> [ ]	sparse-2007-01-12.tar.gz	12-Jan-2007 23:01 	169K
> [ ]	sparse-2007-01-13.tar.gz	13-Jan-2007 23:00 	169K
> [ ]	sparse-2007-01-14.tar.gz	14-Jan-2007 23:00 	169K
> [ ]	sparse-2007-01-15.tar.gz	15-Jan-2007 23:01 	169K
> [ ]	sparse-2007-01-16.tar.gz	16-Jan-2007 00:01 	169K
> [ ]	sparse-latest.tar.gz		16-Jan-2007 00:01 	169K
>         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> ---
> ~Randy
>

Yes, but if you unzip and untar that file, you will see that the sources
are not updated since Nov 6, 2006. So my guess is that the tar is being
built everyday but the checkout is not happening from the correct source
repository.

-Mitesh

Re: How do I get the latest sparse?

[Randy Dunlap]

On Mon, 15 Jan 2007 17:07:10 -0800 Mitesh Shah wrote:

> > http://www.codemonkey.org.uk/projects/git-snapshots/sparse/ says:
> >
> > [ ]	sparse-2007-01-10.tar.gz	10-Jan-2007 23:01 	169K
> > [ ]	sparse-2007-01-11.tar.gz	11-Jan-2007 23:01 	169K
> > [ ]	sparse-2007-01-12.tar.gz	12-Jan-2007 23:01 	169K
> > [ ]	sparse-2007-01-13.tar.gz	13-Jan-2007 23:00 	169K
> > [ ]	sparse-2007-01-14.tar.gz	14-Jan-2007 23:00 	169K
> > [ ]	sparse-2007-01-15.tar.gz	15-Jan-2007 23:01 	169K
> > [ ]	sparse-2007-01-16.tar.gz	16-Jan-2007 00:01 	169K
> > [ ]	sparse-latest.tar.gz		16-Jan-2007 00:01 	169K
> >         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >
> > ---
> > ~Randy
> >
> 
> Yes, but if you unzip and untar that file, you will see that the sources
> are not updated since Nov 6, 2006. So my guess is that the tar is being
> built everyday but the checkout is not happening from the correct source
> repository.

Oh, OK, thanks.  We'll let the codemonkey know.  :)

---
~Randy

Re: How do I get the latest sparse?

[Dave Jones]

On Mon, Jan 15, 2007 at 06:15:15PM -0800, Randy Dunlap wrote:
 > On Mon, 15 Jan 2007 17:07:10 -0800 Mitesh Shah wrote:
 > 
 > > > http://www.codemonkey.org.uk/projects/git-snapshots/sparse/ says:
 > > >
 > > > [ ]	sparse-2007-01-10.tar.gz	10-Jan-2007 23:01 	169K
 > > > [ ]	sparse-2007-01-11.tar.gz	11-Jan-2007 23:01 	169K
 > > > [ ]	sparse-2007-01-12.tar.gz	12-Jan-2007 23:01 	169K
 > > > [ ]	sparse-2007-01-13.tar.gz	13-Jan-2007 23:00 	169K
 > > > [ ]	sparse-2007-01-14.tar.gz	14-Jan-2007 23:00 	169K
 > > > [ ]	sparse-2007-01-15.tar.gz	15-Jan-2007 23:01 	169K
 > > > [ ]	sparse-2007-01-16.tar.gz	16-Jan-2007 00:01 	169K
 > > > [ ]	sparse-latest.tar.gz		16-Jan-2007 00:01 	169K
 > > >         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 > > >
 > > > ---
 > > > ~Randy
 > > >
 > > 
 > > Yes, but if you unzip and untar that file, you will see that the sources
 > > are not updated since Nov 6, 2006. So my guess is that the tar is being
 > > built everyday but the checkout is not happening from the correct source
 > > repository.
 > 
 > Oh, OK, thanks.  We'll let the codemonkey know.  :)

Odd. It's doing a clone of git://git.kernel.org/pub/scm/linux/kernel/git/josh/sparse.git
I'll poke at it some more when I get back from LCA, latency to that
box is pretty awful from here.

		Dave

-- 
http://www.codemonkey.org.uk

Re: How do I get the latest sparse?

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 1525 bytes --]

Dave Jones wrote:
> On Mon, Jan 15, 2007 at 06:15:15PM -0800, Randy Dunlap wrote:
>  > On Mon, 15 Jan 2007 17:07:10 -0800 Mitesh Shah wrote:
>  > 
>  > > > http://www.codemonkey.org.uk/projects/git-snapshots/sparse/ says:
>  > > >
>  > > > [ ]	sparse-2007-01-10.tar.gz	10-Jan-2007 23:01 	169K
>  > > > [ ]	sparse-2007-01-11.tar.gz	11-Jan-2007 23:01 	169K
>  > > > [ ]	sparse-2007-01-12.tar.gz	12-Jan-2007 23:01 	169K
>  > > > [ ]	sparse-2007-01-13.tar.gz	13-Jan-2007 23:00 	169K
>  > > > [ ]	sparse-2007-01-14.tar.gz	14-Jan-2007 23:00 	169K
>  > > > [ ]	sparse-2007-01-15.tar.gz	15-Jan-2007 23:01 	169K
>  > > > [ ]	sparse-2007-01-16.tar.gz	16-Jan-2007 00:01 	169K
>  > > > [ ]	sparse-latest.tar.gz		16-Jan-2007 00:01 	169K
>  > > >         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>  > > >
>  > > > ---
>  > > > ~Randy
>  > > >
>  > > 
>  > > Yes, but if you unzip and untar that file, you will see that the sources
>  > > are not updated since Nov 6, 2006. So my guess is that the tar is being
>  > > built everyday but the checkout is not happening from the correct source
>  > > repository.
>  > 
>  > Oh, OK, thanks.  We'll let the codemonkey know.  :)
> 
> Odd. It's doing a clone of git://git.kernel.org/pub/scm/linux/kernel/git/josh/sparse.git
> I'll poke at it some more when I get back from LCA, latency to that
> box is pretty awful from here.

I just checked, and the snapshots appear to have current bits from git,
definitely newer than November 6.

- Josh Triplett


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

RE: How do I get the latest sparse?

[Mitesh Shah]

Yes, it is fixed now. The snapshots are now matching the repository.

Thanks,

-Mitesh


> -----Original Message-----
> From: Josh Triplett [mailto:josh@freedesktop.org]
> Sent: Tuesday, January 16, 2007 4:14 AM
> To: Dave Jones
> Cc: Randy Dunlap; Mitesh Shah; Linux-Sparse; davej@codemonkey.org.uk
> Subject: Re: How do I get the latest sparse?
> 
> 
> Dave Jones wrote:
> > On Mon, Jan 15, 2007 at 06:15:15PM -0800, Randy Dunlap wrote:
> >  > On Mon, 15 Jan 2007 17:07:10 -0800 Mitesh Shah wrote:
> >  > 
> >  > > > 
> http://www.codemonkey.org.uk/projects/git-snapshots/sparse/ says:
> >  > > >
> >  > > > [ ]	sparse-2007-01-10.tar.gz	10-Jan-2007 
> 23:01 	169K
> >  > > > [ ]	sparse-2007-01-11.tar.gz	11-Jan-2007 
> 23:01 	169K
> >  > > > [ ]	sparse-2007-01-12.tar.gz	12-Jan-2007 
> 23:01 	169K
> >  > > > [ ]	sparse-2007-01-13.tar.gz	13-Jan-2007 
> 23:00 	169K
> >  > > > [ ]	sparse-2007-01-14.tar.gz	14-Jan-2007 
> 23:00 	169K
> >  > > > [ ]	sparse-2007-01-15.tar.gz	15-Jan-2007 
> 23:01 	169K
> >  > > > [ ]	sparse-2007-01-16.tar.gz	16-Jan-2007 
> 00:01 	169K
> >  > > > [ ]	sparse-latest.tar.gz		16-Jan-2007 
> 00:01 	169K
> >  > > >         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >  > > >
> >  > > > ---
> >  > > > ~Randy
> >  > > >
> >  > > 
> >  > > Yes, but if you unzip and untar that file, you will 
> see that the sources
> >  > > are not updated since Nov 6, 2006. So my guess is that 
> the tar is being
> >  > > built everyday but the checkout is not happening from 
> the correct source
> >  > > repository.
> >  > 
> >  > Oh, OK, thanks.  We'll let the codemonkey know.  :)
> > 
> > Odd. It's doing a clone of 
> git://git.kernel.org/pub/scm/linux/kernel/git/josh/sparse.git
> > I'll poke at it some more when I get back from LCA, latency to that
> > box is pretty awful from here.
> 
> I just checked, and the snapshots appear to have current bits 
> from git,
> definitely newer than November 6.
> 
> - Josh Triplett
> 
> 

Re: How do I get the latest sparse?

[Dave Jones]

On Tue, Jan 16, 2007 at 07:40:33AM -0800, Mitesh Shah wrote:
 > Yes, it is fixed now. The snapshots are now matching the repository.

Strange.  I just reran the script by hand rather than by cron yesterday,
so I've no explanation as to what happened before :)

		Dave