Detect unused header files?

Detect unused header files?

[Sam Ravnborg]

In the kernel we would like to avoid all
unused include files.
Especially in the headers we export to userspace.

Are there any easy way we can use sparse to detect
that a specific header file is not used?

I know this will depend on the configuration, but
for the use I have in mind this does matter only a little.

The header files I have in mind will be fully self-contained
as they all include the header files they need to be used.

Sample:

cat foo.h:
#include <linux/types.h>
#include <linux/swab.h>

struct foo {
        __u32 bar;
        __u32 baz;
};

EOF


Here <linux/swab.h> is obviously not used.
And I would like sparse to flag this...

	Sam

Re: Detect unused header files?

[Christopher Li]

On Tue, Jul 28, 2009 at 11:18 AM, Sam Ravnborg<sam@ravnborg.org> wrote:
> In the kernel we would like to avoid all
> unused include files.
> Especially in the headers we export to userspace.
>
> Are there any easy way we can use sparse to detect
> that a specific header file is not used?

I don't know a way to do that with current sparse.
We can implement some code for this.

We need to have a good definition of what is used.

> The header files I have in mind will be fully self-contained
> as they all include the header files they need to be used.

But a lot of case we have the header file is used in *another*
file.

>
> Sample:
>
> cat foo.h:
> #include <linux/types.h>
> #include <linux/swab.h>
>
> struct foo {
>        __u32 bar;
>        __u32 baz;
> };
>
> EOF
>
>
> Here <linux/swab.h> is obviously not used.
> And I would like sparse to flag this...

I don't think it is can be turn on by default.
What if some user space program want to include foo.h
and use the definitional inside "swab.h"? In that case it will
be OK. I don't think we can ban this kind of the usage case.

Looking at the header file itself is not good enough. It also depend
on how the user include it. What macro has been defined.
That will change the behavior of the header file as well.

Chris
--
To unsubscribe from this list: send the line "unsubscribe linux-sparse" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Detect unused header files?

[Robert P. J. Day]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1466 bytes --]

On Tue, 28 Jul 2009, Christopher Li wrote:

> But a lot of case we have the header file is used in *another*
> file.
>
> >
> > Sample:
> >
> > cat foo.h:
> > #include <linux/types.h>
> > #include <linux/swab.h>
> >
> > struct foo {
> >        __u32 bar;
> >        __u32 baz;
> > };
> >
> > EOF
> >
> >
> > Here <linux/swab.h> is obviously not used.
> > And I would like sparse to flag this...
>
> I don't think it is can be turn on by default. What if some user
> space program want to include foo.h and use the definitional inside
> "swab.h"? In that case it will be OK. I don't think we can ban this
> kind of the usage case.

  sure you can.  no one should be counting on the inclusion of header
files by other header files (except possibly in unusual situations,
which i can't even think of at the moment).  if your program needs the
contents of a header file, it's your responsibility to include it.

rday
--


========================================================================
Robert P. J. Day                               Waterloo, Ontario, CANADA

        Linux Consulting, Training and Annoying Kernel Pedantry.

Web page:                                          http://crashcourse.ca
Twitter:                                       http://twitter.com/rpjday
"Kernel Newbie Corner" column @ linux.com:          http://cli.gs/WG6WYX
========================================================================

Re: Detect unused header files?

[Christopher Li]

On Tue, Jul 28, 2009 at 1:36 PM, Robert P. J. Day<rpjday@crashcourse.ca> wrote:
> sure you can.  no one should be counting on the inclusion of header
> files by other header files (except possibly in unusual situations,
> which i can't even think of at the moment).  if your program needs the
> contents of a header file, it's your responsibility to include it.

The case I have in mind is that, some API level header file include some
internal header file for constant defines. For example:

drivers/ieee1394/host.h include drivers/ieee1394/csr.h.

The "csr.h" is just the register define. It is not used in this header file
itself. I did not check very closely, for augments sake pretend those
inline function are not there. It just have register defines.

Should the C program changed to include "csr.h" directly? I don't think so.

Chris
--
To unsubscribe from this list: send the line "unsubscribe linux-sparse" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Detect unused header files?

[Sam Ravnborg]

On Tue, Jul 28, 2009 at 01:49:57PM -0700, Christopher Li wrote:
> On Tue, Jul 28, 2009 at 1:36 PM, Robert P. J. Day<rpjday@crashcourse.ca> wrote:
> > sure you can.  no one should be counting on the inclusion of header
> > files by other header files (except possibly in unusual situations,
> > which i can't even think of at the moment).  if your program needs the
> > contents of a header file, it's your responsibility to include it.
> 
> The case I have in mind is that, some API level header file include some
> internal header file for constant defines. For example:
> 
> drivers/ieee1394/host.h include drivers/ieee1394/csr.h.
> 
> The "csr.h" is just the register define. It is not used in this header file
> itself. I did not check very closely, for augments sake pretend those
> inline function are not there. It just have register defines.
> 
> Should the C program changed to include "csr.h" directly? I don't think so.

Agreed on this one.

So we should add some intelligence when we 'fix' these warnings.
This is sometimes causing us troubles.

I need to see the results before I can judge if this will
be a big issue.

The first challenge will anyway to make all the header files self contained.

	Sam
--
To unsubscribe from this list: send the line "unsubscribe linux-sparse" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Detect unused header files?

[Christopher Li]

On Tue, Jul 28, 2009 at 2:04 PM, Sam Ravnborg<sam@ravnborg.org> wrote:
>
> Agreed on this one.
>
> So we should add some intelligence when we 'fix' these warnings.
> This is sometimes causing us troubles.
>
> I need to see the results before I can judge if this will
> be a big issue.

I don't know. It is not trivial change and lookup_symbol is at the hot path.
Most of the header file does not include unused headers. So I guess amount
the warnings, over 50% will be false positives. We will see.

> The first challenge will anyway to make all the header files self contained.

It is much easier than find out what is not needed.

Can you make a list of all the header files. Then using gcc -c to compile it?
Not self contained header file will failed on compile. Assume you give the right
kernel flags and header file include path.

Chris

Re: Detect unused header files?

[Christopher Li]

Hi Sam,

I create a branch "unused-include-files" for some experiment patch.
Some thing quick and dirty to find out what works and what doesn't.
No where near submitting quality.

http://git.kernel.org/?p=devel/sparse/chrisl/sparse.git;a=shortlog;h=unused-include-files

Currently it just report all unused stream.

for your example foo.h:

#include <linux/types.h>
#include <linux/string.h>

struct foo {
       __u32 bar;
       __u32 baz;
};


$ ./sparse foo.h
unused stream preprocessor(1)
unused stream foo.h(2)
unused stream preprocessor(3)
unused stream /usr/include/linux/stddef.h(6)
unused stream /usr/include/asm/posix_types.h(7)
unused stream /usr/include/asm/types.h(9)
unused stream /usr/include/linux/string.h(11)
unused stream /usr/include/string.h(12)
unused stream /usr/include/features.h(13)
unused stream /usr/include/sys/cdefs.h(14)
unused stream /usr/include/bits/wordsize.h(15)
unused stream /usr/include/gnu/stubs.h(16)
unused stream /usr/include/bits/wordsize.h(17)
unused stream /usr/include/gnu/stubs-32.h(18)
unused stream /usr/lib/gcc/x86_64-redhat-linux/4.3.0//include/stddef.h(19)

Just as I expected. Lost of noise :-)

BTW, use "-vstream" to show the actual symbol usage detail.

Comments?

Chris

Re: Detect unused header files?

[Derek M Jones]

Christopher,

> I create a branch "unused-include-files" for some experiment patch.
> Some thing quick and dirty to find out what works and what doesn't.
> No where near submitting quality.

Sorry I meant to post earlier.  Projects typically contain around 30%
of unused headers.  References to various research articles available
in: c0x.coding-guidelines.com/6.10.2.pdf

-- 
Derek M. Jones                         tel: +44 (0) 1252 520 667
Knowledge Software Ltd                 mailto:derek@knosof.co.uk
Source code analysis                   http://www.knosof.co.uk

Re: Detect unused header files?

[Sam Ravnborg]

On Thu, Jul 30, 2009 at 03:55:46AM -0700, Christopher Li wrote:
> Hi Sam,
> 
> I create a branch "unused-include-files" for some experiment patch.
> Some thing quick and dirty to find out what works and what doesn't.
> No where near submitting quality.
> 
> http://git.kernel.org/?p=devel/sparse/chrisl/sparse.git;a=shortlog;h=unused-include-files
> 
> Currently it just report all unused stream.
> 
> for your example foo.h:
> 
> #include <linux/types.h>
> #include <linux/string.h>
> 
> struct foo {
>        __u32 bar;
>        __u32 baz;
> };
> 
> 
> $ ./sparse foo.h
> unused stream preprocessor(1)
> unused stream foo.h(2)
> unused stream preprocessor(3)
> unused stream /usr/include/linux/stddef.h(6)
> unused stream /usr/include/asm/posix_types.h(7)
> unused stream /usr/include/asm/types.h(9)
> unused stream /usr/include/linux/string.h(11)
> unused stream /usr/include/string.h(12)
> unused stream /usr/include/features.h(13)
> unused stream /usr/include/sys/cdefs.h(14)
> unused stream /usr/include/bits/wordsize.h(15)
> unused stream /usr/include/gnu/stubs.h(16)
> unused stream /usr/include/bits/wordsize.h(17)
> unused stream /usr/include/gnu/stubs-32.h(18)
> unused stream /usr/lib/gcc/x86_64-redhat-linux/4.3.0//include/stddef.h(19)
> 
> Just as I expected. Lost of noise :-)
> 
> BTW, use "-vstream" to show the actual symbol usage detail.
> 
> Comments?
Thanks for hacking this up!

I will try it out during the weekend as work permits. Busy...

	Sam

Re: Detect unused header files?

[Christopher Li]

On Thu, Jul 30, 2009 at 1:36 PM, Sam Ravnborg<sam@ravnborg.org> wrote:
> Thanks for hacking this up!
>
> I will try it out during the weekend as work permits. Busy...

Any updates?

It seems that top level header file including sub component
header file is very common practice, especially on big header
file. From the header file itself is not sufficient to determine
the included header file is a sub component or some thing
new.

Chris

Re: Detect unused header files?

[Sam Ravnborg]

On Tue, Aug 04, 2009 at 02:49:27PM -0700, Christopher Li wrote:
> On Thu, Jul 30, 2009 at 1:36 PM, Sam Ravnborg<sam@ravnborg.org> wrote:
> > Thanks for hacking this up!
> >
> > I will try it out during the weekend as work permits. Busy...
> 
> Any updates?
Sorry - but caught up with day-time job.
 
> It seems that top level header file including sub component
> header file is very common practice, especially on big header
> file. From the header file itself is not sufficient to determine
> the included header file is a sub component or some thing
> new.
In the exported linux-kernel headers this is not common practice - so we will not
be caught by this.
But for general application programming it is useless.

	Sam

Re: Detect unused header files?

[Marko Kreen]

On 8/5/09, Christopher Li <sparse@chrisli.org> wrote:
> On Thu, Jul 30, 2009 at 1:36 PM, Sam Ravnborg<sam@ravnborg.org> wrote:
>  > Thanks for hacking this up!
>  >
>  > I will try it out during the weekend as work permits. Busy...
>
>
> Any updates?
>
>  It seems that top level header file including sub component
>  header file is very common practice, especially on big header
>  file. From the header file itself is not sufficient to determine
>  the included header file is a sub component or some thing
>  new.

Could we have a flag to track top-level headers only?  So that
anything declared in sub-headers will be registered also under
top header file?  This seems more useful scanning mode for regular
user-space code.

-- 
marko

Re: Detect unused header files?

[Robert P. J. Day]

On Thu, 6 Aug 2009, Marko Kreen wrote:

> On 8/5/09, Christopher Li <sparse@chrisli.org> wrote:
> > On Thu, Jul 30, 2009 at 1:36 PM, Sam Ravnborg<sam@ravnborg.org> wrote:
> >  > Thanks for hacking this up!
> >  >
> >  > I will try it out during the weekend as work permits. Busy...
> >
> > Any updates?
> >
> >  It seems that top level header file including sub component
> >  header file is very common practice, especially on big header
> >  file. From the header file itself is not sufficient to determine
> >  the included header file is a sub component or some thing
> >  new.
>
> Could we have a flag to track top-level headers only?  So that
> anything declared in sub-headers will be registered also under top
> header file?  This seems more useful scanning mode for regular
> user-space code.

  has anyone tried cscout?

http://www.spinellis.gr/cscout/

rday
--
========================================================================
Robert P. J. Day                               Waterloo, Ontario, CANADA

        Linux Consulting, Training and Annoying Kernel Pedantry.

Web page:                                          http://crashcourse.ca
Twitter:                                       http://twitter.com/rpjday
"Kernel Newbie Corner" column @ linux.com:          http://cli.gs/WG6WYX
========================================================================

Re: Detect unused header files?

[Kamil Dudka]

On Thu August 6 2009 13:17:02 Robert P. J. Day wrote:
> On Thu, 6 Aug 2009, Marko Kreen wrote:
> > On 8/5/09, Christopher Li <sparse@chrisli.org> wrote:
> > > On Thu, Jul 30, 2009 at 1:36 PM, Sam Ravnborg<sam@ravnborg.org> wrote:
> > >  > Thanks for hacking this up!
> > >  >
> > >  > I will try it out during the weekend as work permits. Busy...
> > >
> > > Any updates?
> > >
> > >  It seems that top level header file including sub component
> > >  header file is very common practice, especially on big header
> > >  file. From the header file itself is not sufficient to determine
> > >  the included header file is a sub component or some thing
> > >  new.
> >
> > Could we have a flag to track top-level headers only?  So that
> > anything declared in sub-headers will be registered also under top
> > header file?  This seems more useful scanning mode for regular
> > user-space code.
>
>   has anyone tried cscout?
>
> http://www.spinellis.gr/cscout/

The homepage looks interesting, but where can I download its sources?

Kamil

Re: Detect unused header files?

[Robert P. J. Day]

On Thu, 6 Aug 2009, Kamil Dudka wrote:

> On Thu August 6 2009 13:17:02 Robert P. J. Day wrote:
> > On Thu, 6 Aug 2009, Marko Kreen wrote:
> > > On 8/5/09, Christopher Li <sparse@chrisli.org> wrote:
> > > > On Thu, Jul 30, 2009 at 1:36 PM, Sam Ravnborg<sam@ravnborg.org> wrote:
> > > >  > Thanks for hacking this up!
> > > >  >
> > > >  > I will try it out during the weekend as work permits. Busy...
> > > >
> > > > Any updates?
> > > >
> > > >  It seems that top level header file including sub component
> > > >  header file is very common practice, especially on big header
> > > >  file. From the header file itself is not sufficient to determine
> > > >  the included header file is a sub component or some thing
> > > >  new.
> > >
> > > Could we have a flag to track top-level headers only?  So that
> > > anything declared in sub-headers will be registered also under top
> > > header file?  This seems more useful scanning mode for regular
> > > user-space code.
> >
> >   has anyone tried cscout?
> >
> > http://www.spinellis.gr/cscout/
>
> The homepage looks interesting, but where can I download its sources?

http://www.spinellis.gr/cscout/download.html

i grabbed a copy of it a few days ago, i just haven't had time to set
it up.  i mention it since one of the things it claims to do is -- ta
da! -- identify unused header files.  but it appears to take some
careful configuration.  i'd like to see someone config it to scan the
current source tree, i just don't think i have the time at the moment.

rday
--

========================================================================
Robert P. J. Day                               Waterloo, Ontario, CANADA

        Linux Consulting, Training and Annoying Kernel Pedantry.

Web page:                                          http://crashcourse.ca
Twitter:                                       http://twitter.com/rpjday
"Kernel Newbie Corner" column @ linux.com:          http://cli.gs/WG6WYX
========================================================================

Re: Detect unused header files?

[Kamil Dudka]

On Thu August 6 2009 14:16:45 Kamil Dudka wrote:
> >   has anyone tried cscout?
> >
> > http://www.spinellis.gr/cscout/
>
> The homepage looks interesting, but where can I download its sources?

FYI we figured out with rday off-list that there are no sources available.

Please leave a note here if you find them.

Kamil

Re: Detect unused header files?

[Michael Stefaniuc]

Kamil Dudka wrote:
> On Thu August 6 2009 14:16:45 Kamil Dudka wrote:
>>>   has anyone tried cscout?
>>>
>>> http://www.spinellis.gr/cscout/
>> The homepage looks interesting, but where can I download its sources?
> 
> FYI we figured out with rday off-list that there are no sources available.
> 
> Please leave a note here if you find them.
It is a proprietary application that is free as in beer to use for
checking open source code. But careful as the application phones home
(see the FAQ). I wanted to use it too on Wine but that stopped me.

bye
	michael

Re: Detect unused header files?

[Robert P. J. Day]

On Thu, 6 Aug 2009, Michael Stefaniuc wrote:

> Kamil Dudka wrote:
> > On Thu August 6 2009 14:16:45 Kamil Dudka wrote:
> >>>   has anyone tried cscout?
> >>>
> >>> http://www.spinellis.gr/cscout/
> >> The homepage looks interesting, but where can I download its sources?
> >
> > FYI we figured out with rday off-list that there are no sources available.
> >
> > Please leave a note here if you find them.

> It is a proprietary application that is free as in beer to use for
> checking open source code. But careful as the application phones
> home (see the FAQ). I wanted to use it too on Wine but that stopped
> me.

  while i understand the dislike OSS folks have for that sort of
thing, is there anything fundamentally harmful with what cscout would
be "sending home" WRT the linux kernel source tree?  i'm not arguing
one way or the other -- i'm just wondering if the benefits of running
cscout on the kernel would outweigh the annoyance of how it behaves.

rday
--

========================================================================
Robert P. J. Day                               Waterloo, Ontario, CANADA

        Linux Consulting, Training and Annoying Kernel Pedantry.

Web page:                                          http://crashcourse.ca
Twitter:                                       http://twitter.com/rpjday
"Kernel Newbie Corner" column @ linux.com:          http://cli.gs/WG6WYX
========================================================================

Re: Detect unused header files?

[Christopher Li]

On Thu, Aug 6, 2009 at 4:14 AM, Marko Kreen<markokr@gmail.com> wrote:
> Could we have a flag to track top-level headers only?  So that
> anything declared in sub-headers will be registered also under
> top header file?  This seems more useful scanning mode for regular
> user-space code.

Yes, we can. It might not help Sam much though because he is
interested to find out unused header file in system header files.

One way to do it is just let stream keep track of its parent and
is_toplevel.

Just add another pass inside show_unused_streams(). Mark all used
stream's parents as used.

Should be straight forward enough to implement base on branch
"unused-include-files". Any brave sparse hackers want to
give it a try?


Chris
--
To unsubscribe from this list: send the line "unsubscribe linux-sparse" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Detect unused header files?

[Sam Ravnborg]

On Tue, Jul 28, 2009 at 01:36:26PM -0700, Christopher Li wrote:
> On Tue, Jul 28, 2009 at 11:18 AM, Sam Ravnborg<sam@ravnborg.org> wrote:
> > In the kernel we would like to avoid all
> > unused include files.
> > Especially in the headers we export to userspace.
> >
> > Are there any easy way we can use sparse to detect
> > that a specific header file is not used?
> 
> I don't know a way to do that with current sparse.
> We can implement some code for this.

Would be great...
If you can give some general into I can maybe give
it a shot albeit I have only very limited sprase
hacking knowledge.

> 
> We need to have a good definition of what is used.

The short version...
The header file that include the file is dependent
on the included file.

So if we:
- use (check/test/reference) a macro from the file => used
- use a typedef/struct
- ...
then we need the included header file.

> 
> > The header files I have in mind will be fully self-contained
> > as they all include the header files they need to be used.
> 
> But a lot of case we have the header file is used in *another*
> file.
We do not care about this case...

The objective is to get the exported kernel headers as lean as possible.
This may occasionally break userland - but until now this has
been manageable.


If we apply this to kernel internal headers then we need to
do a bit of test builds to check things.
This is the same thing we do when we untange the include mess we have
today so we know what to do.
-next is btw a great help here.

If we do this inside spase or using a dedicated backend is not so important.
But adding this as a default=disabled option to sparse would be preferred.

	Sam

Re: Detect unused header files?

[Christopher Li]

On Tue, Jul 28, 2009 at 1:58 PM, Sam Ravnborg<sam@ravnborg.org> wrote:
> Would be great...
> If you can give some general into I can maybe give
> it a shot albeit I have only very limited sprase
> hacking knowledge.

I think the starting point is lookup_symbol().
Currently lookup_symbol does not care about where is
this symbol used. There is "sym->used" already. But it does
not help much in this case because sym->used did not distingish
the usage from the same file vs from other file.  We need to add
information some how. May be one more argument for the source
of the lookup.

>
> So if we:
> - use (check/test/reference) a macro from the file => used
> - use a typedef/struct

lookup_symbol() covers all of those. But there is annoying
part as well. For example. If the same header file get include
twice, the later one will consider using the first one because
it lookup the __HEAD_FILE_NAME__ macro to avoid duplicating
include.

Chris