Kernel API Reference Documentation

Kernel API Reference Documentation

[Lukas Jelinek]

Hello,

a few months ago I looked for something like "Linux Kernel API Reference
Documentation". This search was unsuccessful and somebody recommended me
to generate this documentation from the kernel headers.

I have used Doxygen for this work. But the headers have needed to be
preprocessed by 'sed' using some regexp rules (due to various
incompatible comment formats).

Now I decide to share the result worldwide. The current generated
"Kernel API Reference" can be found at http://www.kernel-api.org.
Although it is very buggy this time I think it may be useful for module
developers.

To allow this work to be better, I suggest to establish some rules for
writing code comments (especially for function prototypes, data
structures etc.) and to add the comments to the kernel headers. The
rules should be chosen carefully to be well accepted by various
documentation generators (at least by Doxygen).

Thank you for your time dedicated to this idea.

Regards

Lukas Jelinek

Re: Kernel API Reference Documentation

[Randy.Dunlap]

On Tue, 27 Jun 2006 21:22:51 +0200 Lukas Jelinek wrote:

> Hello,
> 
> a few months ago I looked for something like "Linux Kernel API Reference
> Documentation". This search was unsuccessful and somebody recommended me
> to generate this documentation from the kernel headers.
> 
> I have used Doxygen for this work. But the headers have needed to be
> preprocessed by 'sed' using some regexp rules (due to various
> incompatible comment formats).
> 
> Now I decide to share the result worldwide. The current generated
> "Kernel API Reference" can be found at http://www.kernel-api.org.
> Although it is very buggy this time I think it may be useful for module
> developers.
> 
> To allow this work to be better, I suggest to establish some rules for
> writing code comments (especially for function prototypes, data
> structures etc.) and to add the comments to the kernel headers. The
> rules should be chosen carefully to be well accepted by various
> documentation generators (at least by Doxygen).

FYI, there are already some kernel-doc rules in
Documentation/kernel-doc-nano-HOWTO.txt.  These rules work with the
doc. generator in the kernel tree (scripts/kernel-doc).
Do you have suggestions for how to make them (the rules) better?
so that the in-tree kernel doc. will improve...

Q2:  what do I get when I download one of the tarballs from kernel-api.org?

Q3:  Can we see your sed scripts?

Thanks,
---
~Randy

Re: Kernel API Reference Documentation

[Lukas Jelinek]

 >
> FYI, there are already some kernel-doc rules in
> Documentation/kernel-doc-nano-HOWTO.txt.  These rules work with the
> doc. generator in the kernel tree (scripts/kernel-doc).
> Do you have suggestions for how to make them (the rules) better?
> so that the in-tree kernel doc. will improve...

These rules seem to be good. I will try to use the generator
(scripts/kernel-doc) and check the result.

But the bigger problem is that many headers are not documented at all.
And some code is documented but not complying the rules.

> 
> Q2:  what do I get when I download one of the tarballs from kernel-api.org?
> 

Each tarball contains exactly the same as can be browsed online at
kernel-api.org. There is no difference.


> Q3:  Can we see your sed scripts?
> 

Yes, here it is (it's really small and mindless):

--- sed script begin ---

/^\(\s\)*#endif/ {
s/\/\*/\/\//
s/\*\///
}

/^\(\s\)*\/\*.*\*\/\(\s\)*$/ {
s/\/\*/\/\/\//
s/\*\///
}

/^.*\/\*.*\*\/\(\s\)*$/ {
s/\/\*/\/\/\/</
s/\*\///
}

s/^\(\s\)*\/\*/\/\*\*\n/

s/^.*\*\//\n\*\//

--- sed script end ---

Re: Kernel API Reference Documentation

[Randy.Dunlap]

On Tue, 27 Jun 2006 22:42:20 +0200 Lukas Jelinek wrote:

>  >
> > FYI, there are already some kernel-doc rules in
> > Documentation/kernel-doc-nano-HOWTO.txt.  These rules work with the
> > doc. generator in the kernel tree (scripts/kernel-doc).
> > Do you have suggestions for how to make them (the rules) better?
> > so that the in-tree kernel doc. will improve...
> 
> These rules seem to be good. I will try to use the generator
> (scripts/kernel-doc) and check the result.
> 
> But the bigger problem is that many headers are not documented at all.
> And some code is documented but not complying the rules.

Yes, well aware of both of those, plus a few minor problems
with scripts/kernel-doc itself.
I just added a minor rules-checker to the script and have
more plans for it.

We try to have new exported interfaces documented (I don't try
on static [private] interfaces).

I have had email with a professor in China who was going to have
some students do some work in this area, but I haven't heard back
from him (over 1 month ago).


> > 
> > Q2:  what do I get when I download one of the tarballs from kernel-api.org?
> > 
> 
> Each tarball contains exactly the same as can be browsed online at
> kernel-api.org. There is no difference.

OK.

> > Q3:  Can we see your sed scripts?
> > 
> 
> Yes, here it is (it's really small and mindless):

Thanks.

---
~Randy

Re: Kernel API Reference Documentation

[Steven Rostedt]

On Tue, 27 Jun 2006, Lukas Jelinek wrote:

> Yes, here it is (it's really small and mindless):
>
> --- sed script begin ---
>
> /^\(\s\)*#endif/ {
> s/\/\*/\/\//
> s/\*\///
> }
>
> /^\(\s\)*\/\*.*\*\/\(\s\)*$/ {
> s/\/\*/\/\/\//
> s/\*\///
> }
>
> /^.*\/\*.*\*\/\(\s\)*$/ {
> s/\/\*/\/\/\/</
> s/\*\///
> }
>
> s/^\(\s\)*\/\*/\/\*\*\n/
>
> s/^.*\*\//\n\*\//
>
> --- sed script end ---
>

Here's a version that gets rid of a lot of confusing backslashes:

/^\(\s\)*#endif/ {
s,/\*,//,
s,\*/,,
}

\,^\(\s\)*/\*.*\*/\(\s\)*$, {
s,/\*,///,
s,\*/,,
}

\,^.*/\*.*\*/\(\s\)*$, {
s,/\*,///<,
s,\*/,,
}

s,^\(\s\)*/\*,/\*\*\n,

s,^.*\*/,\n\*/,


-- Steve

Re: Kernel API Reference Documentation

[Andreas Schwab]

Steven Rostedt <rostedt@goodmis.org> writes:

> Here's a version that gets rid of a lot of confusing backslashes:
>
> /^\(\s\)*#endif/ {

You can get rid of even more of them.

/^\s*#endif/ {

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Kernel API Reference Documentation

[Petr Tesarik]

On Tue, 2006-06-27 at 21:22 +0200, Lukas Jelinek wrote:
> Hello,
> 
> a few months ago I looked for something like "Linux Kernel API Reference
> Documentation". This search was unsuccessful and somebody recommended me
> to generate this documentation from the kernel headers.
> 
> I have used Doxygen for this work. But the headers have needed to be
> preprocessed by 'sed' using some regexp rules (due to various
> incompatible comment formats).
> 
> Now I decide to share the result worldwide. The current generated
> "Kernel API Reference" can be found at http://www.kernel-api.org.
> Although it is very buggy this time I think it may be useful for module
> developers.

I looked at
http://www.kernel-api.org/docs/online/2.6.17/da/dab/structsk__buff.html

and you apparently ignore kernel-doc for structs. Cf.
include/linux/skbuff.h:177 ff.

Regards,
Petr Tesarik

Re: Kernel API Reference Documentation

[Martin Waitz]

[-- Attachment #1: Type: text/plain, Size: 529 bytes --]

hoi :)

when I once experimented with doxygen, I used the following script to
convert some kerneldoc comments to doxygen syntax:

#!/usr/bin/perl -wpi

use strict;

BEGIN { $::state = 0; }

if ($::state == 0) {
	$::state = 1 if /\/\*\*/;
} elsif ($::state == 1) {
	s/(\*\s+)(struct\s+|enum\s+)?\S+ - /$1/;
	s/$/\./ unless /\.$/;
	$::state = 2;
} elsif ($::state == 2) {
	s/(\*\s+)\@(\w+):\s+(.*)/$1\\param $2 $3./;
	s/(\s+)[%&\@](\S+)/$1$2/g;
}
s/\.\.$/./;

$::state = 0 if /\*\//;


-- 
Martin Waitz

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Kernel API Reference Documentation

[Lukas Jelinek]

> 
> I looked at
> http://www.kernel-api.org/docs/online/2.6.17/da/dab/structsk__buff.html
> 
> and you apparently ignore kernel-doc for structs. Cf.
> include/linux/skbuff.h:177 ff.
> 
> Regards,
> Petr Tesarik
> 

There are several problems. The one you describe is probably caused by a
blank line between the struct and the related comment. Doxygen doesn't
recognize it correctly (and simply ignores the comment).

Lukas Jelinek

Re: Kernel API Reference Documentation

[Lukas Jelinek]

> hoi :)
> 
> when I once experimented with doxygen, I used the following script to
> convert some kerneldoc comments to doxygen syntax:
> 
> #!/usr/bin/perl -wpi
> 
> use strict;
> 
> BEGIN { $::state = 0; }
> 
> if ($::state == 0) {
> 	$::state = 1 if /\/\*\*/;
> } elsif ($::state == 1) {
> 	s/(\*\s+)(struct\s+|enum\s+)?\S+ - /$1/;
> 	s/$/\./ unless /\.$/;
> 	$::state = 2;
> } elsif ($::state == 2) {
> 	s/(\*\s+)\@(\w+):\s+(.*)/$1\\param $2 $3./;
> 	s/(\s+)[%&\@](\S+)/$1$2/g;
> }
> s/\.\.$/./;
> 
> $::state = 0 if /\*\//;
> 

A good idea! Thanks. I will try it and compare to the result of my sed
script.

Lukas

Re: Kernel API Reference Documentation

[Randy.Dunlap]

On Wed, 28 Jun 2006 14:22:53 +0200 Lukas Jelinek wrote:

> > 
> > I looked at
> > http://www.kernel-api.org/docs/online/2.6.17/da/dab/structsk__buff.html
> > 
> > and you apparently ignore kernel-doc for structs. Cf.
> > include/linux/skbuff.h:177 ff.
> > 
> > Regards,
> > Petr Tesarik
> > 
> 
> There are several problems. The one you describe is probably caused by a
> blank line between the struct and the related comment. Doxygen doesn't
> recognize it correctly (and simply ignores the comment).

No blank line in this case.

And since you read the kernel-doc HOWTO, you now know that
struct, enum, and typedef may have kernel-doc notations, right?

Thanks,
---
~Randy

Re: Kernel API Reference Documentation

[Petr Tesarik]

On Wed, 2006-06-28 at 09:09 -0700, Randy.Dunlap wrote:
> On Wed, 28 Jun 2006 14:22:53 +0200 Lukas Jelinek wrote:
> 
> > > I looked at
> > > http://www.kernel-api.org/docs/online/2.6.17/da/dab/structsk__buff.html
> > > 
> > > and you apparently ignore kernel-doc for structs. Cf.
> > > include/linux/skbuff.h:177 ff.
> > 
> > There are several problems. The one you describe is probably caused by a
> > blank line between the struct and the related comment. Doxygen doesn't
> > recognize it correctly (and simply ignores the comment).
> 
> No blank line in this case.

Oh, yes, there is a blank line between the comment and the struct. It's
a pitty that someone put much effort into writing a usable description,
which is then not seen. Anyway, should we find all such occurences in
the kernel tree and fix them, or make a workaround for doxygen?

Regards,
Petr

Re: Kernel API Reference Documentation

[Randy.Dunlap]

On Wed, 28 Jun 2006 18:13:35 +0200 Petr Tesarik wrote:

> On Wed, 2006-06-28 at 09:09 -0700, Randy.Dunlap wrote:
> > On Wed, 28 Jun 2006 14:22:53 +0200 Lukas Jelinek wrote:
> > 
> > > > I looked at
> > > > http://www.kernel-api.org/docs/online/2.6.17/da/dab/structsk__buff.html
> > > > 
> > > > and you apparently ignore kernel-doc for structs. Cf.
> > > > include/linux/skbuff.h:177 ff.
> > > 
> > > There are several problems. The one you describe is probably caused by a
> > > blank line between the struct and the related comment. Doxygen doesn't
> > > recognize it correctly (and simply ignores the comment).
> > 
> > No blank line in this case.
> 
> Oh, yes, there is a blank line between the comment and the struct. It's
> a pitty that someone put much effort into writing a usable description,
> which is then not seen. Anyway, should we find all such occurences in
> the kernel tree and fix them, or make a workaround for doxygen?

Which struct are we talking about here?  I missed it.

I guess the easy answer is Both.
However, I'm working on fixing up the kernel tree, so sending
patches is correct IMO.

---
~Randy

Re: Kernel API Reference Documentation

[Lukas Jelinek]

>>>>>I looked at
>>>>>http://www.kernel-api.org/docs/online/2.6.17/da/dab/structsk__buff.html
>>>>>
>>>>>and you apparently ignore kernel-doc for structs. Cf.
>>>>>include/linux/skbuff.h:177 ff.
>>>>
>>>>There are several problems. The one you describe is probably caused by a
>>>>blank line between the struct and the related comment. Doxygen doesn't
>>>>recognize it correctly (and simply ignores the comment).
>>>
>>>No blank line in this case.
>>
>>Oh, yes, there is a blank line between the comment and the struct. It's
>>a pitty that someone put much effort into writing a usable description,
>>which is then not seen. Anyway, should we find all such occurences in
>>the kernel tree and fix them, or make a workaround for doxygen?
> 
> 
> Which struct are we talking about here?  I missed it.
> 
> I guess the easy answer is Both.
> However, I'm working on fixing up the kernel tree, so sending
> patches is correct IMO.
> 

We are currently talking about struct sk_buff. And there _is_ a single
blank line which avoids to make a relation between the struct and the
comment above.

Lukas

Re: Kernel API Reference Documentation

[Randy.Dunlap]

On Wed, 28 Jun 2006 18:41:07 +0200 Lukas Jelinek wrote:

> >>>>>I looked at
> >>>>>http://www.kernel-api.org/docs/online/2.6.17/da/dab/structsk__buff.html
> >>>>>
> >>>>>and you apparently ignore kernel-doc for structs. Cf.
> >>>>>include/linux/skbuff.h:177 ff.
> >>>>
> >>>>There are several problems. The one you describe is probably caused by a
> >>>>blank line between the struct and the related comment. Doxygen doesn't
> >>>>recognize it correctly (and simply ignores the comment).
> >>>
> >>>No blank line in this case.
> >>
> >>Oh, yes, there is a blank line between the comment and the struct. It's
> >>a pitty that someone put much effort into writing a usable description,
> >>which is then not seen. Anyway, should we find all such occurences in
> >>the kernel tree and fix them, or make a workaround for doxygen?
> > 
> > 
> > Which struct are we talking about here?  I missed it.
> > 
> > I guess the easy answer is Both.
> > However, I'm working on fixing up the kernel tree, so sending
> > patches is correct IMO.
> > 
> 
> We are currently talking about struct sk_buff. And there _is_ a single
> blank line which avoids to make a relation between the struct and the
> comment above.

OK, I believed Petr when he stated that, I just wanted to get my
head screwed on straight.  So its the blank line between

 */
and
struct sk_buff {
?

I don't think the kernel-doc rules cover that case, but
scripts/kernel-doc handles it, so the Doxygen parser should handle
it IMO. 

HTH.
---
~Randy

Re: Kernel API Reference Documentation

[Lukas Jelinek]

> 
> 
> OK, I believed Petr when he stated that, I just wanted to get my
> head screwed on straight.  So its the blank line between
> 
>  */
> and
> struct sk_buff {
> ?

Exactly.


> 
> I don't think the kernel-doc rules cover that case, but
> scripts/kernel-doc handles it, so the Doxygen parser should handle
> it IMO. 
> 

Doxygen should handle it but doesn't do so.

Lukas