[ANNOUNCE] various linux kernel devtools : device handling/memory mapping/profiling/etc.

[ANNOUNCE] various linux kernel devtools : device handling/memory mapping/profiling/etc.

[wdebruij]

Hi all,

  During development of a linux kernel network monitoring package (FFPF) I 
created a few kernelspace and cross-userspace/kernelspace tools that I hope 
others can benefit from too.

I haven't packages everything I wrote, just those elements that reduce
kernelsize and development time when multiple people use it. In its current 
state everything is functionally OK, but the code cannot as yet be merged 
into the kernel directly as I developed it outside of the main tree. Instead, 
I hope you will take a look at it and tell me what could be direct kernel 
material, and what should be changed or removed. After that I will supply 
individual patches against the latest version.

very briefly, the toolbox contains the following:

KERNEL specific:

- a generic device file interface, which abstracts away kernelversion 
differences, devfs/mknod/udev differences and memory addressing differences. 
The device API implements most of the standard device file handling code, 
including various memory mapping syscall handlers. With this interface driver 
developers will only have to override small pieces of code, instead of 
building everything from the ground up. I'm using it in 3 different modules, 
already. This is the 'jewel' of the package, and grepping the kernel sources 
I think many drivers could benefit from it (reduced size/complexity). 
Especially for beginner kernel hackers this could reduce the learning curve 
(and for those of you who've been going at it for a long time and forgot: 
it's quite steep :).

- a generic pci initialization interface. Could perhaps be merged with the 
existing PCI subsytem. I needed it for implementing a PCI driver (more below)
- a module API that abstracts away kernel version differences. 

CROSS kernel/userspace (ie, works in both) :
- verbosity level-based debugging, with optional timing/clockcycle and 
location information
- clockcycle profiling: the profiler outputs aggregated stats, such as the 
median to /proc and optionally through the debugging interface above.
- a multi-policy circular buffer. A buffer where the handling code responds to
policy decisions, such as whether writers should be aware of readers and if 
so, how they should respond. I use it for network packet queueing to 
userspace.
- some simpler stuff: hashes, stacks (should probably go).

also included in the distribution are demonstrator modules and userspace 
programs. The pci module, for example, implements a mmapped interface to any 
PCI device's resources. Just supply a vendor id/device id pair on insmod. 
Could in itself perhaps be useful for userspace drivers. I've been using it 
to initialize a network processor board over the PCI bus.

  since most of my software has to be able to run both in kernel- and in 
userspace I cannot simply supply a patch at the moment, I'm sorry. Instead, 
I've packaged the sourcecode (all GPL'ed, naturally) and uploaded it to our 
project's site at sourceforge:

URL: http://osdn.dl.sourceforge.net/sourceforge/ffpf/lkct-1.0.tgz

it's about 35kB. 

Please have a look and send me your remarks. I would find it a shame if I end 
up being the only one using this stuff. I would rather put some extra effort 
into changing it to get it accepted.

cheers,

  Willem

-- 
Willem de Bruijn
+31 6 2695 2446
+31 71 517 7174
wdebruij_at_dds.nl
http://www.wdebruij.dds.nl/

current project : 
Fairly Fast Packet Filter (FFPF)
http://ffpf.sourceforge.net/

Re: [ANNOUNCE] various linux kernel devtools : device handling/memory mapping/profiling/etc.

[Greg KH]

On Mon, Apr 05, 2004 at 07:33:22PM +0200, wdebruij@dds.nl wrote:
> - a generic device file interface, which abstracts away kernelversion 
> differences, devfs/mknod/udev differences and memory addressing differences. 

I don't see anything in there that will work properly for udev.  Am I
just missing the code somewhere?  Remember, for udev to work, you have
to create stuff in sysfs, which I don't see this code doing.

> - a generic pci initialization interface. Could perhaps be merged with the 
> existing PCI subsytem. I needed it for implementing a PCI driver (more below)

Ick, you are using pci_find_device() which is racy, depreciated, and
does not play nice with the rest of the kernel.  Yes, it's the lowest
common denominater accross 2.2, 2.4, and 2.6, but please don't sink to
that level if you don't have to.  For 2.6 it's just not acceptable.

I agree that at times the current kernel driver api learning curve is a
bit steep.  But people are working to reduce that curve where they can,
and it's one of my main priorities for 2.7.  Any help and suggestions
that you might have in that area are greatly appreciated.

thanks,

greg k-h

Re: [ANNOUNCE] various linux kernel devtools : device handling/memory mapping/profiling/etc

[wdebruij]

On Monday 05 April 2004 18:23, Greg KH wrote:
> I don't see anything in there that will work properly for udev.  Am I
> just missing the code somewhere?  Remember, for udev to work, you have
> to create stuff in sysfs, which I don't see this code doing.
indeed, automatic creation of the device files is not yet incorporated under 
udev, but at least it then reverts back to the oldstyle (mknod) device file 
system, right? That's a work in progress as my systems don't actually use 
udev just yet.

> Ick, you are using pci_find_device() which is racy, depreciated, and
> does not play nice with the rest of the kernel.  Yes, it's the lowest
> common denominater accross 2.2, 2.4, and 2.6, but please don't sink to
> that level if you don't have to.  For 2.6 it's just not acceptable.

hmm, really? thanks for the tip. I basically looked at O'Reilly's book when I 
coded that. Do you have a quick alternative for me to use?

>
> I agree that at times the current kernel driver api learning curve is a
> bit steep.  But people are working to reduce that curve where they can,
> and it's one of my main priorities for 2.7.  Any help and suggestions
> that you might have in that area are greatly appreciated.
>
perhaps some of this code (when cleaned up) can serve as a guide. I was 
actually wondering when a 2.7 release was scheduled.

Thanks for taking the time to look at the code,

Willem


ps: my regular smtp server stopped, so I had to copy-paste this into webmail.
Therefore, the in-reply-to, etc. tags are ommitted, possibly causing a
threadbreak. Sorry.

Re: [ANNOUNCE] various linux kernel devtools : device handling/memory mapping/profiling/etc

[Greg KH]

On Mon, Apr 05, 2004 at 09:00:22PM +0200, wdebruij@dds.nl wrote:
> 
> 
> On Monday 05 April 2004 18:23, Greg KH wrote:
> > I don't see anything in there that will work properly for udev.  Am I
> > just missing the code somewhere?  Remember, for udev to work, you have
> > to create stuff in sysfs, which I don't see this code doing.
> indeed, automatic creation of the device files is not yet incorporated under 
> udev,

Huh?

> but at least it then reverts back to the oldstyle (mknod) device file 
> system, right? That's a work in progress as my systems don't actually use 
> udev just yet.

I don't think you understand how udev works.  Who are you thinking does
the mknod of the device files?  udev does it only if it sees a file
called "dev" in sysfs for a device.  Your wrapper does not do this at
all (it could be changed to use class_simple to get it, if you so
desired.)

> > Ick, you are using pci_find_device() which is racy, depreciated, and
> > does not play nice with the rest of the kernel.  Yes, it's the lowest
> > common denominater accross 2.2, 2.4, and 2.6, but please don't sink to
> > that level if you don't have to.  For 2.6 it's just not acceptable.
> 
> hmm, really? thanks for the tip. I basically looked at O'Reilly's book
> when I coded that. Do you have a quick alternative for me to use?

pci_register_driver() as Documentation/pci.txt says to use.  Works just
wonderful from 2.4 to 2.6, and I think it's even been backported to work
on 2.2 if you so desire.

> > I agree that at times the current kernel driver api learning curve is a
> > bit steep.  But people are working to reduce that curve where they can,
> > and it's one of my main priorities for 2.7.  Any help and suggestions
> > that you might have in that area are greatly appreciated.
> >
> perhaps some of this code (when cleaned up) can serve as a guide.

Well, feel free to submit portions of it as patches that clean up the
current API, if you think it will help out any.  In my short glance, I
didn't really see anything that would help out all that much, but I'm
probably missing something.

thanks,

greg k-h

Re: [ANNOUNCE] various linux kernel devtools : device handling/memory mapping/profiling/etc

[Willem de Bruijn]

> pci_register_driver() as Documentation/pci.txt says to use.  Works just
> wonderful from 2.4 to 2.6, and I think it's even been backported to work
> on 2.2 if you so desire.

thanks. I need something that works with a specially patched 
2.3.99-pre3-rmk3-ds1 kernel for an IXP1200 network processor. Porting the 
original patch to 2.6 is simply too much work, so everything *must* be 
backward compatible to this prerelease. Hence the module template files.

>
> Well, feel free to submit portions of it as patches that clean up the
> current API, if you think it will help out any.  In my short glance, I
> didn't really see anything that would help out all that much, but I'm
> probably missing something.
>

AFAIK there the kernel reverts to a standard llseek implementation when this 
operation is left empty. How about having standard fallback behaviour for all 
system call handlers? That's what most of my device code is about: 
implementing regular file access and memory mapping, so that device driver 
writers won't have to do that each time for themselves. 9 times of out 10 
people will want the same .nopage handler, right? Why implement it for each 
driver individually?

having a generic interface does entail that drivers must export the memory 
region information in a standard fashion, which is why I created the 
dev_file_t, dev_file_open_t and dev_file_open_region_t structures (for lack 
of a better name). Once these were finished, adding automatic default 
handling was relatively straightforward. However, as you pointed out, I 
shouldn't have made claims about udev, as I never actually tested that. 

considering udev and devfs are both still being developed (although the latter 
is marked `deprecated'), a single interface to the device file creation, 
would IMO, also be useful, as not all drivers have to be updated or filled 
with KERNEL_VERSION statements. The actual code is quite small (it all 
resides in dev.c), but even without supporting udev I have 3 different cases 
to deal with (2devfs+mknod).

I'm not saying my code should be used for this per se, but I do believe that 
these two features are so generic across drivers that they deserve to be 
included in the main tree somewhere. As for the rest of the package, that 
might not be too useful. I discussed the addition of prepending __FILE__, 
__FUNCTION__ and clockcycles to printk previously and someone else actually 
sent in a patch to printk allowing vsprintf functionality. The buffer 
structure is probably too heavyweight for kernel inclusion, anyway.

Willem

-- 
Dit bericht is gescand op virussen en andere gevaarlijke inhoud door ULCN MailScanner en het bericht lijkt schoon te zijn.
This message has been scanned for viruses and dangerous content by ULCN MailScanner, and is believed to be clean.