Userspace I/O driver core

Userspace I/O driver core

[Greg KH]

A large number of people have expressed interest recently in the
userspace i/o driver core which allows userspace drivers to be written
to handle some types of hardware.

Right now the UIO core is working and in the -mm releases.  It's been
rewritten from the last time patches were posted to lkml and is much
simpler.  It also includes full documentation and two example drivers
and two example userspace programs that test those drivers.

But in order to get this core into the kernel tree, we need to have some
"real" drivers written that use it.  So, for anyone that wants to see
this go into the tree, now is the time to step forward and post your
patches for hardware that this kind of driver interface is needed.

If no such drivers appear, then there is a very slim chance that this
interface will be accepted into the tree.

The patches can be found in the -mm releases or at:
  http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio.patch
    - UIO core
  http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio-documentation.patch
    - UIO documentation
  http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio-dummy.patch
  http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio-irq.patch
    - two example kernel modules and userspace programs showing how to
      use the UIO interface.

If anyone has any questions on how to use this interface, or anything
else about it, please let me and Thomas know.

thanks,

greg k-h

Re: Userspace I/O driver core

[Ben Nizette]

Greg KH wrote:
> But in order to get this core into the kernel tree, we need to have some
> "real" drivers written that use it.  So, for anyone that wants to see
> this go into the tree, now is the time to step forward and post your
> patches for hardware that this kind of driver interface is needed.
>   
We have a product being developed currently for which this interface is 
perfect.

The situation is that we have Linux collecting data from very many 
sources.  The data is processed by throwing it at a memory address at 
which an FPGA lives.  The FPGA processes the data and generates an 
interrupt upon completion at which time the processed data can be read 
back out.  Linux doesn't know anything about the data except it's source 
and destination and, for security reasons, it has to stay that way.  As 
such a formal driver makes little sense: data gets written to a memory 
address and a little while later it is read out again, that's it.  The 
only fly in the ointment is the interrupt.  Before I knew about these 
UIO patches I had written what effectively was a smaller version of UIO 
to handle this interrupt.  With the UIO patches the whole process 
becomes trivial and I (along with my boss) become happy :-)

I shall submit a patch once I move my code over but it's almost not 
worth it, it will be truly trivial.

I can see a similar scenario being played out a lot in industrial 
control and other embedded systems.  For example, if you just want to 
monitor a set of data but interrupt if something critical happens (or 
even just when the data is updated).  All an in-kernel driver would do 
is handle an interrupt and perform copy_{to,from}_user()s but it would 
have to have a fair bit of fluff around it to signal to userland that 
the interrupt had occured.  UIO is a clean, standard and powerful form 
of that fluff.  Congrats to all who worked on her.

Regards,
   Ben.

Re: Userspace I/O driver core

[Avi Kivity]

Greg KH wrote:
> A large number of people have expressed interest recently in the
> userspace i/o driver core which allows userspace drivers to be written
> to handle some types of hardware.
>
> Right now the UIO core is working and in the -mm releases.  It's been
> rewritten from the last time patches were posted to lkml and is much
> simpler.  It also includes full documentation and two example drivers
> and two example userspace programs that test those drivers.
>
> But in order to get this core into the kernel tree, we need to have some
> "real" drivers written that use it.  So, for anyone that wants to see
> this go into the tree, now is the time to step forward and post your
> patches for hardware that this kind of driver interface is needed.
>
>   
[...]

> If anyone has any questions on how to use this interface, or anything
> else about it, please let me and Thomas know.
>
>   

I understand one still has to write a kernel driver to shut up the irq.  
How about writing a small bytecode interpreter to make event than 
unnecessary?

The userspace driver would register a couple of bytecode programs: 
is_interrupt_pending() and disable_interrupt(), which the uio framework 
would call when the interrupt fires.

The bytecode could reuse net/core/filter.c, with the packet replaced by 
the mmio or ioregion, or use something new.

-- 
error compiling committee.c: too many arguments to function

Re: Userspace I/O driver core

[Arjan van de Ven]

> I understand one still has to write a kernel driver to shut up the irq.  
> How about writing a small bytecode interpreter to make event than 
> unnecessary?

if you do that why not do a real driver.

Re: Userspace I/O driver core

[Avi Kivity]

Arjan van de Ven wrote:
>> I understand one still has to write a kernel driver to shut up the irq.  
>> How about writing a small bytecode interpreter to make event than 
>> unnecessary?
>>     
>
> if you do that why not do a real driver.
>
>   

An entire driver in bytecode? that means exposing the entire kernel API 
to the bytecode interpreter.  A monumental task.

Or did I misunderstand you?


-- 
error compiling committee.c: too many arguments to function

Re: Userspace I/O driver core

[Arjan van de Ven]

On Thu, 2006-12-14 at 12:46 +0200, Avi Kivity wrote:
> Arjan van de Ven wrote:
> >> I understand one still has to write a kernel driver to shut up the irq.  
> >> How about writing a small bytecode interpreter to make event than 
> >> unnecessary?
> >>     
> >
> > if you do that why not do a real driver.
> >
> >   
> 
> An entire driver in bytecode? 

no a real, non-bytecode driver.

Re: Userspace I/O driver core

[Avi Kivity]

Arjan van de Ven wrote:
> On Thu, 2006-12-14 at 12:46 +0200, Avi Kivity wrote:
>   
>> Arjan van de Ven wrote:
>>     
>>>> I understand one still has to write a kernel driver to shut up the irq.  
>>>> How about writing a small bytecode interpreter to make event than 
>>>> unnecessary?
>>>>     
>>>>         
>>> if you do that why not do a real driver.
>>>
>>>   
>>>       
>> An entire driver in bytecode? 
>>     
>
> no a real, non-bytecode driver.
>
>   

Isn't the whole point of uio is to avoid writing a kernel mode driver?

As proposed, it doesn't quite accomplish it.  With an additional 
bytecode interpreter, you can have a 100% userspace driver (the bytecode 
interpreter would be part of uio, not the driver).


-- 
error compiling committee.c: too many arguments to function

Re: Userspace I/O driver core

[Greg KH]

On Thu, Dec 14, 2006 at 12:56:29PM +0200, Avi Kivity wrote:
> Arjan van de Ven wrote:
> >On Thu, 2006-12-14 at 12:46 +0200, Avi Kivity wrote:
> >  
> >>Arjan van de Ven wrote:
> >>    
> >>>>I understand one still has to write a kernel driver to shut up the irq. 
> >>>>How about writing a small bytecode interpreter to make event than 
> >>>>unnecessary?
> >>>>    
> >>>>        
> >>>if you do that why not do a real driver.
> >>>
> >>>  
> >>>      
> >>An entire driver in bytecode? 
> >>    
> >
> >no a real, non-bytecode driver.
> >
> >  
> 
> Isn't the whole point of uio is to avoid writing a kernel mode driver?

No.  Did you read the documentation that is written about the uio core?

> As proposed, it doesn't quite accomplish it.  With an additional 
> bytecode interpreter, you can have a 100% userspace driver (the bytecode 
> interpreter would be part of uio, not the driver).

If you want to try to work on something as complex as a bytecode
interpreter that can handle all of the hookups to the pci and other
kernel subsystems that are necessary to get such a driver to work
properly, feel free to.

But until then, you'll have to stick with a tiny kernelspace driver that
handles the basic hardware discovery and initialization logic.

Which, for everyone that I have talked to that needs such a driver, is
not a problem at all.

thanks,

greg k-h

Re: Userspace I/O driver core

[Dr. David Alan Gilbert]

* Arjan van de Ven (arjan@infradead.org) wrote:
> 
> > I understand one still has to write a kernel driver to shut up the irq.  
> > How about writing a small bytecode interpreter to make event than 
> > unnecessary?
> 
> if you do that why not do a real driver.

Because perhaps it is potentially very simple - i.e.
if most of these drivers turn out to be:

if (*loc1 & mask)
{
   *loc2=value;
   flag we have an interrupt
}

then all you actually need to do is provide a way to
specify loc1, mask, loc2 and value.  You could provide
a small handful of mechanisms to suit most simple pieces of hardware
and also provide a definition for the hardware designers to say
'if you make your interrupt registers like this then the software
is dead easy'.  A bytecode interpreter seems a little overkill
unless you think that two or three levels of that type of test/mask
could cope with 90%+ of the cases.
There are probably lots of people reinventing the wheel for simple IO 
boards and the hardware guys will be making it up each time as well.

Dave
-- 
 -----Open up your eyes, open up your mind, open up your code -------   
/ Dr. David Alan Gilbert    | Running GNU/Linux on Alpha,68K| Happy  \ 
\ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/

Re: Userspace I/O driver core

[Hans-Jürgen Koch]

Am Donnerstag, 14. Dezember 2006 10:44 schrieb Avi Kivity:

> 
> I understand one still has to write a kernel driver to shut up the irq.  
> How about writing a small bytecode interpreter to make event than 
> unnecessary?
> 
> The userspace driver would register a couple of bytecode programs: 
> is_interrupt_pending() and disable_interrupt(), which the uio framework 
> would call when the interrupt fires.
> 
> The bytecode could reuse net/core/filter.c, with the packet replaced by 
> the mmio or ioregion, or use something new.
> 

I think this would be overkill. The kernel module you have to write
is _really_ very simple. And it has to be written only once, so even
a manufacturer who employs no experienced kernel developers can
easily outsource that task.

Hans

Re: Userspace I/O driver core

[Avi Kivity]

[why trim the cc?]

Hans-Jürgen Koch wrote:
> Am Donnerstag, 14. Dezember 2006 10:44 schrieb Avi Kivity:
>
>   
>> I understand one still has to write a kernel driver to shut up the irq.  
>> How about writing a small bytecode interpreter to make event than 
>> unnecessary?
>>
>> The userspace driver would register a couple of bytecode programs: 
>> is_interrupt_pending() and disable_interrupt(), which the uio framework 
>> would call when the interrupt fires.
>>
>> The bytecode could reuse net/core/filter.c, with the packet replaced by 
>> the mmio or ioregion, or use something new.
>>
>>     
>
> I think this would be overkill. The kernel module you have to write
> is _really_ very simple. And it has to be written only once, so even
> a manufacturer who employs no experienced kernel developers can
> easily outsource that task.
>
>   

It has to be written once, but compiled for every kernel version and 
$arch out there (for out of tree drivers), or it has to wait for the 
next kernel release and distro sync (for in-tree drivers).

If we make userspace drivers possible, it makes sense that the entire 
driver be in userspace, not just 98.7% of it.



-- 
error compiling committee.c: too many arguments to function

Re: Userspace I/O driver core

[Jan Engelhardt]

> It has to be written once, but compiled for every kernel
> version and $arch out there (for out of tree drivers), or it
> has to wait for the next kernel release and distro sync (for
> in-tree drivers).

Still better than written for every _and_ compiled for every.
But wait, make it simpler: just give the source to the user and
don't bother with precompiling. It's such a PITA anyhow.


	-`J'
-- 

Re: Userspace I/O driver core

[Avi Kivity]

Jan Engelhardt wrote:
>> It has to be written once, but compiled for every kernel
>> version and $arch out there (for out of tree drivers), or it
>> has to wait for the next kernel release and distro sync (for
>> in-tree drivers).
>>     
>
> Still better than written for every _and_ compiled for every.
> But wait, make it simpler: just give the source to the user and
> don't bother with precompiling. It's such a PITA anyhow.
>   

Users don't compile.  They use.  That's why they're called users.

And please don't suggest wrapper scripts to do the compilation.


-- 
error compiling committee.c: too many arguments to function

Re: Userspace I/O driver core

[Jan Engelhardt]

On Dec 14 2006 15:38, Avi Kivity wrote:
> Jan Engelhardt wrote:
>> > It has to be written once, but compiled for every kernel
>> > version and $arch out there (for out of tree drivers), or it
>> > has to wait for the next kernel release and distro sync (for
>> > in-tree drivers).
>> 
>> Still better than written for every _and_ compiled for every.
>> But wait, make it simpler: just give the source to the user and
>> don't bother with precompiling. It's such a PITA anyhow.
>
> Users don't compile.  They use.  That's why they're called users.

I could also take your wording: Users don't install. They use. ...

> And please don't suggest wrapper scripts to do the compilation.

Hell no. I can't already compile VMware without sanitizing the Makefile
beforehand.


	-`J'
-- 

Re: Userspace I/O driver core

[Greg KH]

On Thu, Dec 14, 2006 at 12:48:55PM +0200, Avi Kivity wrote:
> [why trim the cc?]
> 
> Hans-J?rgen Koch wrote:
> >Am Donnerstag, 14. Dezember 2006 10:44 schrieb Avi Kivity:
> >
> >  
> >>I understand one still has to write a kernel driver to shut up the irq.  
> >>How about writing a small bytecode interpreter to make event than 
> >>unnecessary?
> >>
> >>The userspace driver would register a couple of bytecode programs: 
> >>is_interrupt_pending() and disable_interrupt(), which the uio framework 
> >>would call when the interrupt fires.
> >>
> >>The bytecode could reuse net/core/filter.c, with the packet replaced by 
> >>the mmio or ioregion, or use something new.
> >>
> >>    
> >
> >I think this would be overkill. The kernel module you have to write
> >is _really_ very simple. And it has to be written only once, so even
> >a manufacturer who employs no experienced kernel developers can
> >easily outsource that task.
> >
> >  
> 
> It has to be written once, but compiled for every kernel version and 
> $arch out there (for out of tree drivers), or it has to wait for the 
> next kernel release and distro sync (for in-tree drivers).

No, just get the tiny driver into the main kernel tree, like all other
drivers are required to do.

> If we make userspace drivers possible, it makes sense that the entire 
> driver be in userspace, not just 98.7% of it.

If you see a way to do this that is race-free, I know a lot of people
would be glad to see such a patch.

But until then, no, we are not making any such claims of 100% userspace
driver for hardware such as pci devices and other things that this uio
core works with.

thanks,

greg k-h

Re: Userspace I/O driver core

[Alan]

> But in order to get this core into the kernel tree, we need to have some
> "real" drivers written that use it.  So, for anyone that wants to see
> this go into the tree, now is the time to step forward and post your
> patches for hardware that this kind of driver interface is needed.

Might be kind of hairy given uio_read() doesn't even return from the
kernel. This code simply isn't fit for purpose, philosophical debate
aside.

Alan

Re: Userspace I/O driver core

[Thomas Gleixner]

On Thu, 2006-12-14 at 10:52 +0000, Alan wrote:
> Might be kind of hairy given uio_read() doesn't even return from the
> kernel. 

We probably talk about different code here, right ? The one, I'm looking
at returns on each interrupt event.

	tglx

Re: Userspace I/O driver core

[Alan]

On Thu, 14 Dec 2006 12:22:16 +0100
Thomas Gleixner <tglx@linutronix.de> wrote:

> On Thu, 2006-12-14 at 10:52 +0000, Alan wrote:
> > Might be kind of hairy given uio_read() doesn't even return from the
> > kernel. 
> 
> We probably talk about different code here, right ? The one, I'm looking
> at returns on each interrupt event.

The patch Greg posted up has no return inside the while loop.

Alan

Re: Userspace I/O driver core

[Hans-Jürgen Koch]

Am Donnerstag, 14. Dezember 2006 12:39 schrieb Alan:
> On Thu, 14 Dec 2006 12:22:16 +0100
> Thomas Gleixner <tglx@linutronix.de> wrote:
> 
> > On Thu, 2006-12-14 at 10:52 +0000, Alan wrote:
> > > Might be kind of hairy given uio_read() doesn't even return from the
> > > kernel. 
> > 
> > We probably talk about different code here, right ? The one, I'm looking
> > at returns on each interrupt event.
> 
> The patch Greg posted up has no return inside the while loop.
> 

There are three breaks in that while loop, the first makes it return as 
soon as an interrupt occurs.

Hans

Re: Userspace I/O driver core

[Alan]

On Thu, 14 Dec 2006 12:37:16 +0100
Hans-Jürgen Koch <hjk@linutronix.de> wrote:

> There are three breaks in that while loop, the first makes it return as 
> soon as an interrupt occurs.

Doh ignore that I misread it. Perils of reading email before midday

Re: Userspace I/O driver core

[Jan Engelhardt]

>The patches can be found in the -mm releases or at:
>  http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio.patch
>    - UIO core
>  http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio-documentation.patch
>    - UIO documentation
>  http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio-dummy.patch
>  http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio-irq.patch
>    - two example kernel modules and userspace programs showing how to
>      use the UIO interface.
>
>If anyone has any questions on how to use this interface, or anything
>else about it, please let me and Thomas know.

The uio-irq.patch contains SA_INTERRUPT in its code. I thought this flag was
obsoleted by IRQF_* by now?



	-`J'
-- 

Re: Userspace I/O driver core

[linux-os (Dick Johnson)]

On Wed, 13 Dec 2006, Greg KH wrote:

> A large number of people have expressed interest recently in the
> userspace i/o driver core which allows userspace drivers to be written
> to handle some types of hardware.
>
> Right now the UIO core is working and in the -mm releases.  It's been
> rewritten from the last time patches were posted to lkml and is much
> simpler.  It also includes full documentation and two example drivers
> and two example userspace programs that test those drivers.
>
> But in order to get this core into the kernel tree, we need to have some
> "real" drivers written that use it.  So, for anyone that wants to see
> this go into the tree, now is the time to step forward and post your
> patches for hardware that this kind of driver interface is needed.
>
> If no such drivers appear, then there is a very slim chance that this
> interface will be accepted into the tree.
>
> The patches can be found in the -mm releases or at:
>  http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio.patch
>    - UIO core
>  http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio-documentation.patch
>    - UIO documentation
>  http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio-dummy.patch
>  http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio-irq.patch
>    - two example kernel modules and userspace programs showing how to
>      use the UIO interface.
>
> If anyone has any questions on how to use this interface, or anything
> else about it, please let me and Thomas know.
>
> thanks,
>
> greg k-h

Exposing device hardware registers to user-space in contrary to good standards of engineering practice, violates some basic tenets of operating system design, and certainly has no place within a Unix emulation such as Linux. Please understand that even though certain graphical interfaces violate these standards, this does not mean that any such violations are acceptable.

There are well thought-out methods of creating hardware interfaces that have a successfully history of implementation both in Linux and Unix. There are well established APIs that are used to expose devices to user-space with controlled privilege, access mechanisms, and built-in locking to provide atomic access to the functionality of the devices without user-space code needing to understand the device intricacies (and probably getting it wrong).

I recently returned from a conference where somebody had designed a driver that exposes PCI registers and FPGA device registers to user-space. Their problem was how to provide "call-backs" when an interrupt occurred. They were convinced that all they had to do was to have some user-space procedure that could be called when an interrupt occurred. Then their so-called driver would work. They had no clue about the fact that an interrupt can occur at any time not just when somebody is ready and waiting for it, that one usually has sections of code that must not be interrupted, etc. This information was completely lost when talking to this engineer. He had learned that an interrupt service routine is just some code. That's all. Synchronization and atomic operations meant nothing to him. We was a recent college graduate with as Masters Degree so he wasn't uneducated. What he was was uneducated, regardless of his degrees. Driver code needs to be protected from undue user-space interference otherwise the device can't be synchronized, shared, or accessed through the operating system's APIs.

Every time I showed how the driver couldn't work properly, the designer so convinced of his superior methods, would devise a work-around. For instance, to protect a section of code from being modified in an interrupt, the user-space driver was to be executed with iopl(3) and interrupts disabled. To protect the kernel from the ISR being modified or replaced, the code would be checksummed every time an interrupt occurred, etc. I could go on. Drivers have no place user space.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.24 on an i686 machine (5592.72 BogoMips).
New book: http://www.AbominableFirebug.com/
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: Userspace I/O driver core

[Ben Nizette]

linux-os (Dick Johnson) wrote:
> On Wed, 13 Dec 2006, Greg KH wrote:
>>
>> If anyone has any questions on how to use this interface, or anything
>> else about it, please let me and Thomas know.
>>
>> thanks,
>>
>> greg k-h
[snip]
> There are well thought-out methods of creating hardware interfaces that 
 > have a successfully history of implementation both in Linux and Unix.
 > There are well established APIs that are used to expose devices to
 > user-space with controlled privilege, access mechanisms, and built-in
 > locking to provide atomic access to the functionality of the devices
 > without user-space code needing to understand the device intricacies
 > (and probably getting it wrong).
> 
> I recently returned from a conference where somebody had designed a 
 > driver that exposes PCI registers and FPGA device registers to
 > user-space. Their problem was how to provide "call-backs" when an
 > interrupt occurred. They were convinced that all they had to do was
 > to have some user-space procedure that could be called when an
 > interrupt occurred. Then their so-called driver would work. They had
 > no clue about the fact that an interrupt can occur at any time not
 > just when somebody is ready and waiting for it, that one usually has
 > sections of code that must not be interrupted, etc.

This is almost exactly the situation I find myself in and a situation 
for which UIO is perfect.  UIO is not a hole through the kernel in to 
memory, it is a well defined API of the type you describe above (albeit 
not 'established' yet).  UIO interrupts are _handled_ in kernel space 
but subsequently _signalled_ in userspace.  There are no issues of 
kernel code directly calling any userspace functionallity.
 >
 > Driver code needs to be protected from undue user-space interference
 > otherwise the device can't be synchronized, shared, or accessed
 > through the operating system's APIs.

And this is what UIO does, it allows userspace interaction without 
userspace interference.  It provides a safe an sanitized view of the 
hardware to processes which make more sense in userland.
> 
> Every time I showed how the driver couldn't work properly, the 
 > designer so convinced of his superior methods, would devise a
 > work-around. For instance, to protect a section of code from being
 > modified in an interrupt, the user-space driver was to be executed
 > with iopl(3) and interrupts disabled. To protect the kernel from the
 > ISR being modified or replaced, the code would be checksummed every
 > time an interrupt occurred, etc. I could go on. Drivers have no place
 > user space.
> 
No, dumb drivers with dodgy kernel interfaces don't have a place 
_anywhere_.  If this under-educated person was using UIO there would be 
no need for any of his hacks, a userspace driver would be feasible, 
clean, neat and perfectly allowable.

Regards,
	Ben