locking a device for exclusive use by userspace process (or kernel thread)

locking a device for exclusive use by userspace process (or kernel thread)

[Yann Droneaud]

Hi,

I have a device and I have to write a driver that exposes the
following three operations to kernel modules AND to userspace
programs:

- LOCK

LOCK operation is used to get exclusive access to the device.
If exclusive access is already granted, the caller will wait
for it be released.

- PROCESS

PROCESS can be issued only after a successful LOCK operation.
PROCESS operations can be issued multiple times between LOCK
and UNLOCK.

- UNLOCK

UNLOCK operation release the exclusive access to the device.
This operation can be issued only after a successful LOCK
operation.

In kernelspace, the lock would be implemented using
a semaphore (with an initial count of 1). That's the easy
part.

In userspace, it's expected LOCK and UNLOCK can be issued in
different threads within one process, and, if, while holding
the lock, a process die, either being killed or by exit(),
an implicit UNLOCK should be done.

I've initially thought I could represent the lock by a file
descriptor, map LOCK to open() and  UNLOCK() to close() (with
the added benefit that opened file descriptor will be closed
on exit):

  operation    userspace     kernelspace

    LOCK        open()         ->open()
   PROCESS      ioctl()        ->ioctl()
   UNLOCK       close()        ->release()

open() would block if the lock is already held by some other
process (and would not block if O_NONBLOCK or O_NDELAY is used).

Seems a great scheme.

But that doesn't fit well with Unix / POSIX, as both fork()
and dup{,2,3}() or fcntl(, F_DUPFD*,) creates new references
to open()'ed device:

- A child will inherit the file descriptor for the device
after parent fork(). As the child has a file descriptor for
the device, the parent couldn't UNLOCK by calling close() since
the device won't be released until the child also close() its
copy of the file descriptor.

It should be noted that even if the child has the device opened
by the vertue of inheritance, it does not own the lock, so it
should not be allowed to issue PROCESS operation. But at the
same time, it prevents the real owner from releasing the lock
(... I'm seeing dead locks). So who really own the lock ?

It seems awkward to allow an exclusive lock to be shared
between two different process even if they are parents.

- In the very same process, multiple references to the device
could be created with dup(). So close()ing the file descriptor
returned by open() won't UNLOCK the device, as the device will
only be released once the last reference would be close()'d.

It seems awkward to allow an exclusive lock to be "duplicated"
within a process.


I've tried to (ab)use ->flush() to implement UNLOCK operation so
that a call to close() on the device's file descriptor would
UNLOCK, even if a duplicated file descriptor exists in the same
process or in a child. But this has the unfortunate consequence of
allowing to UNLOCK when closing a duplicated file descriptor
(and closing the file descriptor in the child could also mean
UNLOCK).

So in the end, I'm probably not going to use open() for LOCK and
close() for UNLOCK. I think I have to use dedicated LOCK and
UNLOCK ioctls.

But I failed to find a way to be able to release the lock when
the process owning the lock die / exit.

Perhaps I need a mechanism to be notified of the death of the
process owning the lock.

Do you have such mean in mind ?

Or any other reasonable solution to implement the three operations of 
such driver ?

Regards.

-- 
Yann Droneaud
OPTEYA

locking a device for exclusive use by userspace process (or kernel thread)

[Greg KH]

On Wed, Mar 04, 2015 at 06:59:04PM +0100, Yann Droneaud wrote:
> Hi,
> 
> I have a device and I have to write a driver that exposes the
> following three operations to kernel modules AND to userspace
> programs:

<snip>

Why?  What type of device is this?  And who is asking you to do this
homework assignment?

good luck,

greg k-h

locking a device for exclusive use by userspace process (or kernel thread)

[Yann Droneaud]

Hi Greg,

Le mercredi 04 mars 2015 ? 10:11 -0800, Greg KH a ?crit :
> On Wed, Mar 04, 2015 at 06:59:04PM +0100, Yann Droneaud wrote:
> > Hi,
> > 
> > I have a device and I have to write a driver that exposes the
> > following three operations to kernel modules AND to userspace
> > programs:
> 
> <snip>
> 
> Why?

I'm trying to improve the current driver for this device to allow the 
lock to be released when userspace program owning the lock is killed.

>   What type of device is this?

It's a device with some kind of over complicated mailbox.

A userspace program is supposed to feed the mailbox with multiple 
commands and no other userspace program should be allowed to play with 
the device during that time.

>   And who is asking you to do this homework assignment?
> 

It's not a homework assignment. It's not even something a little penguin
ask me to do.

I was so sure I could map the semaphore sematics to open() and close()
and use that file descriptor to represent the lock that I feel terribly 
sorry for not being able to do so.

(At one point I felt like I was fighting against 40 years of Unix, and
such a fight is not going to be won by myself :)

So I'm back to step 0 and looking for a way to be able to release a 
lock in case of a userspace program is killed.

> good luck,
> 

Thanks :)

Regards.

-- 
Yann Droneaud
OPTEYA

locking a device for exclusive use by userspace process (or kernel thread)

[Greg KH]

On Wed, Mar 04, 2015 at 07:38:52PM +0100, Yann Droneaud wrote:
> Hi Greg,
> 
> Le mercredi 04 mars 2015 ? 10:11 -0800, Greg KH a ?crit :
> > On Wed, Mar 04, 2015 at 06:59:04PM +0100, Yann Droneaud wrote:
> > > Hi,
> > > 
> > > I have a device and I have to write a driver that exposes the
> > > following three operations to kernel modules AND to userspace
> > > programs:
> > 
> > <snip>
> > 
> > Why?
> 
> I'm trying to improve the current driver for this device to allow the 
> lock to be released when userspace program owning the lock is killed.
> 
> >   What type of device is this?
> 
> It's a device with some kind of over complicated mailbox.

That's not very descriptive, what _exact_ type of device is this?  What
existing driver supports it?  Pointers to code would be helpful.

> A userspace program is supposed to feed the mailbox with multiple 
> commands and no other userspace program should be allowed to play with 
> the device during that time.

What type of program?  What type of user/kernel interface are you using?
Why wouldn't a simple "only allow one userspace program to open the
device node" work?

> >   And who is asking you to do this homework assignment?
> > 
> 
> It's not a homework assignment. It's not even something a little penguin
> ask me to do.

Who asked you to do this in this specific manner?  What root problem are
you trying to solve?  Why not just prevent userspace from talking to the
device at the same time, you have control over this, right?

> I was so sure I could map the semaphore sematics to open() and close()
> and use that file descriptor to represent the lock that I feel terribly 
> sorry for not being able to do so.
> 
> (At one point I felt like I was fighting against 40 years of Unix, and
> such a fight is not going to be won by myself :)
> 
> So I'm back to step 0 and looking for a way to be able to release a 
> lock in case of a userspace program is killed.

Should be easy to do on the close/release path, why isn't that working?

Or a simple reference count would also work, if you lock it properly.

Again, pointers to code would be best.

thanks,

greg k-h

locking a device for exclusive use by userspace process (or kernel thread)

[Malte Vesper]

Hi Greg, Hi Yann,
I have been actually more naive by implementing it this way (led on by 
ldd3, page 176).
I have the issue for an FPGA board and reconfiguration.
Basically if a program has control of the FPGA it does want the FPGA do 
its computations, without another program changing the design loaded 
onto the FPGA.

For the time being I will settle on the user opening the device to be 
smart enough to not duplicate the fd. But I think there are usecases for 
the requested feature.

regards Malte

On 04/03/15 18:38, Yann Droneaud wrote:
> Hi Greg,
>
> Le mercredi 04 mars 2015 ? 10:11 -0800, Greg KH a ?crit :
>> On Wed, Mar 04, 2015 at 06:59:04PM +0100, Yann Droneaud wrote:
>>> Hi,
>>>
>>> I have a device and I have to write a driver that exposes the
>>> following three operations to kernel modules AND to userspace
>>> programs:
>> <snip>
>>
>> Why?
> I'm trying to improve the current driver for this device to allow the
> lock to be released when userspace program owning the lock is killed.
>
>>    What type of device is this?
> It's a device with some kind of over complicated mailbox.
>
> A userspace program is supposed to feed the mailbox with multiple
> commands and no other userspace program should be allowed to play with
> the device during that time.
>
>>    And who is asking you to do this homework assignment?
>>
> It's not a homework assignment. It's not even something a little penguin
> ask me to do.
>
> I was so sure I could map the semaphore sematics to open() and close()
> and use that file descriptor to represent the lock that I feel terribly
> sorry for not being able to do so.
>
> (At one point I felt like I was fighting against 40 years of Unix, and
> such a fight is not going to be won by myself :)
>
> So I'm back to step 0 and looking for a way to be able to release a
> lock in case of a userspace program is killed.
>
>> good luck,
>>
> Thanks :)
>
> Regards.
>