How to implement a driver's read and write operations with synchronization properly

How to implement a driver's read and write operations with synchronization properly

[Anh Le]

Hi everyone,
I'm trying to write a misc char driver with read and write operations,
and I choose reader writer lock for synchronization between them.
Suppose that when writing 2000 bytes, the write operation is called
twice, each time writing only 1000 bytes. There are 2 ways to
implement synchronization as below:
1. Acquire and release lock everytime the write operation is called.
Same with read operation.
2. Acquire the lock when the write operation is called for the first
time, and release when the whole write procedure is completed. In this
case, the lock is acquired before transfering the 1st byte, and
released after transfering the 2000th byte. Same with read operation.

As how I see it, both of these approaches have problems. The first one
release the lock in between the write operations, so a reader can get
in while the writer has only finished a portion of its work. With the
second approach, I have no way to tell that the write procedure ends
with the 2000th byte. What if there are more?

Could anyone tell me the proper way to deal with this situation?
Thanks in advance.

-- 
Le Quoc Anh

How to implement a driver's read and write operations with synchronization properly

[Jonathan Neuschäfer]

On Tue, Jul 29, 2014 at 03:03:55PM +0700, Anh Le wrote:
> Hi everyone,
> I'm trying to write a misc char driver with read and write operations,
> and I choose reader writer lock for synchronization between them.
> Suppose that when writing 2000 bytes, the write operation is called
> twice, each time writing only 1000 bytes. There are 2 ways to
> implement synchronization as below:

Which "write operation" is called twice in your scenario?

> 1. Acquire and release lock everytime the write operation is called.
> Same with read operation.
> 2. Acquire the lock when the write operation is called for the first
> time, and release when the whole write procedure is completed. In this
> case, the lock is acquired before transfering the 1st byte, and
> released after transfering the 2000th byte. Same with read operation.

If a userspace program writes 1000 bytes at first, how can you know that
it wants to perform another write later on?

> As how I see it, both of these approaches have problems. The first one
> release the lock in between the write operations, so a reader can get
> in while the writer has only finished a portion of its work.

If a userspace program wants to write a chunk of data atomically, it
should use just one call to write(2). (On Linux, one can save some
copying by using writev(2), which writes data from multiple buffers in
one atomic step.)

> With the
> second approach, I have no way to tell that the write procedure ends
> with the 2000th byte. What if there are more?

Again, where does this write procedure run?


Greetings,
Jonathan Neusch?fer

> Could anyone tell me the proper way to deal with this situation?
> Thanks in advance.
> 
> -- 
> Le Quoc Anh

How to implement a driver's read and write operations with synchronization properly

[Pranay Srivastava]

On 29-Jul-2014 1:34 PM, "Anh Le" <anhlq2110@gmail.com> wrote:
>
> Hi everyone,
> I'm trying to write a misc char driver with read and write operations,
> and I choose reader writer lock for synchronization between them.
> Suppose that when writing 2000 bytes, the write operation is called
> twice, each time writing only 1000 bytes. There are 2 ways to
> implement synchronization as below:

Your application is doing 2 writes as I understand. So let's say if only
your process is the only one writing do you need locking?

> 1. Acquire and release lock everytime the write operation is called.
> Same with read operation.
> 2. Acquire the lock when the write operation is called for the first
> time, and release when the whole write procedure is completed. In this
> case, the lock is acquired before transfering the 1st byte, and
> released after transfering the 2000th byte. Same with read operation.
>
How about doing page flipping here? Your locking will be for shorter
duration for reads.

See your read/write as one operation no matter how many bytes. You are
synchronizing for different processes.

> As how I see it, both of these approaches have problems. The first one
> release the lock in between the write operations, so a reader can get
> in while the writer has only finished a portion of its work. With the
> second approach, I have no way to tell that the write procedure ends

One read / write thats it. You make promise at kernel level that you wont
give false data on reads and wont allow parallel write op. Thats all.

How applications honor who goes first is upto them. Say 4 processes writing
on same file no synchronization at user space level is gonna screw data
even when you made a good enough promise that write op wont be in parallel.
The syncing is at 2 places not only kernel.

> with the 2000th byte. What if there are more?
>
> Could anyone tell me the proper way to deal with this situation?
> Thanks in advance.

Just make a good enough promise that data reads would be valid and that
writes wont be screwed up.

If applications are not good citizens then kernel cant do anything except
keep its own promises.

>
> --
> Le Quoc Anh
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140729/b309557b/attachment.html 

How to implement a driver's read and write operations with synchronization properly

[Anh Le]

Thanks, Jonathan, for the quick reply.

On Tue, Jul 29, 2014 at 3:36 PM, Jonathan Neusch?fer
<j.neuschaefer@gmx.net> wrote:
> Which "write operation" is called twice in your scenario?

It's the "write" method of the "struct file_operations" structure. I
define it like this (no synchronizatin yet):
static ssize_t sample_write(struct file *f, const char __user *buf,
size_t len, loff_t *ppos)
{
        printk(KERN_DEBUG "[sample] buf len: %u, *ppos: %u\n", len, *ppos);
        return simple_write_to_buffer(kbuf, 2048, ppos, buf, len);
}

> If a userspace program writes 1000 bytes at first, how can you know that
> it wants to perform another write later on?

I use printk to debug the module. More details in the answer below.

> If a userspace program wants to write a chunk of data atomically, it
> should use just one call to write(2). (On Linux, one can save some
> copying by using writev(2), which writes data from multiple buffers in
> one atomic step.)

I tried the following command: echo $(perl -e "print 'a'x2000") > /dev/sample
and get the following messages from dmesg:
[30884.066433] [sample] buf len: 1008, *ppos: 0
[30884.066451] [sample] buf len: 993, *ppos: 1008

So as I understand my 2001 bytes has been split into 2 chunks, the
first one with 1008 bytes and the second one with 993 bytes, and
therefore the write operation is called 2 times to consume the whole
input.

-- 
Le Quoc Anh

How to implement a driver's read and write operations with synchronization properly

[Pranay Srivastava]

On 29-Jul-2014 5:54 PM, "Anh Le" <anhlq2110@gmail.com> wrote:
>
> Thanks, Jonathan, for the quick reply.
>
> On Tue, Jul 29, 2014 at 3:36 PM, Jonathan Neusch?fer
> <j.neuschaefer@gmx.net> wrote:
> > Which "write operation" is called twice in your scenario?
>
> It's the "write" method of the "struct file_operations" structure. I
> define it like this (no synchronizatin yet):
> static ssize_t sample_write(struct file *f, const char __user *buf,
> size_t len, loff_t *ppos)
> {
>         printk(KERN_DEBUG "[sample] buf len: %u, *ppos: %u\n", len,
*ppos);
>         return simple_write_to_buffer(kbuf, 2048, ppos, buf, len);
> }
>
> > If a userspace program writes 1000 bytes at first, how can you know that
> > it wants to perform another write later on?
>
> I use printk to debug the module. More details in the answer below.
>
> > If a userspace program wants to write a chunk of data atomically, it
> > should use just one call to write(2). (On Linux, one can save some
> > copying by using writev(2), which writes data from multiple buffers in
> > one atomic step.)
>
> I tried the following command: echo $(perl -e "print 'a'x2000") >
/dev/sample
> and get the following messages from dmesg:
> [30884.066433] [sample] buf len: 1008, *ppos: 0
> [30884.066451] [sample] buf len: 993, *ppos: 1008
>
> So as I understand my 2001 bytes has been split into 2 chunks, the

Who do you think did this split? I would say write a simple program and do
only a single write syscall for your buffer size. Just see if that split
happens again.

> first one with 1008 bytes and the second one with 993 bytes, and
> therefore the write operation is called 2 times to consume the whole
> input.
>
> --
> Le Quoc Anh
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140729/c9749bb1/attachment-0001.html 

How to implement a driver's read and write operations with synchronization properly

[Jonathan Neuschäfer]

On Tue, Jul 29, 2014 at 07:15:08PM +0700, Anh Le wrote:
[...]
> I tried the following command: echo $(perl -e "print 'a'x2000") > /dev/sample
> and get the following messages from dmesg:
> [30884.066433] [sample] buf len: 1008, *ppos: 0
> [30884.066451] [sample] buf len: 993, *ppos: 1008
> 
> So as I understand my 2001 bytes has been split into 2 chunks, the
> first one with 1008 bytes and the second one with 993 bytes, and
> therefore the write operation is called 2 times to consume the whole
> input.

I've tried this out myself, and it seems to be an issue with bash:

	$ cat /tmp/aaaa.sh
	#!/bin/sh
	echo aaaaaaa...[trimmed to fit in an e-mail]...aaaaaaa
	$ strace -e trace=write /tmp/aaaa.sh > /dev/null
	write(1, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 2001) = 2001
	+++ exited with 0 +++
	$ strace -e trace=write bash /tmp/aaaa.sh > /dev/null
	write(1, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 1008) = 1008
	write(1, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 993) = 993
	+++ exited with 0 +++

[ My default shell, /bin/sh, is Debian's dash. ]

As you can see, I observed the same pattern of 1008 and 993 bytes.


Greetings,
Jonathan Neusch?fer

How to implement a driver's read and write operations with synchronization properly

[Anh Le]

On Tue, Jul 29, 2014 at 7:47 PM, Jonathan Neusch?fer
<j.neuschaefer@gmx.net> wrote:
> I've tried this out myself, and it seems to be an issue with bash:

> As you can see, I observed the same pattern of 1008 and 993 bytes.

I think you're right. The split is done by the user program, separated
from kernel land. So I guess it's reasonable to use the 1st approach,
which is to always release the lock at the end of the operation. But
still, user programs like bash could have a race problem by spliting
the input, I hope that they can somehow take care of this problem
themselves.

Thanks guys.

-- 
Le Quoc Anh

How to implement a driver's read and write operations with synchronization properly

[Valdis.Kletnieks at vt.edu]

On Tue, 29 Jul 2014 20:59:08 +0700, Anh Le said:

> still, user programs like bash could have a race problem by spliting
> the input, I hope that they can somehow take care of this problem
> themselves.

stdio is *not* always your friend.  fopen/fprintf is prone to splitting on
bugger boundaries without your knowing about it, but most language bindings
(including the Perl you tested with) allow you to use open/write and do the
buffer management yourself

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 848 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140729/d3e06d04/attachment.bin 

How to implement a driver's read and write operations with synchronization properly

[Jonathan Neuschäfer]

On Tue, Jul 29, 2014 at 10:24:41AM -0400, Valdis.Kletnieks at vt.edu wrote:
> On Tue, 29 Jul 2014 20:59:08 +0700, Anh Le said:
> 
> > still, user programs like bash could have a race problem by spliting
> > the input, I hope that they can somehow take care of this problem
> > themselves.
> 
> stdio is *not* always your friend.  fopen/fprintf is prone to splitting on
> bugger boundaries without your knowing about it, but most language bindings
> (including the Perl you tested with) allow you to use open/write and do the
> buffer management yourself
> 

Actually, the Perl's IO layer did pretty little to the result, because
Anh used command line substitution:

	echo $(perl -e "print 'a'x2000") > /dev/sample


Jonathan