Should a block device enforce block atomicity?

Should a block device enforce block atomicity?

[Erez Zilber]

Hi,

I have a question about block devices and whether they are required to
enforce block atomicity:

I read the code of drivers/block/brd.c, and I didn't see any locking
when blocks are read/written. I also looked at the block layer code
that calls brd and didn't find any locking there. Does it mean that
there's no block atomicity (i.e. multiple threads can write a single
block at the same time)? Is there any hidden assumption here? Is this
the responsibility of the application to do that (e.g. not start a
WRITE request before other READ/WRITE requests to the same block were
completed)?

Thanks,
Erez

Re: Should a block device enforce block atomicity?

[Jens Axboe]

On Mon, Jun 30 2008, Erez Zilber wrote:
> Hi,
> 
> I have a question about block devices and whether they are required to
> enforce block atomicity:
> 
> I read the code of drivers/block/brd.c, and I didn't see any locking
> when blocks are read/written. I also looked at the block layer code
> that calls brd and didn't find any locking there. Does it mean that
> there's no block atomicity (i.e. multiple threads can write a single
> block at the same time)? Is there any hidden assumption here? Is this
> the responsibility of the application to do that (e.g. not start a
> WRITE request before other READ/WRITE requests to the same block were
> completed)?

The block layer doesn't give such guarentees, not for "regular" block
devices either. If the IO goes through the page cache then that will
serialize IO to a given page, but with eg O_DIRECT IO, you could have
the same block in flight several times. So if you are doing raw IO, the
application has to ensure ordering of the same block.

-- 
Jens Axboe

Re: Should a block device enforce block atomicity?

[Erez Zilber]

On Mon, Jun 30, 2008 at 9:55 AM, Jens Axboe <jens.axboe@oracle.com> wrote:
> On Mon, Jun 30 2008, Erez Zilber wrote:
>> Hi,
>>
>> I have a question about block devices and whether they are required to
>> enforce block atomicity:
>>
>> I read the code of drivers/block/brd.c, and I didn't see any locking
>> when blocks are read/written. I also looked at the block layer code
>> that calls brd and didn't find any locking there. Does it mean that
>> there's no block atomicity (i.e. multiple threads can write a single
>> block at the same time)? Is there any hidden assumption here? Is this
>> the responsibility of the application to do that (e.g. not start a
>> WRITE request before other READ/WRITE requests to the same block were
>> completed)?
>
> The block layer doesn't give such guarentees, not for "regular" block
> devices either. If the IO goes through the page cache then that will
> serialize IO to a given page, but with eg O_DIRECT IO, you could have
> the same block in flight several times. So if you are doing raw IO, the
> application has to ensure ordering of the same block.
>

So, do you say that people that write applications need to take care
of I/O serialization, and block devices (and the block layer itself)
don't need to care about this problem? I thought that standard disks
guarantee block atomicity (i.e. they don't count on the layers above
them to do that).

Erez

Re: Should a block device enforce block atomicity?

[Boaz Harrosh]

Erez Zilber wrote:
> On Mon, Jun 30, 2008 at 9:55 AM, Jens Axboe <jens.axboe@oracle.com> wrote:
>> On Mon, Jun 30 2008, Erez Zilber wrote:
>>> Hi,
>>>
>>> I have a question about block devices and whether they are required to
>>> enforce block atomicity:
>>>
>>> I read the code of drivers/block/brd.c, and I didn't see any locking
>>> when blocks are read/written. I also looked at the block layer code
>>> that calls brd and didn't find any locking there. Does it mean that
>>> there's no block atomicity (i.e. multiple threads can write a single
>>> block at the same time)? Is there any hidden assumption here? Is this
>>> the responsibility of the application to do that (e.g. not start a
>>> WRITE request before other READ/WRITE requests to the same block were
>>> completed)?
>> The block layer doesn't give such guarentees, not for "regular" block
>> devices either. If the IO goes through the page cache then that will
>> serialize IO to a given page, but with eg O_DIRECT IO, you could have
>> the same block in flight several times. So if you are doing raw IO, the
>> application has to ensure ordering of the same block.
>>
> 
> So, do you say that people that write applications need to take care
> of I/O serialization, and block devices (and the block layer itself)
> don't need to care about this problem? I thought that standard disks
> guarantee block atomicity (i.e. they don't count on the layers above
> them to do that).
> 
> Erez

Don't forget that all IO requests are queued on the device. With a
modern HW and disk you usually have NCQ and most drives will throw
away write request to the same sector if they see a later write to
the same sector in the queue.

That said. There is nothing wrong with writing again and again to 
the same sector on disk. File/record locking is done at the FileSystem
level. An application that wants exclusive write need to open the file
that way. Other wise it could even be written from another machine not
even another thread.

What is it you are concerned with?

Boaz

Re: Should a block device enforce block atomicity?

[Zhao Forrest]

>
> Don't forget that all IO requests are queued on the device. With a
> modern HW and disk you usually have NCQ and most drives will throw
> away write request to the same sector if they see a later write to
> the same sector in the queue.
>
> That said. There is nothing wrong with writing again and again to
> the same sector on disk. File/record locking is done at the FileSystem
> level. An application that wants exclusive write need to open the file
> that way. Other wise it could even be written from another machine not
> even another thread.
>
> What is it you are concerned with?
>
I happen to read the email and have a question, that may not be Erez's
real question :)
Let's suppose the following example:
1 pdflush find a dirty inode and decides to flush a set of dirty pages
to harddrive
2 while this set of dirty pages is being committed to
harddrive(dma_mapping is done, but dirty pages are not really written
to disk media), application/FS is trying to update some pages in this
set of dirty pages.

Then what happens? Will application be put into sleep until page
flushing to disk media is done?

Thanks,
Forrest

Re: Should a block device enforce block atomicity?

[Jens Axboe]

On Mon, Jun 30 2008, Zhao Forrest wrote:
> >
> > Don't forget that all IO requests are queued on the device. With a
> > modern HW and disk you usually have NCQ and most drives will throw
> > away write request to the same sector if they see a later write to
> > the same sector in the queue.
> >
> > That said. There is nothing wrong with writing again and again to
> > the same sector on disk. File/record locking is done at the FileSystem
> > level. An application that wants exclusive write need to open the file
> > that way. Other wise it could even be written from another machine not
> > even another thread.
> >
> > What is it you are concerned with?
> >
> I happen to read the email and have a question, that may not be Erez's
> real question :)
> Let's suppose the following example:
> 1 pdflush find a dirty inode and decides to flush a set of dirty pages
> to harddrive
> 2 while this set of dirty pages is being committed to
> harddrive(dma_mapping is done, but dirty pages are not really written
> to disk media), application/FS is trying to update some pages in this
> set of dirty pages.
> 
> Then what happens? Will application be put into sleep until page
> flushing to disk media is done?

Yes, the page needs to be locked before IO can be issued. So it'll block
on the page lock. That is what I referred to as the page cache providing
this guarentee for you. With raw IO, the application needs to ensure
ordering on its own if it commits IO to the same block more than once.

-- 
Jens Axboe