Cached IO Synchronization Question

Cached IO Synchronization Question

[Matt Klein]

(Please CC me on any responses)

Hi,

I am studying 2.6.27.21 to try to understand the IO synchronization
model used in the Linux kernel.

I have the following question:

How are cached reads/writes synchronized? I assume they are, but I can
not determine how it is done.

1) do_generic_file_read calls find_get_page to get the existing page
cache page if it exists (it uses RCU to prevent the page from being
blown away while reading takes place). Assuming it does, it performs
various checks to make sure the page is up to date before performing an
atomic copy to user mode. The read path does not take the page lock.

2) The write path takes both the inode lock and the page lock to prevent
multiple writers. It also does an atomic copy from user mode into the
page cache page.

>From my reading of the code, the atomic copies are atomic only on a
single processor, not across processors. Right?

If so, what is to prevent one CPU from starting a cached read into a
user mode buffer while another CPU has partially copied data into the
page cache page?

Thanks,
Matt

-- 
Matt Klein
Technical Specialist
Pikewerks
matt.klein@pikewerks.com / 206 327 4515

Re: Cached IO Synchronization Question

[Peter Zijlstra]

On Mon, 2009-04-06 at 14:31 -0700, Matt Klein wrote:
> (Please CC me on any responses)
> 
> Hi,
> 
> I am studying 2.6.27.21 to try to understand the IO synchronization
> model used in the Linux kernel.
> 
> I have the following question:
> 
> How are cached reads/writes synchronized? I assume they are, but I can
> not determine how it is done.
> 
> 1) do_generic_file_read calls find_get_page to get the existing page
> cache page if it exists (it uses RCU to prevent the page from being
> blown away while reading takes place). Assuming it does, it performs
> various checks to make sure the page is up to date before performing an
> atomic copy to user mode. The read path does not take the page lock.
> 
> 2) The write path takes both the inode lock and the page lock to prevent
> multiple writers. It also does an atomic copy from user mode into the
> page cache page.
> 
> From my reading of the code, the atomic copies are atomic only on a
> single processor, not across processors. Right?

Not even that ;-)

> If so, what is to prevent one CPU from starting a cached read into a
> user mode buffer while another CPU has partially copied data into the
> page cache page?

Note how do_generic_file_read() checks PageUptodate() and does a
lock_page_killable() in case its not.

Re: Cached IO Synchronization Question

[Matt Klein]

> 
> Note how do_generic_file_read() checks PageUptodate() and does a
> lock_page_killable() in case its not.
> 
> 
> 

As far as I can tell there is nothing to prevent the page from becoming
out of date (or someone doing a full page cached write) after the
PageUptodate() check returns up to date.

So I have concluded that cached read/write interleaving is not
synchronized. This is the behavior found on Windows.

Re: Cached IO Synchronization Question

[Nick Piggin]

On Thursday 09 April 2009 01:56:38 Matt Klein wrote:
> > 
> > Note how do_generic_file_read() checks PageUptodate() and does a
> > lock_page_killable() in case its not.
> > 
> > 
> > 
> 
> As far as I can tell there is nothing to prevent the page from becoming
> out of date (or someone doing a full page cached write) after the
> PageUptodate() check returns up to date.

Anything which marks the page not uptodate does not change the actual
page data, so the copy will still copy over valid data.

There is otherwise no synchronisation between reads and writes.

All we care about is that the data has been valid at some point in
time.

 
> So I have concluded that cached read/write interleaving is not
> synchronized. This is the behavior found on Windows.

Right.