Why does __do_page_cache_readahead submit READ, not READA?

Why does __do_page_cache_readahead submit READ, not READA?

[Lars Ellenberg]

I naively assumed, from the "readahead" in the name, that readahead
would be submitting READA bios. It does not.

I recently did some statistics on how many READ and READA requests
we actually see on the block device level.
I was suprised that READA is basically only used for file system
internal meta data (and not even for all file systems),
but _never_ for file data.

A simple
	dd if=bigfile of=/dev/null bs=4k count=1
will absolutely cause readahead of the configured amount, no problem.
But on the block device level, these are READ requests, where I'd
expected them to be READA requests, based on the name.

This is because __do_page_cache_readahead() calls read_pages(),
which in turn is mapping->a_ops->readpages(), or, as fallback,
mapping->a_ops->readpage().

On that level, all variants end up submitting as READ.

This may even be intentional.
But if so, I'd like to understand that.

Please, anyone in the know, enlighten me ;)


	Lars

Annecdotical: I've seen an oracle being killed by OOM, because someone
did a grep -r . while accidentally having a bogusly huge readahead set.

Re: Why does __do_page_cache_readahead submit READ, not READA?

[Jens Axboe]

On Wed, Jul 29 2009, Lars Ellenberg wrote:
> I naively assumed, from the "readahead" in the name, that readahead
> would be submitting READA bios. It does not.
> 
> I recently did some statistics on how many READ and READA requests
> we actually see on the block device level.
> I was suprised that READA is basically only used for file system
> internal meta data (and not even for all file systems),
> but _never_ for file data.
> 
> A simple
> 	dd if=bigfile of=/dev/null bs=4k count=1
> will absolutely cause readahead of the configured amount, no problem.
> But on the block device level, these are READ requests, where I'd
> expected them to be READA requests, based on the name.
> 
> This is because __do_page_cache_readahead() calls read_pages(),
> which in turn is mapping->a_ops->readpages(), or, as fallback,
> mapping->a_ops->readpage().
> 
> On that level, all variants end up submitting as READ.
> 
> This may even be intentional.
> But if so, I'd like to understand that.

I don't think it's intentional, and if memory serves, we used to use
READA when submitting read-ahead. Not sure how best to improve the
situation, since (as you describe), we lose the read-ahead vs normal
read at that level. I did some experimentation some time ago for
flagging this, see:

http://git.kernel.dk/?p=linux-2.6-block.git;a=commitdiff;h=16cfe64e3568cda412b3cf6b7b891331946b595e

which should pass down READA properly.

-- 
Jens Axboe

Re: Why does __do_page_cache_readahead submit READ, not READA?

[Chris Mason]

On Wed, Jul 29, 2009 at 11:18:45PM +0200, Jens Axboe wrote:
> On Wed, Jul 29 2009, Lars Ellenberg wrote:
> > I naively assumed, from the "readahead" in the name, that readahead
> > would be submitting READA bios. It does not.
> > 
> > I recently did some statistics on how many READ and READA requests
> > we actually see on the block device level.
> > I was suprised that READA is basically only used for file system
> > internal meta data (and not even for all file systems),
> > but _never_ for file data.
> > 
> > A simple
> > 	dd if=bigfile of=/dev/null bs=4k count=1
> > will absolutely cause readahead of the configured amount, no problem.
> > But on the block device level, these are READ requests, where I'd
> > expected them to be READA requests, based on the name.
> > 
> > This is because __do_page_cache_readahead() calls read_pages(),
> > which in turn is mapping->a_ops->readpages(), or, as fallback,
> > mapping->a_ops->readpage().
> > 
> > On that level, all variants end up submitting as READ.
> > 
> > This may even be intentional.
> > But if so, I'd like to understand that.
> 
> I don't think it's intentional, and if memory serves, we used to use
> READA when submitting read-ahead. Not sure how best to improve the
> situation, since (as you describe), we lose the read-ahead vs normal
> read at that level. I did some experimentation some time ago for
> flagging this, see:
> 
> http://git.kernel.dk/?p=linux-2.6-block.git;a=commitdiff;h=16cfe64e3568cda412b3cf6b7b891331946b595e
> 
> which should pass down READA properly.

One of the problems in the past was that reada would fail if there
wasn't a free request when we actually wanted it to go ahead and wait.
Or something.  We've switched it around a few times I think.

-chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why does __do_page_cache_readahead submit READ, not READA?

[Jens Axboe]

On Wed, Jul 29 2009, Chris Mason wrote:
> On Wed, Jul 29, 2009 at 11:18:45PM +0200, Jens Axboe wrote:
> > On Wed, Jul 29 2009, Lars Ellenberg wrote:
> > > I naively assumed, from the "readahead" in the name, that readahead
> > > would be submitting READA bios. It does not.
> > > 
> > > I recently did some statistics on how many READ and READA requests
> > > we actually see on the block device level.
> > > I was suprised that READA is basically only used for file system
> > > internal meta data (and not even for all file systems),
> > > but _never_ for file data.
> > > 
> > > A simple
> > > 	dd if=bigfile of=/dev/null bs=4k count=1
> > > will absolutely cause readahead of the configured amount, no problem.
> > > But on the block device level, these are READ requests, where I'd
> > > expected them to be READA requests, based on the name.
> > > 
> > > This is because __do_page_cache_readahead() calls read_pages(),
> > > which in turn is mapping->a_ops->readpages(), or, as fallback,
> > > mapping->a_ops->readpage().
> > > 
> > > On that level, all variants end up submitting as READ.
> > > 
> > > This may even be intentional.
> > > But if so, I'd like to understand that.
> > 
> > I don't think it's intentional, and if memory serves, we used to use
> > READA when submitting read-ahead. Not sure how best to improve the
> > situation, since (as you describe), we lose the read-ahead vs normal
> > read at that level. I did some experimentation some time ago for
> > flagging this, see:
> > 
> > http://git.kernel.dk/?p=linux-2.6-block.git;a=commitdiff;h=16cfe64e3568cda412b3cf6b7b891331946b595e
> > 
> > which should pass down READA properly.
> 
> One of the problems in the past was that reada would fail if there
> wasn't a free request when we actually wanted it to go ahead and wait.
> Or something.  We've switched it around a few times I think.

Yes, we did used to do that, whether it was 2.2 or 2.4 I
don't recall :-)

It should be safe to enable know, whether there's a prettier way
than the above, I don't know. It works by detecting the read-ahead
marker, but it's a bit of a fragile design.

-- 
Jens Axboe

Re: Why does __do_page_cache_readahead submit READ, not READA?

[Chris Mason]

On Thu, Jul 30, 2009 at 08:06:49AM +0200, Jens Axboe wrote:
> On Wed, Jul 29 2009, Chris Mason wrote:
> > On Wed, Jul 29, 2009 at 11:18:45PM +0200, Jens Axboe wrote:
> > > On Wed, Jul 29 2009, Lars Ellenberg wrote:
> > > > I naively assumed, from the "readahead" in the name, that readahead
> > > > would be submitting READA bios. It does not.
> > > > 
> > > > I recently did some statistics on how many READ and READA requests
> > > > we actually see on the block device level.
> > > > I was suprised that READA is basically only used for file system
> > > > internal meta data (and not even for all file systems),
> > > > but _never_ for file data.
> > > > 
> > > > A simple
> > > > 	dd if=bigfile of=/dev/null bs=4k count=1
> > > > will absolutely cause readahead of the configured amount, no problem.
> > > > But on the block device level, these are READ requests, where I'd
> > > > expected them to be READA requests, based on the name.
> > > > 
> > > > This is because __do_page_cache_readahead() calls read_pages(),
> > > > which in turn is mapping->a_ops->readpages(), or, as fallback,
> > > > mapping->a_ops->readpage().
> > > > 
> > > > On that level, all variants end up submitting as READ.
> > > > 
> > > > This may even be intentional.
> > > > But if so, I'd like to understand that.
> > > 
> > > I don't think it's intentional, and if memory serves, we used to use
> > > READA when submitting read-ahead. Not sure how best to improve the
> > > situation, since (as you describe), we lose the read-ahead vs normal
> > > read at that level. I did some experimentation some time ago for
> > > flagging this, see:
> > > 
> > > http://git.kernel.dk/?p=linux-2.6-block.git;a=commitdiff;h=16cfe64e3568cda412b3cf6b7b891331946b595e
> > > 
> > > which should pass down READA properly.
> > 
> > One of the problems in the past was that reada would fail if there
> > wasn't a free request when we actually wanted it to go ahead and wait.
> > Or something.  We've switched it around a few times I think.
> 
> Yes, we did used to do that, whether it was 2.2 or 2.4 I
> don't recall :-)
> 
> It should be safe to enable know, whether there's a prettier way
> than the above, I don't know. It works by detecting the read-ahead
> marker, but it's a bit of a fragile design.

I dug through my old email and found this fun bug w/buffer heads and
reada.

1) submit reada ll_rw_block on ext3 directory block
2) decide that we really really need to wait on this block
3) wait_on_buffer(bh) ; check up to date bit when done

The problem in the bugzilla was that reada was returning EAGAIN or
EWOULDBLOCK, and the whole filesystem world expects that if we
wait_on_buffer and don't find the buffer up to date, its time
set things read only and run around screaming.

The expectations in the code at the time were that the caller needs to
be aware the request may fail with EAGAIN/EWOULDBLOCK, but the reality
was that everyone who found that locked buffer also needed to be able to
check for it.  This one bugzilla had a teeny window where the reada
buffer head was leaked to the world.

So, I think we can start using it again if it is just a hint to the
elevator about what to do with the IO, and we never actually turn the
READA into a transient failure (which I think is mostly true today, there
weren't many READA tests in the code I could see).

-chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why does __do_page_cache_readahead submit READ, not READA?

[Jeff Moyer]

Chris Mason <chris.mason@oracle.com> writes:

> On Thu, Jul 30, 2009 at 08:06:49AM +0200, Jens Axboe wrote:
>> On Wed, Jul 29 2009, Chris Mason wrote:
>> > On Wed, Jul 29, 2009 at 11:18:45PM +0200, Jens Axboe wrote:
>> > > On Wed, Jul 29 2009, Lars Ellenberg wrote:
>> > > > I naively assumed, from the "readahead" in the name, that readahead
>> > > > would be submitting READA bios. It does not.
>> > > > 
>> > > > I recently did some statistics on how many READ and READA requests
>> > > > we actually see on the block device level.
>> > > > I was suprised that READA is basically only used for file system
>> > > > internal meta data (and not even for all file systems),
>> > > > but _never_ for file data.
>> > > > 
>> > > > A simple
>> > > > 	dd if=bigfile of=/dev/null bs=4k count=1
>> > > > will absolutely cause readahead of the configured amount, no problem.
>> > > > But on the block device level, these are READ requests, where I'd
>> > > > expected them to be READA requests, based on the name.
>> > > > 
>> > > > This is because __do_page_cache_readahead() calls read_pages(),
>> > > > which in turn is mapping->a_ops->readpages(), or, as fallback,
>> > > > mapping->a_ops->readpage().
>> > > > 
>> > > > On that level, all variants end up submitting as READ.
>> > > > 
>> > > > This may even be intentional.
>> > > > But if so, I'd like to understand that.
>> > > 
>> > > I don't think it's intentional, and if memory serves, we used to use
>> > > READA when submitting read-ahead. Not sure how best to improve the
>> > > situation, since (as you describe), we lose the read-ahead vs normal
>> > > read at that level. I did some experimentation some time ago for
>> > > flagging this, see:
>> > > 
>> > > http://git.kernel.dk/?p=linux-2.6-block.git;a=commitdiff;h=16cfe64e3568cda412b3cf6b7b891331946b595e
>> > > 
>> > > which should pass down READA properly.
>> > 
>> > One of the problems in the past was that reada would fail if there
>> > wasn't a free request when we actually wanted it to go ahead and wait.
>> > Or something.  We've switched it around a few times I think.
>> 
>> Yes, we did used to do that, whether it was 2.2 or 2.4 I
>> don't recall :-)
>> 
>> It should be safe to enable know, whether there's a prettier way
>> than the above, I don't know. It works by detecting the read-ahead
>> marker, but it's a bit of a fragile design.
>
> I dug through my old email and found this fun bug w/buffer heads and
> reada.
>
> 1) submit reada ll_rw_block on ext3 directory block
> 2) decide that we really really need to wait on this block
> 3) wait_on_buffer(bh) ; check up to date bit when done
>
> The problem in the bugzilla was that reada was returning EAGAIN or
> EWOULDBLOCK, and the whole filesystem world expects that if we
> wait_on_buffer and don't find the buffer up to date, its time
> set things read only and run around screaming.
>
> The expectations in the code at the time were that the caller needs to
> be aware the request may fail with EAGAIN/EWOULDBLOCK, but the reality
> was that everyone who found that locked buffer also needed to be able to
> check for it.  This one bugzilla had a teeny window where the reada
> buffer head was leaked to the world.
>
> So, I think we can start using it again if it is just a hint to the
> elevator about what to do with the IO, and we never actually turn the
> READA into a transient failure (which I think is mostly true today, there
> weren't many READA tests in the code I could see).

Well, is it a hint to the elevator or to the driver (or both)?  The one
bug I remember regarding READA failing was due to the FAILFAST bit
getting set for READA I/O, and the powerpath driver returning a failure.
Is that the bug to which you are referring?

Cheers,
Jeff

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why does __do_page_cache_readahead submit READ, not READA?

[Chris Mason]

On Thu, Jul 30, 2009 at 12:47:21PM -0400, Jeff Moyer wrote:
> Chris Mason <chris.mason@oracle.com> writes:
> 
> > On Thu, Jul 30, 2009 at 08:06:49AM +0200, Jens Axboe wrote:
> >> On Wed, Jul 29 2009, Chris Mason wrote:
> >> > On Wed, Jul 29, 2009 at 11:18:45PM +0200, Jens Axboe wrote:
> >> > > On Wed, Jul 29 2009, Lars Ellenberg wrote:
> >> > > > I naively assumed, from the "readahead" in the name, that readahead
> >> > > > would be submitting READA bios. It does not.
> >> > > > 
> >> > > > I recently did some statistics on how many READ and READA requests
> >> > > > we actually see on the block device level.
> >> > > > I was suprised that READA is basically only used for file system
> >> > > > internal meta data (and not even for all file systems),
> >> > > > but _never_ for file data.
> >> > > > 
> >> > > > A simple
> >> > > > 	dd if=bigfile of=/dev/null bs=4k count=1
> >> > > > will absolutely cause readahead of the configured amount, no problem.
> >> > > > But on the block device level, these are READ requests, where I'd
> >> > > > expected them to be READA requests, based on the name.
> >> > > > 
> >> > > > This is because __do_page_cache_readahead() calls read_pages(),
> >> > > > which in turn is mapping->a_ops->readpages(), or, as fallback,
> >> > > > mapping->a_ops->readpage().
> >> > > > 
> >> > > > On that level, all variants end up submitting as READ.
> >> > > > 
> >> > > > This may even be intentional.
> >> > > > But if so, I'd like to understand that.
> >> > > 
> >> > > I don't think it's intentional, and if memory serves, we used to use
> >> > > READA when submitting read-ahead. Not sure how best to improve the
> >> > > situation, since (as you describe), we lose the read-ahead vs normal
> >> > > read at that level. I did some experimentation some time ago for
> >> > > flagging this, see:
> >> > > 
> >> > > http://git.kernel.dk/?p=linux-2.6-block.git;a=commitdiff;h=16cfe64e3568cda412b3cf6b7b891331946b595e
> >> > > 
> >> > > which should pass down READA properly.
> >> > 
> >> > One of the problems in the past was that reada would fail if there
> >> > wasn't a free request when we actually wanted it to go ahead and wait.
> >> > Or something.  We've switched it around a few times I think.
> >> 
> >> Yes, we did used to do that, whether it was 2.2 or 2.4 I
> >> don't recall :-)
> >> 
> >> It should be safe to enable know, whether there's a prettier way
> >> than the above, I don't know. It works by detecting the read-ahead
> >> marker, but it's a bit of a fragile design.
> >
> > I dug through my old email and found this fun bug w/buffer heads and
> > reada.
> >
> > 1) submit reada ll_rw_block on ext3 directory block
> > 2) decide that we really really need to wait on this block
> > 3) wait_on_buffer(bh) ; check up to date bit when done
> >
> > The problem in the bugzilla was that reada was returning EAGAIN or
> > EWOULDBLOCK, and the whole filesystem world expects that if we
> > wait_on_buffer and don't find the buffer up to date, its time
> > set things read only and run around screaming.
> >
> > The expectations in the code at the time were that the caller needs to
> > be aware the request may fail with EAGAIN/EWOULDBLOCK, but the reality
> > was that everyone who found that locked buffer also needed to be able to
> > check for it.  This one bugzilla had a teeny window where the reada
> > buffer head was leaked to the world.
> >
> > So, I think we can start using it again if it is just a hint to the
> > elevator about what to do with the IO, and we never actually turn the
> > READA into a transient failure (which I think is mostly true today, there
> > weren't many READA tests in the code I could see).
> 
> Well, is it a hint to the elevator or to the driver (or both)?

I would say both as long as they don't fail it.  IOW a priority decision
instead of a discard this request at will decision.

> The one
> bug I remember regarding READA failing was due to the FAILFAST bit
> getting set for READA I/O, and the powerpath driver returning a failure.
> Is that the bug to which you are referring?

This was a rhel bug with ext3 and (both dm and powerpath ) multipath, but in
theory it could be triggered on regular drives.  I don't think we ever
managed to, but removing READA definitely fixed it.

It was bug 213921 in the RH bugzilla, and I think it had been fixed in
other ways in mainline by the time we found it.

-chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why does __do_page_cache_readahead submit READ, not READA?

[Wu Fengguang]

On Thu, Jul 30, 2009 at 08:06:49AM +0200, Jens Axboe wrote:
> On Wed, Jul 29 2009, Chris Mason wrote:
> > On Wed, Jul 29, 2009 at 11:18:45PM +0200, Jens Axboe wrote:
> > > On Wed, Jul 29 2009, Lars Ellenberg wrote:
> > > > I naively assumed, from the "readahead" in the name, that readahead
> > > > would be submitting READA bios. It does not.
> > > > 
> > > > I recently did some statistics on how many READ and READA requests
> > > > we actually see on the block device level.
> > > > I was suprised that READA is basically only used for file system
> > > > internal meta data (and not even for all file systems),
> > > > but _never_ for file data.
> > > > 
> > > > A simple
> > > > 	dd if=bigfile of=/dev/null bs=4k count=1
> > > > will absolutely cause readahead of the configured amount, no problem.
> > > > But on the block device level, these are READ requests, where I'd
> > > > expected them to be READA requests, based on the name.
> > > > 
> > > > This is because __do_page_cache_readahead() calls read_pages(),
> > > > which in turn is mapping->a_ops->readpages(), or, as fallback,
> > > > mapping->a_ops->readpage().
> > > > 
> > > > On that level, all variants end up submitting as READ.
> > > > 
> > > > This may even be intentional.
> > > > But if so, I'd like to understand that.
> > > 
> > > I don't think it's intentional, and if memory serves, we used to use
> > > READA when submitting read-ahead. Not sure how best to improve the
> > > situation, since (as you describe), we lose the read-ahead vs normal
> > > read at that level. I did some experimentation some time ago for
> > > flagging this, see:
> > > 
> > > http://git.kernel.dk/?p=linux-2.6-block.git;a=commitdiff;h=16cfe64e3568cda412b3cf6b7b891331946b595e
> > > 
> > > which should pass down READA properly.
> > 
> > One of the problems in the past was that reada would fail if there
> > wasn't a free request when we actually wanted it to go ahead and wait.
> > Or something.  We've switched it around a few times I think.
> 
> Yes, we did used to do that, whether it was 2.2 or 2.4 I
> don't recall :-)
> 
> It should be safe to enable know, whether there's a prettier way
> than the above, I don't know. It works by detecting the read-ahead
> marker, but it's a bit of a fragile design.

Another consideration is io-priority reversion and the overheads
required to avoid it:

        readahead(pages A-Z)    => READA IO for pages A-Z
        <short time later>
        read(page A) => blocked => find the request that contains page A
                                   and requeue/kick it as READ IO

The page-to-request lookups are not always required but nevertheless
the complexity and overheads won't be trivial.

The page-to-request lookup feature would be also useful for "advanced"
features like io-canceling (if implemented, hwpoison could be its
first user ;)

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why does __do_page_cache_readahead submit READ, not READA?

[Jens Axboe]

On Mon, Aug 03 2009, Wu Fengguang wrote:
> On Thu, Jul 30, 2009 at 08:06:49AM +0200, Jens Axboe wrote:
> > On Wed, Jul 29 2009, Chris Mason wrote:
> > > On Wed, Jul 29, 2009 at 11:18:45PM +0200, Jens Axboe wrote:
> > > > On Wed, Jul 29 2009, Lars Ellenberg wrote:
> > > > > I naively assumed, from the "readahead" in the name, that readahead
> > > > > would be submitting READA bios. It does not.
> > > > > 
> > > > > I recently did some statistics on how many READ and READA requests
> > > > > we actually see on the block device level.
> > > > > I was suprised that READA is basically only used for file system
> > > > > internal meta data (and not even for all file systems),
> > > > > but _never_ for file data.
> > > > > 
> > > > > A simple
> > > > > 	dd if=bigfile of=/dev/null bs=4k count=1
> > > > > will absolutely cause readahead of the configured amount, no problem.
> > > > > But on the block device level, these are READ requests, where I'd
> > > > > expected them to be READA requests, based on the name.
> > > > > 
> > > > > This is because __do_page_cache_readahead() calls read_pages(),
> > > > > which in turn is mapping->a_ops->readpages(), or, as fallback,
> > > > > mapping->a_ops->readpage().
> > > > > 
> > > > > On that level, all variants end up submitting as READ.
> > > > > 
> > > > > This may even be intentional.
> > > > > But if so, I'd like to understand that.
> > > > 
> > > > I don't think it's intentional, and if memory serves, we used to use
> > > > READA when submitting read-ahead. Not sure how best to improve the
> > > > situation, since (as you describe), we lose the read-ahead vs normal
> > > > read at that level. I did some experimentation some time ago for
> > > > flagging this, see:
> > > > 
> > > > http://git.kernel.dk/?p=linux-2.6-block.git;a=commitdiff;h=16cfe64e3568cda412b3cf6b7b891331946b595e
> > > > 
> > > > which should pass down READA properly.
> > > 
> > > One of the problems in the past was that reada would fail if there
> > > wasn't a free request when we actually wanted it to go ahead and wait.
> > > Or something.  We've switched it around a few times I think.
> > 
> > Yes, we did used to do that, whether it was 2.2 or 2.4 I
> > don't recall :-)
> > 
> > It should be safe to enable know, whether there's a prettier way
> > than the above, I don't know. It works by detecting the read-ahead
> > marker, but it's a bit of a fragile design.
> 
> Another consideration is io-priority reversion and the overheads
> required to avoid it:
> 
>         readahead(pages A-Z)    => READA IO for pages A-Z
>         <short time later>
>         read(page A) => blocked => find the request that contains page A
>                                    and requeue/kick it as READ IO
> 
> The page-to-request lookups are not always required but nevertheless
> the complexity and overheads won't be trivial.
> 
> The page-to-request lookup feature would be also useful for "advanced"
> features like io-canceling (if implemented, hwpoison could be its
> first user ;)

I added that 3-4 years ago or so, to experiment with in-kernel
cancellation for things like truncate(). Tracking pages is not cheap,
and since the write cancelling wasn't really very sucessful, I didn't go
ahead with it.

So I'm not sure it's a viable alternative, even if we restricted it to
just tracking READA's, for instance.

But I don't think we have any priority inversion to worry about, at
least not from the CFQ perspective.

-- 
Jens Axboe

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why does __do_page_cache_readahead submit READ, not READA?

[Wu Fengguang]

On Mon, Aug 03, 2009 at 03:59:33PM +0800, Jens Axboe wrote:
> On Mon, Aug 03 2009, Wu Fengguang wrote:
> > On Thu, Jul 30, 2009 at 08:06:49AM +0200, Jens Axboe wrote:
> > > On Wed, Jul 29 2009, Chris Mason wrote:
> > > > On Wed, Jul 29, 2009 at 11:18:45PM +0200, Jens Axboe wrote:
> > > > > On Wed, Jul 29 2009, Lars Ellenberg wrote:
> > > > > > I naively assumed, from the "readahead" in the name, that readahead
> > > > > > would be submitting READA bios. It does not.
> > > > > > 
> > > > > > I recently did some statistics on how many READ and READA requests
> > > > > > we actually see on the block device level.
> > > > > > I was suprised that READA is basically only used for file system
> > > > > > internal meta data (and not even for all file systems),
> > > > > > but _never_ for file data.
> > > > > > 
> > > > > > A simple
> > > > > > 	dd if=bigfile of=/dev/null bs=4k count=1
> > > > > > will absolutely cause readahead of the configured amount, no problem.
> > > > > > But on the block device level, these are READ requests, where I'd
> > > > > > expected them to be READA requests, based on the name.
> > > > > > 
> > > > > > This is because __do_page_cache_readahead() calls read_pages(),
> > > > > > which in turn is mapping->a_ops->readpages(), or, as fallback,
> > > > > > mapping->a_ops->readpage().
> > > > > > 
> > > > > > On that level, all variants end up submitting as READ.
> > > > > > 
> > > > > > This may even be intentional.
> > > > > > But if so, I'd like to understand that.
> > > > > 
> > > > > I don't think it's intentional, and if memory serves, we used to use
> > > > > READA when submitting read-ahead. Not sure how best to improve the
> > > > > situation, since (as you describe), we lose the read-ahead vs normal
> > > > > read at that level. I did some experimentation some time ago for
> > > > > flagging this, see:
> > > > > 
> > > > > http://git.kernel.dk/?p=linux-2.6-block.git;a=commitdiff;h=16cfe64e3568cda412b3cf6b7b891331946b595e
> > > > > 
> > > > > which should pass down READA properly.
> > > > 
> > > > One of the problems in the past was that reada would fail if there
> > > > wasn't a free request when we actually wanted it to go ahead and wait.
> > > > Or something.  We've switched it around a few times I think.
> > > 
> > > Yes, we did used to do that, whether it was 2.2 or 2.4 I
> > > don't recall :-)
> > > 
> > > It should be safe to enable know, whether there's a prettier way
> > > than the above, I don't know. It works by detecting the read-ahead
> > > marker, but it's a bit of a fragile design.
> > 
> > Another consideration is io-priority reversion and the overheads
> > required to avoid it:
> > 
> >         readahead(pages A-Z)    => READA IO for pages A-Z
> >         <short time later>
> >         read(page A) => blocked => find the request that contains page A
> >                                    and requeue/kick it as READ IO
> > 
> > The page-to-request lookups are not always required but nevertheless
> > the complexity and overheads won't be trivial.
> > 
> > The page-to-request lookup feature would be also useful for "advanced"
> > features like io-canceling (if implemented, hwpoison could be its
> > first user ;)
> 
> I added that 3-4 years ago or so, to experiment with in-kernel
> cancellation for things like truncate(). Tracking pages is not cheap,
> and since the write cancelling wasn't really very sucessful, I didn't go
> ahead with it.

Ah OK.

> So I'm not sure it's a viable alternative, even if we restricted it to
> just tracking READA's, for instance.

Kind of agreed. I guess it won't benefit too much workloads to default
to READA; for most workloads it would be pure overheads if considering
priority inversion.

> But I don't think we have any priority inversion to worry about, at
> least not from the CFQ perspective.

The priority inversion problem showed up in an early attempt to do
boot time prefetching. I guess this problem was somehow circumvented
by limiting the prefetch depth and do prefetches in original read
order instead of disk location order (Arjan cc'ed).

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why does __do_page_cache_readahead submit READ, not READA?

[Jens Axboe]

On Mon, Aug 03 2009, Wu Fengguang wrote:
> On Mon, Aug 03, 2009 at 03:59:33PM +0800, Jens Axboe wrote:
> > On Mon, Aug 03 2009, Wu Fengguang wrote:
> > > On Thu, Jul 30, 2009 at 08:06:49AM +0200, Jens Axboe wrote:
> > > > On Wed, Jul 29 2009, Chris Mason wrote:
> > > > > On Wed, Jul 29, 2009 at 11:18:45PM +0200, Jens Axboe wrote:
> > > > > > On Wed, Jul 29 2009, Lars Ellenberg wrote:
> > > > > > > I naively assumed, from the "readahead" in the name, that readahead
> > > > > > > would be submitting READA bios. It does not.
> > > > > > > 
> > > > > > > I recently did some statistics on how many READ and READA requests
> > > > > > > we actually see on the block device level.
> > > > > > > I was suprised that READA is basically only used for file system
> > > > > > > internal meta data (and not even for all file systems),
> > > > > > > but _never_ for file data.
> > > > > > > 
> > > > > > > A simple
> > > > > > > 	dd if=bigfile of=/dev/null bs=4k count=1
> > > > > > > will absolutely cause readahead of the configured amount, no problem.
> > > > > > > But on the block device level, these are READ requests, where I'd
> > > > > > > expected them to be READA requests, based on the name.
> > > > > > > 
> > > > > > > This is because __do_page_cache_readahead() calls read_pages(),
> > > > > > > which in turn is mapping->a_ops->readpages(), or, as fallback,
> > > > > > > mapping->a_ops->readpage().
> > > > > > > 
> > > > > > > On that level, all variants end up submitting as READ.
> > > > > > > 
> > > > > > > This may even be intentional.
> > > > > > > But if so, I'd like to understand that.
> > > > > > 
> > > > > > I don't think it's intentional, and if memory serves, we used to use
> > > > > > READA when submitting read-ahead. Not sure how best to improve the
> > > > > > situation, since (as you describe), we lose the read-ahead vs normal
> > > > > > read at that level. I did some experimentation some time ago for
> > > > > > flagging this, see:
> > > > > > 
> > > > > > http://git.kernel.dk/?p=linux-2.6-block.git;a=commitdiff;h=16cfe64e3568cda412b3cf6b7b891331946b595e
> > > > > > 
> > > > > > which should pass down READA properly.
> > > > > 
> > > > > One of the problems in the past was that reada would fail if there
> > > > > wasn't a free request when we actually wanted it to go ahead and wait.
> > > > > Or something.  We've switched it around a few times I think.
> > > > 
> > > > Yes, we did used to do that, whether it was 2.2 or 2.4 I
> > > > don't recall :-)
> > > > 
> > > > It should be safe to enable know, whether there's a prettier way
> > > > than the above, I don't know. It works by detecting the read-ahead
> > > > marker, but it's a bit of a fragile design.
> > > 
> > > Another consideration is io-priority reversion and the overheads
> > > required to avoid it:
> > > 
> > >         readahead(pages A-Z)    => READA IO for pages A-Z
> > >         <short time later>
> > >         read(page A) => blocked => find the request that contains page A
> > >                                    and requeue/kick it as READ IO
> > > 
> > > The page-to-request lookups are not always required but nevertheless
> > > the complexity and overheads won't be trivial.
> > > 
> > > The page-to-request lookup feature would be also useful for "advanced"
> > > features like io-canceling (if implemented, hwpoison could be its
> > > first user ;)
> > 
> > I added that 3-4 years ago or so, to experiment with in-kernel
> > cancellation for things like truncate(). Tracking pages is not cheap,
> > and since the write cancelling wasn't really very sucessful, I didn't go
> > ahead with it.
> 
> Ah OK.
> 
> > So I'm not sure it's a viable alternative, even if we restricted it to
> > just tracking READA's, for instance.
> 
> Kind of agreed. I guess it won't benefit too much workloads to default
> to READA; for most workloads it would be pure overheads if considering
> priority inversion.
> 
> > But I don't think we have any priority inversion to worry about, at
> > least not from the CFQ perspective.
> 
> The priority inversion problem showed up in an early attempt to do
> boot time prefetching. I guess this problem was somehow circumvented
> by limiting the prefetch depth and do prefetches in original read
> order instead of disk location order (Arjan cc'ed).

But was that not due to the prefetcher running at a lower cpu priority?
Just flagging a reada hint will not change your priority in the IO
scheduler, so we should have no priority inversion there.

-- 
Jens Axboe

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why does __do_page_cache_readahead submit READ, not READA?

[Wu Fengguang]

On Mon, Aug 03, 2009 at 05:25:15PM +0800, Jens Axboe wrote:
> On Mon, Aug 03 2009, Wu Fengguang wrote:
> > On Mon, Aug 03, 2009 at 03:59:33PM +0800, Jens Axboe wrote:
> > > On Mon, Aug 03 2009, Wu Fengguang wrote:
> > > > On Thu, Jul 30, 2009 at 08:06:49AM +0200, Jens Axboe wrote:
> > > > > > > read at that level. I did some experimentation some time ago for
> > > > > > > flagging this, see:
> > > > > > > 
> > > > > > > http://git.kernel.dk/?p=linux-2.6-block.git;a=commitdiff;h=16cfe64e3568cda412b3cf6b7b891331946b595e
> > > > > > > 
> > > > > > > which should pass down READA properly.
> > > > > > 
> > > > > > One of the problems in the past was that reada would fail if there
> > > > > > wasn't a free request when we actually wanted it to go ahead and wait.
> > > > > > Or something.  We've switched it around a few times I think.
> > > > > 
> > > > > Yes, we did used to do that, whether it was 2.2 or 2.4 I
> > > > > don't recall :-)
> > > > > 
> > > > > It should be safe to enable know, whether there's a prettier way
> > > > > than the above, I don't know. It works by detecting the read-ahead
> > > > > marker, but it's a bit of a fragile design.
> > > > 
> > > > Another consideration is io-priority reversion and the overheads
> > > > required to avoid it:
> > > > 
> > > >         readahead(pages A-Z)    => READA IO for pages A-Z
> > > >         <short time later>
> > > >         read(page A) => blocked => find the request that contains page A
> > > >                                    and requeue/kick it as READ IO
> > > > 
> > > > The page-to-request lookups are not always required but nevertheless
> > > > the complexity and overheads won't be trivial.
> > > > 
> > > > The page-to-request lookup feature would be also useful for "advanced"
> > > > features like io-canceling (if implemented, hwpoison could be its
> > > > first user ;)
> > > 
> > > I added that 3-4 years ago or so, to experiment with in-kernel
> > > cancellation for things like truncate(). Tracking pages is not cheap,
> > > and since the write cancelling wasn't really very sucessful, I didn't go
> > > ahead with it.
> > 
> > Ah OK.
> > 
> > > So I'm not sure it's a viable alternative, even if we restricted it to
> > > just tracking READA's, for instance.
> > 
> > Kind of agreed. I guess it won't benefit too much workloads to default
> > to READA; for most workloads it would be pure overheads if considering
> > priority inversion.
> > 
> > > But I don't think we have any priority inversion to worry about, at
> > > least not from the CFQ perspective.
> > 
> > The priority inversion problem showed up in an early attempt to do
> > boot time prefetching. I guess this problem was somehow circumvented
> > by limiting the prefetch depth and do prefetches in original read
> > order instead of disk location order (Arjan cc'ed).
> 
> But was that not due to the prefetcher running at a lower cpu priority?

Yes, it is. Thus the priority inversion problem.

> Just flagging a reada hint will not change your priority in the IO
> scheduler, so we should have no priority inversion there.

Ah OK. So READA merely means "don't try hard on error" for now.
Sorry I implicitly associated it with some priority class..

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why does __do_page_cache_readahead submit READ, not READA?

[Jens Axboe]

On Mon, Aug 03 2009, Wu Fengguang wrote:
> On Mon, Aug 03, 2009 at 05:25:15PM +0800, Jens Axboe wrote:
> > On Mon, Aug 03 2009, Wu Fengguang wrote:
> > > On Mon, Aug 03, 2009 at 03:59:33PM +0800, Jens Axboe wrote:
> > > > On Mon, Aug 03 2009, Wu Fengguang wrote:
> > > > > On Thu, Jul 30, 2009 at 08:06:49AM +0200, Jens Axboe wrote:
> > > > > > > > read at that level. I did some experimentation some time ago for
> > > > > > > > flagging this, see:
> > > > > > > > 
> > > > > > > > http://git.kernel.dk/?p=linux-2.6-block.git;a=commitdiff;h=16cfe64e3568cda412b3cf6b7b891331946b595e
> > > > > > > > 
> > > > > > > > which should pass down READA properly.
> > > > > > > 
> > > > > > > One of the problems in the past was that reada would fail if there
> > > > > > > wasn't a free request when we actually wanted it to go ahead and wait.
> > > > > > > Or something.  We've switched it around a few times I think.
> > > > > > 
> > > > > > Yes, we did used to do that, whether it was 2.2 or 2.4 I
> > > > > > don't recall :-)
> > > > > > 
> > > > > > It should be safe to enable know, whether there's a prettier way
> > > > > > than the above, I don't know. It works by detecting the read-ahead
> > > > > > marker, but it's a bit of a fragile design.
> > > > > 
> > > > > Another consideration is io-priority reversion and the overheads
> > > > > required to avoid it:
> > > > > 
> > > > >         readahead(pages A-Z)    => READA IO for pages A-Z
> > > > >         <short time later>
> > > > >         read(page A) => blocked => find the request that contains page A
> > > > >                                    and requeue/kick it as READ IO
> > > > > 
> > > > > The page-to-request lookups are not always required but nevertheless
> > > > > the complexity and overheads won't be trivial.
> > > > > 
> > > > > The page-to-request lookup feature would be also useful for "advanced"
> > > > > features like io-canceling (if implemented, hwpoison could be its
> > > > > first user ;)
> > > > 
> > > > I added that 3-4 years ago or so, to experiment with in-kernel
> > > > cancellation for things like truncate(). Tracking pages is not cheap,
> > > > and since the write cancelling wasn't really very sucessful, I didn't go
> > > > ahead with it.
> > > 
> > > Ah OK.
> > > 
> > > > So I'm not sure it's a viable alternative, even if we restricted it to
> > > > just tracking READA's, for instance.
> > > 
> > > Kind of agreed. I guess it won't benefit too much workloads to default
> > > to READA; for most workloads it would be pure overheads if considering
> > > priority inversion.
> > > 
> > > > But I don't think we have any priority inversion to worry about, at
> > > > least not from the CFQ perspective.
> > > 
> > > The priority inversion problem showed up in an early attempt to do
> > > boot time prefetching. I guess this problem was somehow circumvented
> > > by limiting the prefetch depth and do prefetches in original read
> > > order instead of disk location order (Arjan cc'ed).
> > 
> > But was that not due to the prefetcher running at a lower cpu priority?
> 
> Yes, it is. Thus the priority inversion problem.
> 
> > Just flagging a reada hint will not change your priority in the IO
> > scheduler, so we should have no priority inversion there.
> 
> Ah OK. So READA merely means "don't try hard on error" for now.
> Sorry I implicitly associated it with some priority class..

Well not necessarily, it could also have some priority implications in
the scheduler. My point is just that it need not be severe enough to
introduce priority inversions, so that we need a specific tracking
framework to graduate READA to READ.

-- 
Jens Axboe

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why does __do_page_cache_readahead submit READ, not READA?

[Wu Fengguang]

On Mon, Aug 03, 2009 at 05:37:53PM +0800, Jens Axboe wrote:
> On Mon, Aug 03 2009, Wu Fengguang wrote:
> > On Mon, Aug 03, 2009 at 05:25:15PM +0800, Jens Axboe wrote:
> > > On Mon, Aug 03 2009, Wu Fengguang wrote:
> > > > On Mon, Aug 03, 2009 at 03:59:33PM +0800, Jens Axboe wrote:
> > > > > On Mon, Aug 03 2009, Wu Fengguang wrote:
> > > > > > On Thu, Jul 30, 2009 at 08:06:49AM +0200, Jens Axboe wrote:
> > > > > > > > > read at that level. I did some experimentation some time ago for
> > > > > > > > > flagging this, see:
> > > > > > > > > 
> > > > > > > > > http://git.kernel.dk/?p=linux-2.6-block.git;a=commitdiff;h=16cfe64e3568cda412b3cf6b7b891331946b595e
> > > > > > > > > 
> > > > > > > > > which should pass down READA properly.
> > > > > > > > 
> > > > > > > > One of the problems in the past was that reada would fail if there
> > > > > > > > wasn't a free request when we actually wanted it to go ahead and wait.
> > > > > > > > Or something.  We've switched it around a few times I think.
> > > > > > > 
> > > > > > > Yes, we did used to do that, whether it was 2.2 or 2.4 I
> > > > > > > don't recall :-)
> > > > > > > 
> > > > > > > It should be safe to enable know, whether there's a prettier way
> > > > > > > than the above, I don't know. It works by detecting the read-ahead
> > > > > > > marker, but it's a bit of a fragile design.
> > > > > > 
> > > > > > Another consideration is io-priority reversion and the overheads
> > > > > > required to avoid it:
> > > > > > 
> > > > > >         readahead(pages A-Z)    => READA IO for pages A-Z
> > > > > >         <short time later>
> > > > > >         read(page A) => blocked => find the request that contains page A
> > > > > >                                    and requeue/kick it as READ IO
> > > > > > 
> > > > > > The page-to-request lookups are not always required but nevertheless
> > > > > > the complexity and overheads won't be trivial.
> > > > > > 
> > > > > > The page-to-request lookup feature would be also useful for "advanced"
> > > > > > features like io-canceling (if implemented, hwpoison could be its
> > > > > > first user ;)
> > > > > 
> > > > > I added that 3-4 years ago or so, to experiment with in-kernel
> > > > > cancellation for things like truncate(). Tracking pages is not cheap,
> > > > > and since the write cancelling wasn't really very sucessful, I didn't go
> > > > > ahead with it.
> > > > 
> > > > Ah OK.
> > > > 
> > > > > So I'm not sure it's a viable alternative, even if we restricted it to
> > > > > just tracking READA's, for instance.
> > > > 
> > > > Kind of agreed. I guess it won't benefit too much workloads to default
> > > > to READA; for most workloads it would be pure overheads if considering
> > > > priority inversion.
> > > > 
> > > > > But I don't think we have any priority inversion to worry about, at
> > > > > least not from the CFQ perspective.
> > > > 
> > > > The priority inversion problem showed up in an early attempt to do
> > > > boot time prefetching. I guess this problem was somehow circumvented
> > > > by limiting the prefetch depth and do prefetches in original read
> > > > order instead of disk location order (Arjan cc'ed).
> > > 
> > > But was that not due to the prefetcher running at a lower cpu priority?
> > 
> > Yes, it is. Thus the priority inversion problem.
> > 
> > > Just flagging a reada hint will not change your priority in the IO
> > > scheduler, so we should have no priority inversion there.
> > 
> > Ah OK. So READA merely means "don't try hard on error" for now.
> > Sorry I implicitly associated it with some priority class..
> 
> Well not necessarily, it could also have some priority implications in
> the scheduler. My point is just that it need not be severe enough to
> introduce priority inversions, so that we need a specific tracking
> framework to graduate READA to READ.

Right, that's a good point. 

Thanks,
Fengguang

Re: [dm-devel] Re: Why does __do_page_cache_readahead submit READ, not READA?

[James Bottomley]

On Mon, 2009-08-03 at 09:59 +0200, Jens Axboe wrote:
> On Mon, Aug 03 2009, Wu Fengguang wrote:
> > On Thu, Jul 30, 2009 at 08:06:49AM +0200, Jens Axboe wrote:
> > > On Wed, Jul 29 2009, Chris Mason wrote:
> > > > On Wed, Jul 29, 2009 at 11:18:45PM +0200, Jens Axboe wrote:
> > > > > On Wed, Jul 29 2009, Lars Ellenberg wrote:
> > > > > > I naively assumed, from the "readahead" in the name, that readahead
> > > > > > would be submitting READA bios. It does not.
> > > > > > 
> > > > > > I recently did some statistics on how many READ and READA requests
> > > > > > we actually see on the block device level.
> > > > > > I was suprised that READA is basically only used for file system
> > > > > > internal meta data (and not even for all file systems),
> > > > > > but _never_ for file data.
> > > > > > 
> > > > > > A simple
> > > > > > 	dd if=bigfile of=/dev/null bs=4k count=1
> > > > > > will absolutely cause readahead of the configured amount, no problem.
> > > > > > But on the block device level, these are READ requests, where I'd
> > > > > > expected them to be READA requests, based on the name.
> > > > > > 
> > > > > > This is because __do_page_cache_readahead() calls read_pages(),
> > > > > > which in turn is mapping->a_ops->readpages(), or, as fallback,
> > > > > > mapping->a_ops->readpage().
> > > > > > 
> > > > > > On that level, all variants end up submitting as READ.
> > > > > > 
> > > > > > This may even be intentional.
> > > > > > But if so, I'd like to understand that.
> > > > > 
> > > > > I don't think it's intentional, and if memory serves, we used to use
> > > > > READA when submitting read-ahead. Not sure how best to improve the
> > > > > situation, since (as you describe), we lose the read-ahead vs normal
> > > > > read at that level. I did some experimentation some time ago for
> > > > > flagging this, see:
> > > > > 
> > > > > http://git.kernel.dk/?p=linux-2.6-block.git;a=commitdiff;h=16cfe64e3568cda412b3cf6b7b891331946b595e
> > > > > 
> > > > > which should pass down READA properly.
> > > > 
> > > > One of the problems in the past was that reada would fail if there
> > > > wasn't a free request when we actually wanted it to go ahead and wait.
> > > > Or something.  We've switched it around a few times I think.
> > > 
> > > Yes, we did used to do that, whether it was 2.2 or 2.4 I
> > > don't recall :-)
> > > 
> > > It should be safe to enable know, whether there's a prettier way
> > > than the above, I don't know. It works by detecting the read-ahead
> > > marker, but it's a bit of a fragile design.
> > 
> > Another consideration is io-priority reversion and the overheads
> > required to avoid it:
> > 
> >         readahead(pages A-Z)    => READA IO for pages A-Z
> >         <short time later>
> >         read(page A) => blocked => find the request that contains page A
> >                                    and requeue/kick it as READ IO
> > 
> > The page-to-request lookups are not always required but nevertheless
> > the complexity and overheads won't be trivial.
> > 
> > The page-to-request lookup feature would be also useful for "advanced"
> > features like io-canceling (if implemented, hwpoison could be its
> > first user ;)
> 
> I added that 3-4 years ago or so, to experiment with in-kernel
> cancellation for things like truncate(). Tracking pages is not cheap,
> and since the write cancelling wasn't really very sucessful, I didn't go
> ahead with it.

Cancellation also came up several years ago with aio, which also has
requirements for it.

> So I'm not sure it's a viable alternative, even if we restricted it to
> just tracking READA's, for instance.
> 
> But I don't think we have any priority inversion to worry about, at
> least not from the CFQ perspective.

The basic problem with cancellation when implemented at the storage
layer is that its an unusual operation.  The storage primitives which
implement it aren't often invoked, so there's a lot of wariness to
implementing them in practice.  For example, although SCSI has the abort
function, it's not implemented by a lot of controllers, so we'd have to
drop all pending I/O on the floor with a reset and then try and pick up
the pieces we wanted.  Also, cancellation is racy since you never quite
know if the I/O hit the storage or not.

On the back of this, we thought in 2003 or so that the best way to
implement cancellation was simply to do nothing and have something wait
around for the completion and throw it away.  This has exactly the same
properties as storage implemented cancellation, but the benefit is that
we don't have to perturb the storage stack to do it.

James

Re: [dm-devel] Re: Why does __do_page_cache_readahead submit READ, not READA?

[Jens Axboe]

(not sure why your mailer stripped me from the CC?)


On Mon, Aug 03 2009, James Bottomley wrote:
> On Mon, 2009-08-03 at 09:59 +0200, Jens Axboe wrote:
> > On Mon, Aug 03 2009, Wu Fengguang wrote:
> > > On Thu, Jul 30, 2009 at 08:06:49AM +0200, Jens Axboe wrote:
> > > > On Wed, Jul 29 2009, Chris Mason wrote:
> > > > > On Wed, Jul 29, 2009 at 11:18:45PM +0200, Jens Axboe wrote:
> > > > > > On Wed, Jul 29 2009, Lars Ellenberg wrote:
> > > > > > > I naively assumed, from the "readahead" in the name, that readahead
> > > > > > > would be submitting READA bios. It does not.
> > > > > > > 
> > > > > > > I recently did some statistics on how many READ and READA requests
> > > > > > > we actually see on the block device level.
> > > > > > > I was suprised that READA is basically only used for file system
> > > > > > > internal meta data (and not even for all file systems),
> > > > > > > but _never_ for file data.
> > > > > > > 
> > > > > > > A simple
> > > > > > > 	dd if=bigfile of=/dev/null bs=4k count=1
> > > > > > > will absolutely cause readahead of the configured amount, no problem.
> > > > > > > But on the block device level, these are READ requests, where I'd
> > > > > > > expected them to be READA requests, based on the name.
> > > > > > > 
> > > > > > > This is because __do_page_cache_readahead() calls read_pages(),
> > > > > > > which in turn is mapping->a_ops->readpages(), or, as fallback,
> > > > > > > mapping->a_ops->readpage().
> > > > > > > 
> > > > > > > On that level, all variants end up submitting as READ.
> > > > > > > 
> > > > > > > This may even be intentional.
> > > > > > > But if so, I'd like to understand that.
> > > > > > 
> > > > > > I don't think it's intentional, and if memory serves, we used to use
> > > > > > READA when submitting read-ahead. Not sure how best to improve the
> > > > > > situation, since (as you describe), we lose the read-ahead vs normal
> > > > > > read at that level. I did some experimentation some time ago for
> > > > > > flagging this, see:
> > > > > > 
> > > > > > http://git.kernel.dk/?p=linux-2.6-block.git;a=commitdiff;h=16cfe64e3568cda412b3cf6b7b891331946b595e
> > > > > > 
> > > > > > which should pass down READA properly.
> > > > > 
> > > > > One of the problems in the past was that reada would fail if there
> > > > > wasn't a free request when we actually wanted it to go ahead and wait.
> > > > > Or something.  We've switched it around a few times I think.
> > > > 
> > > > Yes, we did used to do that, whether it was 2.2 or 2.4 I
> > > > don't recall :-)
> > > > 
> > > > It should be safe to enable know, whether there's a prettier way
> > > > than the above, I don't know. It works by detecting the read-ahead
> > > > marker, but it's a bit of a fragile design.
> > > 
> > > Another consideration is io-priority reversion and the overheads
> > > required to avoid it:
> > > 
> > >         readahead(pages A-Z)    => READA IO for pages A-Z
> > >         <short time later>
> > >         read(page A) => blocked => find the request that contains page A
> > >                                    and requeue/kick it as READ IO
> > > 
> > > The page-to-request lookups are not always required but nevertheless
> > > the complexity and overheads won't be trivial.
> > > 
> > > The page-to-request lookup feature would be also useful for "advanced"
> > > features like io-canceling (if implemented, hwpoison could be its
> > > first user ;)
> > 
> > I added that 3-4 years ago or so, to experiment with in-kernel
> > cancellation for things like truncate(). Tracking pages is not cheap,
> > and since the write cancelling wasn't really very sucessful, I didn't go
> > ahead with it.
> 
> Cancellation also came up several years ago with aio, which also has
> requirements for it.

Are they any real-world use cases for cancellation with aio?

> > So I'm not sure it's a viable alternative, even if we restricted it to
> > just tracking READA's, for instance.
> > 
> > But I don't think we have any priority inversion to worry about, at
> > least not from the CFQ perspective.
> 
> The basic problem with cancellation when implemented at the storage
> layer is that its an unusual operation.  The storage primitives which
> implement it aren't often invoked, so there's a lot of wariness to
> implementing them in practice.  For example, although SCSI has the abort
> function, it's not implemented by a lot of controllers, so we'd have to
> drop all pending I/O on the floor with a reset and then try and pick up
> the pieces we wanted.  Also, cancellation is racy since you never quite
> know if the I/O hit the storage or not.
> 
> On the back of this, we thought in 2003 or so that the best way to
> implement cancellation was simply to do nothing and have something wait
> around for the completion and throw it away.  This has exactly the same
> properties as storage implemented cancellation, but the benefit is that
> we don't have to perturb the storage stack to do it.

The approach I took (and exactly for the reasons you outline I think
it's the only feasible one) is to allow cancellations of unstarted IO
only. I can just imagine the pandora box we would open if we where to do
this at the hardware/device level as well, so just chose not to go that
far.

-- 
Jens Axboe

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>