Re: CFQ and dm-crypt

Re: CFQ and dm-crypt

[Milan Broz]

On 10/24/2010 03:51 PM, Richard Kralovic wrote:
> CFQ io scheduler relies on using task_struct current to determine which
> process makes the io request. On the other hand, some dm modules (such
> as dm-crypt) use separate threads for doing io. As CFQ sees only these
> threads, it provides a very poor performance in such a case.
> 
> IMHO the correct solution for this would be to store, for every io
> request, the process that initiated it (and preserve this information
> while the request is processed by device mapper). Would that be feasible?

Yes, this seems to be correct solution. I think this should be
handled by core device-mapper (as you noted, more dm targets using
threads to process.)


> Other possibility is to avoid using separate threads for doing io in dm
> modules. The attached patch (against 2.6.36) modifies dm-crypt in this
> way, what results into much better behavior of cfq (e.g., io priorities
> work correctly).

Sorry, this completely dismantles the way how dm-crypt solves problems
with stacking dm devices.
Basically it reintroduces possible deadlocks for low memory
situations (the reason why there are these threads).

Milan

Re: CFQ and dm-crypt

[Richard Kralovic]

On 10/24/10 18:15, Milan Broz wrote:
> On 10/24/2010 03:51 PM, Richard Kralovic wrote:
>> CFQ io scheduler relies on using task_struct current to determine which
>> process makes the io request. On the other hand, some dm modules (such
>> as dm-crypt) use separate threads for doing io. As CFQ sees only these
>> threads, it provides a very poor performance in such a case.
>>
>> IMHO the correct solution for this would be to store, for every io
>> request, the process that initiated it (and preserve this information
>> while the request is processed by device mapper). Would that be feasible?
> 
> Yes, this seems to be correct solution. I think this should be
> handled by core device-mapper (as you noted, more dm targets using
> threads to process.)

Do you think it is possible to handle this in device-mapper, without any
support from the cfq code?

I also noticed that a solution for this problem was proposed a few years
ago by Hirokazu Takahashi (a patch for linux-2.6.25,
http://lkml.org/lkml/2008/4/22/193), but there was no response to it. Is
such an approach wrong?

>> Other possibility is to avoid using separate threads for doing io in dm
>> modules. The attached patch (against 2.6.36) modifies dm-crypt in this
>> way, what results into much better behavior of cfq (e.g., io priorities
>> work correctly).
> 
> Sorry, this completely dismantles the way how dm-crypt solves problems
> with stacking dm devices.
> Basically it reintroduces possible deadlocks for low memory
> situations (the reason why there are these threads).

Would the problem with deadlock be still present if the io worker queue
was used for writes only, but reads were issued directly? (Even this
would be a significant improvement for people using cfq and a full-disk
encryption over dm-crypt, since asynchronous writes are not supported by
cfq anyway.)

	Richard

Re: CFQ and dm-crypt

[Milan Broz]

On 10/25/2010 11:53 AM, Richard Kralovic wrote:

> Do you think it is possible to handle this in device-mapper, without any
> support from the cfq code?
> 
> I also noticed that a solution for this problem was proposed a few years
> ago by Hirokazu Takahashi (a patch for linux-2.6.25,
> http://lkml.org/lkml/2008/4/22/193), but there was no response to it. Is
> such an approach wrong?

Not sure but it will be discussed now.
There are more situations where the process id is lost because of other queue,
a lot of new recent code (including internal thread in dm-core).

>>> Other possibility is to avoid using separate threads for doing io in dm
>>> modules. The attached patch (against 2.6.36) modifies dm-crypt in this
>>> way, what results into much better behavior of cfq (e.g., io priorities
>>> work correctly).
>>
>> Sorry, this completely dismantles the way how dm-crypt solves problems
>> with stacking dm devices.
>> Basically it reintroduces possible deadlocks for low memory
>> situations (the reason why there are these threads).
> 
> Would the problem with deadlock be still present if the io worker queue
> was used for writes only, but reads were issued directly? (Even this
> would be a significant improvement for people using cfq and a full-disk
> encryption over dm-crypt, since asynchronous writes are not supported by
> cfq anyway.)

Sorry, both must be issued from separate thread, you must not block in
common crypt_map() call:

- READ must first allocate BIO clone for ciphertext data
(it can wait here - memory allocation, imagine it waits for swap -> swap is
on another crypt device -> deadlock)

- WRITES first run encryption thus must allocate memory too.
Moreover if encryption runs in async mode, it can block when encryption queue
is full (waiting for condition triggerred from async callback) -> again,
possible deadlock if in common thread.

Of course it will work most of the time, but the design must be robust even
for not common situations.

Milan

Re: CFQ and dm-crypt

[Jeff Moyer]

Milan Broz <mbroz@redhat.com> writes:

> On 10/25/2010 11:53 AM, Richard Kralovic wrote:
>
>> Do you think it is possible to handle this in device-mapper, without any
>> support from the cfq code?
>> 
>> I also noticed that a solution for this problem was proposed a few years
>> ago by Hirokazu Takahashi (a patch for linux-2.6.25,
>> http://lkml.org/lkml/2008/4/22/193), but there was no response to it. Is
>> such an approach wrong?
>
> Not sure but it will be discussed now.
> There are more situations where the process id is lost because of other queue,
> a lot of new recent code (including internal thread in dm-core).

It would probably be helpful to CC Jens (which I did).

I haven't looked very deeply at the approach in the referenced mail
thread (though I did look far enough to tell that the aio bits were
wrong), but in general I think this sort of thing is worthwhile.  I'm
not sure what the barriers are.  Jens?

Cheers,
Jeff

>>>> Other possibility is to avoid using separate threads for doing io in dm
>>>> modules. The attached patch (against 2.6.36) modifies dm-crypt in this
>>>> way, what results into much better behavior of cfq (e.g., io priorities
>>>> work correctly).
>>>
>>> Sorry, this completely dismantles the way how dm-crypt solves problems
>>> with stacking dm devices.
>>> Basically it reintroduces possible deadlocks for low memory
>>> situations (the reason why there are these threads).
>> 
>> Would the problem with deadlock be still present if the io worker queue
>> was used for writes only, but reads were issued directly? (Even this
>> would be a significant improvement for people using cfq and a full-disk
>> encryption over dm-crypt, since asynchronous writes are not supported by
>> cfq anyway.)
>
> Sorry, both must be issued from separate thread, you must not block in
> common crypt_map() call:
>
> - READ must first allocate BIO clone for ciphertext data
> (it can wait here - memory allocation, imagine it waits for swap -> swap is
> on another crypt device -> deadlock)
>
> - WRITES first run encryption thus must allocate memory too.
> Moreover if encryption runs in async mode, it can block when encryption queue
> is full (waiting for condition triggerred from async callback) -> again,
> possible deadlock if in common thread.
>
> Of course it will work most of the time, but the design must be robust even
> for not common situations.
>
> Milan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: CFQ and dm-crypt

[Vivek Goyal]

On Mon, Oct 25, 2010 at 11:53:39AM +0200, Richard Kralovic wrote:
> On 10/24/10 18:15, Milan Broz wrote:
> > On 10/24/2010 03:51 PM, Richard Kralovic wrote:
> >> CFQ io scheduler relies on using task_struct current to determine which
> >> process makes the io request. On the other hand, some dm modules (such
> >> as dm-crypt) use separate threads for doing io. As CFQ sees only these
> >> threads, it provides a very poor performance in such a case.
> >>
> >> IMHO the correct solution for this would be to store, for every io
> >> request, the process that initiated it (and preserve this information
> >> while the request is processed by device mapper). Would that be feasible?
> > 
> > Yes, this seems to be correct solution. I think this should be
> > handled by core device-mapper (as you noted, more dm targets using
> > threads to process.)
> 

Richard,

So what problem are you facing? I know you are referring to CFQ ioprio not
working with dm targets but how does it impact you? So it is not about
overall disk performance or any slow down with dm-crypt target but just
about prioritizing your IO over other?

> Do you think it is possible to handle this in device-mapper, without any
> support from the cfq code?
> 
> I also noticed that a solution for this problem was proposed a few years
> ago by Hirokazu Takahashi (a patch for linux-2.6.25,
> http://lkml.org/lkml/2008/4/22/193), but there was no response to it. Is
> such an approach wrong?

Conceptually it makes sense to put some kind of info in bio so that we
can associate with right context. I think above thread kind of of died
down.

Re-reading the thread now, and it looks like that Hirokazu also planned
to use this info for associating IO to right cgroup for WRITES. There
was alternative approach of "IO tracking" where IO controller cgroup
info was to be put in page_cgroup structure and once the bio is submitted
to CFQ, it will trace the page/page_cgroup for bio and extract the 
cgroup info and attribute IO to right group.

Storing some info in page_cgroup makes it dependent on memory controller
and which should not be a necessary thing for READS. For WRITES it probably
is still a necessary thing as it also provides (work in progress from greg)
per cgroup dirty ratio.

Storing some kind of io context info in bio makes sense to me. Not sure
if Jens has other ideas.

Thanks
Vivek
 

> 
> >> Other possibility is to avoid using separate threads for doing io in dm
> >> modules. The attached patch (against 2.6.36) modifies dm-crypt in this
> >> way, what results into much better behavior of cfq (e.g., io priorities
> >> work correctly).
> > 
> > Sorry, this completely dismantles the way how dm-crypt solves problems
> > with stacking dm devices.
> > Basically it reintroduces possible deadlocks for low memory
> > situations (the reason why there are these threads).
> 
> Would the problem with deadlock be still present if the io worker queue
> was used for writes only, but reads were issued directly? (Even this
> would be a significant improvement for people using cfq and a full-disk
> encryption over dm-crypt, since asynchronous writes are not supported by
> cfq anyway.)
> 
> 	Richard
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: CFQ and dm-crypt

[Richard Kralovic]

On 10/25/10 22:59, Vivek Goyal wrote:
> Richard,
> 
> So what problem are you facing? I know you are referring to CFQ ioprio not
> working with dm targets but how does it impact you? So it is not about
> overall disk performance or any slow down with dm-crypt target but just
> about prioritizing your IO over other?

The ioprio not working is probably the biggest problem (since it is used
quite a lot for background tasks like desktop indexing services). But
also the overall performance is worse. I didn't do a rigorous
benchmarking, but tried a following simple test to see the impact of my
dm-crypt patch:

test-write:

SIZE=640


KERN=`uname -r`
((time /bin/bash -c "dd if=/dev/zero bs=1M count=64 \
   of=normal.tst oflag=direct") 1>$KERN-write-normal 2>&1) |
((time /bin/bash -c "ionice -c 3 dd if=/dev/zero bs=1M \
   count=64 of=idle.tst oflag=direct") 1>$KERN-write-idle 2>&1)

Times for vanilla kernel (with CFQ) were 5.24s for idle and 5.38s for
normal, times for patched kernel were 4.9s for idle and 3.13s for
normal. A similar test for reading showed even bigger differences:
vanilla kernel has 8.5s for idle as well as 8.5s for normal, patched
kernel has 4.2s for idle and 2.1s for normal.

So it seems that CFQ is behaving really badly if it is not able to see
which process is doing the IO (and sees kcryptd everywhere). As far as I
understood, there is no point in using CFQ in that case and it is much
better to use other scheduler in this situation.


	Richard

Re: CFQ and dm-crypt

[Vivek Goyal]

On Tue, Oct 26, 2010 at 10:37:09AM +0200, Richard Kralovic wrote:
> On 10/25/10 22:59, Vivek Goyal wrote:
> > Richard,
> > 
> > So what problem are you facing? I know you are referring to CFQ ioprio not
> > working with dm targets but how does it impact you? So it is not about
> > overall disk performance or any slow down with dm-crypt target but just
> > about prioritizing your IO over other?
> 
> The ioprio not working is probably the biggest problem (since it is used
> quite a lot for background tasks like desktop indexing services). But
> also the overall performance is worse. I didn't do a rigorous
> benchmarking, but tried a following simple test to see the impact of my
> dm-crypt patch:
> 
> test-write:
> 
> SIZE=640
> 
> 
> KERN=`uname -r`
> ((time /bin/bash -c "dd if=/dev/zero bs=1M count=64 \
>    of=normal.tst oflag=direct") 1>$KERN-write-normal 2>&1) |
> ((time /bin/bash -c "ionice -c 3 dd if=/dev/zero bs=1M \
>    count=64 of=idle.tst oflag=direct") 1>$KERN-write-idle 2>&1)
> 
> Times for vanilla kernel (with CFQ) were 5.24s for idle and 5.38s for
> normal, times for patched kernel were 4.9s for idle and 3.13s for
> normal. A similar test for reading showed even bigger differences:
> vanilla kernel has 8.5s for idle as well as 8.5s for normal, patched
> kernel has 4.2s for idle and 2.1s for normal.
> 
> So it seems that CFQ is behaving really badly if it is not able to see
> which process is doing the IO (and sees kcryptd everywhere). As far as I
> understood, there is no point in using CFQ in that case and it is much
> better to use other scheduler in this situation.

Ok, so are you getting better results with noop and deadline?

So your bigger concerns seems to be not necessarily making ioprio and
class working but why there is a performance drop when dm-crypt starts
submitting IOs with the help of a worker thread and we lose original
context.

If you are getting better numbers say with noop, then I would think that
somehow we are idling a lot in CFQ (with dm-crypt) and it is overshadowing
the benefits of reduced seeks due to idling (if any). 

Is it possible to capture a trace with CFQ using blktrace. Say 30 second
trace for two cases. Vanilla CFQ and patched CFQ with normal case (Will
look into the case of IDLE later). I want to compare two traces and see
what changed in terms of idling.

One explanation could that your workload is sequential (dd case), and
by exposing the context to CFQ you are getting the idling right and
reducing some seeks. By submitting everything from kcryptd, I think it
practically becomes a seeky traffic (read/write intermixed) and increased
seeks reduce throughput. But if this is the case, same should be true
for noop and i do not understand why you would get better performance
with noop.

Anyway, looking at blktrace might give some idea.

Thanks
Vivek