Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Sven Eschenberg]

Erh, no, this would come down to 0.1 seconds (since it's 100th - centi).

And now, it does not start a flush, it only wakes the daemon up for
analysis and consideration. If, and how much is flushed depends on other
tuneables.
Unfortunately it's not that straight forward.

Regards

-Sven

P.S.: Sorry for the duplicate post with the wrong e-mail addy.

On Fri, September 18, 2009 23:09, Arno Wagner wrote:
>
> A pity. Have you tried to set dirty_writeback_centisecs to
> something very low, e.g. 10? If I understand this correctly that would
cause regular flushes to start after 1 sec.
>
>
> Arno
>
>

Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Arno Wagner]

Hmm. It used to be that easy a long time ago. Seems the really 
useful parameters are still hidden. I am more than a bit anoyed
by this. Does anybody here know where the eraly flush timeout
can be found? 

Arno


On Fri, Sep 18, 2009 at 11:21:32PM +0200, Sven Eschenberg wrote:
> Erh, no, this would come down to 0.1 seconds (since it's 100th - centi).
> 
> And now, it does not start a flush, it only wakes the daemon up for
> analysis and consideration. If, and how much is flushed depends on other
> tuneables.
> Unfortunately it's not that straight forward.
> 
> Regards
> 
> -Sven
> 
> P.S.: Sorry for the duplicate post with the wrong e-mail addy.
> 
> On Fri, September 18, 2009 23:09, Arno Wagner wrote:
> >
> > A pity. Have you tried to set dirty_writeback_centisecs to
> > something very low, e.g. 10? If I understand this correctly that would
> cause regular flushes to start after 1 sec.
> >
> >
> > Arno
> >
> >
> 
> 
> 
> 
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
> 

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Sven Eschenberg]

You are not alone there.

But I fear there is no single flush timeout, I was trying to hunt
something like it down for a while without success.

The tuneable you gave determines how often pdflush daemons wake up,
another one (with centisecs in its name) gives a bound (or threshold)
after what time a dirty page is eligable at all (I'd assume that this
one needs to be smaller, to force all pages that went dirty since the
last pdflush wakeup, to be queued for a flush). Another tuneable gives
some sort of a filling ratio of dirty pages (in respect to whole system
memory) at which a flush is done (always? only when a daemon was woken
up beforehand?). Unfortunately each tuneable is documented on it's own,
there is no info, if all additional criteria is or-ed, xor-ed or and-ed
(or even a mixture?) and there is no single equation giving the whole
picture (well except somewhere in the source probably) - at least I
couldn't find one.

Regards

-Sven

P.S.: And I might be wrong, but I fear the I/O scheduling queue
discipline could influence real world disk I/O patterns as well (In
respect to the original post).

On Sat, 2009-09-19 at 02:45 +0200, Arno Wagner wrote:
> Hmm. It used to be that easy a long time ago. Seems the really 
> useful parameters are still hidden. I am more than a bit anoyed
> by this. Does anybody here know where the eraly flush timeout
> can be found? 
> 
> Arno
> 
> 
> On Fri, Sep 18, 2009 at 11:21:32PM +0200, Sven Eschenberg wrote:
> > Erh, no, this would come down to 0.1 seconds (since it's 100th - centi).
> > 
> > And now, it does not start a flush, it only wakes the daemon up for
> > analysis and consideration. If, and how much is flushed depends on other
> > tuneables.
> > Unfortunately it's not that straight forward.
> > 
> > Regards
> > 
> > -Sven
> > 
> > P.S.: Sorry for the duplicate post with the wrong e-mail addy.
> > 
> > On Fri, September 18, 2009 23:09, Arno Wagner wrote:
> > >
> > > A pity. Have you tried to set dirty_writeback_centisecs to
> > > something very low, e.g. 10? If I understand this correctly that would
> > cause regular flushes to start after 1 sec.
> > >
> > >
> > > Arno
> > >
> > >
> > 
> > 
> > 
> > 
> > _______________________________________________
> > dm-crypt mailing list
> > dm-crypt@saout.de
> > http://www.saout.de/mailman/listinfo/dm-crypt
> > 
> 

Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Christian Pernegger]

Sorry for being silent so long, I did a lot of testing ... I've even
set up a dedicated testing box cobbled together from spare parts :)

1) It's not kvm, I took that out of the mix.
2) dm-crypt is at least a co-culprit since lvm on md works fine.
3) Tuning changing the I/O scheduler and/or the /proc/sys/vm/ settings
does little.

So I tried playing with the layering:

a) dm-crypt on lvm on md (1 kcryptd / LV): bad
b) lvm on dm-crypt on md (1 kcryptd): worse
c) lvm on md on dm-crypt: (1 kcryptd / physical disk): excellent

The last case does not have any abnormal starvation issues at all and
performance is near the raw speed for the array. It's a landslide,
even on the dual core testbox (~110 MiB/s per core, max. 60 MB/s per
disk).

If only it weren't so ugly. Data is now written n/(n-1) times in the
whole-stripe case, I don't even want to think about read-modify-write
... Multicore dm-crypt would be really nice to have now ...

I've opened an LKML thread in the hopes that something can be done
outside dm-crypt but so far, not much ...

Thanks,

Chris

Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Rick Moritz]

I think earlier you (or someone else in this thread?) were mentioning that the slowdown is a function of array speed (higher is worse) and processing power (slower is worse).
With AES hardware acceleration coming to mainstream x86 with the next generation of Intel boards, would you expect the issue to be allayed?
Is anyone running one of the first two scenarions on a machine with hardcoded AES instructions?
I expect the last scenario scales up to a core/disk ratio of one? Is the advantage still there, when you've only got one core available? (I expect that to be the case, because the issue appears to be of architectural provenance - somewhere the io-subsystem is inducing wait states...)

On Wed, 21 Oct 2009 12:48:55 +0200 Christian Pernegger <pernegger@gmail.com> wrote:

> Sorry for being silent so long, I did a lot of testing ... I've even
> set up a dedicated testing box cobbled together from spare parts :)
> 
> 1) It's not kvm, I took that out of the mix.
> 2) dm-crypt is at least a co-culprit since lvm on md works fine.
> 3) Tuning changing the I/O scheduler and/or the /proc/sys/vm/ settings
> does little.
> 
> So I tried playing with the layering:
> 
> a) dm-crypt on lvm on md (1 kcryptd / LV): bad
> b) lvm on dm-crypt on md (1 kcryptd): worse
> c) lvm on md on dm-crypt: (1 kcryptd / physical disk): excellent
> 
> The last case does not have any abnormal starvation issues at all and
> performance is near the raw speed for the array. It's a landslide,
> even on the dual core testbox (~110 MiB/s per core, max. 60 MB/s per
> disk).
> 
> If only it weren't so ugly. Data is now written n/(n-1) times in the
> whole-stripe case, I don't even want to think about read-modify-write
> ... Multicore dm-crypt would be really nice to have now ...
> 
> I've opened an LKML thread in the hopes that something can be done
> outside dm-crypt but so far, not much ...
> 
> Thanks,
> 
> Chris

Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Christian Pernegger]

> I think earlier you (or someone else in this thread?) were mentioning that the slowdown is a
> function of array speed (higher is worse) and processing power (slower is worse).

It certainly seems that way, though I don't have enough data-points to
be able to call it a function.
Actually I've tried to find a way to put a hard cap on a block
device's speed in order to test this but there doesn't seem to be an
easy solution, or one that would allow throtteling I/Os to/from the
dm-crypt mapper device itself.

> With AES hardware acceleration coming to mainstream x86 with the next generation of Intel
> boards, would you expect the issue to be allayed?

Maybe. I've no idea how much benefit AES-NI has in practice.

Hopefully that isn't going to be an excuse for not impementing
multi-threaded dm-crypt. Is there a technical/algorithmical reason why
multiple cores couldn't work on a single dm-crypt device? Even if
larger requests can't be (easily) split, maybe unrelated ones could?
Hell, even a fixed mapping of addresses to cores would probably help.

> I expect the last scenario scales up to a core/disk ratio of one?

The test box has only two cores, so that's 2:1. Won't be able to test
1:1 for a while since my only quadcore is in use right now.

> Is the advantage still there, when you've only got one core available?

Is there a way to tell the kernel to disable one core at boot? The
BIOS doesn't have an option for it on this board.

Thanks,

C.

Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Rick Moritz]

> > Is the advantage still there, when you've only got one core available?
> 
> Is there a way to tell the kernel to disable one core at boot? The
> BIOS doesn't have an option for it on this board.

You should be able to build a non-SMP kernel - that ought to restrict the kernel threads to serial execution.

Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Michael Gebetsroither]

* Rick Moritz <rahvin@gmail.com> wrote:
>> > Is the advantage still there, when you've only got one core available?
>> 
>> Is there a way to tell the kernel to disable one core at boot? The
>> BIOS doesn't have an option for it on this board.
>
> You should be able to build a non-SMP kernel - that ought to restrict the kernel threads to serial execution.

booting with maxcpus=0 or nosmp should do the same.

michael
-- 
It's already too late!

Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Christian Pernegger]

> Is the advantage still there, when you've only got one core available?

[Note that this isn't pretending to be a benchmark, just a few
ballpark figures based off a single bonnie++ run each. I'll redo it
properly on the weekend.]

Seq. reads:
2 cores: ~140MiB/s
1 core: ~100MiB/s
(single crypt dev: ~100MB/s)

Seq. writes:
2 cores: 90 MiB/s
1 core: 65 MiB/s
(single crypt dev: ~100MB/s)

Responiveness:
[run 3x dd if=/dev/zero of=testN bs=3M and then try to do something in
another ssh session]
2 cores: barely impacted, maybe 2s for 'top',  5s on 'ls /'
1 core: bad, ~10s for 'top', 35s for 'ls /'
When a command is executed twice in a row (i. e. before everything's
been thrown out of the cache again) it's instant in both cases.
The difference between '1 core' and 'single crypt dev' is hard to
describe. The former feels slooow but steady, the latter bursty.

Chris

[dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Christian Pernegger]

Hi again!

The very short version is that writing a larger stream of data to an
encrypted volume starves all reads on *all* encrypted volumes. Doesn't
matter if they reside on different disks or even different
controllers.

Example: Alice is watching a video (on a windows client using VLC to
access the file on a samba share, in case it matters). Then Bob starts
backing up his /home of multiple gigs using tar, at which point the
video turns into a slideshow. And that's putting it mildly ...

Since the unencrypted volumes are fine it seem dm-crypt is the culprit
but it might as well be any other layer in our storage setup or, more
likely, a combination.

Anyway, on the lowest level are a md-raid1 and two -raid5 arrays,
grouped into lvm VGs raid1 and raid5 respectively. Some of the LVs on
that are encrypted, some aren't. These LVs in turn serve as virtual
disks for a handful of kvm boxes (via virtio). OS is Debian 5.02
(lenny) using the 2.6.30 kernel from backports.org, running on an
Intel C2Q with 8GB of RAM.

I thought 2.6.30 would split the crypto work between the four cores,
but it doesn't. Performance maxes out just over 100MiB/s, which is
close to the crypto performance of one core. (openssl speed
aes-256-cbc gives me ~115MiB/s for one core, ~440MiB/s for all four.)
That's probably kvm's fault, since from the pov of the kernel doing
the I/O a whole virtual machine is just one process ... and all
parallelization, I/O scheduling etc. seems to go out of the window.

One of the VMs is our file server and it's doing fine except that
writes nearly drown out reads, see above. I've tried messing with the
I/O schedulers on the host and guest side, no change. One write stream
is enough to bring the whole disk subsystem to its knees. 15 secs to
execute /bin/ls as soon as one dd if=/dev/zero of=test bs=4M is
running ...

All I can think of at this point is that the disks are "too fast" for
the CPU. The VM sends down write requests continously, all of which
take cycles to encrypt before they even hit the I/O scheduling queue.
Now along comes a lonely read that has to wait ages for decryption
since the crypt processes are so busy with writes. Since the arrays
can write 2-3x faster than the CPU can encrypt, the (I/O scheduling)
queue never comes into action ...

Any ideas on how I could diagnose the problem are very much
appreciated, I'll gladly answer any request for further info.

Thanks,

Chris

Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Arno Wagner]

This sounds more like a general I/O flushing problem.
I uses to use a (slow) MO drive for backup and had this
all the time after the kernel developers removed easy
tuning of the filesystem flush paramaters. (Which I still
think was arrogant and stupid.) With earlier flush 
I did not run into emergency flushes. With the default 
paramaters I always ran into emergency flushes and 
the whole machine would lock up for minutes at a time. 
The lock-up is gone now, likely due to different locking
granularity, but I still miss my manually tuned filesystem
on occasion, when I run into exactly the starvation you 
describe.

It is possible that tuning the paramaters can be done 
again by now, I have not looked into it for some time.

I think the idea was that early writing has a bit lower 
performance, but is easier to interrupt.

So, possibly no connection to dm-crypt, just the
buffer-cache having badly tuned parameters for
this type of operation.

Arno


On Wed, Sep 16, 2009 at 01:54:12AM +0200, Christian Pernegger wrote:
> Hi again!
> 
> The very short version is that writing a larger stream of data to an
> encrypted volume starves all reads on *all* encrypted volumes. Doesn't
> matter if they reside on different disks or even different
> controllers.
> 
> Example: Alice is watching a video (on a windows client using VLC to
> access the file on a samba share, in case it matters). Then Bob starts
> backing up his /home of multiple gigs using tar, at which point the
> video turns into a slideshow. And that's putting it mildly ...
> 
> Since the unencrypted volumes are fine it seem dm-crypt is the culprit
> but it might as well be any other layer in our storage setup or, more
> likely, a combination.
> 
> Anyway, on the lowest level are a md-raid1 and two -raid5 arrays,
> grouped into lvm VGs raid1 and raid5 respectively. Some of the LVs on
> that are encrypted, some aren't. These LVs in turn serve as virtual
> disks for a handful of kvm boxes (via virtio). OS is Debian 5.02
> (lenny) using the 2.6.30 kernel from backports.org, running on an
> Intel C2Q with 8GB of RAM.
> 
> I thought 2.6.30 would split the crypto work between the four cores,
> but it doesn't. Performance maxes out just over 100MiB/s, which is
> close to the crypto performance of one core. (openssl speed
> aes-256-cbc gives me ~115MiB/s for one core, ~440MiB/s for all four.)
> That's probably kvm's fault, since from the pov of the kernel doing
> the I/O a whole virtual machine is just one process ... and all
> parallelization, I/O scheduling etc. seems to go out of the window.
> 
> One of the VMs is our file server and it's doing fine except that
> writes nearly drown out reads, see above. I've tried messing with the
> I/O schedulers on the host and guest side, no change. One write stream
> is enough to bring the whole disk subsystem to its knees. 15 secs to
> execute /bin/ls as soon as one dd if=/dev/zero of=test bs=4M is
> running ...
> 
> All I can think of at this point is that the disks are "too fast" for
> the CPU. The VM sends down write requests continously, all of which
> take cycles to encrypt before they even hit the I/O scheduling queue.
> Now along comes a lonely read that has to wait ages for decryption
> since the crypt processes are so busy with writes. Since the arrays
> can write 2-3x faster than the CPU can encrypt, the (I/O scheduling)
> queue never comes into action ...
> 
> Any ideas on how I could diagnose the problem are very much
> appreciated, I'll gladly answer any request for further info.
> 
> Thanks,
> 
> Chris
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
> 

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Christian Pernegger]

> So, possibly no connection to dm-crypt, just the
> buffer-cache having badly tuned parameters for
> this type of operation.

Quite possible. Normally the block devices kvm provides for guests are
backed by the host's cache, which results in native I/O throughput
(100MiB/s read and write on encrypted partitions) in the guests but
has a major starvation problem when dm-crypt is part of the mix.
Forcing kvm to go directly to the disks bypassing the host's cache
(using drive=...,cache=none) fixes the starvation issue completely but
limits throughput to an appaling 25MB/s read and 15MB/s write ...

Any idea why this setup would behave as it does? How can I check if
flushes really are the problem?

Thanks,

Chris

Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Christian Pernegger]

> Hmm. So without dm-crypt no problem? The main effect of dm-crypt
> is to make the disk appear slower, because of the encryption.

Unfortunately it's not *just* slower. Yes, the arrays would be 2x-3x
as fast without encryption, but that's not the point. It's that I/O
bandwidth isn't shared fairly at all. As if all crypt requests went
into one big FIFO ... (But then, maybe they do.) The stream with the
most/biggest requests then gets all the bandwidth.

> You could try this with a dummy-cipher that does not take encryption time.

Excellent idea, what's the cipher string for that? I couldn't find
anything obvious in /proc/crypto or under
/lib/modules/.../kernel/crypto/.

Thanks,

Chris

Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Arno Wagner]

On Thu, Sep 17, 2009 at 02:36:39AM +0200, Christian Pernegger wrote:
> > Hmm. So without dm-crypt no problem? The main effect of dm-crypt
> > is to make the disk appear slower, because of the encryption.
> 
> Unfortunately it's not *just* slower. Yes, the arrays would be 2x-3x
> as fast without encryption, but that's not the point. It's that I/O
> bandwidth isn't shared fairly at all. As if all crypt requests went
> into one big FIFO ... (But then, maybe they do.) The stream with the
> most/biggest requests then gets all the bandwidth.

Yes. And that may be some s"synergetic" effect triggered by the 
slowness. At least that is one possible explanation.
 
> > You could try this with a dummy-cipher that does not take encryption time.
> 
> Excellent idea, what's the cipher string for that? I couldn't find
> anything obvious in /proc/crypto or under
> /lib/modules/.../kernel/crypto/.

I have not tried this myself, but hit candidates are
"ciper_null" and "ecb(ciper_null)". I have both (kernel 
2.6.30.5)

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Christian Pernegger]

> Yes. And that may be some s"synergetic" effect triggered by the
> slowness. At least that is one possible explanation.

Agreed.

> You could try this with a dummy-cipher that does not take encryption time.
>
> I have not tried this myself, but hit candidates are
> "ciper_null" and "ecb(ciper_null)". I have both (kernel
> 2.6.30.5)

They're there all right, must have overlooked them. However I haven't
managed to construct a cipher spec that'll work with cryptsetup. Maybe
because it takes no key ...
Various tunings of the host's page cache (under /proc/sys/vm/) to make
flushes less bursty didn't help much.

In the meantime I've set up a dedicated kvm guest for these tests,
with one encrypted and one unencrypted scratch volume. Maybe some
systematic benchmarking will point somewhere.

Seems a lot of people are having this problem, since ~2.6.24 at least,
and it seems to get worse the slower the CPUs are, the faster the
disks are. Chosing a faster cipher seems to help.

Chris

Re: [dm-crypt] md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads

[Arno Wagner]

On Fri, Sep 18, 2009 at 02:22:18PM +0200, Christian Pernegger wrote:
> > Yes. And that may be some s"synergetic" effect triggered by the
> > slowness. At least that is one possible explanation.
> 
> Agreed.
> 
> > You could try this with a dummy-cipher that does not take encryption time.
> >
> > I have not tried this myself, but hit candidates are
> > "ciper_null" and "ecb(ciper_null)". I have both (kernel
> > 2.6.30.5)
> 
> They're there all right, must have overlooked them. However I haven't
> managed to construct a cipher spec that'll work with cryptsetup. Maybe
> because it takes no key ...

Possibly. I do not know how to do this either. 

> Various tunings of the host's page cache (under /proc/sys/vm/) to make
> flushes less bursty didn't help much.

A pity. Have you tried to set dirty_writeback_centisecs to
something very low, e.g. 10? If I understand this correctly that
would cause regular flushes to start after 1 sec. 

> In the meantime I've set up a dedicated kvm guest for these tests,
> with one encrypted and one unencrypted scratch volume. Maybe some
> systematic benchmarking will point somewhere.

Good approach. You can also try different speed ciphers.
2fish should be faster than AES.

 > Seems a lot of people are having this problem, since ~2.6.24 at least,
> and it seems to get worse the slower the CPUs are, the faster the
> disks are. Chosing a faster cipher seems to help.

That would be consistent with a backlog before the cipher step
that eventually leads to an emergency flush. One possible solution 
would be to slow down writes before the buffer becomes full. However
I don't think there are provisions for that in the kernel. 

Arno


> Chris
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
> 

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier