[dm-crypt] dm-crypt flush-to-disk freezes

[dm-crypt] dm-crypt flush-to-disk freezes

[Arno Wagner]

Hi all,

now I have been hit with a flush-to-disk problem. 

The set-up is a dm-crypt partition with a Windows XP 
VM and the current VMplayer, all with kernel 2.6.34.4
from kernel.org and current vmware-tools in the VM.

The XP in the VM (which I use when I cannot avoid 
working with Word) frequently hangs for single digit
seconds, on what seems to be flushes-to-disk. 

Moving the VM to an unencrypted partition cures the 
problem, but as I work on confidential stuff, I would 
rather have it encrypted. 

What is the current take on this problem? Any fix 
known or expected for the near future?

Thanks for any insights.

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Heinz Diehl]

On 17.08.2010, Arno Wagner wrote: 

> What is the current take on this problem? Any fix 
> known or expected for the near future?

I don't know, and I'm not a VM proff either. That said, 
does it improve things when you write e.g. the following in your
sysctl.conf:

 vm.dirty_ratio = 10
 vm.dirty_background_ratio = 5

Or you could go even lower using

 vm.dirty_background_bytes = 134217728
 vm.dirty_bytes = 268435456

(but then without setting vm.dirty_ratio and vm.dirty_background).
It causes pdflush to write data more frequently and in smaller amounts,
thus reducing blocking via disk i/o.

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Rick Moritz]

I've had io-locks in 2.6.32, but they seem to have disappeared in 2.6.34 (gentoo hardened-r1).
At least the frequency is drastically less,as previously under heavy IO I would get lock ups weekly, and now under the same load, my kernel hasn't been locking up for two to three weeks.
Of course, this is just anecdotal evidence, and not based on any change-logs...

On Wed, 18 Aug 2010 10:31:00 +0200 Heinz Diehl <htd@fancy-poultry.org> wrote:

> On 17.08.2010, Arno Wagner wrote: 
> 
> > What is the current take on this problem? Any fix 
> > known or expected for the near future?
> 
> I don't know, and I'm not a VM proff either. That said, 
> does it improve things when you write e.g. the following in your
> sysctl.conf:
> 
>  vm.dirty_ratio = 10
>  vm.dirty_background_ratio = 5
> 
> Or you could go even lower using
> 
>  vm.dirty_background_bytes = 134217728
>  vm.dirty_bytes = 268435456
> 
> (but then without setting vm.dirty_ratio and vm.dirty_background).
> It causes pdflush to write data more frequently and in smaller amounts,
> thus reducing blocking via disk i/o.

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Milan Broz]

On 08/17/2010 08:46 PM, Arno Wagner wrote:
> The set-up is a dm-crypt partition with a Windows XP 
> VM and the current VMplayer, all with kernel 2.6.34.4
> from kernel.org and current vmware-tools in the VM.

Can you be more specific?

You have direct partition with LUKS and this is mapped
to vmware directly as disk, right?
No fs in the middle?

Are there other encrypted disks on the same underlying
device?

Underlying disk is not SSD, right?

What is the preemption type set in Linux kernel
(Processor type/Preemption model in kernel config)?

What is the IO scheduler for the underlying device?
(You can try deadline or cfq with some tuning - this is
possible to tune online).

Probably tuning fs inside vm can help too. If it is
massive syncing with lot of data waiting for completion,
it is not good.

btw I hope that in 2.6.36 we switch to global workqueues
per cpu for dm-crypt, unfortunately Alasdair is still waiting
with committing it (patches are ready and I am using it myself).
(It can help or it can make it worse here. It needs testing.)

Milan

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Heinz Diehl]

On 18.08.2010, Milan Broz wrote: 

> btw I hope that in 2.6.36 we switch to global workqueues
> per cpu for dm-crypt, unfortunately Alasdair is still waiting
> with committing it (patches are ready and I am using it myself).
> (It can help or it can make it worse here. It needs testing.)

Where can I get these patches?

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Milan Broz]

On 08/18/2010 03:18 PM, Heinz Diehl wrote:
> On 18.08.2010, Milan Broz wrote: 
> 
>> btw I hope that in 2.6.36 we switch to global workqueues
>> per cpu for dm-crypt, unfortunately Alasdair is still waiting
>> with committing it (patches are ready and I am using it myself).
>> (It can help or it can make it worse here. It needs testing.)
> 
> Where can I get these patches?

Not yet in upstream dm tree still...

So my working version is temporarily here
http://mbroz.fedorapeople.org/dm-crypt/

It basically patch from Andi Kleen with some minor fixes, split
into 4 parts for easy review.
(You have to apply _always_ all 4 patches.)

Version for 2.6.34/35 is split into one file (actually I am using
this version on my stable machine now).
Probably this is what you want.

Please note one thing - it creates threads per CPU, but work
submitted by one cpu should never be processed by other thread.
So it helps if you have parallel application but will not help too
much if all work is committed from one process.
More discussion - see dm-devel archive.

Milan

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Heinz Diehl]

On 18.08.2010, Milan Broz wrote: 

> What is the IO scheduler for the underlying device?
> (You can try deadline or cfq with some tuning - this is
> possible to tune online).

On my system, I get the best results with these (cfq):

echo "1" > /sys/block/sda/queue/iosched/low_latency
echo "8" > /sys/block/sda/queue/iosched/slice_idle
echo "8" > /sys/block/sda/queue/iosched/quantum

On some systems, things improve with slice_idle set to 0.

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Arno Wagner]

On Wed, Aug 18, 2010 at 01:43:13PM +0200, Milan Broz wrote:
> On 08/17/2010 08:46 PM, Arno Wagner wrote:
> > The set-up is a dm-crypt partition with a Windows XP 
> > VM and the current VMplayer, all with kernel 2.6.34.4
> > from kernel.org and current vmware-tools in the VM.
> 
> Can you be more specific?
> 
> You have direct partition with LUKS and this is mapped
> to vmware directly as disk, right?
> No fs in the middle?

This is VMwarePlayer. I do not think it even supports putting
the OS on a partition. So no, there is ext3 in between and the
VM filesystem Image goes into several 2GB files.
 
Ah, forgot one possibly important detail: This is dm-crypt on top
of md RAID1. 

> Are there other encrypted disks on the same underlying
> device?

No and there is no other didk activity at the time of these 
freezes. 

> Underlying disk is not SSD, right?

No.

> What is the preemption type set in Linux kernel
> (Processor type/Preemption model in kernel config)?

Low latency desktop. But the freezes are too long to 
be CPU related and there is heavy disk activity when 
they happen. Therefore mty conclusion that this is
an emegerncy flush.

> What is the IO scheduler for the underlying device?
> (You can try deadline or cfq with some tuning - this is
> possible to tune online).

Now you are getting into performance optimization. This
is not a performance optimization issue, as there is
no other load on the machine when this happens and
the problem is not something being slow. Anyways, I
just used the default, it has worked well for me so far.

Any recommendations for a different one?

> Probably tuning fs inside vm can help too. If it is
> massive syncing with lot of data waiting for completion,
> it is not good.

Should not. However I do not know what stupidity
can be found in the XP filesystem layer and OS. 
I suspect quit a lot. What I am doing when this 
happens is scrollign thhrough a word document that
has been open for some time or making small edits
with the keyboard. I really see no reasons to flush
a lot of data to disk on this, but it seems it
is done here. 

Then there is also the VMware layer, which has its own
write buffering. Turning it off or on seems to have
no effect on the problem, so for the moment I assume
this is not the source of the problem.
 
> btw I hope that in 2.6.36 we switch to global workqueues
> per cpu for dm-crypt, unfortunately Alasdair is still waiting
> with committing it (patches are ready and I am using it myself).
> (It can help or it can make it worse here. It needs testing.)

I expect I can test this ;-)

Anyways, thanks and I think I will just run the OS unencrypted
for the moment. The data still goes into the encrypted partition
and curiously that does not cause problems. Very strange. 

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Milan Broz]

On 08/18/2010 04:12 PM, Arno Wagner wrote:
> On Wed, Aug 18, 2010 at 01:43:13PM +0200, Milan Broz wrote:
>> On 08/17/2010 08:46 PM, Arno Wagner wrote:
>>> The set-up is a dm-crypt partition with a Windows XP 
>>> VM and the current VMplayer, all with kernel 2.6.34.4
>>> from kernel.org and current vmware-tools in the VM.
>>
>> Can you be more specific?
>>
>> You have direct partition with LUKS and this is mapped
>> to vmware directly as disk, right?
>> No fs in the middle?
> 
> This is VMwarePlayer. I do not think it even supports putting
> the OS on a partition. So no, there is ext3 in between and the
> VM filesystem Image goes into several 2GB files.
>  
> Ah, forgot one possibly important detail: This is dm-crypt on top
> of md RAID1. 

This is quite complex setup...
(vmware ws suppported direct access to disk device, it is probably
better setup here. Did they this functionality from player?)

I think ext3 part is problem here too, is it using barriers?
(Try to switch them off, it is mount option - just to see if it helps.)

> Low latency desktop. But the freezes are too long to 
> be CPU related and there is heavy disk activity when 
> they happen. Therefore mty conclusion that this is
> an emegerncy flush.

There can be more problems, dm-crypt itself contains cond_resched()
after every encrypted sector allowing decision of scheduler
to switch there. So dm-crypt istelf should not freeze host system this
way, but it doesn't mean that VM suffers from starvation.

I'll check this, I am using similar system, maybe it uncovers some
other problems...

> Anyways, thanks and I think I will just run the OS unencrypted
> for the moment. The data still goes into the encrypted partition
> and curiously that does not cause problems. Very strange. 

(There is also option to encrypt VM in vmware workstation7 directly.
I wonder if they support it in player :-)

Milan

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Arno Wagner]

On Wed, Aug 18, 2010 at 04:47:24PM +0200, Milan Broz wrote:
> On 08/18/2010 04:12 PM, Arno Wagner wrote:
> > On Wed, Aug 18, 2010 at 01:43:13PM +0200, Milan Broz wrote:
> >> On 08/17/2010 08:46 PM, Arno Wagner wrote:
> >>> The set-up is a dm-crypt partition with a Windows XP 
> >>> VM and the current VMplayer, all with kernel 2.6.34.4
> >>> from kernel.org and current vmware-tools in the VM.
> >>
> >> Can you be more specific?
> >>
> >> You have direct partition with LUKS and this is mapped
> >> to vmware directly as disk, right?
> >> No fs in the middle?
> > 
> > This is VMwarePlayer. I do not think it even supports putting
> > the OS on a partition. So no, there is ext3 in between and the
> > VM filesystem Image goes into several 2GB files.
> >  
> > Ah, forgot one possibly important detail: This is dm-crypt on top
> > of md RAID1. 
> 
> This is quite complex setup...

Yes, come to think of it, it is. It is decepticely easy to
do however, since the layers all have clean interfaces.

> (vmware ws suppported direct access to disk device, it is probably
> better setup here. Did they this functionality from player?)

No, I do not think so.

> I think ext3 part is problem here too, is it using barriers?
> (Try to switch them off, it is mount option - just to see if it helps.)

Will try that. Thanks.
 
> > Low latency desktop. But the freezes are too long to 
> > be CPU related and there is heavy disk activity when 
> > they happen. Therefore mty conclusion that this is
> > an emegerncy flush.
> 
> There can be more problems, dm-crypt itself contains cond_resched()
> after every encrypted sector allowing decision of scheduler
> to switch there. So dm-crypt istelf should not freeze host system this
> way, but it doesn't mean that VM suffers from starvation.

Good idea. I have not noticed freezes when the focus did
not belong to the VM window. Should check that as well,
maybe it really is CPU starvation.

> I'll check this, I am using similar system, maybe it uncovers some
> other problems...
> 
> > Anyways, thanks and I think I will just run the OS unencrypted
> > for the moment. The data still goes into the encrypted partition
> > and curiously that does not cause problems. Very strange. 
> 
> (There is also option to encrypt VM in vmware workstation7 directly.
> I wonder if they support it in player :-)

Don't think so. At least not in any obvious manner.

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Arno Wagner]

On Wed, Aug 18, 2010 at 05:44:19PM +0200, Arno Wagner wrote:
> On Wed, Aug 18, 2010 at 04:47:24PM +0200, Milan Broz wrote:
[...]
> > I think ext3 part is problem here too, is it using barriers?
> > (Try to switch them off, it is mount option - just to see if it helps.)
> 
> Will try that. Thanks.

Preliminary but this seems to help a lot. Not sure the
problem is entirely gone though. But if not it happens 
very rarely now.

What do I lose with barriers off?

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Christoph Anton Mitterer]

On Sun, 2010-08-22 at 21:52 +0200, Arno Wagner wrote:
> What do I lose with barriers off?
data security ;)


In case of the filesystem barriers (not the IO barriers, which are
different AFAIK) they're used to make sure, that the COMMITS in the
journal are written after the journal is correctly flushed out.


Cheers,
Chris.

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Arno Wagner]

On Sun, Aug 22, 2010 at 11:42:44PM +0200, Christoph Anton Mitterer wrote:
> On Sun, 2010-08-22 at 21:52 +0200, Arno Wagner wrote:
> > What do I lose with barriers off?
> data security ;)
> 
> 
> In case of the filesystem barriers (not the IO barriers, which are
> different AFAIK) they're used to make sure, that the COMMITS in the
> journal are written after the journal is correctly flushed out.


If that means I am back to ext2 safety levels, then I can
live with that.

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 326 bytes --]

On Sun, 2010-08-22 at 23:51 +0200, Arno Wagner wrote:
> If that means I am back to ext2 safety levels, then I can
> live with that.
AFAIK it's even worse (potentially).

In case you have a crash,... the kernel will (on remounting) replay the
(also corrupted) journal, thereby destroying even more.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3387 bytes --]

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 75 bytes --]

So I'd suggest to rather disable the journal then...


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3387 bytes --]

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Heinz Diehl]

On 23.08.2010, Arno Wagner wrote: 

> > In case of the filesystem barriers (not the IO barriers, which are
> > different AFAIK) they're used to make sure, that the COMMITS in the
> > journal are written after the journal is correctly flushed out.

> If that means I am back to ext2 safety levels, then I can
> live with that.

You could turn the hardware wb cache of your harddrive off and leave the
barriers off too..

I've been running my disks a liftime without barrier support without any
hassle, because it improves speed and interactivity a lot, but I must 
admit that my machines are all UPS secured.

 /dev/mapper/root on / type xfs (rw,noatime,logbsize=256k,logbufs=2,nobarrier)
 /dev/mapper/home on /home type xfs (rw,noatime,logbsize=256k,logbufs=2,nobarrier)

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Arno Wagner]

On Sun, Aug 22, 2010 at 11:42:44PM +0200, Christoph Anton Mitterer wrote:
> On Sun, 2010-08-22 at 21:52 +0200, Arno Wagner wrote:
> > What do I lose with barriers off?
> data security ;)
> 
> 
> In case of the filesystem barriers (not the IO barriers, which are
> different AFAIK) they're used to make sure, that the COMMITS in the
> journal are written after the journal is correctly flushed out.

Just disabled the journal. That should do it for the moment.

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Milan Broz]

On 08/22/2010 11:42 PM, Christoph Anton Mitterer wrote:
> On Sun, 2010-08-22 at 21:52 +0200, Arno Wagner wrote:
>> What do I lose with barriers off?
> data security ;)
> 
> 
> In case of the filesystem barriers (not the IO barriers, which are
> different AFAIK) they're used to make sure, that the COMMITS in the
> journal are written after the journal is correctly flushed out.

Slight confusion here.

FS uses flush (called by fsync for example), it is currently implemented
using IO barrier.
After this operation, FS code can be sure that preceding barrier
reached disk.
(Device-mapper internally waits for all IO to finish, processing always
one barrier at a time and queuing following requests.)

If you disable it, data simply reach disk later and e.g. unexpected
power loss can cause quite serious data loss.
But you probably see better performance.

So disabling barriers helps in your case? Then probably some tuning
of fs can help also.

(There is ongoing discussion about reimplementing barriers in block
layer, maybe it will slightly help here too.)

Milan

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Arno Wagner]

On Mon, Aug 23, 2010 at 09:13:28AM +0200, Milan Broz wrote:
> On 08/22/2010 11:42 PM, Christoph Anton Mitterer wrote:
> > On Sun, 2010-08-22 at 21:52 +0200, Arno Wagner wrote:
> >> What do I lose with barriers off?
> > data security ;)
> > 
> > 
> > In case of the filesystem barriers (not the IO barriers, which are
> > different AFAIK) they're used to make sure, that the COMMITS in the
> > journal are written after the journal is correctly flushed out.
> 
> Slight confusion here.
> 
> FS uses flush (called by fsync for example), it is currently implemented
> using IO barrier.

Ok. So is a process calls fsync on a file, this happens.

> After this operation, FS code can be sure that preceding barrier
> reached disk.
> (Device-mapper internally waits for all IO to finish, processing always
> one barrier at a time and queuing following requests.)

Seems there is some backlog in my case.

What I find curious is that plain ext3 on RAID1 (same disks,
different partitions) does not cause problems. I would 
expect that an fsync blocks the disk, not only the partition.
Maybe having the system and data on different filesystems just
reduced the backlog enough.

> If you disable it, data simply reach disk later and e.g. unexpected
> power loss can cause quite serious data loss.
> But you probably see better performance.

I went back to ext2 for the time being. This is used only to 
work on Word documents. The data gets copied off imediately 
anyways and I have a backup of the complete VM in case 
something breaks.

> So disabling barriers helps in your case? Then probably some tuning
> of fs can help also.

I would expect that. Especially more aggressive flushing 
should help. Will look into it when I have time. For the moment
I am happy to have a solution that does not increase the 
considerable pain level word manages to cause all by itself 
(I am a LeTeX person....).

> (There is ongoing discussion about reimplementing barriers in block
> layer, maybe it will slightly help here too.)

I find them quite a pain sometimes, especially when writing
large amounts of data. I used to have a fine-tuned parameter set
that managed to almost completely avoid emergency flushes, while 
having minimal impact on performance. Then the kernel devs decided
to take the interface away. I am still mad at them for that.
Should probably look at this again.

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] dm-crypt flush-to-disk freezes

[Heinz Diehl]

On 25.08.2010, Arno Wagner wrote: 

> I went back to ext2 for the time being. This is used only to 
> work on Word documents. The data gets copied off imediately 
> anyways and I have a backup of the complete VM in case 
> something breaks.

You could mount ext3 with "data=journal", this gives identical behaviour.

> Especially more aggressive flushing should help. 
> Will look into it when I have time.

See my reply a few mails above in this thread.