[dm-crypt] md + dm-crypt + lvm2: I/O slows down system, I/O schedule ineffective

[dm-crypt] md + dm-crypt + lvm2: I/O slows down system, I/O schedule ineffective

[Günter Merz]

Hi,

at the moment, I've got the following I/O stack on my system:

filesystems (ext4 and swap)
lvm2
dm-crypt
md raid 5
physical disks

other than that, I'm using a quadcore Athlon II X4 640 and 8 GB RAM.
I'm using a vanilla kernel (I/O scheduler: CFS).

I'm experiencing unresponsiveness and a general system slowdown when I'm unpacking big archives or building a big packages (during builds I'm only using max 2 cores). To counter this, I've been experimenting with nice and ionice. Unfortunately, ionice -c3 before big I/O traffic doesn't seem to make any difference. The responsiveness is still really bad at times.

I've been googling in reading in forums quite a lot and there seems to be an longstanding issue in the linux kernel (https://bugzilla.kernel.org/show_bug.cgi?id=12309) that hasn't been resolved for years.

Huge discussion threads seem to deal with this issue:

http://forums.gentoo.org/viewtopic-t-793263-start-100-postdays-0-postorder-asc-highlight-.html?sid=175e371c81170f23e7116626a6b26755
https://bbs.archlinux.org/viewtopic.php?id=93960

Some are suggesting to use different schedulers (BFQ, BFS).
Others are tinkering with the kernel I/O parameters.

Most of these counter measures seem to have little or no effect to others in follow-ups postings.

At this point I could have said: "Linux doesn't handle I/O well" and grudgingly live with it.

However, a different post caught my attention:

http://www.saout.de/pipermail/dm-crypt/2009-October/000329.html

Christian Pernegger did some tests in which he changed the vfs layers.

My layer (dm-crypt on top of md) came out worst in his tests.
dm-crypt directly on top of the physical disk seemed to be the best option.

Other people suggested that using dm-crypt renders the I/O scheduler of the kernel useless.

Since Christian Pernegger's post in 2009, some things have changed:
dm-crypt has become multi-threaded
the linux kernel has changed (VFS, CFS)

So my questions are:
- does the position of dm-crypt in the vfs layer affect I/O performance significantly nowadays?
- is there an explanation for that?
- can anyone suggest a diagnosis on my system that would prove or disprove that dm-crypt is in the wrong position in the vfs stack before I start reordering it on my system
 		 	   		  

Re: [dm-crypt] md + dm-crypt + lvm2: I/O slows down system, I/O schedule ineffective

[Marc Ballarin]

Hi,

Am 23.02.2012 11:19, schrieb Günter Merz:
> I'm experiencing unresponsiveness and a general system slowdown when I'm unpacking big archives or building a big packages (during builds I'm only using max 2 cores). To counter this, I've been experimenting with nice and ionice. Unfortunately, ionice -c3 before big I/O traffic doesn't seem to make any difference. The responsiveness is still really bad at times.
This sounds like those (more or less) well-known hugepage issues.

You could try "echo madvise > 
/sys/kernel/mm/transparent_hugepage/defrag" (or "echo never...") if your 
kernel is configured to use transparent hugepages (which most modern 
distribution kernels are).

If this is your issue, the bug isn't really caused by dm-crypt, but by 
Linux memory management issues and it is completely unrelated to ionice 
or io scheduler settings.

Regards,
Marc

Re: [dm-crypt] md + dm-crypt + lvm2: I/O slows down system, I/O schedule ineffective

[Alasdair G Kergon]

Check everything is aligned properly.

Maybe test this dm-crypt performance proposal too:
  http://people.redhat.com/mpatocka/patches/kernel/dm-crypt-paralelizace/current/series.html

The last couple of patches in particular are 'unexpected'.

Also experiment with the queue depth of the devices:
  nr_requests (/sys/block/<dev>/queue) is often far too small by default.

Alasdair

Re: [dm-crypt] md + dm-crypt + lvm2: I/O slows down system, I/O schedule ineffective

[Milan Broz]

On 02/23/2012 12:54 PM, Alasdair G Kergon wrote:
> Check everything is aligned properly.
>
> Maybe test this dm-crypt performance proposal too:
>    http://people.redhat.com/mpatocka/patches/kernel/dm-crypt-paralelizace/current/series.html

Just note, this is not yet stable repository and final patches
will change (or perhaps some patches will not be committed,
depends on testing output).

IOW - if you can experiment with your data, try it, otherwise please wait
until it is in upstream repo.

Thanks,
Milan