[dm-crypt] dm-crypt kills NFS performance?

[Randall Cotton <recotton@earthlink.net>]

A few months back I brought up a new NFS server that provides user home
directories for my home network. Using Ubuntu 10.04 LTS I set up a 3-disk
RAID 5 device and then dm-crypt on top of that, and finally LVM on top of
that.

It's been working fine except for one increasingly unacceptable problem.
I've set up a backup system that uses rsync every 5 minutes to take a
snapshot of my home directory and store it on the same disk. This is a
short-term backup solution that lets me get back my work from 5, 10, 15
minutes ago if I accidentally nuke something. Hard links are used to
minimize the space needed for the snapshots (a la Mike Rubel)

So this snapshot routine runs on the server itself. The problem is that
when it does run, NFS service essentially goes into grand mal seizure. And
any NFS client accordingly also goes into seizure with the attendant
/var/log/messages entries such as:

Apr 21 08:52:08 aquamarine kernel: [8073022.344817] nfs: server
pearl.randallcotton.com not responding, still trying
Apr 21 08:52:13 aquamarine kernel: [8073027.401976] nfs: server
pearl.randallcotton.com OK

And so every 5 minutes, I have to sit in front of my workstation while it
seizes up for 10, 15 even 20 seconds sometimes. As the size of my home
directory grows, the problem gets worse. It's already unacceptable. I
didn't have this problem with my previous NFS server.

Now my previous NFS server didn't have either a RAID setup, encryption or
LVM, but I'm guessing that kcryptd is the culprit since it's the biggest
CPU hog by far during the seizures. However, cranking down the priority of
kcryptd (and apparently-related processes crypto and kcryptd_io) and also
cranking down the priority of the rsync process and then ALSO cranking *up*
the priority of the nfsd processes seems to help a little, but not much.

Am I asking too much here (running dm-crypt on an NFS server)? Or does
anyone have an idea how I might adjust things so NFS performance isn't so
hideously cannibalized when kcryptd is busy?

I don't know dm-crypt internals, so I'm flying a little blind here. If
there's something I should study up on to fix this, let me know.

Thanks
R