Re: [dm-crypt] dm-crypt kills NFS performance?

[Arno Wagner <arno@wagner.name>]

This sounds like one more problems with the Big Kernel Lock.
The basic problem (simplified) is that some operations
block everything while running. dm-crypt and RAID makes 
them take longer, but the problem is the blocking. 

You can possibly provoke the same issue by doing a 
  cat /dev/zero > <some file>
on the fileserver, not necessarily to the RAID.
As the process priority does not much influence IO
usage, that does not really help.

As far as I can see, the best current solution is
to wait for kernel 2.6.39 (2.6.38 does already 
help). In the meantime, slowing down rsync i/o (not 
CPU load) could help. You can try something like this:

  rsync --bwlimit=1000 ...

Gr"usse,
Arno





On Thu, Apr 21, 2011 at 12:14:48PM -0500, Randall Cotton wrote:
> A few months back I brought up a new NFS server that provides user home
> directories for my home network. Using Ubuntu 10.04 LTS I set up a 3-disk
> RAID 5 device and then dm-crypt on top of that, and finally LVM on top of
> that.
> 
> It's been working fine except for one increasingly unacceptable problem.
> I've set up a backup system that uses rsync every 5 minutes to take a
> snapshot of my home directory and store it on the same disk. This is a
> short-term backup solution that lets me get back my work from 5, 10, 15
> minutes ago if I accidentally nuke something. Hard links are used to
> minimize the space needed for the snapshots (a la Mike Rubel)
> 
> So this snapshot routine runs on the server itself. The problem is that
> when it does run, NFS service essentially goes into grand mal seizure. And
> any NFS client accordingly also goes into seizure with the attendant
> /var/log/messages entries such as:
> 
> Apr 21 08:52:08 aquamarine kernel: [8073022.344817] nfs: server
> pearl.randallcotton.com not responding, still trying
> Apr 21 08:52:13 aquamarine kernel: [8073027.401976] nfs: server
> pearl.randallcotton.com OK
> 
> And so every 5 minutes, I have to sit in front of my workstation while it
> seizes up for 10, 15 even 20 seconds sometimes. As the size of my home
> directory grows, the problem gets worse. It's already unacceptable. I
> didn't have this problem with my previous NFS server.
> 
> Now my previous NFS server didn't have either a RAID setup, encryption or
> LVM, but I'm guessing that kcryptd is the culprit since it's the biggest
> CPU hog by far during the seizures. However, cranking down the priority of
> kcryptd (and apparently-related processes crypto and kcryptd_io) and also
> cranking down the priority of the rsync process and then ALSO cranking *up*
> the priority of the nfsd processes seems to help a little, but not much.
> 
> Am I asking too much here (running dm-crypt on an NFS server)? Or does
> anyone have an idea how I might adjust things so NFS performance isn't so
> hideously cannibalized when kcryptd is busy?
> 
> I don't know dm-crypt internals, so I'm flying a little blind here. If
> there's something I should study up on to fix this, let me know.
> 
> Thanks
> R
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
> 

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier