[dm-crypt] dm-crypt kills NFS performance?

[dm-crypt] dm-crypt kills NFS performance?

[Randall Cotton]

A few months back I brought up a new NFS server that provides user home
directories for my home network. Using Ubuntu 10.04 LTS I set up a 3-disk
RAID 5 device and then dm-crypt on top of that, and finally LVM on top of
that.

It's been working fine except for one increasingly unacceptable problem.
I've set up a backup system that uses rsync every 5 minutes to take a
snapshot of my home directory and store it on the same disk. This is a
short-term backup solution that lets me get back my work from 5, 10, 15
minutes ago if I accidentally nuke something. Hard links are used to
minimize the space needed for the snapshots (a la Mike Rubel)

So this snapshot routine runs on the server itself. The problem is that
when it does run, NFS service essentially goes into grand mal seizure. And
any NFS client accordingly also goes into seizure with the attendant
/var/log/messages entries such as:

Apr 21 08:52:08 aquamarine kernel: [8073022.344817] nfs: server
pearl.randallcotton.com not responding, still trying
Apr 21 08:52:13 aquamarine kernel: [8073027.401976] nfs: server
pearl.randallcotton.com OK

And so every 5 minutes, I have to sit in front of my workstation while it
seizes up for 10, 15 even 20 seconds sometimes. As the size of my home
directory grows, the problem gets worse. It's already unacceptable. I
didn't have this problem with my previous NFS server.

Now my previous NFS server didn't have either a RAID setup, encryption or
LVM, but I'm guessing that kcryptd is the culprit since it's the biggest
CPU hog by far during the seizures. However, cranking down the priority of
kcryptd (and apparently-related processes crypto and kcryptd_io) and also
cranking down the priority of the rsync process and then ALSO cranking *up*
the priority of the nfsd processes seems to help a little, but not much.

Am I asking too much here (running dm-crypt on an NFS server)? Or does
anyone have an idea how I might adjust things so NFS performance isn't so
hideously cannibalized when kcryptd is busy?

I don't know dm-crypt internals, so I'm flying a little blind here. If
there's something I should study up on to fix this, let me know.

Thanks
R

Re: [dm-crypt] dm-crypt kills NFS performance?

[Arno Wagner]

This sounds like one more problems with the Big Kernel Lock.
The basic problem (simplified) is that some operations
block everything while running. dm-crypt and RAID makes 
them take longer, but the problem is the blocking. 

You can possibly provoke the same issue by doing a 
  cat /dev/zero > <some file>
on the fileserver, not necessarily to the RAID.
As the process priority does not much influence IO
usage, that does not really help.

As far as I can see, the best current solution is
to wait for kernel 2.6.39 (2.6.38 does already 
help). In the meantime, slowing down rsync i/o (not 
CPU load) could help. You can try something like this:

  rsync --bwlimit=1000 ...

Gr"usse,
Arno





On Thu, Apr 21, 2011 at 12:14:48PM -0500, Randall Cotton wrote:
> A few months back I brought up a new NFS server that provides user home
> directories for my home network. Using Ubuntu 10.04 LTS I set up a 3-disk
> RAID 5 device and then dm-crypt on top of that, and finally LVM on top of
> that.
> 
> It's been working fine except for one increasingly unacceptable problem.
> I've set up a backup system that uses rsync every 5 minutes to take a
> snapshot of my home directory and store it on the same disk. This is a
> short-term backup solution that lets me get back my work from 5, 10, 15
> minutes ago if I accidentally nuke something. Hard links are used to
> minimize the space needed for the snapshots (a la Mike Rubel)
> 
> So this snapshot routine runs on the server itself. The problem is that
> when it does run, NFS service essentially goes into grand mal seizure. And
> any NFS client accordingly also goes into seizure with the attendant
> /var/log/messages entries such as:
> 
> Apr 21 08:52:08 aquamarine kernel: [8073022.344817] nfs: server
> pearl.randallcotton.com not responding, still trying
> Apr 21 08:52:13 aquamarine kernel: [8073027.401976] nfs: server
> pearl.randallcotton.com OK
> 
> And so every 5 minutes, I have to sit in front of my workstation while it
> seizes up for 10, 15 even 20 seconds sometimes. As the size of my home
> directory grows, the problem gets worse. It's already unacceptable. I
> didn't have this problem with my previous NFS server.
> 
> Now my previous NFS server didn't have either a RAID setup, encryption or
> LVM, but I'm guessing that kcryptd is the culprit since it's the biggest
> CPU hog by far during the seizures. However, cranking down the priority of
> kcryptd (and apparently-related processes crypto and kcryptd_io) and also
> cranking down the priority of the rsync process and then ALSO cranking *up*
> the priority of the nfsd processes seems to help a little, but not much.
> 
> Am I asking too much here (running dm-crypt on an NFS server)? Or does
> anyone have an idea how I might adjust things so NFS performance isn't so
> hideously cannibalized when kcryptd is busy?
> 
> I don't know dm-crypt internals, so I'm flying a little blind here. If
> there's something I should study up on to fix this, let me know.
> 
> Thanks
> R
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
> 

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] dm-crypt kills NFS performance?

[Milan Broz]

On 04/21/2011 07:52 PM, Arno Wagner wrote:
> 
> This sounds like one more problems with the Big Kernel Lock.
> The basic problem (simplified) is that some operations
> block everything while running. dm-crypt and RAID makes 
> them take longer, but the problem is the blocking. 

It can be many problems, from network card driver to
nfs handling or io scheduler setting.

For dm-crypt (kcryptd) - it doesn't not use BKL at all
and after every sector encryption it calls cond_resched()
(hinting scheduler to switch process if there is other work
even if kernel is compiled with voluntary preemption).
So it should behave correctly under load.

Milan

Re: [dm-crypt] dm-crypt kills NFS performance?

[Arno Wagner]

On Thu, Apr 21, 2011 at 08:06:25PM +0200, Milan Broz wrote:
> On 04/21/2011 07:52 PM, Arno Wagner wrote:
> > 
> > This sounds like one more problems with the Big Kernel Lock.
> > The basic problem (simplified) is that some operations
> > block everything while running. dm-crypt and RAID makes 
> > them take longer, but the problem is the blocking. 
> 
> It can be many problems, from network card driver to
> nfs handling or io scheduler setting.
> 
> For dm-crypt (kcryptd) - it doesn't not use BKL at all
> and after every sector encryption it calls cond_resched()
> (hinting scheduler to switch process if there is other work
> even if kernel is compiled with voluntary preemption).
> So it should behave correctly under load.

Ah. Good to know. 

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier