performance loss with lots of snapshots

performance loss with lots of snapshots

[Russell Coker]

There are two uses of backups, recovering from user errors (IE deleting the 
wrong file) and recovering from sysadmin errors or hardware failures (IE disks 
are dead or wiped).  For the former use I'm mainly using BTRFS snapshots on 
many systems.

A problem that I have had on more than a few occasions (most recently on the 
latest Debian 3.9 kernel) is of severe performance loss.  A few days ago this 
happened on a workstation running an Intel 120G SSD device for the root 
filesystem which was being used for basic workstation tasks (kmail, GIMP, 
OpenOffice, etc).  The /home and / subvols had about 400 snapshots between 
them (which doesn't seem like a huge number) when the system became unusably 
slow while running a scrub from a cron job, programs like GIMP became stuck in 
D state.  The system in question has 8G of RAM and very light load, there 
shouldn't be any reason for it not giving good performance while the scrub was 
in progress and it definitely should have performed well when the scrub was 
cancelled.  But it didn't return to decent performance until I deleted about 
300 snapshots.

This has happened to me often enough that I can probably reproduce it on a VM.  
What kernel should I use for such tests?

If I get a virtual machine in a state where it has ongoing performance 
problems would any of the BTRFS developers like root access to debug it?

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

Re: performance loss with lots of snapshots

[Chris Samuel]

Hiya Russell,

On 10/07/13 12:54, Russell Coker wrote:

> This has happened to me often enough that I can probably reproduce it on a VM.  
> What kernel should I use for such tests?

I'd suggest at least 3.10, and perhaps also Chris Mason's git tree here:

https://git.kernel.org/cgit/linux/kernel/git/mason/linux-btrfs.git/

which is 3.10 plus the latest btrfs code that was merged by Linus into
his master for 3.11 today.

All the best,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

Re: performance loss with lots of snapshots

[Josef Bacik]

On Wed, Jul 10, 2013 at 12:54:44PM +1000, Russell Coker wrote:
> There are two uses of backups, recovering from user errors (IE deleting the 
> wrong file) and recovering from sysadmin errors or hardware failures (IE disks 
> are dead or wiped).  For the former use I'm mainly using BTRFS snapshots on 
> many systems.
> 
> A problem that I have had on more than a few occasions (most recently on the 
> latest Debian 3.9 kernel) is of severe performance loss.  A few days ago this 
> happened on a workstation running an Intel 120G SSD device for the root 
> filesystem which was being used for basic workstation tasks (kmail, GIMP, 
> OpenOffice, etc).  The /home and / subvols had about 400 snapshots between 
> them (which doesn't seem like a huge number) when the system became unusably 
> slow while running a scrub from a cron job, programs like GIMP became stuck in 
> D state.  The system in question has 8G of RAM and very light load, there 
> shouldn't be any reason for it not giving good performance while the scrub was 
> in progress and it definitely should have performed well when the scrub was 
> cancelled.  But it didn't return to decent performance until I deleted about 
> 300 snapshots.
> 
> This has happened to me often enough that I can probably reproduce it on a VM.  
> What kernel should I use for such tests?
> 
> If I get a virtual machine in a state where it has ongoing performance 
> problems would any of the BTRFS developers like root access to debug it?
> 

There is a memory leak-ish with scrub where it doesn't free up the csums it's
looked up until after its done scrubbing an area which can lead to OOM's or
degraded performance.  Btrfs-next has the fix as well as the pull request that
just went to Linus, so pick which one you want and run again and see if that
helps?  I imagine you are probably seeing two things, first that oom'ish
behavior and then some other performance gotcha with a fair number of snapshots,
but just in case.  Thanks,

Josef