Re: performance loss with lots of snapshots

[Josef Bacik <jbacik@fusionio.com>]

On Wed, Jul 10, 2013 at 12:54:44PM +1000, Russell Coker wrote:
> There are two uses of backups, recovering from user errors (IE deleting the 
> wrong file) and recovering from sysadmin errors or hardware failures (IE disks 
> are dead or wiped).  For the former use I'm mainly using BTRFS snapshots on 
> many systems.
> 
> A problem that I have had on more than a few occasions (most recently on the 
> latest Debian 3.9 kernel) is of severe performance loss.  A few days ago this 
> happened on a workstation running an Intel 120G SSD device for the root 
> filesystem which was being used for basic workstation tasks (kmail, GIMP, 
> OpenOffice, etc).  The /home and / subvols had about 400 snapshots between 
> them (which doesn't seem like a huge number) when the system became unusably 
> slow while running a scrub from a cron job, programs like GIMP became stuck in 
> D state.  The system in question has 8G of RAM and very light load, there 
> shouldn't be any reason for it not giving good performance while the scrub was 
> in progress and it definitely should have performed well when the scrub was 
> cancelled.  But it didn't return to decent performance until I deleted about 
> 300 snapshots.
> 
> This has happened to me often enough that I can probably reproduce it on a VM.  
> What kernel should I use for such tests?
> 
> If I get a virtual machine in a state where it has ongoing performance 
> problems would any of the BTRFS developers like root access to debug it?
> 

There is a memory leak-ish with scrub where it doesn't free up the csums it's
looked up until after its done scrubbing an area which can lead to OOM's or
degraded performance.  Btrfs-next has the fix as well as the pull request that
just went to Linus, so pick which one you want and run again and see if that
helps?  I imagine you are probably seeing two things, first that oom'ish
behavior and then some other performance gotcha with a fair number of snapshots,
but just in case.  Thanks,

Josef