Re: snapshot destruction making IO extremely slow

[David Sterba <dsterba@suse.cz>]

On Wed, Mar 25, 2015 at 11:55:36AM +0100, Marc Cousin wrote:
> On 25/03/2015 02:19, David Sterba wrote:
> > Snapper might add to that if you have
> > 
> > EMPTY_PRE_POST_CLEANUP="yes"
> > 
> > as it reads the pre/post snapshots and deletes them if the diff is
> > empty. This adds some IO stress.
> 
> I couldn't find a clear explanation in the documentation. Does it mean
> that when there is absolutely no difference between two snapshots, one
> of them is deleted ?

Only the pre-post snapshots, ie. no timeline or other types (eg.
manually created one).

> And that snapper does a diff between them to
> determine that ?

AFAIK yes.

> If so, yes, I can remove it, I don't care about that :)
> 
> > 
> >> The btrfs cleaner is 100% active:
> >>
> >>  1501 root      20   0       0      0      0 R 100,0  0,0   9:10.40 [btrfs-cleaner]    
> > 
> > That points to the snapshot cleaning, but the cleaner thread does more
> > than that. It may also process delayed file deletion and work scheduled
> > if 'autodefrag' is on.
> 
> autodefrag is activated. These are mechanical drives, so I'd rather keep
> it on, shouldn't I ?

You should (I do have autogefrag on), unless you applications are
latency sensitive and you can measure the difference. Autodefrag tends
to read/write surrounding blocks for random write so it may imply some
seek penalty if the affected block is far from the others.

> >> What is "funny" is that the filesystem seems to be working again when
> >> there is some IO activity and btrfs-cleaner gets to a lower cpu usage
> >> (around 70%).
> > 
> > Possibly a behaviour caused by scheduling (both cpu and io), the other
> > process gets a slice and slows down cleaner that hogs the system.
> 
> I have almost no IO on these disks during the problem (I had put an
> iostat on the first email). Only one CPU core at 100% load. That's why I
> felt it looked more like a locking or serialization issue.

So it would be good to sample the active threads and see where it's
spending the time. It could be the somewhere in the rb-tree representing
extents, but that's a guess.