fixing slow sync(2)

fixing slow sync(2)

[Tomas Carnecky]

The issue with sync(2) being slow has been known for a while now (I can 
find discussions as early as July 2010). But nobody seems to be willing 
to fix it or even figure out out what the actual problem is. I use btrfs 
on two hosts, and the sync(2) syscall is consistently slow. By slow I 
mean 20 seconds on one and 45(!) seconds on the other. Every single 
time, no exceptions.

Is there anything I can do to help track down the cause? I can test 
patches, trace the kernel or use whatever tool is most helpful.

tom

Re: fixing slow sync(2)

[Josef Bacik]

On Sat, Oct 08, 2011 at 05:24:15PM +0200, Tomas Carnecky wrote:
> The issue with sync(2) being slow has been known for a while now (I can  
> find discussions as early as July 2010). But nobody seems to be willing  
> to fix it or even figure out out what the actual problem is. I use btrfs  
> on two hosts, and the sync(2) syscall is consistently slow. By slow I  
> mean 20 seconds on one and 45(!) seconds on the other. Every single  
> time, no exceptions.
>
> Is there anything I can do to help track down the cause? I can test  
> patches, trace the kernel or use whatever tool is most helpful.
>

I think I fixed this, try my git tree

git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-work.git

Let me know if it helps.  And what are you doing when you call sync?  I've not
been able to reproduce this problem so I'm having a hard time nailing down what
it is, so if I can get a reliable way to reproduce it I'll try and figure it
out.  Thanks,

Josef

Re: fixing slow sync(2)

[John]

Resending as plain text.

---------- Forwarded message ----------
=46rom: John <jello@waste.org>
Date: Mon, Oct 10, 2011 at 9:34 PM
Subject: Re: fixing slow sync(2)
To: Josef Bacik <josef@redhat.com>
Cc: Tomas Carnecky <tom@dbservice.com>, linux-btrfs@vger.kernel.org


On Sat, Oct 8, 2011 at 10:35 AM, Josef Bacik <josef@redhat.com> wrote:
>
> I think I fixed this, try my git tree
>
> git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-work.git
>
> Let me know if it helps. =A0And what are you doing when you call sync=
? =A0I've not
> been able to reproduce this problem so I'm having a hard time nailing=
 down what
> it is, so if I can get a reliable way to reproduce it I'll try and fi=
gure it
> out. =A0Thanks,

I've been seeing very slow syncs (and unmounts) and this helps. Under
3.0.0 if I mounted the FS (which is 200G, about 97G used with ~720
snapshots) it took about 3 minutes 40 seconds. If I did another sync
right away it took the same amount of time. With a kernel compiled
from your repo the first time I sync it takes about 1 minute, as soon
as I sync a=A0btrfs-endio-met and=A0btrfs-cache-0 process show up in to=
p
and a lot of IO happens. Once those go away (which takes a couple
minutes) then it takes well under a second. If I unmount and remount
it acts exactly the same. Will those changes be up in 3.1 or 3.2? If
you want me to test anything else, I'm happy to do so.
Thanks much.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: fixing slow sync(2)

[Josef Bacik]

On Mon, Oct 10, 2011 at 09:34:12PM -0500, John wrote:
> On Sat, Oct 8, 2011 at 10:35 AM, Josef Bacik <josef@redhat.com> wrote:
> 
> >
> > I think I fixed this, try my git tree
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-work.git
> >
> > Let me know if it helps.  And what are you doing when you call sync?  I've
> > not
> > been able to reproduce this problem so I'm having a hard time nailing down
> > what
> > it is, so if I can get a reliable way to reproduce it I'll try and figure
> > it
> > out.  Thanks,
> >
> 
> I've been seeing very slow syncs (and unmounts) and this helps. Under 3.0.0
> if I mounted the FS (which is 200G, about 97G used with ~720 snapshots) it
> took about 3 minutes 40 seconds. If I did another sync right away it took
> the same amount of time. With a kernel compiled from your repo the first
> time I sync it takes about 1 minute, as soon as I sync a btrfs-endio-met
> and btrfs-cache-0 process show up in top and a lot of IO happens. Once those
> go away (which takes a couple minutes) then it takes well under a second. If
> I unmount and remount it acts exactly the same. Will those changes be up in
> 3.1 or 3.2? If you want me to test anything else, I'm happy to do so.
> 

They'll show up in 3.2.  Thanks for testing,

Josef

Re: fixing slow sync(2)

[Dan Merillat]

On Sat, Oct 8, 2011 at 11:35 AM, Josef Bacik <josef@redhat.com> wrote:

> I think I fixed this, try my git tree
>
> git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-work.git

I wanted to Ack this as well - 3.1-rc4 was completely unusable when
firefox was running (30+ second pauses to read directories, btrfs
threads were constantly running, even the mouse was jerky due to the
load)

Built from your tree (fa5cf66) and everything works like it should
again.  No load, fast response to IO requests.