Re: Btrfs switches to using mostly one thread

[Chris Mason <chris.mason@oracle.com>]

On Fri, Dec 09, 2011 at 12:05:40PM +0000, Jeremy Sanders wrote:
> On 08/12/11 20:11, Chris Mason wrote:
> >On Thu, Dec 08, 2011 at 05:39:16PM +0000, Jeremy Sanders wrote:
> >>On 08/12/11 17:23, Chris Mason wrote:
> >>>On Thu, Dec 08, 2011 at 04:57:12PM +0000, Jeremy Sanders wrote:
> >>>>On 08/12/11 15:32, Chris Mason wrote:
> >>>>>On Thu, Dec 08, 2011 at 03:19:38PM +0000, Jeremy Sanders wrote:
> >>>>>>Hi - I'm trying out btrfs again, and I see the same old bug in kernel 3.1.4
> >>>>>>(Fedora 16, x86_64, dual-core), where after a few hours of writing, it
> >>>>>>switches from writing with several threads to writing with one:
> >>>>>
> >>>>>Ok, I'll try to reproduce this here.  Could you please do a sysrq-t, I'd
> >>>>>like to see what the other delalloc-writers are doing.
> >>>>
> >>>>I've attached sysrq-t. It looks like it might be truncated at the
> >>>>beginning, however.
> >>>
> >>>/var/log/messages may have the whole thing, please do check.
> >>
> >>That was from /var/log/messages. I think it needs a longer
> >>log_buf_len. Unfortunately the system hasn't come back from its
> >>reboot, so it will have to wait until tomorrow when I can get to it
> >>physically.
> >
> >Ok, this trace shows that we have tar sitting in balance_dirty_pages and
> >we have the single delalloc worker doing requests.  The other delalloc
> >workers don't show up at all.
> >
> >So either they are earlier in the trace or they disappeared somehow.
> >I'll definitely need the full trace if you can send it.
> 
> I've got the full trace now. It's pretty big (430KB), so I've put it
> on the web.
> 
> Here's the state before switching to one thread
> http://www-xray.ast.cam.ac.uk/~jss/data/btrfs-before.txt
> 
> Here it is after it has switched to one thread:
> http://www-xray.ast.cam.ac.uk/~jss/data/btrfs-after.txt

According to this you've only got one delalloc worker.  That would
explain it.  Could you please confirm with ps?

You might be hitting a problem Josef sent patches for, I'll dig in.

-chris