Hang running fs_mark

Hang running fs_mark

[Josef Bacik]

Hello,

Reporting this on behalf of ric.  He was running the following fs_mark command

./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F

Seems it hung and wasn't making any progress.  He managed to get some sysrq-t,
which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom
of the document.  He could ctrl+c and unmount the fs so its not a hard hang.
Looks like we've just locked up behind a page lock somewhere.  I have to run off
to class so I can't look into it too deeply so throwing this out there hoping
somebody else figures it out :).  Thanks,

Josef

Re: Hang running fs_mark

[Josef Bacik]

On Thu, Sep 25, 2008 at 02:37:02PM -0400, Chris Mason wrote:
> On Thu, 2008-09-25 at 13:56 -0400, Josef Bacik wrote:
> > Hello,
> > 
> > Reporting this on behalf of ric.  He was running the following fs_mark command
> > 
> > ./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F
> > 
> > Seems it hung and wasn't making any progress.  He managed to get some sysrq-t,
> > which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom
> > of the document.  He could ctrl+c and unmount the fs so its not a hard hang.
> > Looks like we've just locked up behind a page lock somewhere.  I have to run off
> > to class so I can't look into it too deeply so throwing this out there hoping
> > somebody else figures it out :).  Thanks,
> 
> Which kernel was this?
> 
>

Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he
said, its just shy whatever patches you've pushed into git recently, it has
yan's backref work plus a few other fixes.  Thanks,

Josef 
> 

Re: Hang running fs_mark

[Chris Mason]

On Thu, 2008-09-25 at 13:56 -0400, Josef Bacik wrote:
> Hello,
> 
> Reporting this on behalf of ric.  He was running the following fs_mark command
> 
> ./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F
> 
> Seems it hung and wasn't making any progress.  He managed to get some sysrq-t,
> which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom
> of the document.  He could ctrl+c and unmount the fs so its not a hard hang.
> Looks like we've just locked up behind a page lock somewhere.  I have to run off
> to class so I can't look into it too deeply so throwing this out there hoping
> somebody else figures it out :).  Thanks,

Which kernel was this?

-chris

Re: Hang running fs_mark

[Chris Mason]

On Thu, 2008-09-25 at 14:34 -0400, Josef Bacik wrote:
> On Thu, Sep 25, 2008 at 02:37:02PM -0400, Chris Mason wrote:
> > On Thu, 2008-09-25 at 13:56 -0400, Josef Bacik wrote:
> > > Hello,
> > > 
> > > Reporting this on behalf of ric.  He was running the following fs_mark command
> > > 
> > > ./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F
> > > 
> > > Seems it hung and wasn't making any progress.  He managed to get some sysrq-t,
> > > which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom
> > > of the document.  He could ctrl+c and unmount the fs so its not a hard hang.
> > > Looks like we've just locked up behind a page lock somewhere.  I have to run off
> > > to class so I can't look into it too deeply so throwing this out there hoping
> > > somebody else figures it out :).  Thanks,
> > 
> > Which kernel was this?

> Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he
> said, its just shy whatever patches you've pushed into git recently, it has
> yan's backref work plus a few other fixes.  Thanks,
> 

Well, the trace is strange because it looks like everyone is waiting on
IO.  Any chance once of the procs was spinning in system time?

-chris

Re: Hang running fs_mark

[Ric Wheeler]

Josef Bacik wrote:
> On Thu, Sep 25, 2008 at 02:37:02PM -0400, Chris Mason wrote:
>   
>> On Thu, 2008-09-25 at 13:56 -0400, Josef Bacik wrote:
>>     
>>> Hello,
>>>
>>> Reporting this on behalf of ric.  He was running the following fs_mark command
>>>
>>> ./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F
>>>
>>> Seems it hung and wasn't making any progress.  He managed to get some sysrq-t,
>>> which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom
>>> of the document.  He could ctrl+c and unmount the fs so its not a hard hang.
>>> Looks like we've just locked up behind a page lock somewhere.  I have to run off
>>> to class so I can't look into it too deeply so throwing this out there hoping
>>> somebody else figures it out :).  Thanks,
>>>       
>> Which kernel was this?
>>
>>
>>     
>
> Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he
> said, its just shy whatever patches you've pushed into git recently, it has
> yan's backref work plus a few other fixes.  Thanks,
>
> Josef 
>   

I can fire off a new test with the latest from unstable if that is of 
interest,

ric

Re: Hang running fs_mark

[Ric Wheeler]

Chris Mason wrote:
> On Thu, 2008-09-25 at 14:34 -0400, Josef Bacik wrote:
>   
>> On Thu, Sep 25, 2008 at 02:37:02PM -0400, Chris Mason wrote:
>>     
>>> On Thu, 2008-09-25 at 13:56 -0400, Josef Bacik wrote:
>>>       
>>>> Hello,
>>>>
>>>> Reporting this on behalf of ric.  He was running the following fs_mark command
>>>>
>>>> ./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F
>>>>
>>>> Seems it hung and wasn't making any progress.  He managed to get some sysrq-t,
>>>> which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom
>>>> of the document.  He could ctrl+c and unmount the fs so its not a hard hang.
>>>> Looks like we've just locked up behind a page lock somewhere.  I have to run off
>>>> to class so I can't look into it too deeply so throwing this out there hoping
>>>> somebody else figures it out :).  Thanks,
>>>>         
>>> Which kernel was this?
>>>       
>
>   
>> Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he
>> said, its just shy whatever patches you've pushed into git recently, it has
>> yan's backref work plus a few other fixes.  Thanks,
>>
>>     
>
> Well, the trace is strange because it looks like everyone is waiting on
> IO.  Any chance once of the procs was spinning in system time?
>
> -chris
>
>   

It actually cleaned up very nicely after I killed the fs_mark processes 
& unmounted the btrfs file system. Before doing that, the box was 
sluggish and had the feeling of a system with something that might have 
been spinning (but that is just an observation, not measured in any 
strict sense).

ric

Re: Hang running fs_mark

[Chris Mason]

On Thu, 2008-09-25 at 16:37 -0400, Ric Wheeler wrote:
> Chris Mason wrote:
> > On Thu, 2008-09-25 at 14:34 -0400, Josef Bacik wrote:
> >   
> >> On Thu, Sep 25, 2008 at 02:37:02PM -0400, Chris Mason wrote:
> >>     
> >>> On Thu, 2008-09-25 at 13:56 -0400, Josef Bacik wrote:
> >>>       
> >>>> Hello,
> >>>>
> >>>> Reporting this on behalf of ric.  He was running the following fs_mark command
> >>>>
> >>>> ./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F
> >>>>
> >>>> Seems it hung and wasn't making any progress.  He managed to get some sysrq-t,
> >>>> which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom
> >>>> of the document.  He could ctrl+c and unmount the fs so its not a hard hang.
> >>>> Looks like we've just locked up behind a page lock somewhere.  I have to run off
> >>>> to class so I can't look into it too deeply so throwing this out there hoping
> >>>> somebody else figures it out :).  Thanks,
> >>>>         
> >>> Which kernel was this?

> It actually cleaned up very nicely after I killed the fs_mark processes 
> & unmounted the btrfs file system. Before doing that, the box was 
> sluggish and had the feeling of a system with something that might have 
> been spinning (but that is just an observation, not measured in any 
> strict sense).

Ok, I have that fs_mark test running here.  How far did yours get before
it stopped?

-chris

Re: Hang running fs_mark

[Ric Wheeler]

Chris Mason wrote:
> On Thu, 2008-09-25 at 16:37 -0400, Ric Wheeler wrote:
>   
>> Chris Mason wrote:
>>     
>>> On Thu, 2008-09-25 at 14:34 -0400, Josef Bacik wrote:
>>>   
>>>       
>>>> On Thu, Sep 25, 2008 at 02:37:02PM -0400, Chris Mason wrote:
>>>>     
>>>>         
>>>>> On Thu, 2008-09-25 at 13:56 -0400, Josef Bacik wrote:
>>>>>       
>>>>>           
>>>>>> Hello,
>>>>>>
>>>>>> Reporting this on behalf of ric.  He was running the following fs_mark command
>>>>>>
>>>>>> ./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F
>>>>>>
>>>>>> Seems it hung and wasn't making any progress.  He managed to get some sysrq-t,
>>>>>> which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom
>>>>>> of the document.  He could ctrl+c and unmount the fs so its not a hard hang.
>>>>>> Looks like we've just locked up behind a page lock somewhere.  I have to run off
>>>>>> to class so I can't look into it too deeply so throwing this out there hoping
>>>>>> somebody else figures it out :).  Thanks,
>>>>>>         
>>>>>>             
>>>>> Which kernel was this?
>>>>>           
>
>   
>> It actually cleaned up very nicely after I killed the fs_mark processes 
>> & unmounted the btrfs file system. Before doing that, the box was 
>> sluggish and had the feeling of a system with something that might have 
>> been spinning (but that is just an observation, not measured in any 
>> strict sense).
>>     
>
> Ok, I have that fs_mark test running here.  How far did yours get before
> it stopped?
>
> -chris
>
>   
I had gone (in heavy fsync mode) up to about 8 million files on a 1TB 
s-ata disk:

    17      8064000        20480          5.6         15301404

This is the new (no system sync() call) Chris special fs_mark.  The rate 
had been quite reasonable, starting out at around 160 20k files/sec, 
went under 100 files/sec at around 3 million files and then fell under 
50 files/sec at around 7.5 million before hitting this really low speed 
at just under 8 million.

Maybe it really was not hung, just extremely slow...

ric

Re: Hang running fs_mark

[Chris Mason]

On Thu, 2008-09-25 at 17:10 -0400, Ric Wheeler wrote:

> > Ok, I have that fs_mark test running here.  How far did yours get before
> > it stopped?
> >
> > -chris
> >
> >   
> I had gone (in heavy fsync mode) up to about 8 million files on a 1TB 
> s-ata disk:
> 
>     17      8064000        20480          5.6         15301404
> 
> This is the new (no system sync() call) Chris special fs_mark.  The rate 
> had been quite reasonable, starting out at around 160 20k files/sec, 
> went under 100 files/sec at around 3 million files and then fell under 
> 50 files/sec at around 7.5 million before hitting this really low speed 
> at just under 8 million.
> 
> Maybe it really was not hung, just extremely slow...
> 

I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I
get  155 files/sec ;)  My hope is that we're spinning around due to bad
accounting on the reserved extents, and that Yan's latest patch set will
fix it.

-chris

Re: Hang running fs_mark

[Ric Wheeler]

Chris Mason wrote:
> On Thu, 2008-09-25 at 17:10 -0400, Ric Wheeler wrote:
>
>   
>>> Ok, I have that fs_mark test running here.  How far did yours get before
>>> it stopped?
>>>
>>> -chris
>>>
>>>   
>>>       
>> I had gone (in heavy fsync mode) up to about 8 million files on a 1TB 
>> s-ata disk:
>>
>>     17      8064000        20480          5.6         15301404
>>
>> This is the new (no system sync() call) Chris special fs_mark.  The rate 
>> had been quite reasonable, starting out at around 160 20k files/sec, 
>> went under 100 files/sec at around 3 million files and then fell under 
>> 50 files/sec at around 7.5 million before hitting this really low speed 
>> at just under 8 million.
>>
>> Maybe it really was not hung, just extremely slow...
>>
>>     
>
> I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I
> get  155 files/sec ;)  My hope is that we're spinning around due to bad
> accounting on the reserved extents, and that Yan's latest patch set will
> fix it.
>
> -chris
I can update & restart my test as well. It is an odd box (8 CPUs, only 
1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in 
testing out edge conditions ;-)

ric

Re: Hang running fs_mark

[Chris Mason]

On Thu, 2008-09-25 at 18:58 -0400, Ric Wheeler wrote:
> >
> > I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I
> > get  155 files/sec ;)  My hope is that we're spinning around due to bad
> > accounting on the reserved extents, and that Yan's latest patch set will
> > fix it.
> >
> > -chris
> I can update & restart my test as well. It is an odd box (8 CPUs, only 
> 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in 
> testing out edge conditions ;-)

I'll push out Yan's patches tomorrow.  My box here is at 17.5 million
files and still going at 148 files/sec

-chris

Re: Hang running fs_mark

[Ric Wheeler]

Chris Mason wrote:
> On Thu, 2008-09-25 at 18:58 -0400, Ric Wheeler wrote:
>   
>>> I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I
>>> get  155 files/sec ;)  My hope is that we're spinning around due to bad
>>> accounting on the reserved extents, and that Yan's latest patch set will
>>> fix it.
>>>
>>> -chris
>>>       
>> I can update & restart my test as well. It is an odd box (8 CPUs, only 
>> 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in 
>> testing out edge conditions ;-)
>>     
>
> I'll push out Yan's patches tomorrow.  My box here is at 17.5 million
> files and still going at 148 files/sec
>
> -chris
>
>
>   
Sounds like a plan, thanks!

Ric

Re: Hang running fs_mark

[Chris Mason]

On Thu, 2008-09-25 at 21:01 -0400, Ric Wheeler wrote:
> Chris Mason wrote:
> > On Thu, 2008-09-25 at 18:58 -0400, Ric Wheeler wrote:
> >   
> >>> I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I
> >>> get  155 files/sec ;)  My hope is that we're spinning around due to bad
> >>> accounting on the reserved extents, and that Yan's latest patch set will
> >>> fix it.
> >>>
> >>> -chris
> >>>       
> >> I can update & restart my test as well. It is an odd box (8 CPUs, only 
> >> 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in 
> >> testing out edge conditions ;-)
> >>     
> >
> > I'll push out Yan's patches tomorrow.  My box here is at 17.5 million
> > files and still going at 148 files/sec
> >
> > -chris
> >
> >
> >   
> Sounds like a plan, thanks!

I declined a bit down to 60 files/sec, but overnight made it up to 58 or
so million files without stalling.  It is possible that my metadata
threshold changes caused problems for you, which might explain why my
4GB of ram lasted longer than your 1GB.

I'll try to rework the thresholds.

-chris

Re: Hang running fs_mark

[Chris Mason]

On Thu, 2008-09-25 at 21:01 -0400, Ric Wheeler wrote:
> Chris Mason wrote:
> > On Thu, 2008-09-25 at 18:58 -0400, Ric Wheeler wrote:
> >   
> >>> I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I
> >>> get  155 files/sec ;)  My hope is that we're spinning around due to bad
> >>> accounting on the reserved extents, and that Yan's latest patch set will
> >>> fix it.
> >>>
> >>> -chris
> >>>       
> >> I can update & restart my test as well. It is an odd box (8 CPUs, only 
> >> 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in 
> >> testing out edge conditions ;-)
> >>     
> >
> > I'll push out Yan's patches tomorrow.  My box here is at 17.5 million
> > files and still going at 148 files/sec
> >   
> Sounds like a plan, thanks!

I turned up a long standing balancing bug while testing Yan's latest
code.  It took forever to track down, but I've finally pushed out his
patches.

My fs_mark run never did stall, but I'm retesting now with all of the
latest changes.  You'll need to pull the latest unstable btrfs-progs and
kernel from git.

-chris