Fwd: Current State of BTRFS

Fwd: Current State of BTRFS

[Florian Hofmann]

Hi everybody,

I am using btrfs as my main fs for some time now, but I am
experiencing severe performance drawbacks. I can't qualify the
circumstances, but sometimes during disc access the whole system
freezes for some time.

Maybe somebody could suggest some general things I could try to search
for the problem?

Thanks in advance,
Florian

---

Some background information:

$ mount | grep btrfs
/dev/mapper/sda2crypt on / type btrfs (rw,relatime,compress=lzo,space_cache)

Hmm ... couldn't think of more :(

Re: Fwd: Current State of BTRFS

[Josef Bacik]

On Fri, Feb 08, 2013 at 02:30:04PM -0700, Florian Hofmann wrote:
> Hi everybody,
> 
> I am using btrfs as my main fs for some time now, but I am
> experiencing severe performance drawbacks. I can't qualify the
> circumstances, but sometimes during disc access the whole system
> freezes for some time.
> 
> Maybe somebody could suggest some general things I could try to search
> for the problem?

Can I see

btrfs fi df /mnt/point
btrfs fi show

and then when you are having problems

echo w > /proc/sysrq-trigger

and capture dmesg.  Thanks,

Josef

Re: Fwd: Current State of BTRFS

[Florian Hofmann]

$ btrfs fi df /
Data: total=165.00GB, used=164.19GB
System, DUP: total=32.00MB, used=28.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=2.00GB, used=1.40GB

$ btrfs fi show
failed to read /dev/sr0
Label: none  uuid: b4ec0b14-2a42-47e3-a0cd-1257e789ed25
Total devices 1 FS bytes used 165.59GB
devid    1 size 600.35GB used 169.07GB path /dev/dm-0

Btrfs Btrfs v0.19

---

I just noticed that I can force 'it' by transferring a large file from
my NAS. I did the sysrq-trigger thing, but there is no suspicious
output in dmesg (http://pastebin.com/swrCdC3U).

Anything else?

2013/2/8 Josef Bacik <jbacik@fusionio.com>:
> On Fri, Feb 08, 2013 at 02:30:04PM -0700, Florian Hofmann wrote:
>> Hi everybody,
>>
>> I am using btrfs as my main fs for some time now, but I am
>> experiencing severe performance drawbacks. I can't qualify the
>> circumstances, but sometimes during disc access the whole system
>> freezes for some time.
>>
>> Maybe somebody could suggest some general things I could try to search
>> for the problem?
>
> Can I see
>
> btrfs fi df /mnt/point
> btrfs fi show
>
> and then when you are having problems
>
> echo w > /proc/sysrq-trigger
>
> and capture dmesg.  Thanks,
>
> Josef

Re: Fwd: Current State of BTRFS

[Florian Hofmann]

Oh ... I should have mentioned that btrfs is running on top of LUKS.

2013/2/8 Florian Hofmann <fhofmann@techfak.uni-bielefeld.de>:
> $ btrfs fi df /
> Data: total=165.00GB, used=164.19GB
> System, DUP: total=32.00MB, used=28.00KB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=2.00GB, used=1.40GB
>
> $ btrfs fi show
> failed to read /dev/sr0
> Label: none  uuid: b4ec0b14-2a42-47e3-a0cd-1257e789ed25
> Total devices 1 FS bytes used 165.59GB
> devid    1 size 600.35GB used 169.07GB path /dev/dm-0
>
> Btrfs Btrfs v0.19
>
> ---
>
> I just noticed that I can force 'it' by transferring a large file from
> my NAS. I did the sysrq-trigger thing, but there is no suspicious
> output in dmesg (http://pastebin.com/swrCdC3U).
>
> Anything else?
>
> 2013/2/8 Josef Bacik <jbacik@fusionio.com>:
>> On Fri, Feb 08, 2013 at 02:30:04PM -0700, Florian Hofmann wrote:
>>> Hi everybody,
>>>
>>> I am using btrfs as my main fs for some time now, but I am
>>> experiencing severe performance drawbacks. I can't qualify the
>>> circumstances, but sometimes during disc access the whole system
>>> freezes for some time.
>>>
>>> Maybe somebody could suggest some general things I could try to search
>>> for the problem?
>>
>> Can I see
>>
>> btrfs fi df /mnt/point
>> btrfs fi show
>>
>> and then when you are having problems
>>
>> echo w > /proc/sysrq-trigger
>>
>> and capture dmesg.  Thanks,
>>
>> Josef

Re: Fwd: Current State of BTRFS

[cwillu]

On Fri, Feb 8, 2013 at 4:56 PM, Florian Hofmann
<fhofmann@techfak.uni-bielefeld.de> wrote:
> Oh ... I should have mentioned that btrfs is running on top of LUKS.
>
> 2013/2/8 Florian Hofmann <fhofmann@techfak.uni-bielefeld.de>:
>> $ btrfs fi df /
>> Data: total=165.00GB, used=164.19GB
>> System, DUP: total=32.00MB, used=28.00KB
>> System: total=4.00MB, used=0.00
>> Metadata, DUP: total=2.00GB, used=1.40GB
>>
>> $ btrfs fi show
>> failed to read /dev/sr0
>> Label: none  uuid: b4ec0b14-2a42-47e3-a0cd-1257e789ed25
>> Total devices 1 FS bytes used 165.59GB
>> devid    1 size 600.35GB used 169.07GB path /dev/dm-0
>>
>> Btrfs Btrfs v0.19
>>
>> ---
>>
>> I just noticed that I can force 'it' by transferring a large file from
>> my NAS. I did the sysrq-trigger thing, but there is no suspicious
>> output in dmesg (http://pastebin.com/swrCdC3U).
>>
>> Anything else?

The pastebin didn't include any output from sysrq-w; even if there's
nothing to report there would still be a dozen lines or so per cpu; at
the absolute minimum there should be a line for each time you ran it:

[4477369.680307] SysRq : Show Blocked State

Note that you need to echo as root, or use the keyboard combo
alt-sysrq-w to trigger.

Re: Fwd: Current State of BTRFS

[Florian Hofmann]

I just had another big freeze ... I could change to a terminal and
sysrq-triggering gave this: http://pastebin.com/F1jQqtUQ

2013/2/8 Florian Hofmann <fhofmann@techfak.uni-bielefeld.de>:
> Oh ... I should have mentioned that btrfs is running on top of LUKS.
>
> 2013/2/8 Florian Hofmann <fhofmann@techfak.uni-bielefeld.de>:
>> $ btrfs fi df /
>> Data: total=165.00GB, used=164.19GB
>> System, DUP: total=32.00MB, used=28.00KB
>> System: total=4.00MB, used=0.00
>> Metadata, DUP: total=2.00GB, used=1.40GB
>>
>> $ btrfs fi show
>> failed to read /dev/sr0
>> Label: none  uuid: b4ec0b14-2a42-47e3-a0cd-1257e789ed25
>> Total devices 1 FS bytes used 165.59GB
>> devid    1 size 600.35GB used 169.07GB path /dev/dm-0
>>
>> Btrfs Btrfs v0.19
>>
>> ---
>>
>> I just noticed that I can force 'it' by transferring a large file from
>> my NAS. I did the sysrq-trigger thing, but there is no suspicious
>> output in dmesg (http://pastebin.com/swrCdC3U).
>>
>> Anything else?
>>
>> 2013/2/8 Josef Bacik <jbacik@fusionio.com>:
>>> On Fri, Feb 08, 2013 at 02:30:04PM -0700, Florian Hofmann wrote:
>>>> Hi everybody,
>>>>
>>>> I am using btrfs as my main fs for some time now, but I am
>>>> experiencing severe performance drawbacks. I can't qualify the
>>>> circumstances, but sometimes during disc access the whole system
>>>> freezes for some time.
>>>>
>>>> Maybe somebody could suggest some general things I could try to search
>>>> for the problem?
>>>
>>> Can I see
>>>
>>> btrfs fi df /mnt/point
>>> btrfs fi show
>>>
>>> and then when you are having problems
>>>
>>> echo w > /proc/sysrq-trigger
>>>
>>> and capture dmesg.  Thanks,
>>>
>>> Josef

Re: Fwd: Current State of BTRFS

[Florian Hofmann]

I ran it as root. The first times there was no output whatsoever. This
time triggering gave the SysRq : Show Blocked State line. Strange

2013/2/9 cwillu <cwillu@cwillu.com>:
> On Fri, Feb 8, 2013 at 4:56 PM, Florian Hofmann
> <fhofmann@techfak.uni-bielefeld.de> wrote:
>> Oh ... I should have mentioned that btrfs is running on top of LUKS.
>>
>> 2013/2/8 Florian Hofmann <fhofmann@techfak.uni-bielefeld.de>:
>>> $ btrfs fi df /
>>> Data: total=165.00GB, used=164.19GB
>>> System, DUP: total=32.00MB, used=28.00KB
>>> System: total=4.00MB, used=0.00
>>> Metadata, DUP: total=2.00GB, used=1.40GB
>>>
>>> $ btrfs fi show
>>> failed to read /dev/sr0
>>> Label: none  uuid: b4ec0b14-2a42-47e3-a0cd-1257e789ed25
>>> Total devices 1 FS bytes used 165.59GB
>>> devid    1 size 600.35GB used 169.07GB path /dev/dm-0
>>>
>>> Btrfs Btrfs v0.19
>>>
>>> ---
>>>
>>> I just noticed that I can force 'it' by transferring a large file from
>>> my NAS. I did the sysrq-trigger thing, but there is no suspicious
>>> output in dmesg (http://pastebin.com/swrCdC3U).
>>>
>>> Anything else?
>
> The pastebin didn't include any output from sysrq-w; even if there's
> nothing to report there would still be a dozen lines or so per cpu; at
> the absolute minimum there should be a line for each time you ran it:
>
> [4477369.680307] SysRq : Show Blocked State
>
> Note that you need to echo as root, or use the keyboard combo
> alt-sysrq-w to trigger.

Re: Fwd: Current State of BTRFS

[Marc MERLIN]

On Fri, Feb 08, 2013 at 10:30:04PM +0100, Florian Hofmann wrote:
> Hi everybody,
> 
> I am using btrfs as my main fs for some time now, but I am
> experiencing severe performance drawbacks. I can't qualify the
> circumstances, but sometimes during disc access the whole system
> freezes for some time.
> 
> Maybe somebody could suggest some general things I could try to search
> for the problem?

Are you running on an SSD?

If so, that is likely your problem.

Depending on your SSD:
- it's going bad
- it's running out of blocks to reallocate, or being very slow 
- other firmware issue

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/

Re: Fwd: Current State of BTRFS

[Florian Hofmann]

(Sadly) No SSD.

I just upgraded from kernel version 3.7.5 to 3.7.6 (running Arch
Linux) with no change in behavior.

2013/2/9 Marc MERLIN <marc@merlins.org>:
> On Fri, Feb 08, 2013 at 10:30:04PM +0100, Florian Hofmann wrote:
>> Hi everybody,
>>
>> I am using btrfs as my main fs for some time now, but I am
>> experiencing severe performance drawbacks. I can't qualify the
>> circumstances, but sometimes during disc access the whole system
>> freezes for some time.
>>
>> Maybe somebody could suggest some general things I could try to search
>> for the problem?
>
> Are you running on an SSD?
>
> If so, that is likely your problem.
>
> Depending on your SSD:
> - it's going bad
> - it's running out of blocks to reallocate, or being very slow
> - other firmware issue
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/


2013/2/9 Marc MERLIN <marc@merlins.org>:
> On Fri, Feb 08, 2013 at 10:30:04PM +0100, Florian Hofmann wrote:
>> Hi everybody,
>>
>> I am using btrfs as my main fs for some time now, but I am
>> experiencing severe performance drawbacks. I can't qualify the
>> circumstances, but sometimes during disc access the whole system
>> freezes for some time.
>>
>> Maybe somebody could suggest some general things I could try to search
>> for the problem?
>
> Are you running on an SSD?
>
> If so, that is likely your problem.
>
> Depending on your SSD:
> - it's going bad
> - it's running out of blocks to reallocate, or being very slow
> - other firmware issue
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/

Re: Fwd: Current State of BTRFS

[Josef Bacik]

On Fri, Feb 08, 2013 at 04:12:17PM -0700, Florian Hofmann wrote:
> I ran it as root. The first times there was no output whatsoever. This
> time triggering gave the SysRq : Show Blocked State line. Strange
>

Well that's weird, it looks like you don't have any blocked tasks, so either
sysrq+w is screwing up or you are getting stuck in something CPU intensive.  Try
doing sysrq+w next time it happens again and also run top and see if something
is using up 100% of the CPU.  If it's something chewing up CPU then I'll tell
you how to figure out what's going on.  Thanks,

Josef 

Re: Fwd: Current State of BTRFS

[Leonidas Spyropoulos]

On Mon, Feb 11, 2013 at 4:05 PM, Josef Bacik <jbacik@fusionio.com> wrote:
> On Fri, Feb 08, 2013 at 04:12:17PM -0700, Florian Hofmann wrote:
>> I ran it as root. The first times there was no output whatsoever. This
>> time triggering gave the SysRq : Show Blocked State line. Strange
>>
>
> Well that's weird, it looks like you don't have any blocked tasks, so either
> sysrq+w is screwing up or you are getting stuck in something CPU intensive.  Try
> doing sysrq+w next time it happens again and also run top and see if something
> is using up 100% of the CPU.  If it's something chewing up CPU then I'll tell
> you how to figure out what's going on.  Thanks,
>
I noticed you have the FS mounted with compress flag (compress=lzo).
Could it be that your CPU is bottle-necking the process?

> Josef
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Caution: breathing may be hazardous to your health.

#include <stdio.h>
int main(){printf("%s","\x4c\x65\x6f\x6e\x69\x64\x61\x73");}

Re: Fwd: Current State of BTRFS

[Florian Hofmann]

@Leonidas: I don't think so ... this is running on a i7 core and lzo
is pretty damn fast.

2013/2/11 Leonidas Spyropoulos <artafinde@gmail.com>:
> On Mon, Feb 11, 2013 at 4:05 PM, Josef Bacik <jbacik@fusionio.com> wrote:
>> On Fri, Feb 08, 2013 at 04:12:17PM -0700, Florian Hofmann wrote:
>>> I ran it as root. The first times there was no output whatsoever. This
>>> time triggering gave the SysRq : Show Blocked State line. Strange
>>>
>>
>> Well that's weird, it looks like you don't have any blocked tasks, so either
>> sysrq+w is screwing up or you are getting stuck in something CPU intensive.  Try
>> doing sysrq+w next time it happens again and also run top and see if something
>> is using up 100% of the CPU.  If it's something chewing up CPU then I'll tell
>> you how to figure out what's going on.  Thanks,
>>
> I noticed you have the FS mounted with compress flag (compress=lzo).
> Could it be that your CPU is bottle-necking the process?
>
>> Josef
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Caution: breathing may be hazardous to your health.
>
> #include <stdio.h>
> int main(){printf("%s","\x4c\x65\x6f\x6e\x69\x64\x61\x73");}

Re: Fwd: Current State of BTRFS

[Chris Samuel]

Hi Florian,

On 10/02/13 00:35, Florian Hofmann wrote:

> (Sadly) No SSD.

I think Marc was asking as they can cause issues, so don't be sad. :-)

> I just upgraded from kernel version 3.7.5 to 3.7.6 (running Arch
> Linux) with no change in behavior.

There are rarely any updates to btrfs in stable kernel releases, if you
want to try the current state of the code you'll need to try 3.8-rc7.

cheers,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC