Btrfs performance problem; metadata size to blame?

Btrfs performance problem; metadata size to blame?

[John .]

Hi guys,

My Btrfs fs has a performance problem which I hope you can help me
solve. I have a dataset of around 3.15 TiB, that has lived on a ZFS
volume for almost two years (ZRAID1, 4 2TiB disks). In order to move
to Btrfs I bought myself a 4TiB disk with the idea of buying a new one
next week and balance it to a RAID1 of 2 4TiB disks.

I created a single disk Btrfs volume with the default mkfs options (no
data duplication, metadata duplication on). Next I transferred my
dataset to this disk (no problems there). Today when I tried to create
a directory and I noticed the Btrfs volume was awefully slow; it took
a few seconds to create the directory and a few to delete it (which
should be milliseconds as you know). In fact each and every operation
on the volume grinded the fs down to a halt.

FS information:

# btrfs fi df /storage
Data: total=3.29TB, used=3.15TB
System, DUP: total=8.00MB, used=360.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=4.00GB, used=3.88GB
Metadata: total=8.00MB, used=0.00

# btrfs fi show
Label: 'storage'  uuid: 3fa262cd-baa9-46dc-92a8-318c87166186
        Total devices 1 FS bytes used 3.16TB
        devid    1 size 3.64TB used 3.30TB path /dev/sdb

I suspect my performance blow has everything to do with the abysmally
low amount of space for metadata that is left, but since I am not a
Btrfs guru I don't now whether this is truly the case and/or how to
solve it. btrfs fi balance start -dusage=5 did not help.

Yours,

John

Re: Btrfs performance problem; metadata size to blame?

[Harald Glatt]

On Sun, Apr 28, 2013 at 9:04 PM, John . <btrfsprob@gmail.com> wrote:
> Hi guys,
>
> My Btrfs fs has a performance problem which I hope you can help me
> solve. I have a dataset of around 3.15 TiB, that has lived on a ZFS
> volume for almost two years (ZRAID1, 4 2TiB disks). In order to move
> to Btrfs I bought myself a 4TiB disk with the idea of buying a new one
> next week and balance it to a RAID1 of 2 4TiB disks.
>
> I created a single disk Btrfs volume with the default mkfs options (no
> data duplication, metadata duplication on). Next I transferred my
> dataset to this disk (no problems there). Today when I tried to create
> a directory and I noticed the Btrfs volume was awefully slow; it took
> a few seconds to create the directory and a few to delete it (which
> should be milliseconds as you know). In fact each and every operation
> on the volume grinded the fs down to a halt.
>
> FS information:
>
> # btrfs fi df /storage
> Data: total=3.29TB, used=3.15TB
> System, DUP: total=8.00MB, used=360.00KB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=4.00GB, used=3.88GB
> Metadata: total=8.00MB, used=0.00
>
> # btrfs fi show
> Label: 'storage'  uuid: 3fa262cd-baa9-46dc-92a8-318c87166186
>         Total devices 1 FS bytes used 3.16TB
>         devid    1 size 3.64TB used 3.30TB path /dev/sdb
>
> I suspect my performance blow has everything to do with the abysmally
> low amount of space for metadata that is left, but since I am not a
> Btrfs guru I don't now whether this is truly the case and/or how to
> solve it. btrfs fi balance start -dusage=5 did not help.
>
> Yours,
>
> John
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Try to defragment the root of the volume (e.g the mountpoint). While
it's mounted:
btrfs fi defrag /path/to/mnt

Then try performance again

Re: Btrfs performance problem; metadata size to blame?

[John .]

I use Ubuntu with kernel 3.8.0-19-generic. I also tested with the
latest live disk of Arch Linux; write performance was the same (bad).

My mount options: rw,compress=lzo.
Iotop does not show any strange disk activity.

2013/4/28 Harald Glatt <mail@hachre.de>:
> On Sun, Apr 28, 2013 at 9:10 PM, John . <btrfsprob@gmail.com> wrote:
>> Hi Harald,
>>
>> I did perform a defrag of the volume a few hours ago. This did not
>> make a difference. :(
>>
>> Yours,
>>
>> John
>>
>> 2013/4/28 Harald Glatt <mail@hachre.de>:
>>> On Sun, Apr 28, 2013 at 9:04 PM, John . <btrfsprob@gmail.com> wrote:
>>>> Hi guys,
>>>>
>>>> My Btrfs fs has a performance problem which I hope you can help me
>>>> solve. I have a dataset of around 3.15 TiB, that has lived on a ZFS
>>>> volume for almost two years (ZRAID1, 4 2TiB disks). In order to move
>>>> to Btrfs I bought myself a 4TiB disk with the idea of buying a new one
>>>> next week and balance it to a RAID1 of 2 4TiB disks.
>>>>
>>>> I created a single disk Btrfs volume with the default mkfs options (no
>>>> data duplication, metadata duplication on). Next I transferred my
>>>> dataset to this disk (no problems there). Today when I tried to create
>>>> a directory and I noticed the Btrfs volume was awefully slow; it took
>>>> a few seconds to create the directory and a few to delete it (which
>>>> should be milliseconds as you know). In fact each and every operation
>>>> on the volume grinded the fs down to a halt.
>>>>
>>>> FS information:
>>>>
>>>> # btrfs fi df /storage
>>>> Data: total=3.29TB, used=3.15TB
>>>> System, DUP: total=8.00MB, used=360.00KB
>>>> System: total=4.00MB, used=0.00
>>>> Metadata, DUP: total=4.00GB, used=3.88GB
>>>> Metadata: total=8.00MB, used=0.00
>>>>
>>>> # btrfs fi show
>>>> Label: 'storage'  uuid: 3fa262cd-baa9-46dc-92a8-318c87166186
>>>>         Total devices 1 FS bytes used 3.16TB
>>>>         devid    1 size 3.64TB used 3.30TB path /dev/sdb
>>>>
>>>> I suspect my performance blow has everything to do with the abysmally
>>>> low amount of space for metadata that is left, but since I am not a
>>>> Btrfs guru I don't now whether this is truly the case and/or how to
>>>> solve it. btrfs fi balance start -dusage=5 did not help.
>>>>
>>>> Yours,
>>>>
>>>> John
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>> Try to defragment the root of the volume (e.g the mountpoint). While
>>> it's mounted:
>>> btrfs fi defrag /path/to/mnt
>>>
>>> Then try performance again
>
> What kernel version do you use? What are your mount options? Try to
> run iotop and see if there is any unusual activity...

Re: Btrfs performance problem; metadata size to blame?

[Harald Glatt]

On Sun, Apr 28, 2013 at 9:18 PM, John . <btrfsprob@gmail.com> wrote:
> I use Ubuntu with kernel 3.8.0-19-generic. I also tested with the
> latest live disk of Arch Linux; write performance was the same (bad).
>
> My mount options: rw,compress=lzo.
> Iotop does not show any strange disk activity.
>
> 2013/4/28 Harald Glatt <mail@hachre.de>:
>> On Sun, Apr 28, 2013 at 9:10 PM, John . <btrfsprob@gmail.com> wrote:
>>> Hi Harald,
>>>
>>> I did perform a defrag of the volume a few hours ago. This did not
>>> make a difference. :(
>>>
>>> Yours,
>>>
>>> John
>>>
>>> 2013/4/28 Harald Glatt <mail@hachre.de>:
>>>> On Sun, Apr 28, 2013 at 9:04 PM, John . <btrfsprob@gmail.com> wrote:
>>>>> Hi guys,
>>>>>
>>>>> My Btrfs fs has a performance problem which I hope you can help me
>>>>> solve. I have a dataset of around 3.15 TiB, that has lived on a ZFS
>>>>> volume for almost two years (ZRAID1, 4 2TiB disks). In order to move
>>>>> to Btrfs I bought myself a 4TiB disk with the idea of buying a new one
>>>>> next week and balance it to a RAID1 of 2 4TiB disks.
>>>>>
>>>>> I created a single disk Btrfs volume with the default mkfs options (no
>>>>> data duplication, metadata duplication on). Next I transferred my
>>>>> dataset to this disk (no problems there). Today when I tried to create
>>>>> a directory and I noticed the Btrfs volume was awefully slow; it took
>>>>> a few seconds to create the directory and a few to delete it (which
>>>>> should be milliseconds as you know). In fact each and every operation
>>>>> on the volume grinded the fs down to a halt.
>>>>>
>>>>> FS information:
>>>>>
>>>>> # btrfs fi df /storage
>>>>> Data: total=3.29TB, used=3.15TB
>>>>> System, DUP: total=8.00MB, used=360.00KB
>>>>> System: total=4.00MB, used=0.00
>>>>> Metadata, DUP: total=4.00GB, used=3.88GB
>>>>> Metadata: total=8.00MB, used=0.00
>>>>>
>>>>> # btrfs fi show
>>>>> Label: 'storage'  uuid: 3fa262cd-baa9-46dc-92a8-318c87166186
>>>>>         Total devices 1 FS bytes used 3.16TB
>>>>>         devid    1 size 3.64TB used 3.30TB path /dev/sdb
>>>>>
>>>>> I suspect my performance blow has everything to do with the abysmally
>>>>> low amount of space for metadata that is left, but since I am not a
>>>>> Btrfs guru I don't now whether this is truly the case and/or how to
>>>>> solve it. btrfs fi balance start -dusage=5 did not help.
>>>>>
>>>>> Yours,
>>>>>
>>>>> John
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>> Try to defragment the root of the volume (e.g the mountpoint). While
>>>> it's mounted:
>>>> btrfs fi defrag /path/to/mnt
>>>>
>>>> Then try performance again
>>
>> What kernel version do you use? What are your mount options? Try to
>> run iotop and see if there is any unusual activity...

Try mount -o inode_cache,space_cache,autodefrag - first mount with the
new options might take a while, also there might be disk activity for
a while after mounting with this...

Re: Btrfs performance problem; metadata size to blame?

[John .]

I just started up my usenet reader (which generate a lot of small
files) and transferred two large files (7,5GiB) at the same time.
Performance seems all right again! :D Thanks!

Could you explain to me why each of the options could have a positive
effect on performance? The wiki explains what the option imply, but
not how they could help boost performance.

root@host:/storage# time mkdir test

real 0m0.001s
user 0m0.000s
sys 0m0.000s
root@test:/storage# time rm -Rf test

real 0m0.001s
user 0m0.000s
sys 0m0.000s

Because performance was good again I was able to spam the volume with
data and the metadata size also grew. No problems in that department
either. ;-)

2013/4/28 Harald Glatt <mail@hachre.de>:
> On Sun, Apr 28, 2013 at 9:18 PM, John . <btrfsprob@gmail.com> wrote:
>> I use Ubuntu with kernel 3.8.0-19-generic. I also tested with the
>> latest live disk of Arch Linux; write performance was the same (bad).
>>
>> My mount options: rw,compress=lzo.
>> Iotop does not show any strange disk activity.
>>
>> 2013/4/28 Harald Glatt <mail@hachre.de>:
>>> On Sun, Apr 28, 2013 at 9:10 PM, John . <btrfsprob@gmail.com> wrote:
>>>> Hi Harald,
>>>>
>>>> I did perform a defrag of the volume a few hours ago. This did not
>>>> make a difference. :(
>>>>
>>>> Yours,
>>>>
>>>> John
>>>>
>>>> 2013/4/28 Harald Glatt <mail@hachre.de>:
>>>>> On Sun, Apr 28, 2013 at 9:04 PM, John . <btrfsprob@gmail.com> wrote:
>>>>>> Hi guys,
>>>>>>
>>>>>> My Btrfs fs has a performance problem which I hope you can help me
>>>>>> solve. I have a dataset of around 3.15 TiB, that has lived on a ZFS
>>>>>> volume for almost two years (ZRAID1, 4 2TiB disks). In order to move
>>>>>> to Btrfs I bought myself a 4TiB disk with the idea of buying a new one
>>>>>> next week and balance it to a RAID1 of 2 4TiB disks.
>>>>>>
>>>>>> I created a single disk Btrfs volume with the default mkfs options (no
>>>>>> data duplication, metadata duplication on). Next I transferred my
>>>>>> dataset to this disk (no problems there). Today when I tried to create
>>>>>> a directory and I noticed the Btrfs volume was awefully slow; it took
>>>>>> a few seconds to create the directory and a few to delete it (which
>>>>>> should be milliseconds as you know). In fact each and every operation
>>>>>> on the volume grinded the fs down to a halt.
>>>>>>
>>>>>> FS information:
>>>>>>
>>>>>> # btrfs fi df /storage
>>>>>> Data: total=3.29TB, used=3.15TB
>>>>>> System, DUP: total=8.00MB, used=360.00KB
>>>>>> System: total=4.00MB, used=0.00
>>>>>> Metadata, DUP: total=4.00GB, used=3.88GB
>>>>>> Metadata: total=8.00MB, used=0.00
>>>>>>
>>>>>> # btrfs fi show
>>>>>> Label: 'storage'  uuid: 3fa262cd-baa9-46dc-92a8-318c87166186
>>>>>>         Total devices 1 FS bytes used 3.16TB
>>>>>>         devid    1 size 3.64TB used 3.30TB path /dev/sdb
>>>>>>
>>>>>> I suspect my performance blow has everything to do with the abysmally
>>>>>> low amount of space for metadata that is left, but since I am not a
>>>>>> Btrfs guru I don't now whether this is truly the case and/or how to
>>>>>> solve it. btrfs fi balance start -dusage=5 did not help.
>>>>>>
>>>>>> Yours,
>>>>>>
>>>>>> John
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>> Try to defragment the root of the volume (e.g the mountpoint). While
>>>>> it's mounted:
>>>>> btrfs fi defrag /path/to/mnt
>>>>>
>>>>> Then try performance again
>>>
>>> What kernel version do you use? What are your mount options? Try to
>>> run iotop and see if there is any unusual activity...
>
> Try mount -o inode_cache,space_cache,autodefrag - first mount with the
> new options might take a while, also there might be disk activity for
> a while after mounting with this...

Re: Btrfs performance problem; metadata size to blame?

[Harald Glatt]

On Sun, Apr 28, 2013 at 9:53 PM, John . <btrfsprob@gmail.com> wrote:
> I just started up my usenet reader (which generate a lot of small
> files) and transferred two large files (7,5GiB) at the same time.
> Performance seems all right again! :D Thanks!
>
> Could you explain to me why each of the options could have a positive
> effect on performance? The wiki explains what the option imply, but
> not how they could help boost performance.
>
> root@host:/storage# time mkdir test
>
> real 0m0.001s
> user 0m0.000s
> sys 0m0.000s
> root@test:/storage# time rm -Rf test
>
> real 0m0.001s
> user 0m0.000s
> sys 0m0.000s
>
> Because performance was good again I was able to spam the volume with
> data and the metadata size also grew. No problems in that department
> either. ;-)
>
> 2013/4/28 Harald Glatt <mail@hachre.de>:
>> On Sun, Apr 28, 2013 at 9:18 PM, John . <btrfsprob@gmail.com> wrote:
>>> I use Ubuntu with kernel 3.8.0-19-generic. I also tested with the
>>> latest live disk of Arch Linux; write performance was the same (bad).
>>>
>>> My mount options: rw,compress=lzo.
>>> Iotop does not show any strange disk activity.
>>>
>>> 2013/4/28 Harald Glatt <mail@hachre.de>:
>>>> On Sun, Apr 28, 2013 at 9:10 PM, John . <btrfsprob@gmail.com> wrote:
>>>>> Hi Harald,
>>>>>
>>>>> I did perform a defrag of the volume a few hours ago. This did not
>>>>> make a difference. :(
>>>>>
>>>>> Yours,
>>>>>
>>>>> John
>>>>>
>>>>> 2013/4/28 Harald Glatt <mail@hachre.de>:
>>>>>> On Sun, Apr 28, 2013 at 9:04 PM, John . <btrfsprob@gmail.com> wrote:
>>>>>>> Hi guys,
>>>>>>>
>>>>>>> My Btrfs fs has a performance problem which I hope you can help me
>>>>>>> solve. I have a dataset of around 3.15 TiB, that has lived on a ZFS
>>>>>>> volume for almost two years (ZRAID1, 4 2TiB disks). In order to move
>>>>>>> to Btrfs I bought myself a 4TiB disk with the idea of buying a new one
>>>>>>> next week and balance it to a RAID1 of 2 4TiB disks.
>>>>>>>
>>>>>>> I created a single disk Btrfs volume with the default mkfs options (no
>>>>>>> data duplication, metadata duplication on). Next I transferred my
>>>>>>> dataset to this disk (no problems there). Today when I tried to create
>>>>>>> a directory and I noticed the Btrfs volume was awefully slow; it took
>>>>>>> a few seconds to create the directory and a few to delete it (which
>>>>>>> should be milliseconds as you know). In fact each and every operation
>>>>>>> on the volume grinded the fs down to a halt.
>>>>>>>
>>>>>>> FS information:
>>>>>>>
>>>>>>> # btrfs fi df /storage
>>>>>>> Data: total=3.29TB, used=3.15TB
>>>>>>> System, DUP: total=8.00MB, used=360.00KB
>>>>>>> System: total=4.00MB, used=0.00
>>>>>>> Metadata, DUP: total=4.00GB, used=3.88GB
>>>>>>> Metadata: total=8.00MB, used=0.00
>>>>>>>
>>>>>>> # btrfs fi show
>>>>>>> Label: 'storage'  uuid: 3fa262cd-baa9-46dc-92a8-318c87166186
>>>>>>>         Total devices 1 FS bytes used 3.16TB
>>>>>>>         devid    1 size 3.64TB used 3.30TB path /dev/sdb
>>>>>>>
>>>>>>> I suspect my performance blow has everything to do with the abysmally
>>>>>>> low amount of space for metadata that is left, but since I am not a
>>>>>>> Btrfs guru I don't now whether this is truly the case and/or how to
>>>>>>> solve it. btrfs fi balance start -dusage=5 did not help.
>>>>>>>
>>>>>>> Yours,
>>>>>>>
>>>>>>> John
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>> Try to defragment the root of the volume (e.g the mountpoint). While
>>>>>> it's mounted:
>>>>>> btrfs fi defrag /path/to/mnt
>>>>>>
>>>>>> Then try performance again
>>>>
>>>> What kernel version do you use? What are your mount options? Try to
>>>> run iotop and see if there is any unusual activity...
>>
>> Try mount -o inode_cache,space_cache,autodefrag - first mount with the
>> new options might take a while, also there might be disk activity for
>> a while after mounting with this...

Great, glad it helped!

I'm not dev so I can only give vague and possibly wrong answers here:

- autodefrag: would actually negatively impact immediate performance
but will make a difference compared to no defrag over time
- inode_cache: is apparently caching the latest inode number(?) that
has been used so that whena  new one has to be given it is immediately
available instead of searching for it again
- space_cache: caches the amount of space free, otherwise each space
free 'question' to the volume would require it to recaculate it

If you want better answers you should wait until a dev answers or corrects me :)
Or come onto #btrfs on irc.freenode.net and ask there.

Re: Btrfs performance problem; metadata size to blame?

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 28/04/13 12:57, Harald Glatt wrote:
> If you want better answers ...

There is a lot of good information at the wiki and it does see regular
updates.  For example the performance mount options are on this page:

  https://btrfs.wiki.kernel.org/index.php/Mount_options

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlF9g+wACgkQmOOfHg372QQu6QCffq/cB7GPutTwiAUE0CyTuIJx
Qj8AnjsqxVyPrK5FTDqaLk1d1lsYYB38
=6HN3
-----END PGP SIGNATURE-----

Re: Btrfs performance problem; metadata size to blame?

[cwillu]

On Sun, Apr 28, 2013 at 2:17 PM, Roger Binns <rogerb@rogerbinns.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 28/04/13 12:57, Harald Glatt wrote:
>> If you want better answers ...
>
> There is a lot of good information at the wiki and it does see regular
> updates.  For example the performance mount options are on this page:
>
>   https://btrfs.wiki.kernel.org/index.php/Mount_options
>
> Roger
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
>
> iEYEARECAAYFAlF9g+wACgkQmOOfHg372QQu6QCffq/cB7GPutTwiAUE0CyTuIJx
> Qj8AnjsqxVyPrK5FTDqaLk1d1lsYYB38
> =6HN3
> -----END PGP SIGNATURE-----
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Btrfs performance problem; metadata size to blame?

[cwillu]

[how'd that send button get there]

space_cache is the default, set by mkfs, for a year or so now.  It's
sticky, so even if it wasn't, you'd only need to mount with it once.