Extreme slowdown

Extreme slowdown

[Tobias]

Hi all!

My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and 
deleting is horrible slow.

There are many files and many links on the FS.

# btrfs filesystem df /srv/storage
Data: total=3.09TB, used=3.07TB
System, DUP: total=8.00MB, used=496.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=154.88GB, used=70.56GB
Metadata: total=8.00MB, used=0.00

Maybe it's because there is so much Metadata and it needs so many seeks 
on the discs when deleting?

I'd like to delete some of the old Files but its so horrible slow that i 
think its maybe faster to copy all needed Data to a different Disc, 
killing the FS and move the Files back...

The machine is a QuadCore with 8GB RAM. Kernel is "3.1+for-linus".

Any hints how i could speed it up?

Tobias

Re: Extreme slowdown

[Liu Bo]

On 12/16/2011 02:49 AM, Tobias wrote:
> Hi all!
> 
> My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and
> deleting is horrible slow.
> 
> There are many files and many links on the FS.
> 
> # btrfs filesystem df /srv/storage
> Data: total=3.09TB, used=3.07TB
> System, DUP: total=8.00MB, used=496.00KB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=154.88GB, used=70.56GB
> Metadata: total=8.00MB, used=0.00
> 
> Maybe it's because there is so much Metadata and it needs so many seeks
> on the discs when deleting?
> 
> I'd like to delete some of the old Files but its so horrible slow that i
> think its maybe faster to copy all needed Data to a different Disc,
> killing the FS and move the Files back...
> 
> The machine is a QuadCore with 8GB RAM. Kernel is "3.1+for-linus".
> 
> Any hints how i could speed it up?
> 


I recommend to run defragment on your files,
it should help you, although defrag maybe slow, too.

thanks,
liubo

> Tobias
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Extreme slowdown

[Fajar A. Nugraha]

On Fri, Dec 16, 2011 at 1:49 AM, Tobias <tracer@robotech.de> wrote:
> Hi all!
>
> My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and
> deleting is horrible slow.
>
> There are many files and many links on the FS.
>
> # btrfs filesystem df /srv/storage
> Data: total=3.09TB, used=3.07TB

this is ... what, over 99% full?
The slow down is normal, somewhat. Same thing happens on zfs, which
became slower when usage is above 80-90%.

> Maybe it's because there is so much Metadata and it needs so many seeks on
> the discs when deleting?

I doubt it.

> I'd like to delete some of the old Files but its so horrible slow that i
> think its maybe faster to copy all needed Data to a different Disc, killing
> the FS and move the Files back...
>
> The machine is a QuadCore with 8GB RAM. Kernel is "3.1+for-linus".
>
> Any hints how i could speed it up?

Try:
- mounting it with nodatacow:
https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#Can_copy-on-write_be_turned_off_for_data_blocks.3F
- clobbering a big file:
https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#if_your_device_is_large_.28.3E16gb.29

... until you have at least 20% free space available.

-- 
Fajar

Re: Extreme slowdown

[Chester]

On Thu, Dec 15, 2011 at 8:19 PM, Fajar A. Nugraha <list@fajar.net> wrot=
e:
> On Fri, Dec 16, 2011 at 1:49 AM, Tobias <tracer@robotech.de> wrote:
>> Hi all!
>>
>> My BTRFS-FS ist getting really slow. Reading is ok, writing is slow =
and
>> deleting is horrible slow.
>>
>> There are many files and many links on the FS.
>>
>> # btrfs filesystem df /srv/storage
>> Data: total=3D3.09TB, used=3D3.07TB
>
> this is ... what, over 99% full?
> The slow down is normal, somewhat. Same thing happens on zfs, which
> became slower when usage is above 80-90%.

I don't think "total" actually means "total space available" it
increases as you use up more space.
>
>> Maybe it's because there is so much Metadata and it needs so many se=
eks on
>> the discs when deleting?
>
> I doubt it.
>
>> I'd like to delete some of the old Files but its so horrible slow th=
at i
>> think its maybe faster to copy all needed Data to a different Disc, =
killing
>> the FS and move the Files back...
>>
>> The machine is a QuadCore with 8GB RAM. Kernel is "3.1+for-linus".
>>
>> Any hints how i could speed it up?
>
> Try:
> - mounting it with nodatacow:
> https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#Can_copy-o=
n-write_be_turned_off_for_data_blocks.3F
> - clobbering a big file:
> https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#if_your_de=
vice_is_large_.28.3E16gb.29
>
> ... until you have at least 20% free space available.
>
> --
> Fajar
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs=
" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Extreme slowdown

[Shyam Prasad N]

On 12/16/2011 09:14 AM, Chester wrote:
> On Thu, Dec 15, 2011 at 8:19 PM, Fajar A. Nugraha<list@fajar.net>  wrote:
>> On Fri, Dec 16, 2011 at 1:49 AM, Tobias<tracer@robotech.de>  wrote:
>>> Hi all!
>>>
>>> My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and
>>> deleting is horrible slow.
>>>
>>> There are many files and many links on the FS.
>>>
>>> # btrfs filesystem df /srv/storage
>>> Data: total=3.09TB, used=3.07TB
>> this is ... what, over 99% full?
>> The slow down is normal, somewhat. Same thing happens on zfs, which
>> became slower when usage is above 80-90%.
> I don't think "total" actually means "total space available" it
> increases as you use up more space.
>>> Maybe it's because there is so much Metadata and it needs so many seeks on
>>> the discs when deleting?
>> I doubt it.
>>
>>> I'd like to delete some of the old Files but its so horrible slow that i
>>> think its maybe faster to copy all needed Data to a different Disc, killing
>>> the FS and move the Files back...
>>>
>>> The machine is a QuadCore with 8GB RAM. Kernel is "3.1+for-linus".
>>>
>>> Any hints how i could speed it up?
>> Try:
>> - mounting it with nodatacow:
>> https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#Can_copy-on-write_be_turned_off_for_data_blocks.3F
>> - clobbering a big file:
>> https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#if_your_device_is_large_.28.3E16gb.29
>>
>> ... until you have at least 20% free space available.
>>
>> --
>> Fajar
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
Wouldn't it be more valuable to first collect enough info/data to debug 
the problem, before suggesting ways to get out of this situation?

AFAIK, the slowness could be due to several reasons:
1. Slow disk writes.
2. Too many things to commit to the disk.
3. Too many things to do to actually accomplish the writes.
4. Important threads suspending due to lack of resources.
5. Something else which I've missed above.

Is there some sort of profiling which we can enable, which can give us 
info about the speeds and quantities of read and write traffic on the fs?
Aren't there btrfs statistics that we can print out, which can give us 
these info? If not, we probably should think about adding some.

P.S: btrfs n00b talking!

Regards,
Shyam

Re: Extreme slowdown

[Tobias]

Am 16.12.2011 03:09, schrieb Liu Bo:
> On 12/16/2011 02:49 AM, Tobias wrote:
>> Any hints how i could speed it up?
>>
>
> I recommend to run defragment on your files,
> it should help you, although defrag maybe slow, too.
>

When deleting files does it matter if the data is fragmented?

When i do normal file-access the speed of reading is ok so until not i 
didn't care about fragmentation. Especially because the data isn't 
changes after the first write.

Can i do a defrag on the metadata only?

Tobias

Re: Extreme slowdown

[Tobias]

Am 16.12.2011 05:15, schrieb Shyam Prasad N:
> On 12/16/2011 09:14 AM, Chester wrote:
>> On Thu, Dec 15, 2011 at 8:19 PM, Fajar A. Nugraha<list@fajar.net>  
>> wrote:
>>> On Fri, Dec 16, 2011 at 1:49 AM, Tobias<tracer@robotech.de>  wrote:
>>>> Hi all!
>>>>
>>>> My BTRFS-FS ist getting really slow. Reading is ok, writing is slow 
>>>> and
>>>> deleting is horrible slow.
>>>>
>>>> There are many files and many links on the FS.
>>>>
>>>> # btrfs filesystem df /srv/storage
>>>> Data: total=3.09TB, used=3.07TB
>>> this is ... what, over 99% full?
>>> The slow down is normal, somewhat. Same thing happens on zfs, which
>>> became slower when usage is above 80-90%.
>> I don't think "total" actually means "total space available" it
>> increases as you use up more space.
Chester is right, the disk is a 8TB Array - the fs is less than half filled.

> Wouldn't it be more valuable to first collect enough info/data to 
> debug the problem, before suggesting ways to get out of this situation?
>
> AFAIK, the slowness could be due to several reasons:
> 1. Slow disk writes.
That depends on what is slow for you. Its a SATA-Disc-Array so good 
linear Read/Write and low IOPs (compared to SSDs or SAS-Discs)

> 2. Too many things to commit to the disk.
I think there should be very little data to write/commit when deleting 
files. Remember: writing is slow, but acceptable - deleting is the problem

> 3. Too many things to do to actually accomplish the writes.
I just do rm -rf xxx. What has BTRFS to do to actually delete a file or 
directory?

> Is there some sort of profiling which we can enable, which can give us 
> info about the speeds and quantities of read and write traffic on the fs?
> Aren't there btrfs statistics that we can print out, which can give us 
> these info? If not, we probably should think about adding some.

Good question - maybe a dev can say?

Tobias

Re: Extreme slowdown

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1055 bytes --]

On Fri, Dec 16, 2011 at 09:51:56AM +0100, Tobias wrote:
> Am 16.12.2011 03:09, schrieb Liu Bo:
> >On 12/16/2011 02:49 AM, Tobias wrote:
> >>Any hints how i could speed it up?
> >>
> >
> >I recommend to run defragment on your files,
> >it should help you, although defrag maybe slow, too.
> >
> 
> When deleting files does it matter if the data is fragmented?

   Yes. Deleting a file involves visiting the metadata for every
extent (i.e. fragment) and updating it to reduce the reference count
and possibly free up the space -- which itself involves more writes,
as the free space cache is updated.

> When i do normal file-access the speed of reading is ok so until not
> i didn't care about fragmentation. Especially because the data isn't
> changes after the first write.
> 
> Can i do a defrag on the metadata only?

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
                    --- UNIX:  Italian pen maker. ---                    

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: Extreme slowdown

[Sander]

Tobias wrote (ao):
> >>>On Fri, Dec 16, 2011 at 1:49 AM, Tobias<tracer@robotech.de>  wrote:
> >>>>My BTRFS-FS ist getting really slow. Reading is ok, writing is
> >>>>slow and deleting is horrible slow.
> >>>>
> >>>>There are many files and many links on the FS.

Do you happen to have (many) snapshots? Are btrfs kernel threads using a
lot of cpu?

	Sander

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: Extreme slowdown

[Tobias]

Am 16.12.2011 10:19, schrieb Sander:
> Tobias wrote (ao):
>>>>> On Fri, Dec 16, 2011 at 1:49 AM, Tobias<tracer@robotech.de>   wrote:
>>>>>> My BTRFS-FS ist getting really slow. Reading is ok, writing is
>>>>>> slow and deleting is horrible slow.
>>>>>>
>>>>>> There are many files and many links on the FS.
> Do you happen to have (many) snapshots? Are btrfs kernel threads using a
> lot of cpu?
>
No Snapshots at all. Only compress-force=lzo, nothing else special.

The CPU is almost idle, IOwait is at 100%; looks likes the discs are 
accessing many small blocks. (seeking like hell...)

Output from iostat (average each over 120 seconds)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s 
avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0.00     0.42  195.31  190.44   781.23  1657.90    
12.65    64.91  168.90    3.42  338.60   2.51  96.64
sdb               0.00     0.50   75.00  336.57   300.00  2028.60    
11.32   105.41  255.83    3.16  312.14   2.24  92.38
sdb               0.00     0.62   72.03  441.72   288.11  2164.88     
9.55   116.39  226.27    4.20  262.48   1.89  97.29
sdb               0.00     0.20    1.07  468.74     4.26  2604.99    
11.11   145.07  308.91  581.30  308.29   2.13 100.00
sdb               0.00     0.00   20.00  326.97    80.00  2114.03    
12.65    38.56  111.41   58.02  114.67   2.83  98.28
sdb               0.00     0.00    5.02  563.35    20.08  3102.45    
10.99   137.71  242.26    6.44  244.36   1.70  96.62

Tobias