10 times higher disk load with btrfs

10 times higher disk load with btrfs

[Stefan Priebe]

Hi devs,

while btrfs is now declared as stable ;-) i wanted to retest btrfs on 
our production cluster on 2 out of 54 osds. So if they crash it doesn't 
hurt.

While if those OSDs run XFS have spikes of 20MB/s every 4-7s. The same 
OSDs after formatting them with btrfs have spikes of 190MB/s every 4-7s.

Why does just another filesystem raises the disk load by a factor of 10?

I'm running dumpling.

Greets Stefan

Re: 10 times higher disk load with btrfs

[Stefan Priebe]

Am 05.01.2015 um 19:36 schrieb Stefan Priebe:
> Hi devs,
>
> while btrfs is now declared as stable ;-) i wanted to retest btrfs on
> our production cluster on 2 out of 54 osds. So if they crash it doesn't
> hurt.
>
> While if those OSDs run XFS have spikes of 20MB/s every 4-7s. The same
> OSDs after formatting them with btrfs have spikes of 190MB/s every 4-7s.
>
> Why does just another filesystem raises the disk load by a factor of 10?

OK this seems to happen cause ceph is creating every 5s a new subvolume 
/ snap. Is this really expected / needed?

Stefan

>
> I'm running dumpling.
>
> Greets Stefan

Re: 10 times higher disk load with btrfs

[Sage Weil]

On Mon, 5 Jan 2015, Stefan Priebe wrote:
> 
> Am 05.01.2015 um 19:36 schrieb Stefan Priebe:
> > Hi devs,
> > 
> > while btrfs is now declared as stable ;-) i wanted to retest btrfs on
> > our production cluster on 2 out of 54 osds. So if they crash it doesn't
> > hurt.
> > 
> > While if those OSDs run XFS have spikes of 20MB/s every 4-7s. The same
> > OSDs after formatting them with btrfs have spikes of 190MB/s every 4-7s.
> > 
> > Why does just another filesystem raises the disk load by a factor of 10?
> 
> OK this seems to happen cause ceph is creating every 5s a new subvolume /
> snap. Is this really expected / needed?

You can disable it with

 filestore btrfs snap = false

I'm curious how much this drops the load down; originally the 
snaps were no more expensive than a regular sync but perhaps this 
has changed...

sage

Re: 10 times higher disk load with btrfs

[Stefan Priebe]

Hi Sage,

Am 05.01.2015 um 20:25 schrieb Sage Weil:
> On Mon, 5 Jan 2015, Stefan Priebe wrote:
>>
>> Am 05.01.2015 um 19:36 schrieb Stefan Priebe:
>>> Hi devs,
>>>
>>> while btrfs is now declared as stable ;-) i wanted to retest btrfs on
>>> our production cluster on 2 out of 54 osds. So if they crash it doesn't
>>> hurt.
>>>
>>> While if those OSDs run XFS have spikes of 20MB/s every 4-7s. The same
>>> OSDs after formatting them with btrfs have spikes of 190MB/s every 4-7s.
>>>
>>> Why does just another filesystem raises the disk load by a factor of 10?
>>
>> OK this seems to happen cause ceph is creating every 5s a new subvolume /
>> snap. Is this really expected / needed?
>
> You can disable it with
>
>   filestore btrfs snap = false
>
> I'm curious how much this drops the load down; originally the
> snaps were no more expensive than a regular sync but perhaps this
> has changed...

- with XFS the average write is at 9Mb/s
- with btrfs (filestore_btrfs_snap=true) write is at 40Mb/s
- with btrfs (filestore_btrfs_snap=false) write is at 20Mb/s

Stefan

Re: 10 times higher disk load with btrfs

[Mark Nelson]

On 01/05/2015 02:20 PM, Stefan Priebe wrote:
> Hi Sage,
>
> Am 05.01.2015 um 20:25 schrieb Sage Weil:
>> On Mon, 5 Jan 2015, Stefan Priebe wrote:
>>>
>>> Am 05.01.2015 um 19:36 schrieb Stefan Priebe:
>>>> Hi devs,
>>>>
>>>> while btrfs is now declared as stable ;-) i wanted to retest btrfs on
>>>> our production cluster on 2 out of 54 osds. So if they crash it doesn't
>>>> hurt.
>>>>
>>>> While if those OSDs run XFS have spikes of 20MB/s every 4-7s. The same
>>>> OSDs after formatting them with btrfs have spikes of 190MB/s every
>>>> 4-7s.
>>>>
>>>> Why does just another filesystem raises the disk load by a factor of
>>>> 10?
>>>
>>> OK this seems to happen cause ceph is creating every 5s a new
>>> subvolume /
>>> snap. Is this really expected / needed?
>>
>> You can disable it with
>>
>>   filestore btrfs snap = false
>>
>> I'm curious how much this drops the load down; originally the
>> snaps were no more expensive than a regular sync but perhaps this
>> has changed...
>
> - with XFS the average write is at 9Mb/s
> - with btrfs (filestore_btrfs_snap=true) write is at 40Mb/s
> - with btrfs (filestore_btrfs_snap=false) write is at 20Mb/s

Is that the average and not the spikes?  It looks like before the spikes 
were 20MB/s and 190MB/s?

>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 10 times higher disk load with btrfs

[Stefan Priebe]

Am 05.01.2015 um 21:29 schrieb Mark Nelson:
>
>
> On 01/05/2015 02:20 PM, Stefan Priebe wrote:
>> Hi Sage,
>>
>> Am 05.01.2015 um 20:25 schrieb Sage Weil:
>>> On Mon, 5 Jan 2015, Stefan Priebe wrote:
>>>>
>>>> Am 05.01.2015 um 19:36 schrieb Stefan Priebe:
>>>>> Hi devs,
>>>>>
>>>>> while btrfs is now declared as stable ;-) i wanted to retest btrfs on
>>>>> our production cluster on 2 out of 54 osds. So if they crash it
>>>>> doesn't
>>>>> hurt.
>>>>>
>>>>> While if those OSDs run XFS have spikes of 20MB/s every 4-7s. The same
>>>>> OSDs after formatting them with btrfs have spikes of 190MB/s every
>>>>> 4-7s.
>>>>>
>>>>> Why does just another filesystem raises the disk load by a factor of
>>>>> 10?
>>>>
>>>> OK this seems to happen cause ceph is creating every 5s a new
>>>> subvolume /
>>>> snap. Is this really expected / needed?
>>>
>>> You can disable it with
>>>
>>>   filestore btrfs snap = false
>>>
>>> I'm curious how much this drops the load down; originally the
>>> snaps were no more expensive than a regular sync but perhaps this
>>> has changed...
>>
>> - with XFS the average write is at 9Mb/s
>> - with btrfs (filestore_btrfs_snap=true) write is at 40Mb/s
>> - with btrfs (filestore_btrfs_snap=false) write is at 20Mb/s
>
> Is that the average and not the spikes?  It looks like before the spikes
> were 20MB/s and 190MB/s?

Yes these are average values.

Spikes:
- with XFS the spike write is at 20Mb/s
- with btrfs (filestore_btrfs_snap=true) spike write is 200Mb/s
- with btrfs (filestore_btrfs_snap=false) spike is still 185Mb/s but avg 
is 1/2 (20Mb/s) see above


>
>>
>> Stefan
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 10 times higher disk load with btrfs

[Alexandre DERUMIER]

Hi Stefan,

Do you see a difference if you force filestore journal writeahead for btrfs instead parrallel ?

filestore journal writeahead = 1
filestore journal parallel = 0

----- Mail original -----
De: "Stefan Priebe" <s.priebe@profihost.ag>
À: "Mark Nelson" <mnelson@redhat.com>, "Sage Weil" <sage@newdream.net>
Cc: "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Lundi 5 Janvier 2015 21:33:22
Objet: Re: 10 times higher disk load with btrfs

Am 05.01.2015 um 21:29 schrieb Mark Nelson: 
> 
> 
> On 01/05/2015 02:20 PM, Stefan Priebe wrote: 
>> Hi Sage, 
>> 
>> Am 05.01.2015 um 20:25 schrieb Sage Weil: 
>>> On Mon, 5 Jan 2015, Stefan Priebe wrote: 
>>>> 
>>>> Am 05.01.2015 um 19:36 schrieb Stefan Priebe: 
>>>>> Hi devs, 
>>>>> 
>>>>> while btrfs is now declared as stable ;-) i wanted to retest btrfs on 
>>>>> our production cluster on 2 out of 54 osds. So if they crash it 
>>>>> doesn't 
>>>>> hurt. 
>>>>> 
>>>>> While if those OSDs run XFS have spikes of 20MB/s every 4-7s. The same 
>>>>> OSDs after formatting them with btrfs have spikes of 190MB/s every 
>>>>> 4-7s. 
>>>>> 
>>>>> Why does just another filesystem raises the disk load by a factor of 
>>>>> 10? 
>>>> 
>>>> OK this seems to happen cause ceph is creating every 5s a new 
>>>> subvolume / 
>>>> snap. Is this really expected / needed? 
>>> 
>>> You can disable it with 
>>> 
>>> filestore btrfs snap = false 
>>> 
>>> I'm curious how much this drops the load down; originally the 
>>> snaps were no more expensive than a regular sync but perhaps this 
>>> has changed... 
>> 
>> - with XFS the average write is at 9Mb/s 
>> - with btrfs (filestore_btrfs_snap=true) write is at 40Mb/s 
>> - with btrfs (filestore_btrfs_snap=false) write is at 20Mb/s 
> 
> Is that the average and not the spikes? It looks like before the spikes 
> were 20MB/s and 190MB/s? 

Yes these are average values. 

Spikes: 
- with XFS the spike write is at 20Mb/s 
- with btrfs (filestore_btrfs_snap=true) spike write is 200Mb/s 
- with btrfs (filestore_btrfs_snap=false) spike is still 185Mb/s but avg 
is 1/2 (20Mb/s) see above 


> 
>> 
>> Stefan 
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
>> the body of a message to majordomo@vger.kernel.org 
>> More majordomo info at http://vger.kernel.org/majordomo-info.html 
-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 10 times higher disk load with btrfs

[Stefan Priebe - Profihost AG]

Hi,
Am 06.01.2015 um 04:44 schrieb Alexandre DERUMIER:
> Hi Stefan,
> 
> Do you see a difference if you force filestore journal writeahead for btrfs instead parrallel ?
> 
> filestore journal writeahead = 1
> filestore journal parallel = 0

i already tested filestore btrfs snap = false which automatically
disabled parallel write.

Stefan

> ----- Mail original -----
> De: "Stefan Priebe" <s.priebe@profihost.ag>
> À: "Mark Nelson" <mnelson@redhat.com>, "Sage Weil" <sage@newdream.net>
> Cc: "ceph-devel" <ceph-devel@vger.kernel.org>
> Envoyé: Lundi 5 Janvier 2015 21:33:22
> Objet: Re: 10 times higher disk load with btrfs
> 
> Am 05.01.2015 um 21:29 schrieb Mark Nelson: 
>>
>>
>> On 01/05/2015 02:20 PM, Stefan Priebe wrote: 
>>> Hi Sage, 
>>>
>>> Am 05.01.2015 um 20:25 schrieb Sage Weil: 
>>>> On Mon, 5 Jan 2015, Stefan Priebe wrote: 
>>>>>
>>>>> Am 05.01.2015 um 19:36 schrieb Stefan Priebe: 
>>>>>> Hi devs, 
>>>>>>
>>>>>> while btrfs is now declared as stable ;-) i wanted to retest btrfs on 
>>>>>> our production cluster on 2 out of 54 osds. So if they crash it 
>>>>>> doesn't 
>>>>>> hurt. 
>>>>>>
>>>>>> While if those OSDs run XFS have spikes of 20MB/s every 4-7s. The same 
>>>>>> OSDs after formatting them with btrfs have spikes of 190MB/s every 
>>>>>> 4-7s. 
>>>>>>
>>>>>> Why does just another filesystem raises the disk load by a factor of 
>>>>>> 10? 
>>>>>
>>>>> OK this seems to happen cause ceph is creating every 5s a new 
>>>>> subvolume / 
>>>>> snap. Is this really expected / needed? 
>>>>
>>>> You can disable it with 
>>>>
>>>> filestore btrfs snap = false 
>>>>
>>>> I'm curious how much this drops the load down; originally the 
>>>> snaps were no more expensive than a regular sync but perhaps this 
>>>> has changed... 
>>>
>>> - with XFS the average write is at 9Mb/s 
>>> - with btrfs (filestore_btrfs_snap=true) write is at 40Mb/s 
>>> - with btrfs (filestore_btrfs_snap=false) write is at 20Mb/s 
>>
>> Is that the average and not the spikes? It looks like before the spikes 
>> were 20MB/s and 190MB/s? 
> 
> Yes these are average values. 
> 
> Spikes: 
> - with XFS the spike write is at 20Mb/s 
> - with btrfs (filestore_btrfs_snap=true) spike write is 200Mb/s 
> - with btrfs (filestore_btrfs_snap=false) spike is still 185Mb/s but avg 
> is 1/2 (20Mb/s) see above 
> 
> 
>>
>>>
>>> Stefan 
>>> -- 
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
>>> the body of a message to majordomo@vger.kernel.org 
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html