Ceph RBD performance - random writes

Ceph RBD performance - random writes

[Mark Kirkwood]

[-- Attachment #1: Type: text/plain, Size: 2585 bytes --]

I've been looking at using Ceph RBD as a block store for database use. 
As part of this I'm looking a how (particularly random) IO of smallish 
(4K, 8K) block sizes performs.

I've setup Ceph with a single osd and mon spread over two SSD (Intel 
520) - 2G journal on one and the osd data on the other (xfs filesystem). 
The Intel's are pretty fast, and (despite being shackled by a crappy 
Nvidia SATA controller) fly for random IO.

However I am not seeing that reflected in the RBD case. I have the 
device mounted on the local machine where the osd and mon are running 
(so network performance should not be a factor here).

Here is what I did:

Create a rbd device of 10G and mount on /mnt/vol0:

$ rbd create --size 10240 vol0
$ rbd map vol0
$ mkfx.xfs /dev/rbd0
$ rbd mount /dev/rdb0 /mnt/vol0

Make a file:

$ dd if=/dev/zero of=/mnt/vol0/dump/file bs=4k count=300000 conv=fsync
1228800000 bytes (1.2 GB) copied, 13.4361 s, 91.5 MB/s

Performance ok if file size < journal (2G).

$ dd if=/dev/zero of=/mnt/vol0/dump/file bs=4096k count=200 conv=fsync
838860800 bytes (839 MB) copied, 9.47086 s, 88.6 MB/s

Not so good if file size > journal.

$ dd if=/dev/zero of=/mnt/vol0/dump/file bs=4096k count=1000 conv=fsync
4194304000 bytes (4.2 GB) copied, 279.891 s, 15.0 MB/s

Random writes (see attached file) sync'ed with sync_file_range are ok if 
block size big:

$ ./writetest /mnt/vol0/dump/file 4194304 0 1
random writes: 292 of: 4194304 bytes elapsed: 9.8397s io rate: 30/s 
(118.70 MB/s)

$ ./writetest /mnt/vol0/dump/file 1048576 0 1
random writes: 1171 of: 1048576 bytes elapsed: 10.6042s io rate: 110/s 
(110.43 MB/s)

$ ./writetest /mnt/vol0/dump/file 131072 0 1
random writes: 9375 of: 131072 bytes elapsed: 15.8075s io rate: 593/s 
(74.13 MB/s)


However smallish block size is suicide (trigger suicide assert after a 
while), I see 100 IOPS or less on actual devices, all 100% util:

$ ./writetest /mnt/vol0/dump/file 8192 0 1

I am running into http://tracker.newdream.net/issues/2784 here I think.

Note that the actual SSD are very fast for this when accessed directly:

$ ./writetest /data1/ceph/1/file 8192 0 1
random writes: 1000000 of: 8192 bytes elapsed: 125.7907s io rate: 7950/s 
(62.11 MB/s)


Thanks for your patience in reading so far - some actual questions now :-)

1/ Why is the appending write from dd when the size of file > journal so 
slow, despite reasonably capable storage devices?

2/ Is the sudden dramatic drop in random write performance a 
manifestation of the "small requests  are slow" issue? or is this 
something else?


Thanks

Mark



[-- Attachment #2: ceph.conf.gz --]
[-- Type: application/x-gzip, Size: 1237 bytes --]

[-- Attachment #3: writetest.c.gz --]
[-- Type: application/x-gzip, Size: 1656 bytes --]

Re: Ceph RBD performance - random writes

[Josh Durgin]

On 08/07/2012 10:19 PM, Mark Kirkwood wrote:
> I've been looking at using Ceph RBD as a block store for database use.
> As part of this I'm looking a how (particularly random) IO of smallish
> (4K, 8K) block sizes performs.
>
> I've setup Ceph with a single osd and mon spread over two SSD (Intel
> 520) - 2G journal on one and the osd data on the other (xfs filesystem).
> The Intel's are pretty fast, and (despite being shackled by a crappy
> Nvidia SATA controller) fly for random IO.
>
> However I am not seeing that reflected in the RBD case. I have the
> device mounted on the local machine where the osd and mon are running
> (so network performance should not be a factor here).
>
> Here is what I did:
>
> Create a rbd device of 10G and mount on /mnt/vol0:
>
> $ rbd create --size 10240 vol0
> $ rbd map vol0
> $ mkfx.xfs /dev/rbd0
> $ rbd mount /dev/rdb0 /mnt/vol0
>
> Make a file:
>
> $ dd if=/dev/zero of=/mnt/vol0/dump/file bs=4k count=300000 conv=fsync
> 1228800000 bytes (1.2 GB) copied, 13.4361 s, 91.5 MB/s
>
> Performance ok if file size < journal (2G).
>
> $ dd if=/dev/zero of=/mnt/vol0/dump/file bs=4096k count=200 conv=fsync
> 838860800 bytes (839 MB) copied, 9.47086 s, 88.6 MB/s
>
> Not so good if file size > journal.
>
> $ dd if=/dev/zero of=/mnt/vol0/dump/file bs=4096k count=1000 conv=fsync
> 4194304000 bytes (4.2 GB) copied, 279.891 s, 15.0 MB/s
>
> Random writes (see attached file) sync'ed with sync_file_range are ok if
> block size big:
>
> $ ./writetest /mnt/vol0/dump/file 4194304 0 1
> random writes: 292 of: 4194304 bytes elapsed: 9.8397s io rate: 30/s
> (118.70 MB/s)
>
> $ ./writetest /mnt/vol0/dump/file 1048576 0 1
> random writes: 1171 of: 1048576 bytes elapsed: 10.6042s io rate: 110/s
> (110.43 MB/s)
>
> $ ./writetest /mnt/vol0/dump/file 131072 0 1
> random writes: 9375 of: 131072 bytes elapsed: 15.8075s io rate: 593/s
> (74.13 MB/s)
>
>
> However smallish block size is suicide (trigger suicide assert after a
> while), I see 100 IOPS or less on actual devices, all 100% util:
>
> $ ./writetest /mnt/vol0/dump/file 8192 0 1
>
> I am running into http://tracker.newdream.net/issues/2784 here I think.

This can be a sign of a bug in the underlying filesystem or hardware -
maybe your controller? That assert is hit when a single operation to
the filesystem beneath the osd takes longer than 180 seconds (by
default).

> Note that the actual SSD are very fast for this when accessed directly:
>
> $ ./writetest /data1/ceph/1/file 8192 0 1
> random writes: 1000000 of: 8192 bytes elapsed: 125.7907s io rate: 7950/s
> (62.11 MB/s)
>
>
> Thanks for your patience in reading so far - some actual questions now :-)
>
> 1/ Why is the appending write from dd when the size of file > journal so
> slow, despite reasonably capable storage devices?

It's possible you need to use more threads to have more operations in
flight in to the filestore (the main storage for the osd). Try
something like this in your ceph configuration for the osds:

     osd op threads = 24
     osd disk threads = 24
     filestore op threads = 6
     filestore queue max ops = 24

(from http://www.spinics.net/lists/ceph-devel/msg07128.html)

> 2/ Is the sudden dramatic drop in random write performance a
> manifestation of the "small requests  are slow" issue? or is this
> something else?

It's probably that. Sam's actively looking into it, and once he has
something it will be interesting to see how well it works on your
hardware.

Josh

> Thanks
>
> Mark
>
>

Re: Ceph RBD performance - random writes

[Mark Nelson]

On 8/8/12 1:46 PM, Josh Durgin wrote:
> On 08/07/2012 10:19 PM, Mark Kirkwood wrote:
>> I've been looking at using Ceph RBD as a block store for database use.
>> As part of this I'm looking a how (particularly random) IO of smallish
>> (4K, 8K) block sizes performs.
>>
>> I've setup Ceph with a single osd and mon spread over two SSD (Intel
>> 520) - 2G journal on one and the osd data on the other (xfs filesystem).
>> The Intel's are pretty fast, and (despite being shackled by a crappy
>> Nvidia SATA controller) fly for random IO.
>>
>> However I am not seeing that reflected in the RBD case. I have the
>> device mounted on the local machine where the osd and mon are running
>> (so network performance should not be a factor here).
>>
>> Here is what I did:
>>
>> Create a rbd device of 10G and mount on /mnt/vol0:
>>
>> $ rbd create --size 10240 vol0
>> $ rbd map vol0
>> $ mkfx.xfs /dev/rbd0
>> $ rbd mount /dev/rdb0 /mnt/vol0
>>
>> Make a file:
>>
>> $ dd if=/dev/zero of=/mnt/vol0/dump/file bs=4k count=300000 conv=fsync
>> 1228800000 bytes (1.2 GB) copied, 13.4361 s, 91.5 MB/s
>>
>> Performance ok if file size < journal (2G).
>>
>> $ dd if=/dev/zero of=/mnt/vol0/dump/file bs=4096k count=200 conv=fsync
>> 838860800 bytes (839 MB) copied, 9.47086 s, 88.6 MB/s
>>
>> Not so good if file size > journal.
>>
>> $ dd if=/dev/zero of=/mnt/vol0/dump/file bs=4096k count=1000 conv=fsync
>> 4194304000 bytes (4.2 GB) copied, 279.891 s, 15.0 MB/s
>>
>> Random writes (see attached file) sync'ed with sync_file_range are ok if
>> block size big:
>>
>> $ ./writetest /mnt/vol0/dump/file 4194304 0 1
>> random writes: 292 of: 4194304 bytes elapsed: 9.8397s io rate: 30/s
>> (118.70 MB/s)
>>
>> $ ./writetest /mnt/vol0/dump/file 1048576 0 1
>> random writes: 1171 of: 1048576 bytes elapsed: 10.6042s io rate: 110/s
>> (110.43 MB/s)
>>
>> $ ./writetest /mnt/vol0/dump/file 131072 0 1
>> random writes: 9375 of: 131072 bytes elapsed: 15.8075s io rate: 593/s
>> (74.13 MB/s)
>>
>>
>> However smallish block size is suicide (trigger suicide assert after a
>> while), I see 100 IOPS or less on actual devices, all 100% util:
>>
>> $ ./writetest /mnt/vol0/dump/file 8192 0 1
>>
>> I am running into http://tracker.newdream.net/issues/2784 here I think.
>
> This can be a sign of a bug in the underlying filesystem or hardware -
> maybe your controller? That assert is hit when a single operation to
> the filesystem beneath the osd takes longer than 180 seconds (by
> default).
>
>> Note that the actual SSD are very fast for this when accessed directly:
>>
>> $ ./writetest /data1/ceph/1/file 8192 0 1
>> random writes: 1000000 of: 8192 bytes elapsed: 125.7907s io rate: 7950/s
>> (62.11 MB/s)
>>
>>
>> Thanks for your patience in reading so far - some actual questions now
>> :-)
>>
>> 1/ Why is the appending write from dd when the size of file > journal so
>> slow, despite reasonably capable storage devices?
>
> It's possible you need to use more threads to have more operations in
> flight in to the filestore (the main storage for the osd). Try
> something like this in your ceph configuration for the osds:
>
>      osd op threads = 24
>      osd disk threads = 24
>      filestore op threads = 6
>      filestore queue max ops = 24
>
> (from http://www.spinics.net/lists/ceph-devel/msg07128.html)
>

It's probably worth giving a try, but I haven't had much luck improving 
small IO performance with more threads/queued ops.

>> 2/ Is the sudden dramatic drop in random write performance a
>> manifestation of the "small requests  are slow" issue? or is this
>> something else?
>
> It's probably that. Sam's actively looking into it, and once he has
> something it will be interesting to see how well it works on your
> hardware.

For what it's worth, with mostly default settings I was seeing about 
8MB/s to dell branded samsung SSDs with 4k IOs using rados bench.  That 
was with 256 concurrent client requests.  This is definitely something 
we are working hard on tracking down.

>
> Josh
>
>> Thanks
>>
>> Mark
>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph RBD performance - random writes

[Mark Kirkwood]

On 09/08/12 09:58, Mark Nelson wrote:
>
> For what it's worth, with mostly default settings I was seeing about 
> 8MB/s to dell branded samsung SSDs with 4k IOs using rados bench.  
> That was with 256 concurrent client requests.  This is definitely 
> something we are working hard on tracking down.
>

Right (FWIW I think the Dells are actually Toshiba devices - and are 
pretty good). This is what I get doing the rados bench for 1M and 4K blocks:



$ rados bench -b 1048576 -t 256 -p rbd 100  write
Total writes made:      3879
Write size:             1048576
Bandwidth (MB/sec):     38.414

Stddev Bandwidth:       40.157
Max bandwidth (MB/sec): 174
Min bandwidth (MB/sec): 0
Average Latency:        6.57279
Stddev Latency:         6.20373
Max latency:            28.531
Min latency:            0.435691


$ rados bench -b 4096 -t 256 -p rbd 100  write
Total writes made:      4851
Write size:             4096
Bandwidth (MB/sec):     0.179

Stddev Bandwidth:       0.378904
Max bandwidth (MB/sec): 3.22266
Min bandwidth (MB/sec): 0
Average Latency:        5.60013
Stddev Latency:         8.18634
Max latency:            22.6542
Min latency:            0.020016

Re: Ceph RBD performance - random writes

[Mark Kirkwood]

On 09/08/12 11:36, Mark Kirkwood wrote:
> On 09/08/12 09:58, Mark Nelson wrote:
>>
>> For what it's worth, with mostly default settings I was seeing about 
>> 8MB/s to dell branded samsung SSDs with 4k IOs using rados bench.  
>> That was with 256 concurrent client requests.  This is definitely 
>> something we are working hard on tracking down.
>>
>
> Right (FWIW I think the Dells are actually Toshiba devices - and are 
> pretty good). This is what I get doing the rados bench for 1M and 4K 
> blocks:
>
>

Sorry Josh - I completely missed the word "Samsung" in your 
post...(we've been testing some Dell branded SSD for another project and 
it was like giving blood to get the actual brand and model out of them)...

>
> $ rados bench -b 1048576 -t 256 -p rbd 100  write
> Total writes made:      3879
> Write size:             1048576
> Bandwidth (MB/sec):     38.414
>
> Stddev Bandwidth:       40.157
> Max bandwidth (MB/sec): 174
> Min bandwidth (MB/sec): 0
> Average Latency:        6.57279
> Stddev Latency:         6.20373
> Max latency:            28.531
> Min latency:            0.435691
>
>
> $ rados bench -b 4096 -t 256 -p rbd 100  write
> Total writes made:      4851
> Write size:             4096
> Bandwidth (MB/sec):     0.179
>
> Stddev Bandwidth:       0.378904
> Max bandwidth (MB/sec): 3.22266
> Min bandwidth (MB/sec): 0
> Average Latency:        5.60013
> Stddev Latency:         8.18634
> Max latency:            22.6542
> Min latency:            0.020016
>
>

I tried out a raft of xfs config changes and also made the Ceph journal 
really big (10G):

$ mkfs.xfs -f -l internal,size=1024m -d agcount=4 /dev/sd[b,c]2

+ mount options with nobarrier,logbufs=8

The results improved a little, but still very slow for small request 
sizes...


$ rados bench -b 1048576 -t 256 -p rbd 100  write
Total writes made:      5165
Write size:             1048576
Bandwidth (MB/sec):     50.646

Stddev Bandwidth:       18.8124
Max bandwidth (MB/sec): 205
Min bandwidth (MB/sec): 0
Average Latency:        4.93633
Stddev Latency:         0.549063
Max latency:            5.58999
Min latency:            1.90631


$ rados bench -b 4096 -t 256 -p rbd 100  write
Write size:             4096
Bandwidth (MB/sec):     0.316

Stddev Bandwidth:       0.874144
Max bandwidth (MB/sec): 3.96484
Min bandwidth (MB/sec): 0
Average Latency:        3.16257
Stddev Latency:         11.9167
Max latency:            58.1325
Min latency:            0.017823

Re: Ceph RBD performance - random writes

[Mark Kirkwood]

On 09/08/12 12:43, Mark Kirkwood wrote:
>
>>
>
>
> I tried out a raft of xfs config changes and also made the Ceph 
> journal really big (10G):
>
> $ mkfs.xfs -f -l internal,size=1024m -d agcount=4 /dev/sd[b,c]2
>
> + mount options with nobarrier,logbufs=8
>
> The results improved a little, but still very slow for small request 
> sizes...

Some more careful analysis showed that all the benefit derived from the 
ceph storage reinit after the filesystem was remade, so going back 
gradually to the default filesystem options (mkfs.xfs, default mount 
with noatime, discard) and 2G journal results in the same numbers as I 
posted with the tweaked settings.

So sorry, appears to be nothing gained (on this system anyway) from said 
tweaking.

Regards

Mark

Re: Ceph RBD performance - random writes

[Mark Nelson]

On 8/8/12 10:54 PM, Mark Kirkwood wrote:
> On 09/08/12 12:43, Mark Kirkwood wrote:
>>
>>>
>>
>>
>> I tried out a raft of xfs config changes and also made the Ceph
>> journal really big (10G):
>>
>> $ mkfs.xfs -f -l internal,size=1024m -d agcount=4 /dev/sd[b,c]2
>>
>> + mount options with nobarrier,logbufs=8
>>
>> The results improved a little, but still very slow for small request
>> sizes...
>
> Some more careful analysis showed that all the benefit derived from the
> ceph storage reinit after the filesystem was remade, so going back
> gradually to the default filesystem options (mkfs.xfs, default mount
> with noatime, discard) and 2G journal results in the same numbers as I
> posted with the tweaked settings.
>
> So sorry, appears to be nothing gained (on this system anyway) from said
> tweaking.
>
> Regards
>
> Mark

Hi Mark,

Would you mind installing and running collectl during your test?  I 
think it's in the apt repositories now in 12.04.

Try "collectl -sD -oT --dskfilt sd<N>,sd<M>" where the dskfilt options 
are the devices for your OSD(s).  I'd like to see what the device wait 
and svc times are like on your setup in both cases.

Thanks,
Mark

Re: Ceph RBD performance - random writes

[Mark Kirkwood]

On 09/08/12 23:42, Mark Nelson wrote:
> On 8/8/12 10:54 PM, Mark Kirkwood wrote:
>> On 09/08/12 12:43, Mark Kirkwood wrote:
>>>
>>>>
>>>
>>>
>>> I tried out a raft of xfs config changes and also made the Ceph
>>> journal really big (10G):
>>>
>>> $ mkfs.xfs -f -l internal,size=1024m -d agcount=4 /dev/sd[b,c]2
>>>
>>> + mount options with nobarrier,logbufs=8
>>>
>>> The results improved a little, but still very slow for small request
>>> sizes...
>>
>> Some more careful analysis showed that all the benefit derived from the
>> ceph storage reinit after the filesystem was remade, so going back
>> gradually to the default filesystem options (mkfs.xfs, default mount
>> with noatime, discard) and 2G journal results in the same numbers as I
>> posted with the tweaked settings.
>>
>> So sorry, appears to be nothing gained (on this system anyway) from said
>> tweaking.
>>
>> Regards
>>
>> Mark
>
> Hi Mark,
>
> Would you mind installing and running collectl during your test? I 
> think it's in the apt repositories now in 12.04.
>
> Try "collectl -sD -oT --dskfilt sd<N>,sd<M>" where the dskfilt options 
> are the devices for your OSD(s).  I'd like to see what the device wait 
> and svc times are like on your setup in both cases.


Ok, yeah it is in the 12.04 repo - will do.

There could well be an additional factor connected with xfs and lots of 
files on these Intel 520s - I have just had a conversation with a 
workmate who switched xfs to ext4 due to this. I will see if ext4 or 
btrfs (scary) do any better on these drives...

Cheers

Mark

Re: Ceph RBD performance - random writes

[Mark Kirkwood]

On 10/08/12 11:31, Mark Kirkwood wrote:
>
>
> There could well be an additional factor connected with xfs and lots 
> of files on these Intel 520s - I have just had a conversation with a 
> workmate who switched xfs to ext4 due to this. I will see if ext4 or 
> btrfs (scary) do any better on these drives...
>

Actually that seems to be a different issue (mass deletion of many small 
files)...I am not seeing significant differences using ext4 or btrfs. So 
back to using xfs again :-)

One thing that *does* seems to have made a difference is upgrading from 
0.48 to 0.50 (also Ubuntu kernel patched from 3.2.0-27 to 3.2.0-29):

$ rados bench -b 4096 -t 256 -p rbd 100  write
Total writes made:      65202
Write size:             4096
Bandwidth (MB/sec):     2.534

Stddev Bandwidth:       0.851288
Max bandwidth (MB/sec): 4.67969
Min bandwidth (MB/sec): 0
Average Latency:        0.39456
Stddev Latency:         0.283639
Max latency:            4.07996
Min latency:            0.022397

(Recall Bandwidth was about 0.3 MB/sec previously). Typical iostat looks 
much happier:

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz 
avgqu-sz   await r_await w_await  svctm  %util
sdc               0.00     0.00    0.00  364.00     0.00 4.14    
23.30     0.06    0.18    0.00    0.18   0.18   6.40
sdb               0.00  1006.00    0.00 1091.00     0.00 10.77    
20.21    12.77   11.66    0.00   11.66   0.66  72.00


I'll get collectl info using this version unless Mark especially wants 
the data for 0.48 (but it *looks* like some nice improvements are in 
there already with the later version).

Cheers

Mark

Re: Ceph RBD performance - random writes

[Matthew Richardson]

[-- Attachment #1: Type: text/plain, Size: 577 bytes --]

On 08/08/12 06:19, Mark Kirkwood wrote:
> I've been looking at using Ceph RBD as a block store for database use.
> As part of this I'm looking a how (particularly random) IO of smallish
> (4K, 8K) block sizes performs.

It may or may not be related, but have a look at the message thread here:

http://www.mail-archive.com/ceph-devel@vger.kernel.org/msg07678.html

It looks like there might be a specific bug relating to random writes.

Matthew

-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]