Raid5 slower with bcache

Raid5 slower with bcache

[Javier Marcet]

Hi,

I've been using bcache since last summer, caching a 4x3TB raid5.

Last week one of my disks died so after replacing it, I also got a new
SSD and began to do some tests. There is something which, while not
proving anything, worries me anyway. Better than describing it, just
see this (caches were flushed before each hdparm):

 # hdparm -tT /dev/md0

/dev/md0:
 Timing cached reads:   23860 MB in  2.00 seconds = 11941.98 MB/sec
 Timing buffered disk reads: 996 MB in  3.00 seconds = 331.85 MB/sec

# hdparm -tT /dev/bcache0

/dev/bcache0:
 Timing cached reads:   22944 MB in  2.00 seconds = 11482.63 MB/sec
 Timing buffered disk reads: 488 MB in  3.00 seconds = 162.49 MB/sec

This is without any caching devices attached, that is, in pass-through
mode. My md raid5 is bottlenecked by the 3Gbps of the SATA controller
but with bcache, even in pass-through mode, seems to have a degrading
effect.

With a caching device attached, I know random I/O improves, but it
seems I'm degrading sequential access considerably. See now with a
fresh 128GB Crucial M4 SSD attached:

# hdparm -tT /dev/sdb

/dev/sdb:
 Timing cached reads:   23964 MB in  2.00 seconds = 11994.69 MB/sec
 Timing buffered disk reads: 1594 MB in  3.00 seconds = 530.72 MB/sec

 # hdparm -tT /dev/bcache0

/dev/bcache0:
 Timing cached reads:   23830 MB in  2.00 seconds = 11927.29 MB/sec
 Timing buffered disk reads: 626 MB in  3.00 seconds = 208.64 MB/sec

# hdparm -tT /dev/bcache0

/dev/bcache0:
 Timing cached reads:   24156 MB in  2.00 seconds = 12089.18 MB/sec
 Timing buffered disk reads: 542 MB in  3.05 seconds = 177.53 MB/sec

I would like to try more suited testing tools directly over the RAID,
without any bcache but it looks like I'd need to recreate the whole
array, sigh.

I would appreciate any more info about these numbers.


-- 
Javier Marcet <jmarcet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>

Re: Raid5 slower with bcache

[Kent Overstreet]

On Wed, Mar 27, 2013 at 11:03:52AM +0100, Javier Marcet wrote:
> Hi,
> 
> I've been using bcache since last summer, caching a 4x3TB raid5.
> 
> Last week one of my disks died so after replacing it, I also got a new
> SSD and began to do some tests. There is something which, while not
> proving anything, worries me anyway. Better than describing it, just
> see this (caches were flushed before each hdparm):

I think I may have finally figured this one out - bcache wasn't setting
readahead_pages, and raid5/6 needs it to be set based on the number of
devices in the array. Can you try the version I just pushed?

> 
>  # hdparm -tT /dev/md0
> 
> /dev/md0:
>  Timing cached reads:   23860 MB in  2.00 seconds = 11941.98 MB/sec
>  Timing buffered disk reads: 996 MB in  3.00 seconds = 331.85 MB/sec
> 
> # hdparm -tT /dev/bcache0
> 
> /dev/bcache0:
>  Timing cached reads:   22944 MB in  2.00 seconds = 11482.63 MB/sec
>  Timing buffered disk reads: 488 MB in  3.00 seconds = 162.49 MB/sec
> 
> This is without any caching devices attached, that is, in pass-through
> mode. My md raid5 is bottlenecked by the 3Gbps of the SATA controller
> but with bcache, even in pass-through mode, seems to have a degrading
> effect.
> 
> With a caching device attached, I know random I/O improves, but it
> seems I'm degrading sequential access considerably. See now with a
> fresh 128GB Crucial M4 SSD attached:
> 
> # hdparm -tT /dev/sdb
> 
> /dev/sdb:
>  Timing cached reads:   23964 MB in  2.00 seconds = 11994.69 MB/sec
>  Timing buffered disk reads: 1594 MB in  3.00 seconds = 530.72 MB/sec
> 
>  # hdparm -tT /dev/bcache0
> 
> /dev/bcache0:
>  Timing cached reads:   23830 MB in  2.00 seconds = 11927.29 MB/sec
>  Timing buffered disk reads: 626 MB in  3.00 seconds = 208.64 MB/sec
> 
> # hdparm -tT /dev/bcache0
> 
> /dev/bcache0:
>  Timing cached reads:   24156 MB in  2.00 seconds = 12089.18 MB/sec
>  Timing buffered disk reads: 542 MB in  3.05 seconds = 177.53 MB/sec
> 
> I would like to try more suited testing tools directly over the RAID,
> without any bcache but it looks like I'd need to recreate the whole
> array, sigh.
> 
> I would appreciate any more info about these numbers.
> 
> 
> -- 
> Javier Marcet <jmarcet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid5 slower with bcache

[Javier Marcet]

On Wed, Mar 27, 2013 at 7:25 PM, Kent Overstreet <koverstreet-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:

> > I've been using bcache since last summer, caching a 4x3TB raid5.
> >
> > Last week one of my disks died so after replacing it, I also got a new
> > SSD and began to do some tests. There is something which, while not
> > proving anything, worries me anyway. Better than describing it, just
> > see this (caches were flushed before each hdparm):
>
> I think I may have finally figured this one out - bcache wasn't setting
> readahead_pages, and raid5/6 needs it to be set based on the number of
> devices in the array. Can you try the version I just pushed?

Sure. It's compiling right now, give me a few minutes and I'll report back.


--
Javier Marcet <jmarcet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>

Re: Raid5 slower with bcache

[Javier Marcet]

On Wed, Mar 27, 2013 at 7:25 PM, Kent Overstreet <koverstreet-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:

>> I've been using bcache since last summer, caching a 4x3TB raid5.
>>
>> Last week one of my disks died so after replacing it, I also got a new
>> SSD and began to do some tests. There is something which, while not
>> proving anything, worries me anyway. Better than describing it, just
>> see this (caches were flushed before each hdparm):
>
> I think I may have finally figured this one out - bcache wasn't setting
> readahead_pages, and raid5/6 needs it to be set based on the number of
> devices in the array. Can you try the version I just pushed?

Well, it improves it a little bit but there is still a wide gap, 1/3
worse than the raid directly:

# hdparm -tT /dev/md0

/dev/md0:
 Timing cached reads:   23742 MB in  2.00 seconds = 11882.23 MB/sec
 Timing buffered disk reads: 1110 MB in  3.00 seconds = 369.83 MB/sec

# hdparm -tT /dev/md0

/dev/md0:
 Timing cached reads:   24350 MB in  2.00 seconds = 12188.02 MB/sec
 Timing buffered disk reads: 1118 MB in  3.00 seconds = 372.39 MB/sec

# hdparm -tT /dev/bcache0

/dev/bcache0:
 Timing cached reads:   23404 MB in  2.00 seconds = 11713.20 MB/sec
 Timing buffered disk reads: 724 MB in  3.00 seconds = 240.98 MB/sec

# hdparm -tT /dev/bcache0

/dev/bcache0:
 Timing cached reads:   22638 MB in  2.00 seconds = 11330.39 MB/sec
 Timing buffered disk reads: 694 MB in  3.02 seconds = 230.08 MB/sec

# hdparm -tT /dev/bcache0

/dev/bcache0:
 Timing cached reads:   23916 MB in  2.00 seconds = 11969.22 MB/sec
 Timing buffered disk reads: 686 MB in  3.00 seconds = 228.53 MB/sec


-- 
Javier Marcet <jmarcet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>