Poor performance with bcache write-back mode

Poor performance with bcache write-back mode

[Patrick Ng]

Hi all,

I have installed Linux kernel 3.9.2 in my Fedora 18.  When I ran FIO in bcache write-through mode, the performance was good, but running a similar test in bcache write-back mode, I got very poor performance.
Did I miss any options when setting the write-back mode?

Environment
===========
500G backing store at /dev/sdb1
150G SSD at /dev/sdc1

Write-through mode
==================
make-bcache -B /dev/sdb1 -C /dev/sdc1
echo /dev/sdb1 > /sys/fs/bcache/register
echo /dev/sdc1 > /sys/fs/bcache/register
mkfs.ext4 /dev/bcache0
mount /dev/bcache0 /mnt/bcache


Running FIO with random, 100% read, iosize=4096, queue depth=16
the result is approx. 65k IOPS


Write-backe mode (setup is almost the same as write-through except the extra step at the end "echo writeback ...")
================
make-bcache -B /dev/sdb1 -C /dev/sdc1
echo /dev/sdb1 > /sys/fs/bcache/register
echo /dev/sdc1 > /sys/fs/bcache/register
mkfs.ext4 /dev/bcache0
mount /dev/bcache0 /mnt/bcache
echo writeback > /sys/block/bcache0/bcache/cache_mode

Running FIO with random, 100% read, iosize=4096, queue depth=16
the result is approx. 12k IOPS

Running FIO with random, 80% read 20% write, iosize=4096, queue depth=16
the result is approx. 2k IOPS

Thanks,
Patrick 

Re: Poor performance with bcache write-back mode

[Patrick Ng]

Hi Robert,

I have checked that the cache partition (/dev/sdc1) starts at 2048, it should be properly aligned.
Similarly for the backing store partition (/dev/sdb1), it starts at 2048 at well.
 
Thanks,
Patrick
>________________________________
> From: Robert Frånlund <robert.franlund-Hou3YEnI4S0Iwnw1HX0dmQ@public.gmane.org>
>To: Patrick Ng <ppng-/E1597aS9LQAvxtiuMwx3w@public.gmane.org> 
>Cc: "linux-bcache-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" <linux-bcache-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> 
>Sent: Friday, May 17, 2013 2:22 AM
>Subject: Re: Poor performance with bcache write-back mode
> 
>
>
>Hi,
>
>
>Have you aligned your partitions on the SSD to at-least 4 KiB block-size, even better with 1 MiB?
>
>
>Best regards,
>Robert
>
>
>
>
>
>
>
>2013/5/16 Patrick Ng <ppng-/E1597aS9LQAvxtiuMwx3w@public.gmane.org>
>
>Hi all,
>>
>>I have installed Linux kernel 3.9.2 in my Fedora 18.  When I ran FIO in bcache write-through mode, the performance was good, but running a similar test in bcache write-back mode, I got very poor performance.
>>Did I miss any options when setting the write-back mode?
>>
>>Environment
>>===========
>>500G backing store at /dev/sdb1
>>150G SSD at /dev/sdc1
>>
>>Write-through mode
>>==================
>>make-bcache -B /dev/sdb1 -C /dev/sdc1
>>echo /dev/sdb1 > /sys/fs/bcache/register
>>echo /dev/sdc1 > /sys/fs/bcache/register
>>mkfs.ext4 /dev/bcache0
>>mount /dev/bcache0 /mnt/bcache
>>
>>
>>Running FIO with random, 100% read, iosize=4096, queue depth=16
>>the result is approx. 65k IOPS
>>
>>
>>Write-backe mode (setup is almost the same as write-through except the extra step at the end "echo writeback ...")
>>================
>>make-bcache -B /dev/sdb1 -C /dev/sdc1
>>echo /dev/sdb1 > /sys/fs/bcache/register
>>echo /dev/sdc1 > /sys/fs/bcache/register
>>mkfs.ext4 /dev/bcache0
>>mount /dev/bcache0 /mnt/bcache
>>echo writeback > /sys/block/bcache0/bcache/cache_mode
>>
>>Running FIO with random, 100% read, iosize=4096, queue depth=16
>>the result is approx. 12k IOPS
>>
>>Running FIO with random, 80% read 20% write, iosize=4096, queue depth=16
>>the result is approx. 2k IOPS
>>
>>Thanks,
>>Patrick 
>>--
>>To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>>the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
> 

Re: Poor performance with bcache write-back mode

[Gabriel de Perthuis]

On Fri, 17 May 2013 13:13:04 -0700, Patrick Ng wrote:
> Hi Robert,
> 
> I have checked that the cache partition (/dev/sdc1) starts at 2048, it should be properly aligned.
> Similarly for the backing store partition (/dev/sdb1), it starts at 2048 at well.

The bcache superblock shifts the start of your partition by 8k though.
I don't know if the upper layers compensate for that, but just in case,
you may want to add `--data_offset 2048` to the make-bcache command line
(1M alignment).

Re: Poor performance with bcache write-back mode

[Robert Frånlund]

Hi,

Have you aligned your partitions on the SSD to at-least 4 KiB
block-size, even better with 1 MiB?

Best regards,
Robert

2013/5/16 Patrick Ng <ppng-/E1597aS9LQAvxtiuMwx3w@public.gmane.org>:
> Hi all,
>
> I have installed Linux kernel 3.9.2 in my Fedora 18.  When I ran FIO in bcache write-through mode, the performance was good, but running a similar test in bcache write-back mode, I got very poor performance.
> Did I miss any options when setting the write-back mode?
>
> Environment
> ===========
> 500G backing store at /dev/sdb1
> 150G SSD at /dev/sdc1
>
> Write-through mode
> ==================
> make-bcache -B /dev/sdb1 -C /dev/sdc1
> echo /dev/sdb1 > /sys/fs/bcache/register
> echo /dev/sdc1 > /sys/fs/bcache/register
> mkfs.ext4 /dev/bcache0
> mount /dev/bcache0 /mnt/bcache
>
>
> Running FIO with random, 100% read, iosize=4096, queue depth=16
> the result is approx. 65k IOPS
>
>
> Write-backe mode (setup is almost the same as write-through except the extra step at the end "echo writeback ...")
> ================
> make-bcache -B /dev/sdb1 -C /dev/sdc1
> echo /dev/sdb1 > /sys/fs/bcache/register
> echo /dev/sdc1 > /sys/fs/bcache/register
> mkfs.ext4 /dev/bcache0
> mount /dev/bcache0 /mnt/bcache
> echo writeback > /sys/block/bcache0/bcache/cache_mode
>
> Running FIO with random, 100% read, iosize=4096, queue depth=16
> the result is approx. 12k IOPS
>
> Running FIO with random, 80% read 20% write, iosize=4096, queue depth=16
> the result is approx. 2k IOPS
>
> Thanks,
> Patrick

Re: Poor performance with bcache write-back mode

[Kent Overstreet]

On Thu, May 16, 2013 at 02:51:46PM -0700, Patrick Ng wrote:
> Hi all,
> 
> I have installed Linux kernel 3.9.2 in my Fedora 18.  When I ran FIO in bcache write-through mode, the performance was good, but running a similar test in bcache write-back mode, I got very poor performance.
> Did I miss any options when setting the write-back mode?
> 
> Environment
> ===========
> 500G backing store at /dev/sdb1
> 150G SSD at /dev/sdc1
> 
> Write-through mode
> ==================
> make-bcache -B /dev/sdb1 -C /dev/sdc1
> echo /dev/sdb1 > /sys/fs/bcache/register
> echo /dev/sdc1 > /sys/fs/bcache/register
> mkfs.ext4 /dev/bcache0
> mount /dev/bcache0 /mnt/bcache
> 
> 
> Running FIO with random, 100% read, iosize=4096, queue depth=16
> the result is approx. 65k IOPS
> 
> 
> Write-backe mode (setup is almost the same as write-through except the extra step at the end "echo writeback ...")
> ================
> make-bcache -B /dev/sdb1 -C /dev/sdc1
> echo /dev/sdb1 > /sys/fs/bcache/register
> echo /dev/sdc1 > /sys/fs/bcache/register
> mkfs.ext4 /dev/bcache0
> mount /dev/bcache0 /mnt/bcache
> echo writeback > /sys/block/bcache0/bcache/cache_mode
> 
> Running FIO with random, 100% read, iosize=4096, queue depth=16
> the result is approx. 12k IOPS
> 
> Running FIO with random, 80% read 20% write, iosize=4096, queue depth=16
> the result is approx. 2k IOPS

You're probably getting bit by the congested bypass - try setting
congested_read_threshold_us and congested_write_threshold_us both to 0.

I wish there was a better solution to the congestion bypass stuff, it's
a hard problem...

Re: Poor performance with bcache write-back mode

[Patrick Ng]

Kent Overstreet <koverstreet@...> writes:

> 
> On Thu, May 16, 2013 at 02:51:46PM -0700, Patrick Ng wrote:
> > Hi all,
> > 
> > I have installed Linux kernel 3.9.2 in my Fedora 18.  When I ran FIO in 
bcache write-through mode, the
> performance was good, but running a similar test in bcache write-back 
mode, I got very poor performance.
> > Did I miss any options when setting the write-back mode?
> > 
> > Environment
> > ===========
> > 500G backing store at /dev/sdb1
> > 150G SSD at /dev/sdc1
> > 
> > Write-through mode
> > ==================
> > make-bcache -B /dev/sdb1 -C /dev/sdc1
> > echo /dev/sdb1 > /sys/fs/bcache/register
> > echo /dev/sdc1 > /sys/fs/bcache/register
> > mkfs.ext4 /dev/bcache0
> > mount /dev/bcache0 /mnt/bcache
> > 
> > 
> > Running FIO with random, 100% read, iosize=4096, queue depth=16
> > the result is approx. 65k IOPS
> > 
> > 
> > Write-backe mode (setup is almost the same as write-through except the 
extra step at the end "echo
> writeback ...")
> > ================
> > make-bcache -B /dev/sdb1 -C /dev/sdc1
> > echo /dev/sdb1 > /sys/fs/bcache/register
> > echo /dev/sdc1 > /sys/fs/bcache/register
> > mkfs.ext4 /dev/bcache0
> > mount /dev/bcache0 /mnt/bcache
> > echo writeback > /sys/block/bcache0/bcache/cache_mode
> > 
> > Running FIO with random, 100% read, iosize=4096, queue depth=16
> > the result is approx. 12k IOPS
> > 
> > Running FIO with random, 80% read 20% write, iosize=4096, queue depth=16
> > the result is approx. 2k IOPS
> 
> You're probably getting bit by the congested bypass - try setting
> congested_read_threshold_us and congested_write_threshold_us both to 0.
> 
> I wish there was a better solution to the congestion bypass stuff, it's
> a hard problem...
> 

Hi Kent, thank you for responding to my question.

I have tried setting both congested_read_threshold_us and 
congested_write_threshold_us to zero.

The result of FIO random 100% read test in WriteBack mode has improved and 
the IOPS is as good as in WriteThur mode.

The result of FIO random 80% read 20% write remain the same with no 
improvement.

It appears that the option does not help on the WRITE?

Thanks,
Patrick