Performance testing with bcache

Performance testing with bcache

[Andrew Martin]

Hello,

I am working on setting up bcache on Ubuntu 13.10 (with kernel 3.11.0-17) 
for read performance testing in writethrough mode. I've written a test script 
which uses dd to write data to the bcache device, read it back in, and then 
attempt to read it again (to measure cache speed):
# write the data to the disk
dd if=/dev/zero of=/path/to/bcache/mount/file.raw bs=512 count=2

# read the data (using iflag=nocache to make sure it gets into bcache's 
# cache and doesn't use the kernel's cache) 
dd if=/path/to/bcache/mount/file.raw of=/dev/null iflag=nocache

# now perform the read, which should read from bcache's cache
dd if=/path/to/bcache/mount/file.raw of=/dev/null iflag=nocache

However, if I cat /sys/block/bcache0/bcache/cache/cache_available_percent
before and after performing the read, the value is always 100. Would using
iflag=direct be better than iflag=nocache in this case? Is there a better
way to force data into the cache for this type of performance benchmarking?

Thanks,

Andrew Martin

Re: Performance testing with bcache

[Sitsofe Wheeler]

On Thu, Mar 13, 2014 at 03:12:16PM -0500, Andrew Martin wrote:
> 
> I am working on setting up bcache on Ubuntu 13.10 (with kernel 3.11.0-17) 
> for read performance testing in writethrough mode. I've written a test script 
> which uses dd to write data to the bcache device, read it back in, and then 
> attempt to read it again (to measure cache speed):
> # write the data to the disk
> dd if=/dev/zero of=/path/to/bcache/mount/file.raw bs=512 count=2

This will only tell you sequential speeds of small blocks. You may want
to use a tool like fio to let you generate different types of IO
patterns to see what the impact is. For example, the difference between
an hdd's and an sdd's top large block sequential speeds isn't nearly as
dramatic as the difference between their top small block random
speeds...

> # read the data (using iflag=nocache to make sure it gets into bcache's 
> # cache and doesn't use the kernel's cache) 
> dd if=/path/to/bcache/mount/file.raw of=/dev/null iflag=nocache
> 
> # now perform the read, which should read from bcache's cache
> dd if=/path/to/bcache/mount/file.raw of=/dev/null iflag=nocache

This may not perform as you expect it to do. My testing shows that the
read cache bcache is a bit quirky and to see it being used you have to
ensure that the blocks being read the second time are the exact same
size as the blocks that were read the first time. I think that
particular issue is alluded to in TROUBLESHOOTING PERFORMANCE on
http://evilpiepirate.org/git/linux-bcache.git/tree/Documentation/bcache.txt?h=bcache-dev#n126
(Still getting cache misses, of the same data). I would advise you to
warm the cache by doing writes as that doesn't seem to show the problem
but also see below.

> However, if I cat /sys/block/bcache0/bcache/cache/cache_available_percent
> before and after performing the read, the value is always 100. Would using
> iflag=direct be better than iflag=nocache in this case? Is there a better
> way to force data into the cache for this type of performance benchmarking?

The amount free isn't as important as the cache hits/misses ratio.
Additionally I would recommend reading the TROUBLESHOOTING PERFORMANCE
section mentioned above and posting your findings here because the
bcache defaults often defeat first attempt synthetic benchmarking...

-- 
Sitsofe | http://sucs.org/~sits/

Re: Performance testing with bcache

[Andrew Martin]

----- Original Message -----
> From: "Sitsofe Wheeler" <sitsofe@yahoo.com>
...
> > However, if I cat /sys/block/bcache0/bcache/cache/cache_available_percent
> > before and after performing the read, the value is always 100. Would using
> > iflag=direct be better than iflag=nocache in this case? Is there a better
> > way to force data into the cache for this type of performance benchmarking?
> 
> The amount free isn't as important as the cache hits/misses ratio.
> Additionally I would recommend reading the TROUBLESHOOTING PERFORMANCE
> section mentioned above and posting your findings here because the
> bcache defaults often defeat first attempt synthetic benchmarking...

Thanks, I will try fio with these recommended steps!

Andrew

Re: Performance testing with bcache

[Andrew Martin]

----- Original Message -----
> From: "Andrew Martin" <amartin@xes-inc.com>
> ----- Original Message -----
> > From: "Sitsofe Wheeler" <sitsofe@yahoo.com>
> ...
> > > However, if I cat /sys/block/bcache0/bcache/cache/cache_available_percent
> > > before and after performing the read, the value is always 100. Would
> > > using
> > > iflag=direct be better than iflag=nocache in this case? Is there a better
> > > way to force data into the cache for this type of performance
> > > benchmarking?
> > 
> > The amount free isn't as important as the cache hits/misses ratio.
> > Additionally I would recommend reading the TROUBLESHOOTING PERFORMANCE
> > section mentioned above and posting your findings here because the
> > bcache defaults often defeat first attempt synthetic benchmarking...
> 
> Thanks, I will try fio with these recommended steps!
> 
Sitsofe,

Do you have an example fio job file that has worked well for testing bcache?

Thanks,

Andrew

Re: Performance testing with bcache

[Sitsofe Wheeler]

On Fri, Mar 14, 2014 at 04:16:27PM -0500, Andrew Martin wrote:
> 
> Do you have an example fio job file that has worked well for testing
> bcache?

Try these as a starting point:
benchmark-seq.fio:
; Script to gather general sequential write performance statistics
; Basic usage:
; BS="4k" DEV="sdf" RT=90 SEED=21 TYPE="ssd" fio benchmark-seq.fio

[global]
filename=/dev/${DEV}
runtime=${RT}
bs=${BS}
iodepth=4
ioengine=libaio
direct=1
stonewall
size=${SIZE}
randseed=${SEED}
[write]
rw=write

benchmark-rand.fio:
; Script to gather general random write performance statistics
; Basic usage:
; BS="4k" DEV="sdf" RT=90 SEED=21 TYPE="ssd" fio benchmark-rand.fio

[global]
filename=/dev/${DEV}
runtime=${RT}
bs=${BS}
iodepth=4
ioengine=libaio
direct=1
stonewall
size=${SIZE}
randseed=${SEED}
[randwrite]
rw=randwrite

After making the above files do
export RT=30; export SEED=21; export SIZE="10G"; export DEV="bcache0"
for b in "4" "16" "64"; do export BS="${b}k"; for job in "benchmark-seq" "benchmark-rand"; do fio $job.fio; done; done

The first time round get the stats where the cache is not being used.
Next time round set the appropriate cache options (easier said than done
:-), run a single benchmark-seq.fio job (I'd recommend 64k blocksize) to
warm the cache then rerun the above.

Let us know what you did and how it went!

-- 
Sitsofe | http://sucs.org/~sits/

Re: Performance testing with bcache

[Andrew Martin]

Sitsofe,

----- Original Message -----
> From: "Sitsofe Wheeler" <sitsofe@yahoo.com>
> To: "Andrew Martin" <amartin@xes-inc.com>
> Cc: linux-bcache@vger.kernel.org
> Sent: Saturday, March 15, 2014 12:52:15 PM
> Subject: Re: Performance testing with bcache
> 
> On Fri, Mar 14, 2014 at 04:16:27PM -0500, Andrew Martin wrote:
> > 
> > Do you have an example fio job file that has worked well for testing
> > bcache?
> 
> Try these as a starting point:
> benchmark-seq.fio:
> ; Script to gather general sequential write performance statistics
> ; Basic usage:
> ; BS="4k" DEV="sdf" RT=90 SEED=21 TYPE="ssd" fio benchmark-seq.fio

I ran the example job files with the commands you posted, but the output
bandwidth seems extremely low:
http://pastebin.com/13RBYwwp
http://pastebin.com/bsp9KbDq

fio job file:
http://pastebin.com/zfVJdjMQ

Am I doing something wrong when running fio?

Thanks,

Andrew

Re: Performance testing with bcache

[Kent Overstreet]

How much data is being bypassed?

On Fri, Apr 4, 2014 at 10:06 AM, Andrew Martin <amartin@xes-inc.com> wrote:
> Sitsofe,
>
> ----- Original Message -----
>> From: "Sitsofe Wheeler" <sitsofe@yahoo.com>
>> To: "Andrew Martin" <amartin@xes-inc.com>
>> Cc: linux-bcache@vger.kernel.org
>> Sent: Saturday, March 15, 2014 12:52:15 PM
>> Subject: Re: Performance testing with bcache
>>
>> On Fri, Mar 14, 2014 at 04:16:27PM -0500, Andrew Martin wrote:
>> >
>> > Do you have an example fio job file that has worked well for testing
>> > bcache?
>>
>> Try these as a starting point:
>> benchmark-seq.fio:
>> ; Script to gather general sequential write performance statistics
>> ; Basic usage:
>> ; BS="4k" DEV="sdf" RT=90 SEED=21 TYPE="ssd" fio benchmark-seq.fio
>
> I ran the example job files with the commands you posted, but the output
> bandwidth seems extremely low:
> http://pastebin.com/13RBYwwp
> http://pastebin.com/bsp9KbDq
>
> fio job file:
> http://pastebin.com/zfVJdjMQ
>
> Am I doing something wrong when running fio?
>
> Thanks,
>
> Andrew
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Performance testing with bcache

[Sitsofe Wheeler]

On Fri, Apr 04, 2014 at 12:10:53PM -0700, Kent Overstreet wrote:
> How much data is being bypassed?
> 
> On Fri, Apr 4, 2014 at 10:06 AM, Andrew Martin <amartin@xes-inc.com> wrote:
> >
> > Am I doing something wrong when running fio?

Andrew, 

There's not enough information to be able to say if something is going
wrong at this stage as I don't know:
a) The speed of the backing device (also the block device name (e.g.
sda/md0 helps later)) with a given IO pattern
b) The speed of the front device (block name also helps later) with a
given IO pattern
c) What bcache settings you have set
d) What type of IO was done to the bcache device and at which points
(e.g. was writeback happening at that point?)
e) What statistics bcache is reporting

Initially while you're getting used to this I'd stick to just running
one block size (4k). So something like:
export RT=30; export SEED=21; export SIZE="5G"; export DEV="bcache0"; BS="4k"

Ensure you have read and implemented the appropriate pieces mentioned in
the TROUBLESHOOTING PERFORMANCE section of
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/bcache.txt#n130
otherwise results can be unexpected when benchmarking...

Something like the following would be easier to follow:
Set the cache_mode to none. Run (and record)
grep -r . /sys/block/bcache0/bcache/
Clear the bcache total stats (see clear_stats)
View and record the bcache backing device total stats then run
fio benchmark-rand.fio
View and record the bcache backing device total stats
Set the cache_mode to writeback. Run (and record)
grep -r . /sys/block/bcache0/bcache/
Clear the bcache total stats then run
View and record the bcache backing device total stats then run
fio benchmark-rand.fio
View and record the bcache backing device total stats

If everything is working (and the front device is faster for random
writes) I'd expect the with writeback fio job to be noticibly faster
than none fio job (how much faster is setup dependent). Only if my
expectation was correct would I repeat the above with the
benchmark-seq.fio job...

-- 
Sitsofe | http://sucs.org/~sits/