Inconsistent Performance

Inconsistent Performance

[Andrew Thrift]

Hi,

We are noticing some odd performance characteristics with bcache on 3.10 
Mainline kernel.

We are using Intel DC S3700 SSD and Seagate Constellation ES.2 HDD

Using FIO to perform a 4K random write test with QD=32 and 4 instances 
we get:

RAW SSD:    55.5K iops

RAW HDD:   540 iops

Fresh bcache on top of SSD/HDD [writeback]:    36.8K iops

Which is not great, but OK.

However if I run the FIO test a number of times, letting it write out 
the data to the backing device after each run, after a few runs I will 
notice the IOPS drop.  When the benchmark starts it will sit around 14K 
IOPS, then as soon as it starts writing to the backing device this will 
drop to around 5K IOPS. On the subsequent runs it will never go above 5K 
IOPS.



Has there been a performance regression on 3.10 ?



Regards,





Andrew

Re: Inconsistent Performance

[Kent Overstreet]

On Mon, Jul 29, 2013 at 03:32:49PM +1200, Andrew Thrift wrote:
> Hi,
> 
> We are noticing some odd performance characteristics with bcache on
> 3.10 Mainline kernel.
> 
> We are using Intel DC S3700 SSD and Seagate Constellation ES.2 HDD
> 
> Using FIO to perform a 4K random write test with QD=32 and 4
> instances we get:
> 
> RAW SSD:    55.5K iops
> 
> RAW HDD:   540 iops
> 
> Fresh bcache on top of SSD/HDD [writeback]:    36.8K iops
> 
> Which is not great, but OK.
> 
> However if I run the FIO test a number of times, letting it write
> out the data to the backing device after each run, after a few runs
> I will notice the IOPS drop.  When the benchmark starts it will sit
> around 14K IOPS, then as soon as it starts writing to the backing
> device this will drop to around 5K IOPS. On the subsequent runs it
> will never go above 5K IOPS.

Hmm... that sounds like something an SSD would do, but it sounds like
you've checked that performance to the bare SSD isn't degrading, and
that is a rather large regression.

> Has there been a performance regression on 3.10 ?

Certainly not impossible.

Could you try doing some profiling with perf and seeing if anything
shows up? Also, try flipping off discards if you have them on.

It's just write performance that's degrading, not read?

Re: Inconsistent Performance

[Kent Overstreet]

On Mon, Jul 29, 2013 at 03:32:49PM +1200, Andrew Thrift wrote:
> Hi,
> 
> We are noticing some odd performance characteristics with bcache on
> 3.10 Mainline kernel.
> 
> We are using Intel DC S3700 SSD and Seagate Constellation ES.2 HDD
> 
> Using FIO to perform a 4K random write test with QD=32 and 4
> instances we get:
> 
> RAW SSD:    55.5K iops
> 
> RAW HDD:   540 iops
> 
> Fresh bcache on top of SSD/HDD [writeback]:    36.8K iops
> 
> Which is not great, but OK.
> 
> However if I run the FIO test a number of times, letting it write
> out the data to the backing device after each run, after a few runs
> I will notice the IOPS drop.  When the benchmark starts it will sit
> around 14K IOPS, then as soon as it starts writing to the backing
> device this will drop to around 5K IOPS. On the subsequent runs it
> will never go above 5K IOPS.
> 
> 
> 
> Has there been a performance regression on 3.10 ?

I don't seem to be able to reproduce this in the bcache-testing branch
(but I haven't tried stock 3.10, and also the SSD in my test machine
isn't near as fast as yours).

Could you try the testing branch and see if you can reproduce it there?