Performance Comparison among EnhanceIO, bcache and dm-cache.

Performance Comparison among EnhanceIO, bcache and dm-cache.

[OS Engineering]

Hi Jens,

In continuation with our previous communication, we have carried out performance comparison among EnhanceIO, bcache and dm-cache.

We found that EnhanceIO provides better throughput on zipf workload (with theta=1.2) in comparison to bcache and dm-cache for write through caches.
However, for write back caches, we found that dm-cache had best throughput followed by EnhanceIO and then bcache. Dm-cache commits on-disk metadata every time a REQ_SYNC or REQ_FUA bio is written. If no such requests are made then it commits metadata once every second. If power is lost, it may lose some recent writes. However, EnhanceIO and bcache do not acknowledge IO completion until both IO and metadata hits the SSD. Hence, EnhanceIO and bcache provide higher data integrity at a cost of performance.

The fio config and setup information follows:
HDD              : 100GB
SSD              :  20GB
Mode             : write through / write back
Cache block_size : 4KB for bcache, EnhanceIO and 256KB for dm-cache

The other options have been left to their default values.

Note:
1) In case of dm-cache, we used 2 partitions of same SSD with 1GB partition as metadata and 20GB partition as caching device. This has been done so as to ensure a fair comparison as EnhanceIO and bcache do not have a separate metadata device.

2) In order to ensure proper cache warm up, We have turned off sequential bypass in bcache. This does not impact our performance results as they are taken for random workload.

For each test, we first performed a warm up run using the following fio options:
fio --direct=1 --size=90% --filesize=20G --blocksize=4k --ioengine=libaio --rw=rw --rwmixread=100 --rwmixwrite=0 --iodepth=8 ...

Then, we performed our actual run with the following fio options:
fio --direct=1 --size=100% --filesize=20G --blocksize=4k --ioengine=libaio --rw=randrw --rwmixread=90 --rwmixwrite=10 --iodepth=8 --numjobs=4 --random_distribution=zipf:1.2 ...

============================= Write Through ===============================
Type      Read Latency(ms)   Write Latency(ms)    Read(MB/s)    Write(MB/s)
===========================================================================
EnhanceIO      1.58              16.53               32.91       3.65
bcache         0.58              31.05               27.17       3.02
dm-cache       0.24              27.45               31.05       3.44

============================= Write Back ==================================
Type      Read Latency(ms)    Write Latency(ms)    Read(MB/s)   Write(MB/s)
===========================================================================
EnhanceIO      0.34               4.98               138.72      15.40
bcache         0.95               1.76               106.82      11.85
dm-cache       0.58               0.55               193.76      21.52

============================ Base Line ====================================
Type      Read Latency(ms)    Write Latency(ms)    Read(MB/s)   Write(MB/s)
===========================================================================
HDD            6.22              27.23                13.51       1.49
SSD            0.47               0.42               235.87      26.21

We have written scripts that aid in cache creation, deletion and performance run for all these caching solutions. These scripts can be found at:
https://github.com/stec-inc/EnhanceIO/tree/master/performance_test

Thanks and Regards,
sTec Team

PROPRIETARY-CONFIDENTIAL INFORMATION INCLUDED



This electronic transmission, and any documents attached hereto, may contain confidential, proprietary and/or legally privileged information. The information is intended only for use by the recipient named above. If you received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of information received in error is strictly prohibited, and violators will be pursued legally.

Re: Performance Comparison among EnhanceIO, bcache and dm-cache.

[Amit Kale]

On Tuesday 11 Jun 2013, OS Engineering wrote:
> Hi Jens,
> 
> In continuation with our previous communication, we have carried out
> performance comparison among EnhanceIO, bcache and dm-cache.
> 
> We found that EnhanceIO provides better throughput on zipf workload (with
> theta=1.2) in comparison to bcache and dm-cache for write through caches.
> However, for write back caches, we found that dm-cache had best throughput
> followed by EnhanceIO and then bcache. Dm-cache commits on-disk metadata
> every time a REQ_SYNC or REQ_FUA bio is written. If no such requests are
> made then it commits metadata once every second. If power is lost, it may
> lose some recent writes. However, EnhanceIO and bcache do not acknowledge
> IO completion until both IO and metadata hits the SSD. Hence, EnhanceIO
> and bcache provide higher data integrity at a cost of performance.

So it won't be fair to compare them with dm-cache in write-back mode since 
guarantees are different. I am sure if similar (SYNC/FUA enforcement for 
persistence) guarantees are implemented in bcache/EnhanceIO, they'll offer a 
much better performance. 

> 
> The fio config and setup information follows:
> HDD              : 100GB
> SSD              :  20GB
> Mode             : write through / write back
> Cache block_size : 4KB for bcache, EnhanceIO and 256KB for dm-cache
> 
> The other options have been left to their default values.
> 
> Note:
> 1) In case of dm-cache, we used 2 partitions of same SSD with 1GB partition
> as metadata and 20GB partition as caching device. This has been done so as
> to ensure a fair comparison as EnhanceIO and bcache do not have a separate
> metadata device.
> 
> 2) In order to ensure proper cache warm up, We have turned off sequential
> bypass in bcache. This does not impact our performance results as they are
> taken for random workload.
> 
> For each test, we first performed a warm up run using the following fio
> options: fio --direct=1 --size=90% --filesize=20G --blocksize=4k
> --ioengine=libaio --rw=rw --rwmixread=100 --rwmixwrite=0 --iodepth=8 ...
> 
> Then, we performed our actual run with the following fio options:
> fio --direct=1 --size=100% --filesize=20G --blocksize=4k --ioengine=libaio
> --rw=randrw --rwmixread=90 --rwmixwrite=10 --iodepth=8 --numjobs=4
> --random_distribution=zipf:1.2 ...

Did you experiment a little with varying iodepth and numjobs? Is this the best 
combination you found out? The numbers below appear low considering a 90% hit 
ratio for EnhanceIO. SSD baseline performance shown below appears low. However 
it should not affect this comparison. All caching solutions are being subjected 
to the same ratio of HDD/SSD performance.

> 
> ============================= Write Through ===============================
> Type      Read Latency(ms)   Write Latency(ms)    Read(MB/s)    Write(MB/s)
> ===========================================================================
> EnhanceIO      1.58              16.53               32.91       3.65
> bcache         0.58              31.05               27.17       3.02
> dm-cache       0.24              27.45               31.05       3.44

EnhanceIO shows much higher read latency here. Is that because of READFILLs ? 
Write latency, read-write throughputs are good.

> 
> ============================= Write Back ==================================
> Type      Read Latency(ms)    Write Latency(ms)    Read(MB/s)   Write(MB/s)
> ===========================================================================
> EnhanceIO      0.34               4.98               138.72      15.40
> bcache         0.95               1.76               106.82      11.85
> dm-cache       0.58               0.55               193.76      21.52

Here EnhanceIO's read latency is better compared the other two. Write latency 
is larger than the other two. 

-Amit

> 
> ============================ Base Line ====================================
> Type      Read Latency(ms)    Write Latency(ms)    Read(MB/s)   Write(MB/s)
> ===========================================================================
> HDD            6.22              27.23                13.51       1.49
> SSD            0.47               0.42               235.87      26.21
> 
> We have written scripts that aid in cache creation, deletion and
> performance run for all these caching solutions. These scripts can be
> found at:
> https://github.com/stec-inc/EnhanceIO/tree/master/performance_test
> 
> Thanks and Regards,
> sTec Team
> 
> PROPRIETARY-CONFIDENTIAL INFORMATION INCLUDED
> 
> This electronic transmission, and any documents attached hereto, may
> contain confidential, proprietary and/or legally privileged information.
> The information is intended only for use by the recipient named above. If
> you received this electronic message in error, please notify the sender
> and delete the electronic message. Any disclosure, copying, distribution,
> or use of the contents of information received in error is strictly
> prohibited, and violators will be pursued legally. --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

RE: Performance Comparison among EnhanceIO, bcache and dm-cache.

[OS Engineering]

> -----Original Message-----
> From: Amit Kale [mailto:amitkale@geeksofpune.in]
> Sent: Wednesday, June 12, 2013 10:28 AM
> To: OS Engineering
> Cc: koverstreet@google.com; linux-bcache@vger.kernel.org;
> thornber@redhat.com; dm-devel@redhat.com; LKML; Jens Axboe; Padmini
> Balasubramaniyan; Amit Phansalkar
> Subject: Re: Performance Comparison among EnhanceIO, bcache and dm-
> cache.
> 
> On Tuesday 11 Jun 2013, OS Engineering wrote:
> > Hi Jens,
> >
> > In continuation with our previous communication, we have carried out
> > performance comparison among EnhanceIO, bcache and dm-cache.
> >
> > We found that EnhanceIO provides better throughput on zipf workload
> > (with
> > theta=1.2) in comparison to bcache and dm-cache for write through caches.
> > However, for write back caches, we found that dm-cache had best
> > throughput followed by EnhanceIO and then bcache. Dm-cache commits
> > on-disk metadata every time a REQ_SYNC or REQ_FUA bio is written. If
> > no such requests are made then it commits metadata once every second.
> > If power is lost, it may lose some recent writes. However, EnhanceIO
> > and bcache do not acknowledge IO completion until both IO and metadata
> > hits the SSD. Hence, EnhanceIO and bcache provide higher data integrity at
> a cost of performance.
> 
> So it won't be fair to compare them with dm-cache in write-back mode since
> guarantees are different. I am sure if similar (SYNC/FUA enforcement for
> persistence) guarantees are implemented in bcache/EnhanceIO, they'll offer
> a much better performance.
> 
> >
> > The fio config and setup information follows:
> > HDD              : 100GB
> > SSD              :  20GB
> > Mode             : write through / write back
> > Cache block_size : 4KB for bcache, EnhanceIO and 256KB for dm-cache
> >
> > The other options have been left to their default values.
> >
> > Note:
> > 1) In case of dm-cache, we used 2 partitions of same SSD with 1GB partition
> > as metadata and 20GB partition as caching device. This has been done so as
> > to ensure a fair comparison as EnhanceIO and bcache do not have a
> separate
> > metadata device.
> >
> > 2) In order to ensure proper cache warm up, We have turned off
> sequential
> > bypass in bcache. This does not impact our performance results as they are
> > taken for random workload.
> >
> > For each test, we first performed a warm up run using the following fio
> > options: fio --direct=1 --size=90% --filesize=20G --blocksize=4k
> > --ioengine=libaio --rw=rw --rwmixread=100 --rwmixwrite=0 --iodepth=8 ...
> >
> > Then, we performed our actual run with the following fio options:
> > fio --direct=1 --size=100% --filesize=20G --blocksize=4k --ioengine=libaio
> > --rw=randrw --rwmixread=90 --rwmixwrite=10 --iodepth=8 --numjobs=4
> > --random_distribution=zipf:1.2 ...
> 
> Did you experiment a little with varying iodepth and numjobs? Is this the
> best
> combination you found out? The numbers below appear low considering a
> 90% hit
> ratio for EnhanceIO. SSD baseline performance shown below appears low.
> However
> it should not affect this comparison. All caching solutions are being subjected
> to the same ratio of HDD/SSD performance.

We have chosen a representative workload for our tests, we could do more experiments  
that tests SSD to its performance limits.  


> >
> > ============================= Write Through ===============================
> > Type      Read Latency(ms)   Write Latency(ms)    Read(MB/s)    Write(MB/s)
> > ========================================================================
> > EnhanceIO      1.58                             16.53                           32.91               3.65
> > bcache             0.58                             31.05                           27.17                3.02
> > dm-cache       0.24                             27.45                           31.05                3.44
> 
> EnhanceIO shows much higher read latency here. Is that because of
> READFILLs ?
> Write latency, read-write throughputs are good.
> 

Yes, READFILL's (SSD writes at the time of read of an uncached block) is the reason for higher read latency in EnhanceIO. 


> >
> > ============================= Write Back ==================================
> > Type      Read Latency(ms)    Write Latency(ms)    Read(MB/s)   Write(MB/s)
> >
>>==========================================================================
> > EnhanceIO      0.34                         4.98                                 138.72                15.40
> > bcache             0.95                          1.76                                 106.82                11.85
> > dm-cache       0.58                          0.55                                 193.76                21.52
> 
> Here EnhanceIO's read latency is better compared the other two. Write
> latency
> is larger than the other two.
> 
> -Amit
 
EnhanceIO has higher write latencies as it acknowledges IO completion only when both data and metadata has
hit the SSD.

> >
> > ============================ Base Line ====================================
> > Type      Read Latency(ms)    Write Latency(ms)    Read(MB/s)   Write(MB/s)
>> ==========================================================================
> > HDD            6.22                              27.23                               13.51                1.49
> > SSD             0.47                                0.42                             235.87              26.21
> >
> > We have written scripts that aid in cache creation, deletion and
> > performance run for all these caching solutions. These scripts can be
> > found at:
> > https://github.com/stec-inc/EnhanceIO/tree/master/performance_test
> >
> > Thanks and Regards,
> > sTec Team

Re: Performance Comparison among EnhanceIO, bcache and dm-cache.

[Joe Thornber]

On Tue, Jun 11, 2013 at 03:05:07PM +0000, OS Engineering wrote:

...

> Dm-cache commits on-disk metadata every time a REQ_SYNC or REQ_FUA
> bio is written. If no such requests are made then it commits
> metadata once every second. If power is lost, it may lose some
> recent writes.

Not true (though it is true for thinp, which may be where you got this
idea?).  For caching we have to commit whenever data is moved about,
otherwise a crash could result in us reading data that is not just out
of date (acceptable for some), but used to belong to a totally
different part of the device (always unacceptable).

- Joe

Re: [dm-devel] Performance Comparison among EnhanceIO, bcache and dm-cache.

[Michael Fortson]

On Mon, Jun 17, 2013 at 6:29 AM, Joe Thornber <thornber@redhat.com> wrote:
> On Tue, Jun 11, 2013 at 03:05:07PM +0000, OS Engineering wrote:
>
> ...
>
>> Dm-cache commits on-disk metadata every time a REQ_SYNC or REQ_FUA
>> bio is written. If no such requests are made then it commits
>> metadata once every second. If power is lost, it may lose some
>> recent writes.
>
> Not true (though it is true for thinp, which may be where you got this
> idea?).  For caching we have to commit whenever data is moved about,
> otherwise a crash could result in us reading data that is not just out
> of date (acceptable for some), but used to belong to a totally
> different part of the device (always unacceptable).
>
> - Joe

(Apologies if this is received multiple times -- I forgot to disable
HTML the first time)

Hi Joe,

In Documentation/device-mapper/cache.txt, is this text out of date?

     On-disk metadata is committed every time a REQ_SYNC or REQ_FUA
     bio is written.  If no such requests are made then commits will
     occur every second.  This means the cache behaves like a physical
     disk that has a write cache (the same is true of the
     thin-provisioning target).  If power is lost you may lose some
     recent writes.  The metadata should always be consistent in spite
     of any crash.

You say that this is the case for thinp, however.  Does that mean that
it's not possible to get stable writes with dm-thin without using
cache-control commands?  I don't see any documentation to this effect.
 If so, are there any plans to change this behavior (or make it
configurable)?

Thanks,
-Michael

RE: Performance Comparison among EnhanceIO, bcache and dm-cache.

[OS Engineering]

>On Jun 11, 2013 11:06 AM, "OS Engineering" <osengineering@stec-inc.com> wrote:
>>
>> Hi Jens,
>>
>> In continuation with our previous communication, we have carried out performance comparison among EnhanceIO, bcache and dm-cache.

>How reproducible are these results?  Any chance you could do 5-10 runs to get the avg and stddev?  May (or may not) prove interesting.

Hi mike,

We found that, in case of write through caches, the test results were consistent for all caching solution with EnhanceIO providing better throughput in comparison to bcache and dm-cache. However, under write-back mode, dm-cache showed a large standard deviation. The results for EnhanceIO and bcache were consistent with EnhanceIO providing higher throughput when compared to bcache.

We have performed 5 runs for all our performance tests and observed their mean and standard deviations. 
The test results can be found at: https://gist.github.com/sanoj-stec/5858574
The test scripts can be found at:  https://github.com/stec-inc/EnhanceIO/tree/master/performance_test

Regards,
Sanoj