dm-cache performance behaviour

dm-cache performance behaviour

[Andreas Herrmann]

Hi,

I've recently looked at performance behaviour of dm-cache and bcache.
I've repeatedly observed very low performance with dm-cache in
different tests. (Similar tests with bcache showed no such oddities.)

To rule out user errors that might have caused this, I shortly describe
what I've done and observed.

- tested kernel version: 4.5.0

- backing device: 1.5 TB spinning drive

- caching device: 128 GB SSD (used for metadata and cache and size
  of metadata part calculated based on
  https://www.redhat.com/archives/dm-devel/2012-December/msg00046.html)

- my test procedure consisted of a sequence of tests performing fio
  runs with different data sets, fio randread performance (bandwidth
  and IOPS) were compared, fio was invoked using something like

  fio --directory=/cached-device --rw=randread --name=fio-1 \
    --size=50G --group_reporting --ioengine=libaio \
    --direct=1 --iodepth=1 --runtime=40 --numjobs=1

  I've iterated over 10 runs for each of numjobs=1,2,3 and varied the
  name parameter to operate with different data sets.

  This procedure implied that with 3 jobs the underlying data set for
  the test consisted of 3 files with 50G each which exceeds the size
  of the caching device.

- Between some tests I've tried to empty the cache. For dm-cache I did
  this by unmounting the "compound" cache device, switching to cleaner
  target, zeroing metadata part of the caching device, recreating
  caching device and finally recreating the compound cache device
  (during this procedure I kept the backing device unmodified).

  I used dmsetup status to check for success of this operation
  (checking for #used_cache_blocks).
  If there is an easier way to do this please let me know -- If it's
  documented I've missed it.

- dm-cache parameters:
  * cache_mode: writeback
  * block size: 512 sectors
  * migration_threshold 2048 (default)

I've observed two oddities:

  (1) Only fio tests with the first data set created (and thus
  initially occupying the cache) showed decent performance
  results. Subsequent fio tests with another data set showed poor
  performance. I think this indicates that SMQ policy does not
  properly promote/demote data to/from caching device in my tests.

  (2) I've seen results where performance was actually below "native"
  (w/o caching) performance of the backing device. I think that this
  should not happen. If a data access falls back to the backing device
  due to a cache miss I would have expected to see almost the
  performance of the backing device. Maybe this points to a
  performance issue in SMQ -- spending too much time in policy code
  before falling back to the backing device.

I've tried to figure out what actually happened in SMQ code in these
cases - but eventually dismissed this. Next I want to check whether
there might be a flaw in my test setup/dm-cache configuration.

My understanding is that there are just two tunables for SMQ. Cache
block size (in sectors) and migration_threshold. So far I've sticked
to the defaults or to what I've found documented elsewhere. Are there
any recommendations for these values depending on the caching/backing
device sizes etc.?


Thanks,

Andreas

PS: Too keep this email short I'll put more details of my test
    procedure and a list of results in a follow-up mail to this one.

Re: dm-cache performance behaviour

[Zdenek Kabelac]

Dne 5.4.2016 v 09:12 Andreas Herrmann napsal(a):
> Hi,
>
> I've recently looked at performance behaviour of dm-cache and bcache.
> I've repeatedly observed very low performance with dm-cache in
> different tests. (Similar tests with bcache showed no such oddities.)
>
> To rule out user errors that might have caused this, I shortly describe
> what I've done and observed.
>
> - tested kernel version: 4.5.0
>
> - backing device: 1.5 TB spinning drive
>
> - caching device: 128 GB SSD (used for metadata and cache and size
>    of metadata part calculated based on
>    https://www.redhat.com/archives/dm-devel/2012-December/msg00046.html)
>
> - my test procedure consisted of a sequence of tests performing fio
>    runs with different data sets, fio randread performance (bandwidth
>    and IOPS) were compared, fio was invoked using something like
>
>    fio --directory=/cached-device --rw=randread --name=fio-1 \
>      --size=50G --group_reporting --ioengine=libaio \
>      --direct=1 --iodepth=1 --runtime=40 --numjobs=1
>
>    I've iterated over 10 runs for each of numjobs=1,2,3 and varied the
>    name parameter to operate with different data sets.
>
>    This procedure implied that with 3 jobs the underlying data set for
>    the test consisted of 3 files with 50G each which exceeds the size
>    of the caching device.
>
> - Between some tests I've tried to empty the cache. For dm-cache I did
>    this by unmounting the "compound" cache device, switching to cleaner
>    target, zeroing metadata part of the caching device, recreating
>    caching device and finally recreating the compound cache device
>    (during this procedure I kept the backing device unmodified).
>
>    I used dmsetup status to check for success of this operation
>    (checking for #used_cache_blocks).
>    If there is an easier way to do this please let me know -- If it's
>    documented I've missed it.
>
> - dm-cache parameters:
>    * cache_mode: writeback
>    * block size: 512 sectors
>    * migration_threshold 2048 (default)
>
> I've observed two oddities:
>
>    (1) Only fio tests with the first data set created (and thus
>    initially occupying the cache) showed decent performance
>    results. Subsequent fio tests with another data set showed poor
>    performance. I think this indicates that SMQ policy does not
>    properly promote/demote data to/from caching device in my tests.
>
>    (2) I've seen results where performance was actually below "native"
>    (w/o caching) performance of the backing device. I think that this
>    should not happen. If a data access falls back to the backing device
>    due to a cache miss I would have expected to see almost the
>    performance of the backing device. Maybe this points to a
>    performance issue in SMQ -- spending too much time in policy code
>    before falling back to the backing device.
>
> I've tried to figure out what actually happened in SMQ code in these
> cases - but eventually dismissed this. Next I want to check whether
> there might be a flaw in my test setup/dm-cache configuration.

Hi

The dm-cache SMQ/MQ is a 'slow moving' hot-spot cache.

So before the block is 'promoted' to the cache - there needs to be a reason 
for it - and it's not a plain single read.

So if the other cache promotes the block to the cache with a single block 
access you may observe different performance.

dm-cache is not targeted for 'quick' promoting of read blocks into a cache - 
rather 'slow' moving of often used blocks.

Unsure how that fits your testing environment and what you try to actually test?

Regards

PS: 256K dm-cache blocks size is quite large - it really depends upon workload 
- min supported size is 32K - lvm2 defaults to 64K...

Zdenek

Re: dm-cache performance behaviour

[Pasi Kärkkäinen]

On Tue, Apr 05, 2016 at 10:36:12AM +0200, Zdenek Kabelac wrote:
> Dne 5.4.2016 v 09:12 Andreas Herrmann napsal(a):
> >Hi,
> >
> >I've recently looked at performance behaviour of dm-cache and bcache.
> >I've repeatedly observed very low performance with dm-cache in
> >different tests. (Similar tests with bcache showed no such oddities.)
> >
> >To rule out user errors that might have caused this, I shortly describe
> >what I've done and observed.
> >
> >- tested kernel version: 4.5.0
> >
> >- backing device: 1.5 TB spinning drive
> >
> >- caching device: 128 GB SSD (used for metadata and cache and size
> >   of metadata part calculated based on
> >   https://www.redhat.com/archives/dm-devel/2012-December/msg00046.html)
> >
> >- my test procedure consisted of a sequence of tests performing fio
> >   runs with different data sets, fio randread performance (bandwidth
> >   and IOPS) were compared, fio was invoked using something like
> >
> >   fio --directory=/cached-device --rw=randread --name=fio-1 \
> >     --size=50G --group_reporting --ioengine=libaio \
> >     --direct=1 --iodepth=1 --runtime=40 --numjobs=1
> >
> >   I've iterated over 10 runs for each of numjobs=1,2,3 and varied the
> >   name parameter to operate with different data sets.
> >
> >   This procedure implied that with 3 jobs the underlying data set for
> >   the test consisted of 3 files with 50G each which exceeds the size
> >   of the caching device.
> >
> >- Between some tests I've tried to empty the cache. For dm-cache I did
> >   this by unmounting the "compound" cache device, switching to cleaner
> >   target, zeroing metadata part of the caching device, recreating
> >   caching device and finally recreating the compound cache device
> >   (during this procedure I kept the backing device unmodified).
> >
> >   I used dmsetup status to check for success of this operation
> >   (checking for #used_cache_blocks).
> >   If there is an easier way to do this please let me know -- If it's
> >   documented I've missed it.
> >
> >- dm-cache parameters:
> >   * cache_mode: writeback
> >   * block size: 512 sectors
> >   * migration_threshold 2048 (default)
> >
> >I've observed two oddities:
> >
> >   (1) Only fio tests with the first data set created (and thus
> >   initially occupying the cache) showed decent performance
> >   results. Subsequent fio tests with another data set showed poor
> >   performance. I think this indicates that SMQ policy does not
> >   properly promote/demote data to/from caching device in my tests.
> >
> >   (2) I've seen results where performance was actually below "native"
> >   (w/o caching) performance of the backing device. I think that this
> >   should not happen. If a data access falls back to the backing device
> >   due to a cache miss I would have expected to see almost the
> >   performance of the backing device. Maybe this points to a
> >   performance issue in SMQ -- spending too much time in policy code
> >   before falling back to the backing device.
> >
> >I've tried to figure out what actually happened in SMQ code in these
> >cases - but eventually dismissed this. Next I want to check whether
> >there might be a flaw in my test setup/dm-cache configuration.
> 
> Hi
> 
> The dm-cache SMQ/MQ is a 'slow moving' hot-spot cache.
> 
> So before the block is 'promoted' to the cache - there needs to be a
> reason for it - and it's not a plain single read.
> 
> So if the other cache promotes the block to the cache with a single
> block access you may observe different performance.
> 
> dm-cache is not targeted for 'quick' promoting of read blocks into a
> cache - rather 'slow' moving of often used blocks.
> 

Are there currently any plans to add some kind of write-back caching of new blocks (writes) to dm-cache ? 

Earlier I did some benchmarks with dm-cache aswell, and it didn't perform well with write-back caching of new blocks,
while for example enhanceIO performed nicely.. (and yes I understand currently dm-cache isn't even supposed to help much with this use case).


Thanks,

-- Pasi

Re: dm-cache performance behaviour

[Andreas Herrmann]

As mentioned, here is a rather long mail containing a description of
what I've tested and a typical sample from the test results gathered
with this test procedure.

Andreas

--

test system, software:
- Intel(R) Core(TM) i7-3770S CPU @ 3.10GHz
- 32 GB DDR3 (1333 MHz) RAM
- Ubuntu 14.04 LTS
- Linux kernel 4.5.0
- Backing device: SAMSUNG HD155UI (1.5 TB), entire disk used
- Caching device: SAMSUNG SSD 830 (128 GB), entire disk used
  (in case of dm-cache used for both metadata cache data)
- fio (flexible I/O tester) as of 2.2.9-37-g0e1c4
- ext4 file system on compound cache device
- CFQ I/O scheduler for caching and backing device

dmcache:
- writeback cache mode
- SMQ cache policy
- cache block size: 512 sectors, resulting in cache blocks of size
  256KB

bcache:
- writeback cache mode
- LRU cache policy

test procedure:
- Create compound cache device
- Test A: fio test using data set 1 (using empty cache
  device, it includes creation of data set 1)
- Test B: fio test using data set 1 for the second
  successive time
- Test C: fio test using data set 2 (it includes creation
  of data set 2)
- Test D: fio test using data set 2 for the second
  successive time
- Test E: fio test using data set 2 for the third
  successive time
- Test F: fio test using data set 3 (it includes creation
  of data set 3)
- Clear cache (remove and recreate caching device, re-attach it to
  compound cache device)
- Test G: fio test using data set 1 (using empty cache)
- Copy data set 1 into temporary directory on test disk to clobber
  cache
- Test H: fio test using data set 1 (after cache was
  clobbered with data from data set 1)
- Test I: fio test using data set 2 (after cache was
  clobbered and test H)

fio options and arguments:
-directory=<test dir on cached device>
-name=<file-name-prefix>
-rw=randread 
-size=50G
-group_reporting
-ioengine=libaio
-direct=1
-iodepth=1
-runtime=40
-numjobs=<1,2,3>

-> block size 4k
-> each of above tests did 10 iterations for 1 job, 10 iterations for
   2 jobs, 10 iterations fo 3 jobs using the same data set.
-> one data set for one test consists of 3 data files (created by
   fio), each having size 50G, thus all 3 files exceed size of caching
   disk

Following 3 tables contain typical bandwidth measurements (in KB/s) of
one run of above test procedure. As reference "native" (ie. w/o
storage caching) performance for both SSD and HD are provided in all
tables. This native performance was gathered with test A from the test
procedure. In case of SSD --size parameter was set to 40G otherwise
all 3 files would not have fit on the device.

First table shows results for test A, B, C for both dm-cache and
bcache.  Remaining tests E-I are shown in second table for dm-cache
and in third table for bcache.

    i = iteration, n = numjobs, shown is bandwidth in KB/s

       | native results|        dm-cache       |       bcache
 i | n |1.5TB HD|  SSD | test A| test B| test C| test A| test B| test C 
-----------------------------------------------------------------------
 1 | 1 | 353.06 | 20568| 1154.7| 1819.6| 394.79|  351.6|  6111 |  377.2 
 2 | 1 | 348.25 | 20571| 1265.2| 1896.6| 395.46|  701.9|  6393 |  748.8 
 3 | 1 | 352.36 | 20573| 1298.2| 1917.1| 395.05| 1047.3|  6681 | 1120.4 
 4 | 1 | 349.60 | 20542| 1494.5| 1919.2| 394.54| 1389.3|  6962 | 1483.4 
 5 | 1 | 353.02 | 20554| 1703.6| 1921.1| 394.52| 1724.7|  7247 | 1841.7 
 6 | 1 | 352.04 | 20564| 1745.1| 1938.2| 381.38| 2060.1|  7517 | 2200.2 
 7 | 1 | 349.19 | 20585| 1929.5| 1957.6| 399.18| 2383.6|  7805 | 2556.5 
 8 | 1 | 350.14 | 20584| 2047.6| 1935.6| 406.40| 2712.5|  8082 | 2908.5 
 9 | 1 | 348.25 | 20573| 2231.3| 1951.2| 407.63| 3034.3|  8350 | 3260.5 
10 | 1 | 349.99 | 20579| 2336.2| 2001.7| 403.35| 3355.5|  8622 | 3612.2 
11 | 2 | 324.67 | 38590| 2904.6| 2643.1| 369.60| 3694.8| 10769 | 3991.5 
12 | 2 | 327.30 | 38466| 2865.3| 2601.1| 373.93| 4033.4| 11085 | 4369.1 
13 | 2 | 329.87 | 38550| 2882.2| 2594.9| 382.51| 4371.9| 11395 | 4743.9 
14 | 2 | 328.70 | 38459| 2893.3| 2614.1| 386.55| 4708.9| 11717 | 5118.6 
15 | 2 | 324.66 | 38468| 2918.2| 2641.8| 393.40| 5038.3| 12033 | 5491.8 
16 | 2 | 324.78 | 38514| 2907.8| 2605.4| 400.04| 5379.1| 12336 | 5861.2 
17 | 2 | 322.55 | 38519| 2923.5| 2684.6| 405.56| 5704.7| 12646 | 6227.9 
18 | 2 | 325.76 | 38519| 2927.1| 2689.4| 408.50| 6034.9| 12955 | 6585.4 
19 | 2 | 323.51 | 38482| 3005.5| 2662.3| 411.20| 6360.4| 13263 | 6942.3 
20 | 2 | 327.49 | 38509| 3016.3| 2714.3| 420.88| 6684.7| 13564 | 7306.2 
21 | 3 | 337.04 | 56139| 2549.8| 2512.5| 430.07| 5205.9| 12577 | 5659.6 
22 | 3 | 338.73 | 56210| 2462.2| 2459.1| 428.68| 5542.3| 12899 | 6040.3 
23 | 3 | 339.08 | 56020| 2442.8| 2519.6| 430.82| 5893.2| 13182 | 6417.4 
24 | 3 | 340.26 | 56142| 2360.3| 2526.4| 432.77| 6230.7| 13546 | 6796.2 
25 | 3 | 333.83 | 56081| 2303.1| 2508.4| 434.03| 6571.1| 13865 | 7170.2 
26 | 3 | 338.32 | 56195| 2344.8| 2508.5| 431.66| 6909.8| 14182 | 7540.5 
27 | 3 | 339.73 | 56165| 2351.4| 2540.4| 434.86| 7227.1| 14494 | 7915.4 
28 | 3 | 331.52 | 56216| 2354.5| 2489.7| 430.30| 7570.4| 14812 | 8279.1 
29 | 3 | 334.47 | 56083| 2339.9| 2490.4| 441.79| 7908.7| 15129 | 8640.6 
30 | 3 | 347.99 | 56089| 2418.5| 2490.3| 446.06| 8241.4| 15443 | 9008.8 

       | native results|                      dm-cache
 i | n |1.5TB HD|  SSD | test D| test E| test F| test G| test H | test I 
------------------------------------------------------------------------
 1 | 1 | 353.06 | 20568| 461.6 | 521.6 | 390.2 |  328.8| 1098.7 | 222.5
 2 | 1 | 348.25 | 20571| 466.6 | 524.2 | 389.2 |  184.6| 1093.5 | 229.1
 3 | 1 | 352.36 | 20573| 485.7 | 521.9 | 389.0 |  480.0| 1095.8 | 225.7
 4 | 1 | 349.60 | 20542| 497.3 | 525.1 | 389.4 |  579.3| 1096.7 | 230.1
 5 | 1 | 353.02 | 20554| 501.5 | 527.4 | 388.9 |  649.5| 1100.6 | 226.9
 6 | 1 | 352.04 | 20564| 504.0 | 528.8 | 369.9 |  714.7| 1095.2 | 224.9
 7 | 1 | 349.19 | 20585| 507.2 | 530.8 | 398.1 |  766.9| 1095.2 | 225.9
 8 | 1 | 350.14 | 20584| 497.9 | 530.4 | 402.3 |  817.1| 1096.6 | 226.5
 9 | 1 | 348.25 | 20573| 516.6 | 535.7 | 399.5 |  847.7| 1098.2 | 226.8
10 | 1 | 349.99 | 20579| 518.1 | 534.2 | 397.1 |  877.8| 1098.7 | 232.6
11 | 2 | 324.67 | 38590| 449.8 | 472.9 | 373.4 |  888.7| 1201.1 | 261.1
12 | 2 | 327.30 | 38466| 447.3 | 473.0 | 377.1 |  898.9| 1205.7 | 265.9
13 | 2 | 329.87 | 38550| 451.9 | 473.9 | 386.3 |  932.4| 1204.1 | 264.9
14 | 2 | 328.70 | 38459| 451.0 | 475.0 | 394.9 |  961.1| 1205.3 | 265.7
15 | 2 | 324.66 | 38468| 456.5 | 476.6 | 401.2 |  993.7| 1210.7 | 268.6
16 | 2 | 324.78 | 38514| 459.4 | 487.0 | 409.0 | 1034.9| 1206.2 | 269.7
17 | 2 | 322.55 | 38519| 456.1 | 498.9 | 411.7 | 1062.3| 1204.9 | 273.3
18 | 2 | 325.76 | 38519| 458.3 | 500.2 | 422.0 | 1081.6| 1204.6 | 280.5
19 | 2 | 323.51 | 38482| 457.1 | 498.3 | 430.5 | 1100.3| 1207.5 | 279.0
20 | 2 | 327.49 | 38509| 455.8 | 500.4 | 428.7 | 1106.8| 1203.6 | 285.7
21 | 3 | 337.04 | 56139| 462.6 | 481.5 | 439.0 | 1060.9| 1250.2 | 312.7
22 | 3 | 338.73 | 56210| 462.9 | 483.3 | 437.2 | 1095.7| 1253.9 | 306.8
23 | 3 | 339.08 | 56020| 469.5 | 489.1 | 437.8 | 1163.7| 1248.9 | 304.7
24 | 3 | 340.26 | 56142| 462.1 | 487.1 | 440.1 | 1189.9| 1252.1 | 303.0
25 | 3 | 333.83 | 56081| 465.1 | 490.8 | 436.9 | 1217.7| 1248.6 | 302.6
26 | 3 | 338.32 | 56195| 465.1 | 489.8 | 433.4 | 1234.8| 1248.7 | 308.3
27 | 3 | 339.73 | 56165| 464.6 | 490.7 | 437.0 | 1256.1| 1253.3 | 305.4
28 | 3 | 331.52 | 56216| 463.2 | 496.7 | 435.6 | 1281.8| 1250.5 | 307.1
29 | 3 | 334.47 | 56083| 469.4 | 493.3 | 434.6 | 1302.6| 1249.4 | 308.9
30 | 3 | 347.99 | 56089| 464.0 | 501.4 | 442.5 | 1321.5| 1250.9 | 305.4

       | native results|                      bcache
 i | n |1.5TB HD|  SSD | test D| test E| test F| test G| test H| test I 
-----------------------------------------------------------------------
 1 | 1 | 353.06 | 20568|  6652 | 11905 |  390.3|  389.9|  6995 |  376.8
 2 | 1 | 348.25 | 20571|  6964 | 12182 |  776.5|  779.3|  7339 |  750.3
 3 | 1 | 352.36 | 20573|  7275 | 12451 | 1161.2| 1164.1|  7677 | 1120.2
 4 | 1 | 349.60 | 20542|  7585 | 12723 | 1541.6| 1547.2|  8020 | 1487.1
 5 | 1 | 353.02 | 20554|  7889 | 12989 | 1916.4| 1924.4|  8357 | 1850.7
 6 | 1 | 352.04 | 20564|  8192 | 13255 | 2287.6| 2301.6|  8688 | 2210.6
 7 | 1 | 349.19 | 20585|  8496 | 13520 | 2658.5| 2677.5|  9023 | 2568.8
 8 | 1 | 350.14 | 20584|  8795 | 13783 | 3029.3| 3053.7|  9359 | 2922.7
 9 | 1 | 348.25 | 20573|  9093 | 14045 | 3397.9| 3423.3|  9687 | 3278.7
10 | 1 | 349.99 | 20579|  9392 | 14305 | 3764.5| 3796.8| 10014 | 3632.8
11 | 2 | 324.67 | 38590| 11773 | 18720 | 4135.8| 4177.6| 12288 | 4015.5
12 | 2 | 327.30 | 38466| 12128 | 19046 | 4499.6| 4554.4| 12631 | 4394.3
13 | 2 | 329.87 | 38550| 12480 | 19369 | 4859.4| 4928.1| 12975 | 4770.4
14 | 2 | 328.70 | 38459| 12822 | 19696 | 5218.2| 5298.8| 13315 | 5142.5
15 | 2 | 324.66 | 38468| 13170 | 20014 | 5575.6| 5667.6| 13651 | 5510.6
16 | 2 | 324.78 | 38514| 13510 | 20333 | 5928.9| 6031.7| 13994 | 5876.4
17 | 2 | 322.55 | 38519| 13848 | 20650 | 6279.7| 6391.6| 14321 | 6239.1
18 | 2 | 325.76 | 38519| 14187 | 20960 | 6626.7| 6745.1| 14645 | 6595.5
19 | 2 | 323.51 | 38482| 14522 | 21271 | 6972.2| 7093.9| 14964 | 6954.6
20 | 2 | 327.49 | 38509| 14850 | 21575 | 7317.6| 7452.5| 15280 | 7305.4
21 | 3 | 337.04 | 56139| 13800 | 21069 | 5811.8| 5914.9| 14496 | 5732.5
22 | 3 | 338.73 | 56210| 14158 | 21402 | 6199.7| 6305.7| 14858 | 6113.1
23 | 3 | 339.08 | 56020| 14510 | 21727 | 6585.1| 6695.2| 15214 | 6494.2
24 | 3 | 340.26 | 56142| 14860 | 22053 | 6972.6| 7079.6| 15570 | 6870.9
25 | 3 | 333.83 | 56081| 15206 | 22373 | 7356.4| 7464.9| 15929 | 7244.4
26 | 3 | 338.32 | 56195| 15545 | 22691 | 7735.1| 7845.5| 16276 | 7616.8
27 | 3 | 339.73 | 56165| 15890 | 23017 | 8113.1| 8225.6| 16627 | 7988.6
28 | 3 | 331.52 | 56216| 16230 | 23323 | 8496.4| 8600.7| 16973 | 8353.7
29 | 3 | 334.47 | 56083| 16562 | 23636 | 8864.9| 8973.5| 17323 | 8722.5
30 | 3 | 347.99 | 56089| 16899 | 23945 | 9235.8| 9343.1| 17663 | 9081.4

Re: dm-cache performance behaviour

[Andreas Herrmann]

On Tue, Apr 05, 2016 at 10:36:12AM +0200, Zdenek Kabelac wrote:
> Dne 5.4.2016 v 09:12 Andreas Herrmann napsal(a):
> >Hi,
> >
> >I've recently looked at performance behaviour of dm-cache and bcache.
> >I've repeatedly observed very low performance with dm-cache in
> >different tests. (Similar tests with bcache showed no such oddities.)
> >
> >To rule out user errors that might have caused this, I shortly describe
> >what I've done and observed.
> >
> >- tested kernel version: 4.5.0
> >
> >- backing device: 1.5 TB spinning drive
> >
> >- caching device: 128 GB SSD (used for metadata and cache and size
> >   of metadata part calculated based on
> >   https://www.redhat.com/archives/dm-devel/2012-December/msg00046.html)
> >
> >- my test procedure consisted of a sequence of tests performing fio
> >   runs with different data sets, fio randread performance (bandwidth
> >   and IOPS) were compared, fio was invoked using something like
> >
> >   fio --directory=/cached-device --rw=randread --name=fio-1 \
> >     --size=50G --group_reporting --ioengine=libaio \
> >     --direct=1 --iodepth=1 --runtime=40 --numjobs=1
> >
> >   I've iterated over 10 runs for each of numjobs=1,2,3 and varied the
> >   name parameter to operate with different data sets.
> >
> >   This procedure implied that with 3 jobs the underlying data set for
> >   the test consisted of 3 files with 50G each which exceeds the size
> >   of the caching device.
> >
> >- Between some tests I've tried to empty the cache. For dm-cache I did
> >   this by unmounting the "compound" cache device, switching to cleaner
> >   target, zeroing metadata part of the caching device, recreating
> >   caching device and finally recreating the compound cache device
> >   (during this procedure I kept the backing device unmodified).
> >
> >   I used dmsetup status to check for success of this operation
> >   (checking for #used_cache_blocks).
> >   If there is an easier way to do this please let me know -- If it's
> >   documented I've missed it.
> >
> >- dm-cache parameters:
> >   * cache_mode: writeback
> >   * block size: 512 sectors
> >   * migration_threshold 2048 (default)
> >
> >I've observed two oddities:
> >
> >   (1) Only fio tests with the first data set created (and thus
> >   initially occupying the cache) showed decent performance
> >   results. Subsequent fio tests with another data set showed poor
> >   performance. I think this indicates that SMQ policy does not
> >   properly promote/demote data to/from caching device in my tests.
> >
> >   (2) I've seen results where performance was actually below "native"
> >   (w/o caching) performance of the backing device. I think that this
> >   should not happen. If a data access falls back to the backing device
> >   due to a cache miss I would have expected to see almost the
> >   performance of the backing device. Maybe this points to a
> >   performance issue in SMQ -- spending too much time in policy code
> >   before falling back to the backing device.
> >
> >I've tried to figure out what actually happened in SMQ code in these
> >cases - but eventually dismissed this. Next I want to check whether
> >there might be a flaw in my test setup/dm-cache configuration.
> 
> Hi
> 
> The dm-cache SMQ/MQ is a 'slow moving' hot-spot cache.

Yep that is mentioned in some places in the source code with the
hot-spot handling stuff.

> So before the block is 'promoted' to the cache - there needs to be a
> reason for it - and it's not a plain single read.

It's not obvious to me when a block finally gets promoted. I had the
impression that once the cache is filled with data, getting new data
into the cache takes quite some time.

> So if the other cache promotes the block to the cache with a single
> block access you may observe different performance.

Yep, that is what my measurements suggest.

> dm-cache is not targeted for 'quick' promoting of read blocks into a
> cache - rather 'slow' moving of often used blocks.

If I completely abandon to use a set of test files (which defined
hotspot blocks initially) and switch to a new set of test files this
"slow" moving of often used (in the past) blocks might be the cause of
the lower than expected (by me) performance in my tests. Would it be
possible to tune this behaviour to allow quicker promotion if a user
thinks he requires it for his workload?

> Unsure how that fits your testing environment and what you try to
> actually test?

Worst results for spinning disks are random accesses. I've seen some
dm-cache benchmark results (fio randread) that showed lower
performance than the underlying backing device itself. That was the
trigger for me to take a closer look at dm-cache and bcache and to do
some performance measurements esp. with random read I/O pattern.

I've observed two oddities (from my point of view) and either they are
due to setup errors, wrong expectations, or point to real issues that
might be worth to be looked at or to be aware of.
I think at least its worth to share my testing results.

> Regards
> 
> PS: 256K dm-cache blocks size is quite large - it really depends
> upon workload - min supported size is 32K - lvm2 defaults to 64K...

I had chosen 512 as block size because documenation mentioned it.

I've kicked off a test with the minimum block size.
Let's see whether that changes anything.


Thanks,

Andreas

Re: dm-cache performance behaviour

[Andreas Herrmann]

On Tue, Apr 05, 2016 at 03:31:09PM +0200, Andreas Herrmann wrote:

-8<-

> test procedure:
> - Create compound cache device
> - Test A: fio test using data set 1 (using empty cache
>   device, it includes creation of data set 1)
> - Test B: fio test using data set 1 for the second
>   successive time
> - Test C: fio test using data set 2 (it includes creation
>   of data set 2)
> - Test D: fio test using data set 2 for the second
>   successive time
> - Test E: fio test using data set 2 for the third
>   successive time
> - Test F: fio test using data set 3 (it includes creation
>   of data set 3)
> - Clear cache (remove and recreate caching device, re-attach it to
>   compound cache device)
> - Test G: fio test using data set 1 (using empty cache)
> - Copy data set 1 into temporary directory on test disk to clobber
>   cache
> - Test H: fio test using data set 1 (after cache was
>   clobbered with data from data set 1)
> - Test I: fio test using data set 2 (after cache was
>   clobbered and test H)

-8<-

> First table shows results for test A, B, C for both dm-cache and
> bcache.  Remaining tests E-I are shown in second table for dm-cache
> and in third table for bcache.
> 
>     i = iteration, n = numjobs, shown is bandwidth in KB/s
> 
>        | native results|        dm-cache       |       bcache
>  i | n |1.5TB HD|  SSD | test A| test B| test C| test A| test B| test C 
> -----------------------------------------------------------------------
>  1 | 1 | 353.06 | 20568| 1154.7| 1819.6| 394.79|  351.6|  6111 |  377.2 
>  2 | 1 | 348.25 | 20571| 1265.2| 1896.6| 395.46|  701.9|  6393 |  748.8 
>  3 | 1 | 352.36 | 20573| 1298.2| 1917.1| 395.05| 1047.3|  6681 | 1120.4 
>  4 | 1 | 349.60 | 20542| 1494.5| 1919.2| 394.54| 1389.3|  6962 | 1483.4 
>  5 | 1 | 353.02 | 20554| 1703.6| 1921.1| 394.52| 1724.7|  7247 | 1841.7 
>  6 | 1 | 352.04 | 20564| 1745.1| 1938.2| 381.38| 2060.1|  7517 | 2200.2 
>  7 | 1 | 349.19 | 20585| 1929.5| 1957.6| 399.18| 2383.6|  7805 | 2556.5 
>  8 | 1 | 350.14 | 20584| 2047.6| 1935.6| 406.40| 2712.5|  8082 | 2908.5 
>  9 | 1 | 348.25 | 20573| 2231.3| 1951.2| 407.63| 3034.3|  8350 | 3260.5 
> 10 | 1 | 349.99 | 20579| 2336.2| 2001.7| 403.35| 3355.5|  8622 | 3612.2 
> 11 | 2 | 324.67 | 38590| 2904.6| 2643.1| 369.60| 3694.8| 10769 | 3991.5 
> 12 | 2 | 327.30 | 38466| 2865.3| 2601.1| 373.93| 4033.4| 11085 | 4369.1 
> 13 | 2 | 329.87 | 38550| 2882.2| 2594.9| 382.51| 4371.9| 11395 | 4743.9 
> 14 | 2 | 328.70 | 38459| 2893.3| 2614.1| 386.55| 4708.9| 11717 | 5118.6 
> 15 | 2 | 324.66 | 38468| 2918.2| 2641.8| 393.40| 5038.3| 12033 | 5491.8 
> 16 | 2 | 324.78 | 38514| 2907.8| 2605.4| 400.04| 5379.1| 12336 | 5861.2 
> 17 | 2 | 322.55 | 38519| 2923.5| 2684.6| 405.56| 5704.7| 12646 | 6227.9 
> 18 | 2 | 325.76 | 38519| 2927.1| 2689.4| 408.50| 6034.9| 12955 | 6585.4 
> 19 | 2 | 323.51 | 38482| 3005.5| 2662.3| 411.20| 6360.4| 13263 | 6942.3 
> 20 | 2 | 327.49 | 38509| 3016.3| 2714.3| 420.88| 6684.7| 13564 | 7306.2 
> 21 | 3 | 337.04 | 56139| 2549.8| 2512.5| 430.07| 5205.9| 12577 | 5659.6 
> 22 | 3 | 338.73 | 56210| 2462.2| 2459.1| 428.68| 5542.3| 12899 | 6040.3 
> 23 | 3 | 339.08 | 56020| 2442.8| 2519.6| 430.82| 5893.2| 13182 | 6417.4 
> 24 | 3 | 340.26 | 56142| 2360.3| 2526.4| 432.77| 6230.7| 13546 | 6796.2 
> 25 | 3 | 333.83 | 56081| 2303.1| 2508.4| 434.03| 6571.1| 13865 | 7170.2 
> 26 | 3 | 338.32 | 56195| 2344.8| 2508.5| 431.66| 6909.8| 14182 | 7540.5 
> 27 | 3 | 339.73 | 56165| 2351.4| 2540.4| 434.86| 7227.1| 14494 | 7915.4 
> 28 | 3 | 331.52 | 56216| 2354.5| 2489.7| 430.30| 7570.4| 14812 | 8279.1 
> 29 | 3 | 334.47 | 56083| 2339.9| 2490.4| 441.79| 7908.7| 15129 | 8640.6 
> 30 | 3 | 347.99 | 56089| 2418.5| 2490.3| 446.06| 8241.4| 15443 | 9008.8 
> 
>        | native results|                      dm-cache
>  i | n |1.5TB HD|  SSD | test D| test E| test F| test G| test H | test I 
> ------------------------------------------------------------------------
>  1 | 1 | 353.06 | 20568| 461.6 | 521.6 | 390.2 |  328.8| 1098.7 | 222.5
>  2 | 1 | 348.25 | 20571| 466.6 | 524.2 | 389.2 |  184.6| 1093.5 | 229.1
>  3 | 1 | 352.36 | 20573| 485.7 | 521.9 | 389.0 |  480.0| 1095.8 | 225.7
>  4 | 1 | 349.60 | 20542| 497.3 | 525.1 | 389.4 |  579.3| 1096.7 | 230.1
>  5 | 1 | 353.02 | 20554| 501.5 | 527.4 | 388.9 |  649.5| 1100.6 | 226.9
>  6 | 1 | 352.04 | 20564| 504.0 | 528.8 | 369.9 |  714.7| 1095.2 | 224.9
>  7 | 1 | 349.19 | 20585| 507.2 | 530.8 | 398.1 |  766.9| 1095.2 | 225.9
>  8 | 1 | 350.14 | 20584| 497.9 | 530.4 | 402.3 |  817.1| 1096.6 | 226.5
>  9 | 1 | 348.25 | 20573| 516.6 | 535.7 | 399.5 |  847.7| 1098.2 | 226.8
> 10 | 1 | 349.99 | 20579| 518.1 | 534.2 | 397.1 |  877.8| 1098.7 | 232.6
> 11 | 2 | 324.67 | 38590| 449.8 | 472.9 | 373.4 |  888.7| 1201.1 | 261.1
> 12 | 2 | 327.30 | 38466| 447.3 | 473.0 | 377.1 |  898.9| 1205.7 | 265.9
> 13 | 2 | 329.87 | 38550| 451.9 | 473.9 | 386.3 |  932.4| 1204.1 | 264.9
> 14 | 2 | 328.70 | 38459| 451.0 | 475.0 | 394.9 |  961.1| 1205.3 | 265.7
> 15 | 2 | 324.66 | 38468| 456.5 | 476.6 | 401.2 |  993.7| 1210.7 | 268.6
> 16 | 2 | 324.78 | 38514| 459.4 | 487.0 | 409.0 | 1034.9| 1206.2 | 269.7
> 17 | 2 | 322.55 | 38519| 456.1 | 498.9 | 411.7 | 1062.3| 1204.9 | 273.3
> 18 | 2 | 325.76 | 38519| 458.3 | 500.2 | 422.0 | 1081.6| 1204.6 | 280.5
> 19 | 2 | 323.51 | 38482| 457.1 | 498.3 | 430.5 | 1100.3| 1207.5 | 279.0
> 20 | 2 | 327.49 | 38509| 455.8 | 500.4 | 428.7 | 1106.8| 1203.6 | 285.7
> 21 | 3 | 337.04 | 56139| 462.6 | 481.5 | 439.0 | 1060.9| 1250.2 | 312.7
> 22 | 3 | 338.73 | 56210| 462.9 | 483.3 | 437.2 | 1095.7| 1253.9 | 306.8
> 23 | 3 | 339.08 | 56020| 469.5 | 489.1 | 437.8 | 1163.7| 1248.9 | 304.7
> 24 | 3 | 340.26 | 56142| 462.1 | 487.1 | 440.1 | 1189.9| 1252.1 | 303.0
> 25 | 3 | 333.83 | 56081| 465.1 | 490.8 | 436.9 | 1217.7| 1248.6 | 302.6
> 26 | 3 | 338.32 | 56195| 465.1 | 489.8 | 433.4 | 1234.8| 1248.7 | 308.3
> 27 | 3 | 339.73 | 56165| 464.6 | 490.7 | 437.0 | 1256.1| 1253.3 | 305.4
> 28 | 3 | 331.52 | 56216| 463.2 | 496.7 | 435.6 | 1281.8| 1250.5 | 307.1
> 29 | 3 | 334.47 | 56083| 469.4 | 493.3 | 434.6 | 1302.6| 1249.4 | 308.9
> 30 | 3 | 347.99 | 56089| 464.0 | 501.4 | 442.5 | 1321.5| 1250.9 | 305.4


Potential issue 1:

It's hard to get data replaced (slow promotion/demotion) in the cache.
Illustrated by comparing results for tests A+B with what happened for
tests C+D. Same action, creating a bunch of test files, then doing
random read accesses and run a subsequent test with same set of data
files. But once it was done with an empty cache (tests A+B) and one
time with an already fully utilized cache (tests C+D). And even doing
a third round with the second data set (test E) doesn't substantially
improve performance.

Potential issue 2:

The very last test on the list -- test I -- showed performance that
was significantly below performance of the backing device.



Regards,

Andreas

Re: dm-cache performance behaviour

[Mike Snitzer]

On Tue, Apr 05 2016 at 10:05am -0400,
Andreas Herrmann <aherrmann@suse.com> wrote:

> On Tue, Apr 05, 2016 at 10:36:12AM +0200, Zdenek Kabelac wrote:
> > Dne 5.4.2016 v 09:12 Andreas Herrmann napsal(a):
> > >Hi,
> > >
> > >I've recently looked at performance behaviour of dm-cache and bcache.
> > >I've repeatedly observed very low performance with dm-cache in
> > >different tests. (Similar tests with bcache showed no such oddities.)
> > >
> > >To rule out user errors that might have caused this, I shortly describe
> > >what I've done and observed.
> > >
> > >- tested kernel version: 4.5.0
> > >
> > >- backing device: 1.5 TB spinning drive
> > >
> > >- caching device: 128 GB SSD (used for metadata and cache and size
> > >   of metadata part calculated based on
> > >   https://www.redhat.com/archives/dm-devel/2012-December/msg00046.html)
> > >
> > >- my test procedure consisted of a sequence of tests performing fio
> > >   runs with different data sets, fio randread performance (bandwidth
> > >   and IOPS) were compared, fio was invoked using something like
> > >
> > >   fio --directory=/cached-device --rw=randread --name=fio-1 \
> > >     --size=50G --group_reporting --ioengine=libaio \
> > >     --direct=1 --iodepth=1 --runtime=40 --numjobs=1
> > >
> > >   I've iterated over 10 runs for each of numjobs=1,2,3 and varied the
> > >   name parameter to operate with different data sets.
> > >
> > >   This procedure implied that with 3 jobs the underlying data set for
> > >   the test consisted of 3 files with 50G each which exceeds the size
> > >   of the caching device.
> > >
> > >- Between some tests I've tried to empty the cache. For dm-cache I did
> > >   this by unmounting the "compound" cache device, switching to cleaner
> > >   target, zeroing metadata part of the caching device, recreating
> > >   caching device and finally recreating the compound cache device
> > >   (during this procedure I kept the backing device unmodified).
> > >
> > >   I used dmsetup status to check for success of this operation
> > >   (checking for #used_cache_blocks).
> > >   If there is an easier way to do this please let me know -- If it's
> > >   documented I've missed it.
> > >
> > >- dm-cache parameters:
> > >   * cache_mode: writeback
> > >   * block size: 512 sectors
> > >   * migration_threshold 2048 (default)
> > >
> > >I've observed two oddities:
> > >
> > >   (1) Only fio tests with the first data set created (and thus
> > >   initially occupying the cache) showed decent performance
> > >   results. Subsequent fio tests with another data set showed poor
> > >   performance. I think this indicates that SMQ policy does not
> > >   properly promote/demote data to/from caching device in my tests.
> > >
> > >   (2) I've seen results where performance was actually below "native"
> > >   (w/o caching) performance of the backing device. I think that this
> > >   should not happen. If a data access falls back to the backing device
> > >   due to a cache miss I would have expected to see almost the
> > >   performance of the backing device. Maybe this points to a
> > >   performance issue in SMQ -- spending too much time in policy code
> > >   before falling back to the backing device.
> > >
> > >I've tried to figure out what actually happened in SMQ code in these
> > >cases - but eventually dismissed this. Next I want to check whether
> > >there might be a flaw in my test setup/dm-cache configuration.
> > 
> > Hi
> > 
> > The dm-cache SMQ/MQ is a 'slow moving' hot-spot cache.
> 
> Yep that is mentioned in some places in the source code with the
> hot-spot handling stuff.
> 
> > So before the block is 'promoted' to the cache - there needs to be a
> > reason for it - and it's not a plain single read.
> 
> It's not obvious to me when a block finally gets promoted. I had the
> impression that once the cache is filled with data, getting new data
> into the cache takes quite some time.
> 
> > So if the other cache promotes the block to the cache with a single
> > block access you may observe different performance.
> 
> Yep, that is what my measurements suggest.
> 
> > dm-cache is not targeted for 'quick' promoting of read blocks into a
> > cache - rather 'slow' moving of often used blocks.
> 
> If I completely abandon to use a set of test files (which defined
> hotspot blocks initially) and switch to a new set of test files this
> "slow" moving of often used (in the past) blocks might be the cause of
> the lower than expected (by me) performance in my tests. Would it be
> possible to tune this behaviour to allow quicker promotion if a user
> thinks he requires it for his workload?
> 
> > Unsure how that fits your testing environment and what you try to
> > actually test?
> 
> Worst results for spinning disks are random accesses. I've seen some
> dm-cache benchmark results (fio randread) that showed lower
> performance than the underlying backing device itself. That was the
> trigger for me to take a closer look at dm-cache and bcache and to do
> some performance measurements esp. with random read I/O pattern.
> 
> I've observed two oddities (from my point of view) and either they are
> due to setup errors, wrong expectations, or point to real issues that
> might be worth to be looked at or to be aware of.
> I think at least its worth to share my testing results.
> 
> > Regards
> > 
> > PS: 256K dm-cache blocks size is quite large - it really depends
> > upon workload - min supported size is 32K - lvm2 defaults to 64K...
> 
> I had chosen 512 as block size because documenation mentioned it.
> 
> I've kicked off a test with the minimum block size.
> Let's see whether that changes anything.

Are you using smq or mq cache policy?  Please use smq.  It is much
better about adapting to changing workloads.  mq has since been
converted over to an alias for smq (in Linux 4.6-rc1).

As for your randread fio test, there needs to be some amount of
redundant access.  randread on its own won't give you that.  fio does
have random_distribution (see zipf and pareto, afaik zipf being more
useful.. but I never actually got a compelling fio commandline together
that made use of random_distribution to simulate hotspots).

Anyway, as Zdenek effectively: said dm-cache isn't a writecache.  If you
need a writecache then bcache is the only option as of now.  Though
there is an emerging DM writecache target that has stalled but can be
revisited, see:
http://git.kernel.org/cgit/linux/kernel/git/snitzer/linux.git/log/?h=writecache

Re: dm-cache performance behaviour

[Andreas Herrmann]

On Tue, Apr 05, 2016 at 12:12:27PM -0400, Mike Snitzer wrote:
> On Tue, Apr 05 2016 at 10:05am -0400,
> Andreas Herrmann <aherrmann@suse.com> wrote:
> 
> > On Tue, Apr 05, 2016 at 10:36:12AM +0200, Zdenek Kabelac wrote:
> > > Dne 5.4.2016 v 09:12 Andreas Herrmann napsal(a):
> > > >Hi,
> > > >
> > > >I've recently looked at performance behaviour of dm-cache and bcache.
> > > >I've repeatedly observed very low performance with dm-cache in
> > > >different tests. (Similar tests with bcache showed no such oddities.)
> > > >
> > > >To rule out user errors that might have caused this, I shortly describe
> > > >what I've done and observed.
> > > >
> > > >- tested kernel version: 4.5.0
> > > >
> > > >- backing device: 1.5 TB spinning drive
> > > >
> > > >- caching device: 128 GB SSD (used for metadata and cache and size
> > > >   of metadata part calculated based on
> > > >   https://www.redhat.com/archives/dm-devel/2012-December/msg00046.html)
> > > >
> > > >- my test procedure consisted of a sequence of tests performing fio
> > > >   runs with different data sets, fio randread performance (bandwidth
> > > >   and IOPS) were compared, fio was invoked using something like
> > > >
> > > >   fio --directory=/cached-device --rw=randread --name=fio-1 \
> > > >     --size=50G --group_reporting --ioengine=libaio \
> > > >     --direct=1 --iodepth=1 --runtime=40 --numjobs=1
> > > >
> > > >   I've iterated over 10 runs for each of numjobs=1,2,3 and varied the
> > > >   name parameter to operate with different data sets.
> > > >
> > > >   This procedure implied that with 3 jobs the underlying data set for
> > > >   the test consisted of 3 files with 50G each which exceeds the size
> > > >   of the caching device.
> > > >
> > > >- Between some tests I've tried to empty the cache. For dm-cache I did
> > > >   this by unmounting the "compound" cache device, switching to cleaner
> > > >   target, zeroing metadata part of the caching device, recreating
> > > >   caching device and finally recreating the compound cache device
> > > >   (during this procedure I kept the backing device unmodified).
> > > >
> > > >   I used dmsetup status to check for success of this operation
> > > >   (checking for #used_cache_blocks).
> > > >   If there is an easier way to do this please let me know -- If it's
> > > >   documented I've missed it.
> > > >
> > > >- dm-cache parameters:
> > > >   * cache_mode: writeback
> > > >   * block size: 512 sectors
> > > >   * migration_threshold 2048 (default)
> > > >
> > > >I've observed two oddities:
> > > >
> > > >   (1) Only fio tests with the first data set created (and thus
> > > >   initially occupying the cache) showed decent performance
> > > >   results. Subsequent fio tests with another data set showed poor
> > > >   performance. I think this indicates that SMQ policy does not
> > > >   properly promote/demote data to/from caching device in my tests.
> > > >
> > > >   (2) I've seen results where performance was actually below "native"
> > > >   (w/o caching) performance of the backing device. I think that this
> > > >   should not happen. If a data access falls back to the backing device
> > > >   due to a cache miss I would have expected to see almost the
> > > >   performance of the backing device. Maybe this points to a
> > > >   performance issue in SMQ -- spending too much time in policy code
> > > >   before falling back to the backing device.
> > > >
> > > >I've tried to figure out what actually happened in SMQ code in these
> > > >cases - but eventually dismissed this. Next I want to check whether
> > > >there might be a flaw in my test setup/dm-cache configuration.
> > > 
> > > Hi
> > > 
> > > The dm-cache SMQ/MQ is a 'slow moving' hot-spot cache.
> > 
> > Yep that is mentioned in some places in the source code with the
> > hot-spot handling stuff.
> > 
> > > So before the block is 'promoted' to the cache - there needs to be a
> > > reason for it - and it's not a plain single read.
> > 
> > It's not obvious to me when a block finally gets promoted. I had the
> > impression that once the cache is filled with data, getting new data
> > into the cache takes quite some time.
> > 
> > > So if the other cache promotes the block to the cache with a single
> > > block access you may observe different performance.
> > 
> > Yep, that is what my measurements suggest.
> > 
> > > dm-cache is not targeted for 'quick' promoting of read blocks into a
> > > cache - rather 'slow' moving of often used blocks.
> > 
> > If I completely abandon to use a set of test files (which defined
> > hotspot blocks initially) and switch to a new set of test files this
> > "slow" moving of often used (in the past) blocks might be the cause of
> > the lower than expected (by me) performance in my tests. Would it be
> > possible to tune this behaviour to allow quicker promotion if a user
> > thinks he requires it for his workload?
> > 
> > > Unsure how that fits your testing environment and what you try to
> > > actually test?
> > 
> > Worst results for spinning disks are random accesses. I've seen some
> > dm-cache benchmark results (fio randread) that showed lower
> > performance than the underlying backing device itself. That was the
> > trigger for me to take a closer look at dm-cache and bcache and to do
> > some performance measurements esp. with random read I/O pattern.
> > 
> > I've observed two oddities (from my point of view) and either they are
> > due to setup errors, wrong expectations, or point to real issues that
> > might be worth to be looked at or to be aware of.
> > I think at least its worth to share my testing results.
> > 
> > > Regards
> > > 
> > > PS: 256K dm-cache blocks size is quite large - it really depends
> > > upon workload - min supported size is 32K - lvm2 defaults to 64K...
> > 
> > I had chosen 512 as block size because documenation mentioned it.
> > 
> > I've kicked off a test with the minimum block size.
> > Let's see whether that changes anything.
> 
> Are you using smq or mq cache policy?  Please use smq.  It is much
> better about adapting to changing workloads.  mq has since been
> converted over to an alias for smq (in Linux 4.6-rc1).

I've used smq.

> As for your randread fio test, there needs to be some amount of
> redundant access.  randread on its own won't give you that.

Yep.

> fio does have random_distribution (see zipf and pareto, afaik zipf
> being more useful.. but I never actually got a compelling fio
> commandline together that made use of random_distribution to
> simulate hotspots).

Thanks for the hint. (So far I haven't modified fio's
random_distribution option.)

Out of curiosity: what do you use for performance tests of dm-cache
(e.g. to track regressions) to simulate hot-spots -- some private
scripts?

> Anyway, as Zdenek effectively: said dm-cache isn't a writecache.  If you
> need a writecache then bcache is the only option as of now.  Though
> there is an emerging DM writecache target that has stalled but can be
> revisited, see:
> http://git.kernel.org/cgit/linux/kernel/git/snitzer/linux.git/log/?h=writecache


Thanks,

Andreas

Re: dm-cache performance behaviour

[Kent Overstreet]

On Tue, Apr 05, 2016 at 12:12:27PM -0400, Mike Snitzer wrote:
> Anyway, as Zdenek effectively: said dm-cache isn't a writecache.  If you
> need a writecache then bcache is the only option as of now.  Though
> there is an emerging DM writecache target that has stalled but can be
> revisited, see:
> http://git.kernel.org/cgit/linux/kernel/git/snitzer/linux.git/log/?h=writecache

bcache is still more than that, it's a combined read/writeback cache :) much
like how today we only have one pagecache for both read caching and write
caching. Main trick is you need a pretty sophisticated btree to pull that off
with good performance.

Re: dm-cache performance behaviour

[Eric Wheeler]

On Tue, 5 Apr 2016, Andreas Herrmann wrote:
> test system, software:
> - Intel(R) Core(TM) i7-3770S CPU @ 3.10GHz
> - 32 GB DDR3 (1333 MHz) RAM
> - Ubuntu 14.04 LTS
> - Linux kernel 4.5.0
> - Backing device: SAMSUNG HD155UI (1.5 TB), entire disk used
> - Caching device: SAMSUNG SSD 830 (128 GB), entire disk used
>   (in case of dm-cache used for both metadata cache data)
> - fio (flexible I/O tester) as of 2.2.9-37-g0e1c4
> - ext4 file system on compound cache device
> - CFQ I/O scheduler for caching and backing device
> 
> dmcache:
> - writeback cache mode
> - SMQ cache policy
> - cache block size: 512 sectors, resulting in cache blocks of size
>   256KB

You might consider tweaking {read,write}_promote_adjustment with 
	dmsetup message /dev/mapper/foo 0 read_promote_adjustment X 
or at initial table creation time to shorten cache heat time.

You could use this with LVM tools:
  lvchange --cachesettings 'read_promote_adjustment = 1' vg/cachedvol

See descriptions here:
  
https://www.kernel.org/doc/Documentation/device-mapper/cache-policies.txt

When we used dm-cache I would set the promote values to 1 (or 0?) to heat 
up the cache and then increment them after the cache filled.

> bcache:
> - writeback cache mode
> - LRU cache policy

bcache defaults to 512k bucket size iirc, you'll want to specify -b 256k 
on your make-bcache call for an apples-to-apples comparison.  FWIW, I like 
the 64k size best for our workload.

--
Eric Wheeler


> test procedure:
> - Create compound cache device
> - Test A: fio test using data set 1 (using empty cache
>   device, it includes creation of data set 1)
> - Test B: fio test using data set 1 for the second
>   successive time
> - Test C: fio test using data set 2 (it includes creation
>   of data set 2)
> - Test D: fio test using data set 2 for the second
>   successive time
> - Test E: fio test using data set 2 for the third
>   successive time
> - Test F: fio test using data set 3 (it includes creation
>   of data set 3)
> - Clear cache (remove and recreate caching device, re-attach it to
>   compound cache device)
> - Test G: fio test using data set 1 (using empty cache)
> - Copy data set 1 into temporary directory on test disk to clobber
>   cache
> - Test H: fio test using data set 1 (after cache was
>   clobbered with data from data set 1)
> - Test I: fio test using data set 2 (after cache was
>   clobbered and test H)
> 
> fio options and arguments:
> -directory=<test dir on cached device>
> -name=<file-name-prefix>
> -rw=randread 
> -size=50G
> -group_reporting
> -ioengine=libaio
> -direct=1
> -iodepth=1
> -runtime=40
> -numjobs=<1,2,3>
> 
> -> block size 4k
> -> each of above tests did 10 iterations for 1 job, 10 iterations for
>    2 jobs, 10 iterations fo 3 jobs using the same data set.
> -> one data set for one test consists of 3 data files (created by
>    fio), each having size 50G, thus all 3 files exceed size of caching
>    disk
> 
> Following 3 tables contain typical bandwidth measurements (in KB/s) of
> one run of above test procedure. As reference "native" (ie. w/o
> storage caching) performance for both SSD and HD are provided in all
> tables. This native performance was gathered with test A from the test
> procedure. In case of SSD --size parameter was set to 40G otherwise
> all 3 files would not have fit on the device.
> 
> First table shows results for test A, B, C for both dm-cache and
> bcache.  Remaining tests E-I are shown in second table for dm-cache
> and in third table for bcache.
> 
>     i = iteration, n = numjobs, shown is bandwidth in KB/s
> 
>        | native results|        dm-cache       |       bcache
>  i | n |1.5TB HD|  SSD | test A| test B| test C| test A| test B| test C 
> -----------------------------------------------------------------------
>  1 | 1 | 353.06 | 20568| 1154.7| 1819.6| 394.79|  351.6|  6111 |  377.2 
>  2 | 1 | 348.25 | 20571| 1265.2| 1896.6| 395.46|  701.9|  6393 |  748.8 
>  3 | 1 | 352.36 | 20573| 1298.2| 1917.1| 395.05| 1047.3|  6681 | 1120.4 
>  4 | 1 | 349.60 | 20542| 1494.5| 1919.2| 394.54| 1389.3|  6962 | 1483.4 
>  5 | 1 | 353.02 | 20554| 1703.6| 1921.1| 394.52| 1724.7|  7247 | 1841.7 
>  6 | 1 | 352.04 | 20564| 1745.1| 1938.2| 381.38| 2060.1|  7517 | 2200.2 
>  7 | 1 | 349.19 | 20585| 1929.5| 1957.6| 399.18| 2383.6|  7805 | 2556.5 
>  8 | 1 | 350.14 | 20584| 2047.6| 1935.6| 406.40| 2712.5|  8082 | 2908.5 
>  9 | 1 | 348.25 | 20573| 2231.3| 1951.2| 407.63| 3034.3|  8350 | 3260.5 
> 10 | 1 | 349.99 | 20579| 2336.2| 2001.7| 403.35| 3355.5|  8622 | 3612.2 
> 11 | 2 | 324.67 | 38590| 2904.6| 2643.1| 369.60| 3694.8| 10769 | 3991.5 
> 12 | 2 | 327.30 | 38466| 2865.3| 2601.1| 373.93| 4033.4| 11085 | 4369.1 
> 13 | 2 | 329.87 | 38550| 2882.2| 2594.9| 382.51| 4371.9| 11395 | 4743.9 
> 14 | 2 | 328.70 | 38459| 2893.3| 2614.1| 386.55| 4708.9| 11717 | 5118.6 
> 15 | 2 | 324.66 | 38468| 2918.2| 2641.8| 393.40| 5038.3| 12033 | 5491.8 
> 16 | 2 | 324.78 | 38514| 2907.8| 2605.4| 400.04| 5379.1| 12336 | 5861.2 
> 17 | 2 | 322.55 | 38519| 2923.5| 2684.6| 405.56| 5704.7| 12646 | 6227.9 
> 18 | 2 | 325.76 | 38519| 2927.1| 2689.4| 408.50| 6034.9| 12955 | 6585.4 
> 19 | 2 | 323.51 | 38482| 3005.5| 2662.3| 411.20| 6360.4| 13263 | 6942.3 
> 20 | 2 | 327.49 | 38509| 3016.3| 2714.3| 420.88| 6684.7| 13564 | 7306.2 
> 21 | 3 | 337.04 | 56139| 2549.8| 2512.5| 430.07| 5205.9| 12577 | 5659.6 
> 22 | 3 | 338.73 | 56210| 2462.2| 2459.1| 428.68| 5542.3| 12899 | 6040.3 
> 23 | 3 | 339.08 | 56020| 2442.8| 2519.6| 430.82| 5893.2| 13182 | 6417.4 
> 24 | 3 | 340.26 | 56142| 2360.3| 2526.4| 432.77| 6230.7| 13546 | 6796.2 
> 25 | 3 | 333.83 | 56081| 2303.1| 2508.4| 434.03| 6571.1| 13865 | 7170.2 
> 26 | 3 | 338.32 | 56195| 2344.8| 2508.5| 431.66| 6909.8| 14182 | 7540.5 
> 27 | 3 | 339.73 | 56165| 2351.4| 2540.4| 434.86| 7227.1| 14494 | 7915.4 
> 28 | 3 | 331.52 | 56216| 2354.5| 2489.7| 430.30| 7570.4| 14812 | 8279.1 
> 29 | 3 | 334.47 | 56083| 2339.9| 2490.4| 441.79| 7908.7| 15129 | 8640.6 
> 30 | 3 | 347.99 | 56089| 2418.5| 2490.3| 446.06| 8241.4| 15443 | 9008.8 
> 
>        | native results|                      dm-cache
>  i | n |1.5TB HD|  SSD | test D| test E| test F| test G| test H | test I 
> ------------------------------------------------------------------------
>  1 | 1 | 353.06 | 20568| 461.6 | 521.6 | 390.2 |  328.8| 1098.7 | 222.5
>  2 | 1 | 348.25 | 20571| 466.6 | 524.2 | 389.2 |  184.6| 1093.5 | 229.1
>  3 | 1 | 352.36 | 20573| 485.7 | 521.9 | 389.0 |  480.0| 1095.8 | 225.7
>  4 | 1 | 349.60 | 20542| 497.3 | 525.1 | 389.4 |  579.3| 1096.7 | 230.1
>  5 | 1 | 353.02 | 20554| 501.5 | 527.4 | 388.9 |  649.5| 1100.6 | 226.9
>  6 | 1 | 352.04 | 20564| 504.0 | 528.8 | 369.9 |  714.7| 1095.2 | 224.9
>  7 | 1 | 349.19 | 20585| 507.2 | 530.8 | 398.1 |  766.9| 1095.2 | 225.9
>  8 | 1 | 350.14 | 20584| 497.9 | 530.4 | 402.3 |  817.1| 1096.6 | 226.5
>  9 | 1 | 348.25 | 20573| 516.6 | 535.7 | 399.5 |  847.7| 1098.2 | 226.8
> 10 | 1 | 349.99 | 20579| 518.1 | 534.2 | 397.1 |  877.8| 1098.7 | 232.6
> 11 | 2 | 324.67 | 38590| 449.8 | 472.9 | 373.4 |  888.7| 1201.1 | 261.1
> 12 | 2 | 327.30 | 38466| 447.3 | 473.0 | 377.1 |  898.9| 1205.7 | 265.9
> 13 | 2 | 329.87 | 38550| 451.9 | 473.9 | 386.3 |  932.4| 1204.1 | 264.9
> 14 | 2 | 328.70 | 38459| 451.0 | 475.0 | 394.9 |  961.1| 1205.3 | 265.7
> 15 | 2 | 324.66 | 38468| 456.5 | 476.6 | 401.2 |  993.7| 1210.7 | 268.6
> 16 | 2 | 324.78 | 38514| 459.4 | 487.0 | 409.0 | 1034.9| 1206.2 | 269.7
> 17 | 2 | 322.55 | 38519| 456.1 | 498.9 | 411.7 | 1062.3| 1204.9 | 273.3
> 18 | 2 | 325.76 | 38519| 458.3 | 500.2 | 422.0 | 1081.6| 1204.6 | 280.5
> 19 | 2 | 323.51 | 38482| 457.1 | 498.3 | 430.5 | 1100.3| 1207.5 | 279.0
> 20 | 2 | 327.49 | 38509| 455.8 | 500.4 | 428.7 | 1106.8| 1203.6 | 285.7
> 21 | 3 | 337.04 | 56139| 462.6 | 481.5 | 439.0 | 1060.9| 1250.2 | 312.7
> 22 | 3 | 338.73 | 56210| 462.9 | 483.3 | 437.2 | 1095.7| 1253.9 | 306.8
> 23 | 3 | 339.08 | 56020| 469.5 | 489.1 | 437.8 | 1163.7| 1248.9 | 304.7
> 24 | 3 | 340.26 | 56142| 462.1 | 487.1 | 440.1 | 1189.9| 1252.1 | 303.0
> 25 | 3 | 333.83 | 56081| 465.1 | 490.8 | 436.9 | 1217.7| 1248.6 | 302.6
> 26 | 3 | 338.32 | 56195| 465.1 | 489.8 | 433.4 | 1234.8| 1248.7 | 308.3
> 27 | 3 | 339.73 | 56165| 464.6 | 490.7 | 437.0 | 1256.1| 1253.3 | 305.4
> 28 | 3 | 331.52 | 56216| 463.2 | 496.7 | 435.6 | 1281.8| 1250.5 | 307.1
> 29 | 3 | 334.47 | 56083| 469.4 | 493.3 | 434.6 | 1302.6| 1249.4 | 308.9
> 30 | 3 | 347.99 | 56089| 464.0 | 501.4 | 442.5 | 1321.5| 1250.9 | 305.4
> 
>        | native results|                      bcache
>  i | n |1.5TB HD|  SSD | test D| test E| test F| test G| test H| test I 
> -----------------------------------------------------------------------
>  1 | 1 | 353.06 | 20568|  6652 | 11905 |  390.3|  389.9|  6995 |  376.8
>  2 | 1 | 348.25 | 20571|  6964 | 12182 |  776.5|  779.3|  7339 |  750.3
>  3 | 1 | 352.36 | 20573|  7275 | 12451 | 1161.2| 1164.1|  7677 | 1120.2
>  4 | 1 | 349.60 | 20542|  7585 | 12723 | 1541.6| 1547.2|  8020 | 1487.1
>  5 | 1 | 353.02 | 20554|  7889 | 12989 | 1916.4| 1924.4|  8357 | 1850.7
>  6 | 1 | 352.04 | 20564|  8192 | 13255 | 2287.6| 2301.6|  8688 | 2210.6
>  7 | 1 | 349.19 | 20585|  8496 | 13520 | 2658.5| 2677.5|  9023 | 2568.8
>  8 | 1 | 350.14 | 20584|  8795 | 13783 | 3029.3| 3053.7|  9359 | 2922.7
>  9 | 1 | 348.25 | 20573|  9093 | 14045 | 3397.9| 3423.3|  9687 | 3278.7
> 10 | 1 | 349.99 | 20579|  9392 | 14305 | 3764.5| 3796.8| 10014 | 3632.8
> 11 | 2 | 324.67 | 38590| 11773 | 18720 | 4135.8| 4177.6| 12288 | 4015.5
> 12 | 2 | 327.30 | 38466| 12128 | 19046 | 4499.6| 4554.4| 12631 | 4394.3
> 13 | 2 | 329.87 | 38550| 12480 | 19369 | 4859.4| 4928.1| 12975 | 4770.4
> 14 | 2 | 328.70 | 38459| 12822 | 19696 | 5218.2| 5298.8| 13315 | 5142.5
> 15 | 2 | 324.66 | 38468| 13170 | 20014 | 5575.6| 5667.6| 13651 | 5510.6
> 16 | 2 | 324.78 | 38514| 13510 | 20333 | 5928.9| 6031.7| 13994 | 5876.4
> 17 | 2 | 322.55 | 38519| 13848 | 20650 | 6279.7| 6391.6| 14321 | 6239.1
> 18 | 2 | 325.76 | 38519| 14187 | 20960 | 6626.7| 6745.1| 14645 | 6595.5
> 19 | 2 | 323.51 | 38482| 14522 | 21271 | 6972.2| 7093.9| 14964 | 6954.6
> 20 | 2 | 327.49 | 38509| 14850 | 21575 | 7317.6| 7452.5| 15280 | 7305.4
> 21 | 3 | 337.04 | 56139| 13800 | 21069 | 5811.8| 5914.9| 14496 | 5732.5
> 22 | 3 | 338.73 | 56210| 14158 | 21402 | 6199.7| 6305.7| 14858 | 6113.1
> 23 | 3 | 339.08 | 56020| 14510 | 21727 | 6585.1| 6695.2| 15214 | 6494.2
> 24 | 3 | 340.26 | 56142| 14860 | 22053 | 6972.6| 7079.6| 15570 | 6870.9
> 25 | 3 | 333.83 | 56081| 15206 | 22373 | 7356.4| 7464.9| 15929 | 7244.4
> 26 | 3 | 338.32 | 56195| 15545 | 22691 | 7735.1| 7845.5| 16276 | 7616.8
> 27 | 3 | 339.73 | 56165| 15890 | 23017 | 8113.1| 8225.6| 16627 | 7988.6
> 28 | 3 | 331.52 | 56216| 16230 | 23323 | 8496.4| 8600.7| 16973 | 8353.7
> 29 | 3 | 334.47 | 56083| 16562 | 23636 | 8864.9| 8973.5| 17323 | 8722.5
> 30 | 3 | 347.99 | 56089| 16899 | 23945 | 9235.8| 9343.1| 17663 | 9081.4
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: dm-cache performance behaviour

[Vasiliy Tolstov]

2016-04-07 4:24 GMT+03:00 Eric Wheeler <bcache@lists.ewheeler.net>:
> You might consider tweaking {read,write}_promote_adjustment with
>         dmsetup message /dev/mapper/foo 0 read_promote_adjustment X
> or at initial table creation time to shorten cache heat time.
>
> You could use this with LVM tools:
>   lvchange --cachesettings 'read_promote_adjustment = 1' vg/cachedvol
>
> See descriptions here:
>
> https://www.kernel.org/doc/Documentation/device-mapper/cache-policies.txt
>
> When we used dm-cache I would set the promote values to 1 (or 0?) to heat
> up the cache and then increment them after the cache filled.
>
>> bcache:
>> - writeback cache mode
>> - LRU cache policy
>
> bcache defaults to 512k bucket size iirc, you'll want to specify -b 256k
> on your make-bcache call for an apples-to-apples comparison.  FWIW, I like
> the 64k size best for our workload.
>
> --
> Eric Wheeler


Does somebody can test bcache vs dm-writeboost vs dm-cache on latest
stable kernels ?


-- 
Vasiliy Tolstov,
e-mail: v.tolstov@selfip.ru