dm-cache policy mq: promotion very picky?

dm-cache policy mq: promotion very picky?

[Pierre Beck]

Hi,

experimenting with dm-cache, I came across the following issue:

Having a rather large SSD available, I wanted to promote sequential I/O 
just like random I/O. Recurring sequential I/O is just as bad in a 
multi-user scenario, especially with unpromoted random I/O mixed in. So 
my dmsetup line looks like this:

# dmsetup create hybridStorage --table "0 7771064048 cache 
/dev/disk/by-id/xxx-part4 /dev/disk/by-id/yyy-part3 /dev/md/zzz 512 1 
writethrough mq 4 sequential_threshold 4294967295 random_threshold 0";

(I would appreciate a parameter to just turn random I/O detection off, btw.)

On top of "hybridStorage" an ext4 with external journal. Then, I repeat:

# echo 1 > /proc/sys/vm/drop_caches; pv smallwrite.0 > /dev/null

And watch a dstat with all relevant partitions. The 70 MB file 
"smallwrite.0" is always being read from HDD and never gets promoted. 
Only a few blocks got promoted at some point in the past.

# echo 1 > /proc/sys/vm/drop_caches; ioping -R smallwrite.0

This doesn't promote blocks, either, although it is random I/O.

# echo 1 > /proc/sys/vm/drop_caches; ioping -s 1m -R smallwrite.0

This does promote a few blocks every iteration.

# dd if=/dev/zero of=testfile bs=4096 count=100000; sync

This file was promoted halfway. Again, just piping it with pv doesn't 
promote any blocks. Piping with "cat" does promote a few blocks every 
iteration.

All in all, dm-cache policy mq seems to be very hesitant (way too 
hesitant for my taste) promoting blocks, even with sequential I/O allowed.

Maybe I'm misunderstanding how the heuristics work - what makes mq pick 
a block for promotion?

Regards,

Pierre Beck

Re: dm-cache policy mq: promotion very picky?

[Joe Thornber]

On Mon, May 13, 2013 at 11:23:04PM +0200, Pierre Beck wrote:
> Maybe I'm misunderstanding how the heuristics work - what makes mq
> pick a block for promotion?

Hi Pierre,

The mq policy is indeed very picky.  The first thing I'd suggest is
you pick up my latest patches if you haven't already:

http://device-mapper.org/patches/3.10/

You just need the dm-cache*.patch ones, apply in the order given in
the series file.

At it's heart mq is v. simple.  It divides time up into slices called
'ticks'.  Hit counts are maintained for all blocks in the cache, and a
group of blocks that we're thinking of promoting.  A cache/origin
block can only be hit once in each tick.  When a candidate block's hit
count goes over a threshold it gets promoted.

The threshold depends on the hit counts of the entries in the cache.

At the moment hit counts don't degrade with time, that's something I
need to add to allow the cache to be more responsive to changing work
loads.


If you start with a discarded device (eg, you've run mkfs).  Then the
cache should fill up v. quickly with my latest patches.  Perhaps it
would be worth verifying this before you start investigating the
effects of sequential io?

- Joe

Re: dm-cache policy mq: promotion very picky?

[Darrick J. Wong]

On Tue, May 14, 2013 at 02:44:53PM +0100, Joe Thornber wrote:
> On Mon, May 13, 2013 at 11:23:04PM +0200, Pierre Beck wrote:
> > Maybe I'm misunderstanding how the heuristics work - what makes mq
> > pick a block for promotion?
> 
> Hi Pierre,
> 
> The mq policy is indeed very picky.  The first thing I'd suggest is
> you pick up my latest patches if you haven't already:
> 
> http://device-mapper.org/patches/3.10/

Hm, it looks like all but two of the dmcache patches are in.  The two patches
to mq aren't (yet) there.

> You just need the dm-cache*.patch ones, apply in the order given in
> the series file.
> 
> At it's heart mq is v. simple.  It divides time up into slices called
> 'ticks'.  Hit counts are maintained for all blocks in the cache, and a
> group of blocks that we're thinking of promoting.  A cache/origin
> block can only be hit once in each tick.  When a candidate block's hit
> count goes over a threshold it gets promoted.
> 
> The threshold depends on the hit counts of the entries in the cache.
> 
> At the moment hit counts don't degrade with time, that's something I
> need to add to allow the cache to be more responsive to changing work
> loads.
> 
> 
> If you start with a discarded device (eg, you've run mkfs).  Then the
> cache should fill up v. quickly with my latest patches.  Perhaps it
> would be worth verifying this before you start investigating the
> effects of sequential io?

I'm also curious about what happens if, say, you have a discard-enabled
zerofree that can "discard" just the unallocated blocks.  That probably only
helps if you can discard pieces big enough to fit one of dmcache's discard
regions, right?

--D
> 
> - Joe
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: dm-cache policy mq: promotion very picky?

[Joe Thornber]

On Tue, May 14, 2013 at 11:16:55AM -0700, Darrick J. Wong wrote:
> On Tue, May 14, 2013 at 02:44:53PM +0100, Joe Thornber wrote:
> > On Mon, May 13, 2013 at 11:23:04PM +0200, Pierre Beck wrote:
> > > Maybe I'm misunderstanding how the heuristics work - what makes mq
> > > pick a block for promotion?
> > 
> > Hi Pierre,
> > 
> > The mq policy is indeed very picky.  The first thing I'd suggest is
> > you pick up my latest patches if you haven't already:
> > 
> > http://device-mapper.org/patches/3.10/
> 
> Hm, it looks like all but two of the dmcache patches are in.  The two patches
> to mq aren't (yet) there.

y, unfortunately there's a fixme in there that frightened Alasdair,
and I don't have any free capacity to spend on dm-cache atm.

> > If you start with a discarded device (eg, you've run mkfs).  Then the
> > cache should fill up v. quickly with my latest patches.  Perhaps it
> > would be worth verifying this before you start investigating the
> > effects of sequential io?
> 
> I'm also curious about what happens if, say, you have a discard-enabled
> zerofree that can "discard" just the unallocated blocks.  That probably only
> helps if you can discard pieces big enough to fit one of dmcache's discard
> regions, right?

Yep.  But your fs will be trying to reduce fragmentation of free
space, so hopefully the pieces will be big enough.

- Joe