dm-cache with zero hit rate

dm-cache with zero hit rate

[Steinar H. Gunderson]

Hi,

I recently set up dm-cache below my main LVM, so that /dev/md2 (a RAID-0 of
two SSDs) has an LVM with /dev/mapper/cache-blocks and
/dev/mapper/cache-metadata, which then back /dev/md1 (a RAID-6 of rotating
disks), where my main LVM, including the root device etc., lies.

After a fair amount of fighting with udev and initramfs-tools, plus upping
the block size to 2048 since 512 complained about not enough RAM (on a 24GB
machine!), this seems to boot up and work, but I seem to get absolutely zero
cache hits.

dmsetup table for the device:

  cache: 0 23440891904 cache 254:0 254:1 9:1 2048 1 writeback default 4 random_threshold 8 sequential_threshold 512

dmsetup status for it:

  cache: 0 23440891904 cache 913/8192 0 170976 0 9614 0 0 0 0 0 2 migration_threshold 2048 4 random_threshold 8 sequential_threshold 512

Note in particular 0/170976 cached reads and 0/9614 cached writes.

Is this normal? Is there any good reason why an LVM on top of a dm-cache
device would not be supported? Or do I just need to wait a few more
hours/weeks/days until it starts being aggressive?

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: dm-cache with zero hit rate

[Mike Snitzer]

On Wed, Jul 31 2013 at  2:50pm -0400,
Steinar H. Gunderson <sgunderson@bigfoot.com> wrote:

> Hi,
> 
> I recently set up dm-cache below my main LVM, so that /dev/md2 (a RAID-0 of
> two SSDs) has an LVM with /dev/mapper/cache-blocks and
> /dev/mapper/cache-metadata, which then back /dev/md1 (a RAID-6 of rotating
> disks), where my main LVM, including the root device etc., lies.
> 
> After a fair amount of fighting with udev and initramfs-tools, plus upping
> the block size to 2048 since 512 complained about not enough RAM (on a 24GB
> machine!), this seems to boot up and work, but I seem to get absolutely zero
> cache hits.

Kudos to you for hacking that to work.

> dmsetup table for the device:
> 
>   cache: 0 23440891904 cache 254:0 254:1 9:1 2048 1 writeback default 4 random_threshold 8 sequential_threshold 512
> 
> dmsetup status for it:
> 
>   cache: 0 23440891904 cache 913/8192 0 170976 0 9614 0 0 0 0 0 2 migration_threshold 2048 4 random_threshold 8 sequential_threshold 512
> 
> Note in particular 0/170976 cached reads and 0/9614 cached writes.

Yeah, no promotions, translates to all misses.
 
> Is this normal? Is there any good reason why an LVM on top of a dm-cache
> device would not be supported? Or do I just need to wait a few more
> hours/weeks/days until it starts being aggressive?

Please see the recent "dm-cache warming thread":
https://www.redhat.com/archives/dm-devel/2013-July/msg00133.html

That isn't to say we cannot take steps to be more aggressive; but we'll
need more context for what you're doing.

A normal system boot is likely predominantly read once, and/or (as Joe
pointed out) the page cache could be masking subsequent reads.

If you're doing write heavy workloads are they being elided by the
sequential_threshold?

Try a git checkout, and switch branches a few times (e.g. checkout v3.1,
then v3.8, then v3.2, then v3.9, then v3.1, etc).

Re: dm-cache with zero hit rate

[Steinar H. Gunderson]

On Wed, Jul 31, 2013 at 04:25:22PM -0400, Mike Snitzer wrote:
>> After a fair amount of fighting with udev and initramfs-tools, plus upping
>> the block size to 2048 since 512 complained about not enough RAM (on a 24GB
>> machine!), this seems to boot up and work, but I seem to get absolutely zero
>> cache hits.
> Kudos to you for hacking that to work.

It's actually not that hard; once things stabilize and I know this is
actually the solution I want to have, I plan to sit down and document it, and
it's not going to be a long document.

> Please see the recent "dm-cache warming thread":
> https://www.redhat.com/archives/dm-devel/2013-July/msg00133.html
> 
> That isn't to say we cannot take steps to be more aggressive; but we'll
> need more context for what you're doing.
> 
> A normal system boot is likely predominantly read once, and/or (as Joe
> pointed out) the page cache could be masking subsequent reads.
> 
> If you're doing write heavy workloads are they being elided by the
> sequential_threshold?
> 
> Try a git checkout, and switch branches a few times (e.g. checkout v3.1,
> then v3.8, then v3.2, then v3.9, then v3.1, etc).

OK, so context: This is my personal do-everything server, so it does all
kinds of things; web serving (both pages, scripts and mirrors), large files,
database work, email (local, not IMAP), backup through rsync, etc.. Although
it has 24GB of RAM, the disk set is 8TB (and the cache disks ~450GB), so one
would think it found _something_ relevant to cache. In particular, my
Maildirs would seem like a prime candidate :-) There's I/O going on all the
time, and I'm a bit surprised that not even any writes are cached. 

I did the git checkout as you asked, 10–15 times with different version
numbers, and although the git command seemed to create I/O (ie., the page
cache wasn't gobbling it all up, although it was mostly write I/O), but it
still didn't put anything in the cache.

I see the thread you're referring to says “too low”, but I really have zero,
even after almost 40 hours of normal operation:

  cache: 0 23440891904 cache 913/8192 0 6888144 0 1653677 0 0 0 0 0 2 migration_threshold 2048 4 random_threshold 8 sequential_threshold 512

Could the large block size be related to this?

/* Steinar */
-- 
Homepage: http://www.sesse.net/

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: dm-cache with zero hit rate

[Mike Snitzer]

On Fri, Aug 02 2013 at  4:49am -0400,
Steinar H. Gunderson <sgunderson@bigfoot.com> wrote:

> On Wed, Jul 31, 2013 at 04:25:22PM -0400, Mike Snitzer wrote:
> >> After a fair amount of fighting with udev and initramfs-tools, plus upping
> >> the block size to 2048 since 512 complained about not enough RAM (on a 24GB
> >> machine!), this seems to boot up and work, but I seem to get absolutely zero
> >> cache hits.
> > Kudos to you for hacking that to work.
> 
> It's actually not that hard; once things stabilize and I know this is
> actually the solution I want to have, I plan to sit down and document it, and
> it's not going to be a long document.
> 
> > Please see the recent "dm-cache warming thread":
> > https://www.redhat.com/archives/dm-devel/2013-July/msg00133.html
> > 
> > That isn't to say we cannot take steps to be more aggressive; but we'll
> > need more context for what you're doing.
> > 
> > A normal system boot is likely predominantly read once, and/or (as Joe
> > pointed out) the page cache could be masking subsequent reads.
> > 
> > If you're doing write heavy workloads are they being elided by the
> > sequential_threshold?
> > 
> > Try a git checkout, and switch branches a few times (e.g. checkout v3.1,
> > then v3.8, then v3.2, then v3.9, then v3.1, etc).
> 
> OK, so context: This is my personal do-everything server, so it does all
> kinds of things; web serving (both pages, scripts and mirrors), large files,
> database work, email (local, not IMAP), backup through rsync, etc.. Although
> it has 24GB of RAM, the disk set is 8TB (and the cache disks ~450GB), so one
> would think it found _something_ relevant to cache. In particular, my
> Maildirs would seem like a prime candidate :-) There's I/O going on all the
> time, and I'm a bit surprised that not even any writes are cached. 

Yes, it is surprising.  Curious to know if the promotions aren't
happening due to the IO scheduler somehow merging all your random small
IO.  We don't yet have a descrete counter to show the number of
migrations that were skipped due to sequential_threshold but that is
something we can add.

But you can effectively disable the sequential_threshold by setting it
really high, e.g.:

dmsetup message cache 0 sequential_threshold 16384

> I did the git checkout as you asked, 10–15 times with different version
> numbers, and although the git command seemed to create I/O (ie., the page
> cache wasn't gobbling it all up, although it was mostly write I/O), but it
> still didn't put anything in the cache.

Please write a file that is smaller than your specified
sequential_threshold, and then read it numerous times via direct IO,
e.g.:

dd if=<your file> of=/dev/null iflag=direct bs=16K
 
> I see the thread you're referring to says “too low”, but I really have zero,
> even after almost 40 hours of normal operation:
> 
>   cache: 0 23440891904 cache 913/8192 0 6888144 0 1653677 0 0 0 0 0 2 migration_threshold 2048 4 random_threshold 8 sequential_threshold 512
> 
> Could the large block size be related to this?

No, a large cache block size just brings a lot more into the cache when
something was deemed worthy of promotion.

We need to better understand/diagnose _why_ nothing is getting promoted,
to that end Heinz and Joe will collaborate on the development of a
"cache_dump" utility to allow visibility into the mq policy's hit counts
and hints associated with each block.

But in the meantime, cranking up sequential_threshold may help.

Mike

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: dm-cache with zero hit rate

[Steinar H. Gunderson]

On Fri, Aug 02, 2013 at 09:23:44AM -0400, Mike Snitzer wrote:
> Yes, it is surprising.  Curious to know if the promotions aren't
> happening due to the IO scheduler somehow merging all your random small
> IO.  We don't yet have a descrete counter to show the number of
> migrations that were skipped due to sequential_threshold but that is
> something we can add.
> 
> But you can effectively disable the sequential_threshold by setting it
> really high, e.g.:
> 
> dmsetup message cache 0 sequential_threshold 16384

I tried this, and it still doesn't appear to promote anything at all:

cache: 0 23440891904 cache 913/8192 0 7021239 0 2049048 0 0 0 0 0 2 migration_threshold 2048 4 random_threshold 8 sequential_threshold 16384

It's only been running for a few minutes, though.

FWIW, earlier I ran it on only one single partition, and then it worked.
So it's not like my kernel is completely broken, at least.

> Please write a file that is smaller than your specified
> sequential_threshold, and then read it numerous times via direct IO,
> e.g.:
> 
> dd if=<your file> of=/dev/null iflag=direct bs=16K

I did, with a 16 kB file (that should certainly be small enough, right?),
executing the dd command 10000 times. Still nothing cached.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: dm-cache with zero hit rate

[Amy Rupp]

[-- Attachment #1.1: Type: text/plain, Size: 1929 bytes --]

Is it possible to use different input files (unique files, anyway) of the
same size, maybe even on different file systems if available?  The creation
of these could be scripted along with a dd - don't remove the file
immediately after the dd, though.  It might be even better if you could
issue multiple reads in parallel, though that will be hard to do if it's a
really small file.


On Fri, Aug 2, 2013 at 8:58 AM, Steinar H. Gunderson <sgunderson@bigfoot.com
> wrote:

> On Fri, Aug 02, 2013 at 09:23:44AM -0400, Mike Snitzer wrote:
> > Yes, it is surprising.  Curious to know if the promotions aren't
> > happening due to the IO scheduler somehow merging all your random small
> > IO.  We don't yet have a descrete counter to show the number of
> > migrations that were skipped due to sequential_threshold but that is
> > something we can add.
> >
> > But you can effectively disable the sequential_threshold by setting it
> > really high, e.g.:
> >
> > dmsetup message cache 0 sequential_threshold 16384
>
> I tried this, and it still doesn't appear to promote anything at all:
>
> cache: 0 23440891904 cache 913/8192 0 7021239 0 2049048 0 0 0 0 0 2
> migration_threshold 2048 4 random_threshold 8 sequential_threshold 16384
>
> It's only been running for a few minutes, though.
>
> FWIW, earlier I ran it on only one single partition, and then it worked.
> So it's not like my kernel is completely broken, at least.
>
> > Please write a file that is smaller than your specified
> > sequential_threshold, and then read it numerous times via direct IO,
> > e.g.:
> >
> > dd if=<your file> of=/dev/null iflag=direct bs=16K
>
> I did, with a 16 kB file (that should certainly be small enough, right?),
> executing the dd command 10000 times. Still nothing cached.
>
> /* Steinar */
> --
> Homepage: http://www.sesse.net/
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

[-- Attachment #1.2: Type: text/html, Size: 2722 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]