* dm-cache with zero hit rate @ 2013-07-31 18:50 Steinar H. Gunderson 2013-07-31 20:25 ` Mike Snitzer 0 siblings, 1 reply; 6+ messages in thread From: Steinar H. Gunderson @ 2013-07-31 18:50 UTC (permalink / raw) To: dm-devel Hi, I recently set up dm-cache below my main LVM, so that /dev/md2 (a RAID-0 of two SSDs) has an LVM with /dev/mapper/cache-blocks and /dev/mapper/cache-metadata, which then back /dev/md1 (a RAID-6 of rotating disks), where my main LVM, including the root device etc., lies. After a fair amount of fighting with udev and initramfs-tools, plus upping the block size to 2048 since 512 complained about not enough RAM (on a 24GB machine!), this seems to boot up and work, but I seem to get absolutely zero cache hits. dmsetup table for the device: cache: 0 23440891904 cache 254:0 254:1 9:1 2048 1 writeback default 4 random_threshold 8 sequential_threshold 512 dmsetup status for it: cache: 0 23440891904 cache 913/8192 0 170976 0 9614 0 0 0 0 0 2 migration_threshold 2048 4 random_threshold 8 sequential_threshold 512 Note in particular 0/170976 cached reads and 0/9614 cached writes. Is this normal? Is there any good reason why an LVM on top of a dm-cache device would not be supported? Or do I just need to wait a few more hours/weeks/days until it starts being aggressive? /* Steinar */ -- Homepage: http://www.sesse.net/ ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: dm-cache with zero hit rate 2013-07-31 18:50 dm-cache with zero hit rate Steinar H. Gunderson @ 2013-07-31 20:25 ` Mike Snitzer 2013-08-02 8:49 ` Steinar H. Gunderson 0 siblings, 1 reply; 6+ messages in thread From: Mike Snitzer @ 2013-07-31 20:25 UTC (permalink / raw) To: Steinar H. Gunderson; +Cc: Heinz Mauelshagen, dm-devel, ejt On Wed, Jul 31 2013 at 2:50pm -0400, Steinar H. Gunderson <sgunderson@bigfoot.com> wrote: > Hi, > > I recently set up dm-cache below my main LVM, so that /dev/md2 (a RAID-0 of > two SSDs) has an LVM with /dev/mapper/cache-blocks and > /dev/mapper/cache-metadata, which then back /dev/md1 (a RAID-6 of rotating > disks), where my main LVM, including the root device etc., lies. > > After a fair amount of fighting with udev and initramfs-tools, plus upping > the block size to 2048 since 512 complained about not enough RAM (on a 24GB > machine!), this seems to boot up and work, but I seem to get absolutely zero > cache hits. Kudos to you for hacking that to work. > dmsetup table for the device: > > cache: 0 23440891904 cache 254:0 254:1 9:1 2048 1 writeback default 4 random_threshold 8 sequential_threshold 512 > > dmsetup status for it: > > cache: 0 23440891904 cache 913/8192 0 170976 0 9614 0 0 0 0 0 2 migration_threshold 2048 4 random_threshold 8 sequential_threshold 512 > > Note in particular 0/170976 cached reads and 0/9614 cached writes. Yeah, no promotions, translates to all misses. > Is this normal? Is there any good reason why an LVM on top of a dm-cache > device would not be supported? Or do I just need to wait a few more > hours/weeks/days until it starts being aggressive? Please see the recent "dm-cache warming thread": https://www.redhat.com/archives/dm-devel/2013-July/msg00133.html That isn't to say we cannot take steps to be more aggressive; but we'll need more context for what you're doing. A normal system boot is likely predominantly read once, and/or (as Joe pointed out) the page cache could be masking subsequent reads. If you're doing write heavy workloads are they being elided by the sequential_threshold? Try a git checkout, and switch branches a few times (e.g. checkout v3.1, then v3.8, then v3.2, then v3.9, then v3.1, etc). ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: dm-cache with zero hit rate 2013-07-31 20:25 ` Mike Snitzer @ 2013-08-02 8:49 ` Steinar H. Gunderson 2013-08-02 13:23 ` Mike Snitzer 0 siblings, 1 reply; 6+ messages in thread From: Steinar H. Gunderson @ 2013-08-02 8:49 UTC (permalink / raw) To: Mike Snitzer; +Cc: Heinz Mauelshagen, dm-devel, ejt On Wed, Jul 31, 2013 at 04:25:22PM -0400, Mike Snitzer wrote: >> After a fair amount of fighting with udev and initramfs-tools, plus upping >> the block size to 2048 since 512 complained about not enough RAM (on a 24GB >> machine!), this seems to boot up and work, but I seem to get absolutely zero >> cache hits. > Kudos to you for hacking that to work. It's actually not that hard; once things stabilize and I know this is actually the solution I want to have, I plan to sit down and document it, and it's not going to be a long document. > Please see the recent "dm-cache warming thread": > https://www.redhat.com/archives/dm-devel/2013-July/msg00133.html > > That isn't to say we cannot take steps to be more aggressive; but we'll > need more context for what you're doing. > > A normal system boot is likely predominantly read once, and/or (as Joe > pointed out) the page cache could be masking subsequent reads. > > If you're doing write heavy workloads are they being elided by the > sequential_threshold? > > Try a git checkout, and switch branches a few times (e.g. checkout v3.1, > then v3.8, then v3.2, then v3.9, then v3.1, etc). OK, so context: This is my personal do-everything server, so it does all kinds of things; web serving (both pages, scripts and mirrors), large files, database work, email (local, not IMAP), backup through rsync, etc.. Although it has 24GB of RAM, the disk set is 8TB (and the cache disks ~450GB), so one would think it found _something_ relevant to cache. In particular, my Maildirs would seem like a prime candidate :-) There's I/O going on all the time, and I'm a bit surprised that not even any writes are cached. I did the git checkout as you asked, 10–15 times with different version numbers, and although the git command seemed to create I/O (ie., the page cache wasn't gobbling it all up, although it was mostly write I/O), but it still didn't put anything in the cache. I see the thread you're referring to says “too low”, but I really have zero, even after almost 40 hours of normal operation: cache: 0 23440891904 cache 913/8192 0 6888144 0 1653677 0 0 0 0 0 2 migration_threshold 2048 4 random_threshold 8 sequential_threshold 512 Could the large block size be related to this? /* Steinar */ -- Homepage: http://www.sesse.net/ -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: dm-cache with zero hit rate 2013-08-02 8:49 ` Steinar H. Gunderson @ 2013-08-02 13:23 ` Mike Snitzer 2013-08-02 13:58 ` Steinar H. Gunderson 0 siblings, 1 reply; 6+ messages in thread From: Mike Snitzer @ 2013-08-02 13:23 UTC (permalink / raw) To: Steinar H. Gunderson; +Cc: Heinz Mauelshagen, dm-devel, ejt On Fri, Aug 02 2013 at 4:49am -0400, Steinar H. Gunderson <sgunderson@bigfoot.com> wrote: > On Wed, Jul 31, 2013 at 04:25:22PM -0400, Mike Snitzer wrote: > >> After a fair amount of fighting with udev and initramfs-tools, plus upping > >> the block size to 2048 since 512 complained about not enough RAM (on a 24GB > >> machine!), this seems to boot up and work, but I seem to get absolutely zero > >> cache hits. > > Kudos to you for hacking that to work. > > It's actually not that hard; once things stabilize and I know this is > actually the solution I want to have, I plan to sit down and document it, and > it's not going to be a long document. > > > Please see the recent "dm-cache warming thread": > > https://www.redhat.com/archives/dm-devel/2013-July/msg00133.html > > > > That isn't to say we cannot take steps to be more aggressive; but we'll > > need more context for what you're doing. > > > > A normal system boot is likely predominantly read once, and/or (as Joe > > pointed out) the page cache could be masking subsequent reads. > > > > If you're doing write heavy workloads are they being elided by the > > sequential_threshold? > > > > Try a git checkout, and switch branches a few times (e.g. checkout v3.1, > > then v3.8, then v3.2, then v3.9, then v3.1, etc). > > OK, so context: This is my personal do-everything server, so it does all > kinds of things; web serving (both pages, scripts and mirrors), large files, > database work, email (local, not IMAP), backup through rsync, etc.. Although > it has 24GB of RAM, the disk set is 8TB (and the cache disks ~450GB), so one > would think it found _something_ relevant to cache. In particular, my > Maildirs would seem like a prime candidate :-) There's I/O going on all the > time, and I'm a bit surprised that not even any writes are cached. Yes, it is surprising. Curious to know if the promotions aren't happening due to the IO scheduler somehow merging all your random small IO. We don't yet have a descrete counter to show the number of migrations that were skipped due to sequential_threshold but that is something we can add. But you can effectively disable the sequential_threshold by setting it really high, e.g.: dmsetup message cache 0 sequential_threshold 16384 > I did the git checkout as you asked, 10–15 times with different version > numbers, and although the git command seemed to create I/O (ie., the page > cache wasn't gobbling it all up, although it was mostly write I/O), but it > still didn't put anything in the cache. Please write a file that is smaller than your specified sequential_threshold, and then read it numerous times via direct IO, e.g.: dd if=<your file> of=/dev/null iflag=direct bs=16K > I see the thread you're referring to says “too low”, but I really have zero, > even after almost 40 hours of normal operation: > > cache: 0 23440891904 cache 913/8192 0 6888144 0 1653677 0 0 0 0 0 2 migration_threshold 2048 4 random_threshold 8 sequential_threshold 512 > > Could the large block size be related to this? No, a large cache block size just brings a lot more into the cache when something was deemed worthy of promotion. We need to better understand/diagnose _why_ nothing is getting promoted, to that end Heinz and Joe will collaborate on the development of a "cache_dump" utility to allow visibility into the mq policy's hit counts and hints associated with each block. But in the meantime, cranking up sequential_threshold may help. Mike -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: dm-cache with zero hit rate 2013-08-02 13:23 ` Mike Snitzer @ 2013-08-02 13:58 ` Steinar H. Gunderson 2013-08-02 15:20 ` Amy Rupp 0 siblings, 1 reply; 6+ messages in thread From: Steinar H. Gunderson @ 2013-08-02 13:58 UTC (permalink / raw) To: Mike Snitzer; +Cc: Heinz Mauelshagen, dm-devel, ejt On Fri, Aug 02, 2013 at 09:23:44AM -0400, Mike Snitzer wrote: > Yes, it is surprising. Curious to know if the promotions aren't > happening due to the IO scheduler somehow merging all your random small > IO. We don't yet have a descrete counter to show the number of > migrations that were skipped due to sequential_threshold but that is > something we can add. > > But you can effectively disable the sequential_threshold by setting it > really high, e.g.: > > dmsetup message cache 0 sequential_threshold 16384 I tried this, and it still doesn't appear to promote anything at all: cache: 0 23440891904 cache 913/8192 0 7021239 0 2049048 0 0 0 0 0 2 migration_threshold 2048 4 random_threshold 8 sequential_threshold 16384 It's only been running for a few minutes, though. FWIW, earlier I ran it on only one single partition, and then it worked. So it's not like my kernel is completely broken, at least. > Please write a file that is smaller than your specified > sequential_threshold, and then read it numerous times via direct IO, > e.g.: > > dd if=<your file> of=/dev/null iflag=direct bs=16K I did, with a 16 kB file (that should certainly be small enough, right?), executing the dd command 10000 times. Still nothing cached. /* Steinar */ -- Homepage: http://www.sesse.net/ ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: dm-cache with zero hit rate 2013-08-02 13:58 ` Steinar H. Gunderson @ 2013-08-02 15:20 ` Amy Rupp 0 siblings, 0 replies; 6+ messages in thread From: Amy Rupp @ 2013-08-02 15:20 UTC (permalink / raw) To: device-mapper development; +Cc: Heinz Mauelshagen, ejt, Mike Snitzer [-- Attachment #1.1: Type: text/plain, Size: 1929 bytes --] Is it possible to use different input files (unique files, anyway) of the same size, maybe even on different file systems if available? The creation of these could be scripted along with a dd - don't remove the file immediately after the dd, though. It might be even better if you could issue multiple reads in parallel, though that will be hard to do if it's a really small file. On Fri, Aug 2, 2013 at 8:58 AM, Steinar H. Gunderson <sgunderson@bigfoot.com > wrote: > On Fri, Aug 02, 2013 at 09:23:44AM -0400, Mike Snitzer wrote: > > Yes, it is surprising. Curious to know if the promotions aren't > > happening due to the IO scheduler somehow merging all your random small > > IO. We don't yet have a descrete counter to show the number of > > migrations that were skipped due to sequential_threshold but that is > > something we can add. > > > > But you can effectively disable the sequential_threshold by setting it > > really high, e.g.: > > > > dmsetup message cache 0 sequential_threshold 16384 > > I tried this, and it still doesn't appear to promote anything at all: > > cache: 0 23440891904 cache 913/8192 0 7021239 0 2049048 0 0 0 0 0 2 > migration_threshold 2048 4 random_threshold 8 sequential_threshold 16384 > > It's only been running for a few minutes, though. > > FWIW, earlier I ran it on only one single partition, and then it worked. > So it's not like my kernel is completely broken, at least. > > > Please write a file that is smaller than your specified > > sequential_threshold, and then read it numerous times via direct IO, > > e.g.: > > > > dd if=<your file> of=/dev/null iflag=direct bs=16K > > I did, with a 16 kB file (that should certainly be small enough, right?), > executing the dd command 10000 times. Still nothing cached. > > /* Steinar */ > -- > Homepage: http://www.sesse.net/ > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > [-- Attachment #1.2: Type: text/html, Size: 2722 bytes --] [-- Attachment #2: Type: text/plain, Size: 0 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-08-02 15:20 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-07-31 18:50 dm-cache with zero hit rate Steinar H. Gunderson 2013-07-31 20:25 ` Mike Snitzer 2013-08-02 8:49 ` Steinar H. Gunderson 2013-08-02 13:23 ` Mike Snitzer 2013-08-02 13:58 ` Steinar H. Gunderson 2013-08-02 15:20 ` Amy Rupp
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.