[linux-lvm] Strange dm-cache activity at startup

[linux-lvm] Strange dm-cache activity at startup

[Stefan Ring]

I have a fairly simple dm-cache setup with one PV on an HDD partition
as the origin block device, and the cache LVs live on a partition on
an SSD. The cache is in writethrough mode. Since I have no prior
experience with this, I basically followed the lvmcache man page when
creating this setup a few months ago.

Every morning at startup, the system immediately starts prolonged
activity on the devices involved in the cache setup. iostat shows
reading activity from the cache device and constant write activity on
the HDD. The HDD is fully utilized, and this goes on for more than an
hour. After that, everything goes into what I would consider the
"normal" state with only the IO activity going on that I actively
throw at it.

Is this normal/expected? What does it do? Why does it write back huge
amounts of data to the HDD after each system start?

Kernel is currently 3.17.8-200.fc20.x86_64 (Fedora)
Userland has been upgraded to fc22 meanwhile

origin:
└─sda6                          8:6    0 894.6G  0 part
  └─vg_zfs-lv_zfsdisk_corig   253:2    0 894.6G  0 lvm
    └─vg_zfs-lv_zfsdisk       253:3    0 894.6G  0 lvm

cache:
└─sdb5                          8:21   0 233.7G  0 part
  ├─vg_zfs-lv_cachedata_cdata 253:0    0 213.1G  0 lvm
  │ └─vg_zfs-lv_zfsdisk       253:3    0 894.6G  0 lvm
  └─vg_zfs-lv_cachedata_cmeta 253:1    0   9.4G  0 lvm
    └─vg_zfs-lv_zfsdisk       253:3    0 894.6G  0 lvm

$ sudo dmsetup ls --tree
vg_zfs-lv_zfsdisk (253:3)
 ├─vg_zfs-lv_zfsdisk_corig (253:2)
 │  └─ (8:6)
 ├─vg_zfs-lv_cachedata_cdata (253:0)
 │  └─ (8:21)
 └─vg_zfs-lv_cachedata_cmeta (253:1)
    └─ (8:21)

$ sudo dmsetup table vg_zfs-lv_zfsdisk
0 1876041728 cache 253:1 253:0 253:2 128 1 writethrough mq 0

Re: [linux-lvm] Strange dm-cache activity at startup

[Zdenek Kabelac]

Dne 10.9.2015 v 11:30 Stefan Ring napsal(a):
> I have a fairly simple dm-cache setup with one PV on an HDD partition
> as the origin block device, and the cache LVs live on a partition on
> an SSD. The cache is in writethrough mode. Since I have no prior
> experience with this, I basically followed the lvmcache man page when
> creating this setup a few months ago.
>
> Every morning at startup, the system immediately starts prolonged
> activity on the devices involved in the cache setup. iostat shows
> reading activity from the cache device and constant write activity on
> the HDD. The HDD is fully utilized, and this goes on for more than an
> hour. After that, everything goes into what I would consider the
> "normal" state with only the IO activity going on that I actively
> throw at it.
>
> Is this normal/expected? What does it do? Why does it write back huge
> amounts of data to the HDD after each system start?
>
> Kernel is currently 3.17.8-200.fc20.x86_64 (Fedora)
> Userland has been upgraded to fc22 meanwhile

Please first try to use the latest kernel.
Kernel 3.17 is almost a year old.

Lots of optimizations went in meanwhile.
It's pointless to hunt already fixed bug.

Zdenek

Re: [linux-lvm] Strange dm-cache activity at startup

[Stefan Ring]

On Thu, Sep 10, 2015 at 2:59 PM, Zdenek Kabelac <zkabelac@redhat.com> wrote:
> Please first try to use the latest kernel.
> Kernel 3.17 is almost a year old.
>
> Lots of optimizations went in meanwhile.
> It's pointless to hunt already fixed bug.

No problem. I'll soon upgrade to 4.2 because I'm eager to try smq.
I've been held back by a zfsonlinux bug that has prevented an upgrade
and has only very recently been fixed.

Re: [linux-lvm] Strange dm-cache activity at startup

[Zdenek Kabelac]

Dne 10.9.2015 v 15:11 Stefan Ring napsal(a):
> On Thu, Sep 10, 2015 at 2:59 PM, Zdenek Kabelac <zkabelac@redhat.com> wrote:
>> Please first try to use the latest kernel.
>> Kernel 3.17 is almost a year old.
>>
>> Lots of optimizations went in meanwhile.
>> It's pointless to hunt already fixed bug.
>
> No problem. I'll soon upgrade to 4.2 because I'm eager to try smq.
> I've been held back by a zfsonlinux bug that has prevented an upgrade
> and has only very recently been fixed.


For playing with smq - you definitely need the latest lvm2 version as well.

Zdenek

Re: [linux-lvm] Strange dm-cache activity at startup

[Joe Thornber]

On Thu, Sep 10, 2015 at 11:30:27AM +0200, Stefan Ring wrote:
> Is this normal/expected? What does it do? Why does it write back huge
> amounts of data to the HDD after each system start?

No, this is not expected.  The most likely cause is your system was
not shutdown cleanly so dm-cache has no choice but to assume all
blocks are dirty and need writing back.  Make sure your volumes are
being deactivated as part of shutdown.

- Joe

Re: [linux-lvm] Strange dm-cache activity at startup

[John Stoffel]

>>>>> "Joe" == Joe Thornber <thornber@redhat.com> writes:

Joe> On Thu, Sep 10, 2015 at 11:30:27AM +0200, Stefan Ring wrote:
>> Is this normal/expected? What does it do? Why does it write back huge
>> amounts of data to the HDD after each system start?

Joe> No, this is not expected.  The most likely cause is your system was
Joe> not shutdown cleanly so dm-cache has no choice but to assume all
Joe> blocks are dirty and need writing back.  Make sure your volumes are
Joe> being deactivated as part of shutdown.

How would you double check to make sure this happening properly in the
logs?  I'm interested in using an SSD to cache my home file server
/home and /local directories to make NFS access faster... but I've
been hesitant to upgrade since I'd hate to lose all this data.

John

Re: [linux-lvm] Strange dm-cache activity at startup

[Simon O'Riordan]

Avoid losing data by imaging the drive using Clonezilla. I do it all the 
time.

On 11/09/2015 14:28, John Stoffel wrote:
>>>>>> "Joe" == Joe Thornber <thornber@redhat.com> writes:
> Joe> On Thu, Sep 10, 2015 at 11:30:27AM +0200, Stefan Ring wrote:
>>> Is this normal/expected? What does it do? Why does it write back huge
>>> amounts of data to the HDD after each system start?
> Joe> No, this is not expected.  The most likely cause is your system was
> Joe> not shutdown cleanly so dm-cache has no choice but to assume all
> Joe> blocks are dirty and need writing back.  Make sure your volumes are
> Joe> being deactivated as part of shutdown.
>
> How would you double check to make sure this happening properly in the
> logs?  I'm interested in using an SSD to cache my home file server
> /home and /local directories to make NFS access faster... but I've
> been hesitant to upgrade since I'd hate to lose all this data.
>
> John
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Strange dm-cache activity at startup

[John Stoffel]

Simon,
I have multiple terabytes of file storage at home.  I'm looking to use
an SSD to speed up access to the more frequently used data.  Don't
have the cash to upgrade to a full SSD environment.  

Simon> Avoid losing data by imaging the drive using Clonezilla. I do it all the 
Simon> time.

Simon> On 11/09/2015 14:28, John Stoffel wrote:
>>>>>>> "Joe" == Joe Thornber <thornber@redhat.com> writes:
Joe> On Thu, Sep 10, 2015 at 11:30:27AM +0200, Stefan Ring wrote:
>>>> Is this normal/expected? What does it do? Why does it write back huge
>>>> amounts of data to the HDD after each system start?
Joe> No, this is not expected.  The most likely cause is your system was
Joe> not shutdown cleanly so dm-cache has no choice but to assume all
Joe> blocks are dirty and need writing back.  Make sure your volumes are
Joe> being deactivated as part of shutdown.
>> 
>> How would you double check to make sure this happening properly in the
>> logs?  I'm interested in using an SSD to cache my home file server
>> /home and /local directories to make NFS access faster... but I've
>> been hesitant to upgrade since I'd hate to lose all this data.
>> 
>> John
>> 
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Simon> _______________________________________________
Simon> linux-lvm mailing list
Simon> linux-lvm@redhat.com
Simon> https://www.redhat.com/mailman/listinfo/linux-lvm
Simon> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Strange dm-cache activity at startup

[Stefan Ring]

On Fri, Sep 11, 2015 at 2:51 PM, Joe Thornber <thornber@redhat.com> wrote:
> On Thu, Sep 10, 2015 at 11:30:27AM +0200, Stefan Ring wrote:
>> Is this normal/expected? What does it do? Why does it write back huge
>> amounts of data to the HDD after each system start?
>
> No, this is not expected.  The most likely cause is your system was
> not shutdown cleanly so dm-cache has no choice but to assume all
> blocks are dirty and need writing back.  Make sure your volumes are
> being deactivated as part of shutdown.

I would have hoped that the distro scripts would take care of this.
Apparently not, for whatever reason :(. I have since upgraded to
kernel 4.2, and the behavior has not changed. I'll try to reboot today
instead of shutdown and see if anything changes.

Re: [linux-lvm] Strange dm-cache activity at startup

[Stefan Ring]

On Wed, Sep 16, 2015 at 9:48 AM, Stefan Ring <stefanrin@gmail.com> wrote:
> I would have hoped that the distro scripts would take care of this.
> Apparently not, for whatever reason :(. I have since upgraded to
> kernel 4.2, and the behavior has not changed. I'll try to reboot today
> instead of shutdown and see if anything changes.

Nothing has changed. However, I noticed this in dmesg during startup:

device-mapper: cache: You have created a cache device with a lot of
individual cache blocks (3491776)
All these mappings can consume a lot of kernel memory, and take some
time to read/write.
Please consider increasing the cache block size to reduce the overall
cache block count.

Now I'll have to see what to do about that.

[linux-lvm] LVM cache (or dm-cache) shutdown

[Ian Pilcher]

On 09/11/2015 07:51 AM, Joe Thornber wrote:
> No, this is not expected.  The most likely cause is your system was
> not shutdown cleanly so dm-cache has no choice but to assume all
> blocks are dirty and need writing back.  Make sure your volumes are
> being deactivated as part of shutdown.

What exactly is required to cleanly shutdown an LVM cache or dm-cache
device?

AFAICT, systemd does a DM_DEV_REMOVE ioctl on all device mapper devices
during shutdown/reboot.  Is this sufficient?

Thanks!

-- 
========================================================================
Ian Pilcher                                         arequipeno@gmail.com
-------- "I grew up before Mark Zuckerberg invented friendship" --------
========================================================================

Re: [linux-lvm] LVM cache (or dm-cache) shutdown

[Zdenek Kabelac]

Dne 16.10.2015 v 19:54 Ian Pilcher napsal(a):
> On 09/11/2015 07:51 AM, Joe Thornber wrote:
>> No, this is not expected.  The most likely cause is your system was
>> not shutdown cleanly so dm-cache has no choice but to assume all
>> blocks are dirty and need writing back.  Make sure your volumes are
>> being deactivated as part of shutdown.
>
> What exactly is required to cleanly shutdown an LVM cache or dm-cache
> device?
>
> AFAICT, systemd does a DM_DEV_REMOVE ioctl on all device mapper devices
> during shutdown/reboot.  Is this sufficient?


ONLY if the give device are UNUSED.

If the volume is e.g. mounted - then 'remove' is irrelevant.

You can't remove device with open count != 0.
(check with:  dmsetup info -c)

Zdenek