[linux-lvm] Mounted, but idle LVM volume causes constant disk writes

[linux-lvm] Mounted, but idle LVM volume causes constant disk writes

[Bostjan Skufca]

[-- Attachment #1: Type: text/plain, Size: 810 bytes --]

Hello,

If LVM2 volume is mounted (ext4 fs), why is there constant write activity
to the devices below (sw raid in this case) despite not using the mounted
volume at all?

I am seing this on multiple hosts. On this particular one I am seeing
around 20 writes per second on average, but they occur in batches of around
50-90 writeIOps on every 3-4 seconds. If I mount the same filesystem (ext4)
on sw raid1 without LVM in between, I see the expected 0 write IOps when
not using the filesystem.

Kernel versions where I noticed this: 3.0.13 (sysrescd) 3.4.1, 3.9.4
(custom compiled). I noticed it on all systems where lvm is in use.

Thank you for the hints,
b.


PS: I did search the web first, but I could not find the answer. Sorry if
this has been answered before, maybe I was searching for the wrong terms.

[-- Attachment #2: Type: text/html, Size: 1069 bytes --]

Re: [linux-lvm] Mounted, but idle LVM volume causes constant disk writes

[Greg Zapp]

[-- Attachment #1: Type: text/plain, Size: 1474 bytes --]

This is just a general stab, but has the filesystem been fully initialized
on those?  The default for ext4 is to do a lazy initialization of the
filesystem and journal when you create it.  Depending on the size of the
file system this can last for some time.  To prevent this use:
mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0

-Greg


On Tue, Jun 4, 2013 at 5:46 PM, Bostjan Skufca <bostjan@a2o.si> wrote:

> Hello,
>
> If LVM2 volume is mounted (ext4 fs), why is there constant write activity
> to the devices below (sw raid in this case) despite not using the mounted
> volume at all?
>
> I am seing this on multiple hosts. On this particular one I am seeing
> around 20 writes per second on average, but they occur in batches of around
> 50-90 writeIOps on every 3-4 seconds. If I mount the same filesystem (ext4)
> on sw raid1 without LVM in between, I see the expected 0 write IOps when
> not using the filesystem.
>
> Kernel versions where I noticed this: 3.0.13 (sysrescd) 3.4.1, 3.9.4
> (custom compiled). I noticed it on all systems where lvm is in use.
>
> Thank you for the hints,
> b.
>
>
> PS: I did search the web first, but I could not find the answer. Sorry if
> this has been answered before, maybe I was searching for the wrong terms.
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/html, Size: 2307 bytes --]

Re: [linux-lvm] Mounted, but idle LVM volume causes constant disk writes

[Bostjan Skufca]

[-- Attachment #1: Type: text/plain, Size: 2012 bytes --]

Greg, spot on!

I did found it a bit weird that mkfs on 5tb volume was faster than usual
(well, I did notice I didn't notice inode tables counter being slow) :)
I have to check other hosts though.

Thank you!
b.



On 4 June 2013 12:59, Greg Zapp <greg.zapp@gmail.com> wrote:

> This is just a general stab, but has the filesystem been fully initialized
> on those?  The default for ext4 is to do a lazy initialization of the
> filesystem and journal when you create it.  Depending on the size of the
> file system this can last for some time.  To prevent this use:
> mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0
>
> -Greg
>
>
> On Tue, Jun 4, 2013 at 5:46 PM, Bostjan Skufca <bostjan@a2o.si> wrote:
>
>> Hello,
>>
>> If LVM2 volume is mounted (ext4 fs), why is there constant write activity
>> to the devices below (sw raid in this case) despite not using the mounted
>> volume at all?
>>
>> I am seing this on multiple hosts. On this particular one I am seeing
>> around 20 writes per second on average, but they occur in batches of around
>> 50-90 writeIOps on every 3-4 seconds. If I mount the same filesystem (ext4)
>> on sw raid1 without LVM in between, I see the expected 0 write IOps when
>> not using the filesystem.
>>
>> Kernel versions where I noticed this: 3.0.13 (sysrescd) 3.4.1, 3.9.4
>> (custom compiled). I noticed it on all systems where lvm is in use.
>>
>> Thank you for the hints,
>> b.
>>
>>
>> PS: I did search the web first, but I could not find the answer. Sorry if
>> this has been answered before, maybe I was searching for the wrong terms.
>>
>>
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/html, Size: 3454 bytes --]

Re: [linux-lvm] Mounted, but idle LVM volume causes constant disk writes

[Tomas Vanderka]

On 06/04/2013 07:46 AM, Bostjan Skufca wrote:
> Hello,
> 
> If LVM2 volume is mounted (ext4 fs), why is there constant write activity to the devices below (sw raid in this case) despite not using the mounted volume at all?
> 
> I am seing this on multiple hosts. On this particular one I am seeing around 20 writes per second on average, but they occur in batches of around 50-90 writeIOps on every
> 3-4 seconds. If I mount the same filesystem (ext4) on sw raid1 without LVM in between, I see the expected 0 write IOps when not using the filesystem.
> 

Just out of curiosity I looked at my systems and I don't see any such IO, even for mounted FS.

You can try running following commands to log writes and where are they comming from (show the result via dmesg)

echo 1 > /proc/sys/vm/block_dump; sleep 10; echo 0 > /proc/sys/vm/block_dump;

By doing so I noticed dm-thin writing to my otherwise unused dm thin pool every second. After looking at the code in dm-thin.c, it looks like the kernel is periodically
commiting metadata every second even if there was no IO to the pool in the time period. I think process_deferred_bios function should probably check if anything changed
before commiting metadata unconditionally when 1s timer is up. Looks like dm-cache-target.c is doing it right.

t.