[linux-lvm] LVM & fstrim behaviour (Fedora 19)

[linux-lvm] LVM & fstrim behaviour (Fedora 19)

[Jorge Fábregas]

Hi,

I'm testing fstrim on an LVM volume but it only seems to work the first
time I run it (just after mounting the volume).  When I run "fstrim -v
/mnt" for the first time it prints all the free blocks that is trimming
(almost all of the filesystem as it is empty) but subsequent runs just
output "0 bytes trimmed" no matter how many files I create/sync & delete
afterwards.  The "issue_discards" is set in lvm.conf.

If I do this over the raw device (/dev/sda3, same ext4 filesystem) I get
the output corresponding to the last deleted files every time I run fstrim.

## THIN ##

I also created a thin pool and a logical volume on that pool.  If I
mount this volume with "discard", I can see that TRIM is working by
doing an "lvs vgthin" (the Data Usage% grows as I create files & shrinks
as I delete files).  However, If I mount it without the "discard" option
(in order to use fstrim) I only see the data-usage reduction when I run
fstrim *just* for the first time.

Any help will be appreciated.

Thanks,
Jorge

Re: [linux-lvm] LVM & fstrim behaviour (Fedora 19)

[Lukáš Czerner]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2373 bytes --]

On Tue, 17 Sep 2013, Jorge Fï¿œbregas wrote:

> Date: Tue, 17 Sep 2013 14:38:08 -0400
> From: Jorge Fï¿œbregas <jorge.fabregas@gmail.com>
> Reply-To: LVM general discussion and development <linux-lvm@redhat.com>
> To: linux-lvm@redhat.com
> Subject: [linux-lvm] LVM & fstrim behaviour (Fedora 19)
> 
> Hi,
> 
> I'm testing fstrim on an LVM volume but it only seems to work the first
> time I run it (just after mounting the volume).  When I run "fstrim -v
> /mnt" for the first time it prints all the free blocks that is trimming
> (almost all of the filesystem as it is empty) but subsequent runs just
> output "0 bytes trimmed" no matter how many files I create/sync & delete
> afterwards.  The "issue_discards" is set in lvm.conf.
> 
> If I do this over the raw device (/dev/sda3, same ext4 filesystem) I get
> the output corresponding to the last deleted files every time I run fstrim.
> 
> ## THIN ##
> 
> I also created a thin pool and a logical volume on that pool.  If I
> mount this volume with "discard", I can see that TRIM is working by
> doing an "lvs vgthin" (the Data Usage% grows as I create files & shrinks
> as I delete files).  However, If I mount it without the "discard" option
> (in order to use fstrim) I only see the data-usage reduction when I run
> fstrim *just* for the first time.
> 
> Any help will be appreciated.

Can you blktrace output for the devivce you're doing the fstrim on ?
Both when you run fstrim for the first time and then when you run it
again after releasing some blocks from the file system.

blktrace -d /dev/sda -o - | blkparse -i -

viz. man blktrace

Can you share how many a what size are the files you're releasing
before running fstrim again ? Are you using sync after removing
those files and before running fstrim (that's pretty important since
blocks are not released instantly from the ext4 file system - but you
can force it with sync).

I can confirm that fstrim on ext4 works as expected with loop image
and I do not see behaviour described by Jorge. I'll try thinp as
well to see what's going on.

Thanks!
-Lukas


> 
> Thanks,
> Jorge
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 

Re: [linux-lvm] LVM & fstrim behaviour (Fedora 19)

[Lukáš Czerner]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4020 bytes --]

On Tue, 24 Sep 2013, Lukďż˝ Czerner wrote:

> Date: Tue, 24 Sep 2013 15:20:13 +0200 (CEST)
> From: Lukďż˝ Czerner <lczerner@redhat.com>
> Reply-To: LVM general discussion and development <linux-lvm@redhat.com>
> To: Jorge F�bregas <jorge.fabregas@gmail.com>
> Cc: linux-lvm@redhat.com
> Subject: Re: [linux-lvm] LVM & fstrim behaviour (Fedora 19)
> 
> On Tue, 17 Sep 2013, Jorge F�bregas wrote:
> 
> > Date: Tue, 17 Sep 2013 14:38:08 -0400
> > From: Jorge F�bregas <jorge.fabregas@gmail.com>
> > Reply-To: LVM general discussion and development <linux-lvm@redhat.com>
> > To: linux-lvm@redhat.com
> > Subject: [linux-lvm] LVM & fstrim behaviour (Fedora 19)
> > 
> > Hi,
> > 
> > I'm testing fstrim on an LVM volume but it only seems to work the first
> > time I run it (just after mounting the volume).  When I run "fstrim -v
> > /mnt" for the first time it prints all the free blocks that is trimming
> > (almost all of the filesystem as it is empty) but subsequent runs just
> > output "0 bytes trimmed" no matter how many files I create/sync & delete
> > afterwards.  The "issue_discards" is set in lvm.conf.
> > 
> > If I do this over the raw device (/dev/sda3, same ext4 filesystem) I get
> > the output corresponding to the last deleted files every time I run fstrim.
> > 
> > ## THIN ##
> > 
> > I also created a thin pool and a logical volume on that pool.  If I
> > mount this volume with "discard", I can see that TRIM is working by
> > doing an "lvs vgthin" (the Data Usage% grows as I create files & shrinks
> > as I delete files).  However, If I mount it without the "discard" option
> > (in order to use fstrim) I only see the data-usage reduction when I run
> > fstrim *just* for the first time.
> > 
> > Any help will be appreciated.
> 
> Can you blktrace output for the devivce you're doing the fstrim on ?
> Both when you run fstrim for the first time and then when you run it
> again after releasing some blocks from the file system.
> 
> blktrace -d /dev/sda -o - | blkparse -i -
> 
> viz. man blktrace
> 
> Can you share how many a what size are the files you're releasing
> before running fstrim again ? Are you using sync after removing
> those files and before running fstrim (that's pretty important since
> blocks are not released instantly from the ext4 file system - but you
> can force it with sync).
> 
> I can confirm that fstrim on ext4 works as expected with loop image
> and I do not see behaviour described by Jorge. I'll try thinp as
> well to see what's going on.

I've tried it with thinp targer and it works as expected:


# mkfs.ext4 /dev/lvm_pool/thin_lv
# mount /dev/lvm_pool/thin_lv /mnt/test
# dd if=/dev/zero of=/mnt/test/file bs=1M count=1024
# dd if=/dev/zero of=/mnt/test/file1 bs=1M count=1024
# dd if=/dev/zero of=/mnt/test/file2 bs=1M count=1024
# sync
# lvs
pool    lvm_pool twi-a-tz-   9,00g              35,83                        
thin_lv lvm_pool Vwi-aotz-  50,00g pool          6,45                        

# rm -f /mnt/test/file
# sync
# fstrim -v /mnt/test
/mnt/test: 47 GiB (50508640256 bytes) trimmed

# lvs
pool    lvm_pool twi-a-tz-   9.00g              33.54                        
thin_lv lvm_pool Vwi-aotz-  50.00g pool          6.04                        

# rm -f /mnt/test/file*
# sync
# fstrim -v /mnt/test
/mnt/test: 47 GiB (50508640256 bajtďż˝) trimmed

# lvs
pool    lvm_pool twi-a-tz-   9.00g              16.44                        
thin_lv lvm_pool Vwi-aotz-  50.00g pool          2.96                        

# lvm version
LVM version:     2.02.98(2) (2012-10-15)
Library version: 1.02.77 (2012-10-15)
Driver version:  4.24.0

-Lukas


> 
> Thanks!
> -Lukas
> 
> 
> > 
> > Thanks,
> > Jorge
> > 
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> > 

Re: [linux-lvm] LVM & fstrim behaviour (Fedora 19)

[Jorge Fábregas]

On 09/24/2013 09:20 AM, Luk� Czerner wrote:
> I can confirm that fstrim on ext4 works as expected with loop image
> and I do not see behaviour described by Jorge. I'll try thinp as
> well to see what's going on.

Hi Luk�,

I'm sorry I forgot to update this thread.  Fstrim over LVM is working
totally fine.  After a bunch of updates and a restart it started working
for me again.  Now the issue is really with thin LVs for which I opened
a bugzilla last week:

https://bugzilla.redhat.com/show_bug.cgi?id=1009546

There you would see all the details (file sizes, package versions etc).
 I can recreate the problem 100% of the time.

Please let me know if you still need any more information.

Thanks,
Jorge

Re: [linux-lvm] LVM & fstrim behaviour (Fedora 19)

[Jorge Fábregas]

I followed your example and trims seems to work (based on the lvs
output) but as you can see, there seems to be something wrong with the
"fstrim -v" output (it shows as if nothing is being trimmed).


# lvs vgthin1
  lvol1      vgthin1 Vwi-aotz- 100.78g lvthinpool   3.20

  lvthinpool vgthin1 twi-a-tz- 100.68g              3.20

# mount /var/lib/libvirt/images
# fstrim -v /var/lib/libvirt/images
/var/lib/libvirt/images: 97.6 GiB (104814780416 bytes) trimmed

# dd if=/dev/zero of=/var/lib/libvirt/images/file bs=1M count=1024
# dd if=/dev/zero of=/var/lib/libvirt/images/file1 bs=1M count=1024
# dd if=/dev/zero of=/var/lib/libvirt/images/file2 bs=1M count=1024

# sync
# lvs vgthin1
  lvol1      vgthin1 Vwi-aotz- 100.78g lvthinpool   6.18

  lvthinpool vgthin1 twi-a-tz- 100.68g              6.18

# rm -f /var/lib/libvirt/images/file
# sync
# fstrim -v /var/lib/libvirt/images
/var/lib/libvirt/images: 0 B (0 bytes) trimmed

# lvs vgthin1
  lvol1      vgthin1 Vwi-aotz- 100.78g lvthinpool   5.19

  lvthinpool vgthin1 twi-a-tz- 100.68g              5.19

# rm -f /var/lib/libvirt/images/file*
# sync
# fstrim -v /var/lib/libvirt/images
/var/lib/libvirt/images: 0 B (0 bytes) trimmed

# lvs vgthin1

  lvol1      vgthin1 Vwi-aotz- 100.78g lvthinpool   3.20

  lvthinpool vgthin1 twi-a-tz- 100.68g              3.20

# lvm version
  LVM version:     2.02.98(2) (2012-10-15)
  Library version: 1.02.77 (2012-10-15)
  Driver version:  4.24.0

# dd if=/dev/zero of=/var/lib/libvirt/images/tempfile bs=1M count=40000
# lvs vgthin1
  lvol1      vgthin1 Vwi-aotz- 100.78g lvthinpool   41.84

  lvthinpool vgthin1 twi-a-tz- 100.68g              41.89

# rm -f /var/lib/libvirt/images/tempfile
# sync
# fstrim -v /var/lib/libvirt/images
/var/lib/libvirt/images: 0 B (0 bytes) trimmed

# lvs vgthin1

  lvol1      vgthin1 Vwi-aotz- 100.78g lvthinpool   3.08

  lvthinpool vgthin1 twi-a-tz- 100.68g              3.09


As you may see, this is the behaviour I'm getting now (fstrim reports
nothing being trimmed although it is being trimmed).  I can't recreate
the original problem when I opened the bug (fstrim reporting always 124
MiB).  I'm not sure if any update had to do anything with it.

Re: [linux-lvm] LVM & fstrim behaviour (Fedora 19)

[Jorge Fábregas]

On 09/24/2013 11:52 AM, Jorge F�bregas wrote:
>  I can't recreate the original problem when I opened the bug (fstrim reporting
>  always 124 MiB).  I'm not sure if any update had to do anything with it.

Arrrgh.  Please discard my last email (with my sample run). I totaly
forgot that I had "discard" as a mount option in /etc/fstab (I did this
as fstrim wasn't properly working).  I'll remove the option now and will
perform the tests again.

Re: [linux-lvm] LVM & fstrim behaviour (Fedora 19)

[Jorge Fábregas]

Here's without discard mount option in fstab:

http://paste.fedoraproject.org/41849/80039018/

As you can see fstrim is not actually working at all.

Regards,
Jorge

Re: [linux-lvm] LVM & fstrim behaviour (Fedora 19)

[Jorge Fábregas]

ok Luk� I think I found where the problem is. I just realized that I
created that filesystem without journaling (in order to increase perform
for the VM disks).  I created it with:

mkfs.ext4 -O ^has_journal /dev/vgthin1/lvol2

I went back and created the filesystem again (on a thin lvol) with
default options and indeed fstrim works as expected.  I created ANOTHER
thinvol (without journaling) and fstrim doesn't work.

Is this a bug or does fstrim requires ext4 with journal by design?

My apologies for not stating this since the beginning!

-- 
Jorge

Re: [linux-lvm] LVM & fstrim behaviour (Fedora 19)

[Lukáš Czerner]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1497 bytes --]

On Tue, 24 Sep 2013, Jorge F�bregas wrote:

> Date: Tue, 24 Sep 2013 14:03:12 -0400
> From: Jorge F�bregas <jorge.fabregas@gmail.com>
> To: linux-lvm@redhat.com
> Cc: Lukďż˝ Czerner <lczerner@redhat.com>
> Subject: Re: [linux-lvm] LVM & fstrim behaviour (Fedora 19)
> 
> ok Lukďż˝ I think I found where the problem is. I just realized that I
> created that filesystem without journaling (in order to increase perform
> for the VM disks).  I created it with:
> 
> mkfs.ext4 -O ^has_journal /dev/vgthin1/lvol2
> 
> I went back and created the filesystem again (on a thin lvol) with
> default options and indeed fstrim works as expected.  I created ANOTHER
> thinvol (without journaling) and fstrim doesn't work.
> 
> Is this a bug or does fstrim requires ext4 with journal by design?

Oh that's useful information :) Yes, it is a bug and here is
untested fix.


diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
index a41e3ba..4d113ef 100644
--- a/fs/ext4/mballoc.c
+++ b/fs/ext4/mballoc.c
@@ -4794,8 +4794,8 @@ do_more:
 					 " group:%d block:%d count:%lu failed"
 					 " with %d", block_group, bit, count,
 					 err);
-		}
-
+		} else
+			EXT4_MB_GRP_CLEAR_TRIMMED(e4b.bd_info);
 
 		ext4_lock_group(sb, block_group);
 		mb_clear_bits(bitmap_bh->b_data, bit, count_clusters);

> 
> My apologies for not stating this since the beginning!

No problem, important is that we've got to the bottom of this.
Thanks for reporting it!

-Lukas

> 
> 

Re: [linux-lvm] LVM & fstrim behaviour (Fedora 19)

[Jorge Fábregas]

On 09/25/2013 04:39 AM, Luk� Czerner wrote:
> Oh that's useful information :) Yes, it is a bug and here is
> untested fix.
<snip>

That's great to know and also that it wasn't something crazy on my setup :)

Thank you for the help!

Best regards,
Jorge