[linux-lvm] lvremove does not pass discards if volume is part of thin pool

[linux-lvm] lvremove does not pass discards if volume is part of thin pool

[vaLentin chernoZemski]

Hi folks,

I am experiencing issues with LVM thin pool and discards that should be 
passed down during lvremove but they are not.

Setup looks like this:

Sparse file -> Loop device -> PV -> VG -> Thin Pool -> LV

If we mount -o discard LV and fill it with data which is later deleted 
sparse file shrinks back ~ to the original size.

However if we directly lvremove LV, the sparse file does not shrink back 
its size so we are forced to use fallocate (which is slow).

According to the docs lvremove should issue discards to the underlying 
device but it appears that this is not the case if LV is part of thin pool

lsblk -D shows DISC-ZERO as 0 for tpool tmeta and tdata devices and all 
their childs which is strange.

[root@testbed ~]# lsblk -D | grep ^NAME ; lsblk -D | grep -A8 
$group-thingroup_tmeta
NAME                                                       DISC-ALN 
DISC-GRAN DISC-MAX DISC-ZERO
|-testgroup-thingroup_tmeta (dm-33)                               0 
    4K       4G         1
| `-testgroup-thingroup-tpool (dm-35)                             0 
   64K      64K         0
|   |-testgroup-thingroup (dm-36)                                 0 
   64K      64K         0
|   `-testgroup-testvol (dm-37)                                   0 
   64K      64K         0
`-testgroup-thingroup_tdata (dm-34)                               0 
    4K       4G         1
   `-testgroup-thingroup-tpool (dm-35)                             0 
    64K      64K         0
     |-testgroup-thingroup (dm-36)                                 0 
    64K      64K         0
     `-testgroup-testvol (dm-37)                                   0 
    64K      64K         0

Kernel version we are using is 3.12.x.

Linux 3.2 - discard support for loop devices - 
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef

Linux 3.4 - discard support for thin pool - 
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=104655fd4dcebd50068ef30253a001da72e3a081

/etc/lvm/lvm.conf is configured as follows

     issue_discards = 1
     thin_pool_discards = "passdown"

Here is a sample script that demonstrate the issue. Note the output 
after the second lvremove where size of "loop-block-device-file" remains 
unchanged even volume is removed.

set -e
file=/root/testfile
group=testgroup
thingroup=thingroup
volume_name=testvol
volume_size=200M

grep -i disca /etc/lvm/lvm.conf  | grep -v \#
rpm -qa | grep lvm2
uname -r

[[ -f ${file} ]] && unlink ${file}
truncate ${file} --size 10G
loopdev=$(losetup -f --show ${file})
pvcreate --metadatasize 1M ${loopdev}
vgcreate ${group} -s 1MiB ${loopdev}
pe_size=$(vgdisplay "/dev/${group}" | grep 'PE Size' | awk '{print $3}')
thin_size=$(echo "$(vgdisplay "/dev/${group}" | grep 'Free  PE' | awk 
'{print $5}')*${pe_size}-180" | bc -l)
lvcreate --ignoreactivationskip -Z n -L ${thin_size}m -T 
"/dev/${group}/${thingroup}"
lvcreate --ignoreactivationskip -V${volume_size} -T 
"${group}/${thingroup}" -n "${volume_name}"
mkfs.ext4 /dev/$group/$volume_name
sync && du -hs $file
lvs $group
lsblk -D | grep ^NAME ; lsblk -D | grep -A8 $group-thingroup_tmeta
sync && du -hs $file
mkdir -p /mnt/tmp/
mount -o discard /dev/$group/$volume_name /mnt/tmp/
dd if=/dev/zero of=/mnt/tmp/fill_file count=100 bs=1M
sync && du -hs $file
umount /mnt/tmp/
sync && du -hs $file
mount -o discard /dev/$group/$volume_name /mnt/tmp/
rm -f /mnt/tmp/fill_file
sync && du -hs $file
umount /mnt/tmp/
sync && du -hs $file
lvremove -f /dev/$group/$volume_name
lvcreate --ignoreactivationskip -V${volume_size} -T 
"${group}/${thingroup}" -n "${volume_name}"
mkfs.ext4 /dev/$group/$volume_name
lvs $group
lsblk -D | grep ^NAME ; lsblk -D | grep -A8 $group-thingroup_tmeta
sync && du -hs $file
mkdir -p /mnt/tmp/
mount -o discard /dev/$group/$volume_name /mnt/tmp/
dd if=/dev/zero of=/mnt/tmp/fill_file count=100 bs=1M
sync && du -hs $file
umount /mnt/tmp/
sync && du -hs $file
lvremove -f $group/$volume_name
echo "==== AFTER THIS LVREMOVE size should shrink but it does not ==="
sync && du -hs $file
vgchange -Kan $group
sync && du -hs $file
losetup -d $loopdev
sync && du -hs $file
echo "==== FALLOCATE does its job well but that's not the point ===="
fallocate -d $file
sync && du -hs $file

Any assistance will be highly appreciated.

Thanks,

vaLentin

Re: [linux-lvm] lvremove does not pass discards if volume is part of thin pool

[Mike Snitzer]

On Mon, Aug 10 2015 at  4:56am -0400,
vaLentin chernoZemski <valentin@siteground.com> wrote:

> Hi folks,
> 
> I am experiencing issues with LVM thin pool and discards that should
> be passed down during lvremove but they are not.
> 
> Setup looks like this:
> 
> Sparse file -> Loop device -> PV -> VG -> Thin Pool -> LV
> 
> If we mount -o discard LV and fill it with data which is later
> deleted sparse file shrinks back ~ to the original size.
> 
> However if we directly lvremove LV, the sparse file does not shrink
> back its size so we are forced to use fallocate (which is slow).
> 
> According to the docs lvremove should issue discards to the
> underlying device but it appears that this is not the case if LV is
> part of thin pool
> 
> lsblk -D shows DISC-ZERO as 0 for tpool tmeta and tdata devices and
> all their childs which is strange.
> 
> [root@testbed ~]# lsblk -D | grep ^NAME ; lsblk -D | grep -A8
> $group-thingroup_tmeta
> NAME                                                       DISC-ALN
> DISC-GRAN DISC-MAX DISC-ZERO
> |-testgroup-thingroup_tmeta (dm-33)                               0
> 4K       4G         1
> | `-testgroup-thingroup-tpool (dm-35)                             0
> 64K      64K         0
> |   |-testgroup-thingroup (dm-36)                                 0
> 64K      64K         0
> |   `-testgroup-testvol (dm-37)                                   0
> 64K      64K         0
> `-testgroup-thingroup_tdata (dm-34)                               0
> 4K       4G         1
>   `-testgroup-thingroup-tpool (dm-35)                             0
> 64K      64K         0
>     |-testgroup-thingroup (dm-36)                                 0
> 64K      64K         0
>     `-testgroup-testvol (dm-37)                                   0
> 64K      64K         0
> 
> Kernel version we are using is 3.12.x.
> 
> Linux 3.2 - discard support for loop devices - http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
> 
> Linux 3.4 - discard support for thin pool - http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=104655fd4dcebd50068ef30253a001da72e3a081

Plese verify your kernel has this commit:
19fa1a6756e ("dm thin: fix discard support to a previously shared block")

But it doesn't look like you're using snapshots so this may not matter.

> /etc/lvm/lvm.conf is configured as follows
> 
>     issue_discards = 1
>     thin_pool_discards = "passdown"
> 
> Here is a sample script that demonstrate the issue. Note the output
> after the second lvremove where size of "loop-block-device-file"
> remains unchanged even volume is removed.
> 
> set -e
> file=/root/testfile
> group=testgroup
> thingroup=thingroup
> volume_name=testvol
> volume_size=200M
> 
> grep -i disca /etc/lvm/lvm.conf  | grep -v \#
> rpm -qa | grep lvm2
> uname -r
> 
> [[ -f ${file} ]] && unlink ${file}
> truncate ${file} --size 10G
> loopdev=$(losetup -f --show ${file})
> pvcreate --metadatasize 1M ${loopdev}
> vgcreate ${group} -s 1MiB ${loopdev}
> pe_size=$(vgdisplay "/dev/${group}" | grep 'PE Size' | awk '{print $3}')
> thin_size=$(echo "$(vgdisplay "/dev/${group}" | grep 'Free  PE' |
> awk '{print $5}')*${pe_size}-180" | bc -l)
> lvcreate --ignoreactivationskip -Z n -L ${thin_size}m -T
> "/dev/${group}/${thingroup}"
> lvcreate --ignoreactivationskip -V${volume_size} -T
> "${group}/${thingroup}" -n "${volume_name}"
> mkfs.ext4 /dev/$group/$volume_name
> sync && du -hs $file
> lvs $group
> lsblk -D | grep ^NAME ; lsblk -D | grep -A8 $group-thingroup_tmeta
> sync && du -hs $file
> mkdir -p /mnt/tmp/
> mount -o discard /dev/$group/$volume_name /mnt/tmp/
> dd if=/dev/zero of=/mnt/tmp/fill_file count=100 bs=1M
> sync && du -hs $file
> umount /mnt/tmp/
> sync && du -hs $file
> mount -o discard /dev/$group/$volume_name /mnt/tmp/
> rm -f /mnt/tmp/fill_file
> sync && du -hs $file
> umount /mnt/tmp/
> sync && du -hs $file
> lvremove -f /dev/$group/$volume_name
> lvcreate --ignoreactivationskip -V${volume_size} -T
> "${group}/${thingroup}" -n "${volume_name}"
> mkfs.ext4 /dev/$group/$volume_name
> lvs $group
> lsblk -D | grep ^NAME ; lsblk -D | grep -A8 $group-thingroup_tmeta
> sync && du -hs $file
> mkdir -p /mnt/tmp/
> mount -o discard /dev/$group/$volume_name /mnt/tmp/
> dd if=/dev/zero of=/mnt/tmp/fill_file count=100 bs=1M
> sync && du -hs $file
> umount /mnt/tmp/
> sync && du -hs $file
> lvremove -f $group/$volume_name
> echo "==== AFTER THIS LVREMOVE size should shrink but it does not ==="

If you do have the patch I referenced above then something else is going
on.  You should probably run with: lvremove -vvvv to see if lvm is
actually issuing a discard.  Or you could use blktrace to see if the
thin device you're removing is actually receiving a discard.

Re: [linux-lvm] lvremove does not pass discards if volume is part of thin pool

[vaLentin chernoZemski]

> Plese verify your kernel has this commit:
> 19fa1a6756e ("dm thin: fix discard support to a previously shared block")
>
> But it doesn't look like you're using snapshots so this may not matter.

The kernel we are using includes the changes listed in that commit.

> If you do have the patch I referenced above then something else is going
> on.  You should probably run with: lvremove -vvvv to see if lvm is
> actually issuing a discard.  Or you could use blktrace to see if the
> thin device you're removing is actually receiving a discard.

lsblk -D shows DISC-ZERO as 1 for loop dev, thingroup_tmeta and 
thingroup_tdata

However it shows DISC-ZERO as 0 for thingroup-tpool in both tmeta and 
tdata sections and all its child devices.

It appears to me that for some reason device mapper or kernel (not sure 
which should do that) are not advertising _tpool_ tmeta and tdata 
devices to support discards (as confirmed by lsblk). That's why during 
lvremove lvm skips issuing discards on those devices.

The only references in lvremove -f -vvvv that are stating discards are those

#libdm-deptree.c:2681     Suppressed testgroup-thingroup (253:36) 
identical table reload.
#ioctl/libdm-iface.c:1795         dm status   (253:35) ON   [16384] (*1)
#libdm-deptree.c:1444         Thin pool transaction id: 3 status: 3 
32/2560 1679/160928 - rw discard_passdown.
#ioctl/libdm-iface.c:1795         dm message   (253:35) OF  delete 1 
[16384] (*1)
#ioctl/libdm-iface.c:1795         dm message   (253:35) OF 
set_transaction_id 3 4 [16384] (*1)
#ioctl/libdm-iface.c:1795         dm status   (253:35) ON   [16384] (*1)
#libdm-deptree.c:1444         Thin pool transaction id: 4 status: 4 
18/2560 0/160928 - rw discard_passdown.
#activate/dev_manager.c:3127         Creating CLEAN tree for thingroup.
#activate/dev_manager.c:1789         Getting device info for 
testgroup-thingroup 
[LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool]
#ioctl/libdm-iface.c:1795         dm info 
LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool OF 
   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:36) OF   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:35) OF   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:34) OF   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:33) OF   [16384] (*1)

Those are the name of the devices with those minor major numbers:

testgroup-thingroup (253:36)
  `-testgroup-thingroup-tpool (253:35)
     |-testgroup-thingroup_tdata (253:34)
     |  `- (7:2)
     `-testgroup-thingroup_tmeta (253:33)
        `- (7:2)
testgroup-testvol (253:37)
  `-testgroup-thingroup-tpool (253:35)
     |-testgroup-thingroup_tdata (253:34)
     |  `- (7:2)
     `-testgroup-thingroup_tmeta (253:33)
        `- (7:2)

And this is the output of lsblk -D

NAME                                                       DISC-ALN 
DISC-GRAN DISC-MAX DISC-ZERO
|-testgroup-thingroup_tmeta (dm-33)                               0 
    4K       4G         1
| `-testgroup-thingroup-tpool (dm-35)                             0 
   64K      64K         0
|   |-testgroup-thingroup (dm-36)                                 0 
   64K      64K         0
|   `-testgroup-testvol (dm-37)                                   0 
   64K      64K         0
`-testgroup-thingroup_tdata (dm-34)                               0 
    4K       4G         1
   `-testgroup-thingroup-tpool (dm-35)                             0 
    64K      64K         0
     |-testgroup-thingroup (dm-36)                                 0 
    64K      64K         0
     `-testgroup-testvol (dm-37)                                   0 
    64K      64K         0

I am yet to test with latest kernel, libdevmapper, and lvm2 but if you 
have any other ideas I will be happy to hear.

Sincerely,

vaLentin

Re: [linux-lvm] lvremove does not pass discards if volume is part of thin pool

[Mike Snitzer]

On Tue, Aug 11 2015 at  4:07am -0400,
vaLentin chernoZemski <valentin@siteground.com> wrote:

> >Plese verify your kernel has this commit:
> >19fa1a6756e ("dm thin: fix discard support to a previously shared block")
> >
> >But it doesn't look like you're using snapshots so this may not matter.
> 
> The kernel we are using includes the changes listed in that commit.
> 
> >If you do have the patch I referenced above then something else is going
> >on.  You should probably run with: lvremove -vvvv to see if lvm is
> >actually issuing a discard.  Or you could use blktrace to see if the
> >thin device you're removing is actually receiving a discard.
> 
> lsblk -D shows DISC-ZERO as 1 for loop dev, thingroup_tmeta and
> thingroup_tdata
> 
> However it shows DISC-ZERO as 0 for thingroup-tpool in both tmeta
> and tdata sections and all its child devices.

DISC-ZERO is discard_zeroes_data.  DM thinp disables that.  It doesn't
mean discard aren't enabled.  DISC-MAX and DISC-GRAN would need to be 0
for discards to be disabled.
 
> It appears to me that for some reason device mapper or kernel (not
> sure which should do that) are not advertising _tpool_ tmeta and
> tdata devices to support discards (as confirmed by lsblk). That's
> why during lvremove lvm skips issuing discards on those devices.

Nope, that isn't it.  The pool and thin device are advertising
discards.  You should verify that the pool is configured to passdown
discards to the underlying loop device.  Run: dmsetup table
You should see 'discard_passdown' -- which gets enabled by default if
the thin-pool's underlying data device supports discards.

> The only references in lvremove -f -vvvv that are stating discards are those
> 
> #libdm-deptree.c:2681     Suppressed testgroup-thingroup (253:36)
> identical table reload.
> #ioctl/libdm-iface.c:1795         dm status   (253:35) ON   [16384] (*1)
> #libdm-deptree.c:1444         Thin pool transaction id: 3 status: 3
> 32/2560 1679/160928 - rw discard_passdown.
> #ioctl/libdm-iface.c:1795         dm message   (253:35) OF  delete 1
> [16384] (*1)
> #ioctl/libdm-iface.c:1795         dm message   (253:35) OF
> set_transaction_id 3 4 [16384] (*1)
> #ioctl/libdm-iface.c:1795         dm status   (253:35) ON   [16384] (*1)
> #libdm-deptree.c:1444         Thin pool transaction id: 4 status: 4
> 18/2560 0/160928 - rw discard_passdown.
> #activate/dev_manager.c:3127         Creating CLEAN tree for thingroup.
> #activate/dev_manager.c:1789         Getting device info for
> testgroup-thingroup [LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool]
> #ioctl/libdm-iface.c:1795         dm info LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool
> OF   [16384] (*1)
> #ioctl/libdm-iface.c:1795         dm deps   (253:36) OF   [16384] (*1)
> #ioctl/libdm-iface.c:1795         dm deps   (253:35) OF   [16384] (*1)
> #ioctl/libdm-iface.c:1795         dm deps   (253:34) OF   [16384] (*1)
> #ioctl/libdm-iface.c:1795         dm deps   (253:33) OF   [16384] (*1)

Looking above it is clear that discard_passdown _is_ enabled.

I'll have to defer to the lvm2 developers, I thought we added explicit
logging when lvm2 issues discards (as part of lvremove, etc) -- Peter,
and/or Alasdair?

Re: [linux-lvm] lvremove does not pass discards if volume is part of thin pool

[vaLentin chernoZemski]

Thanks for getting back to me again Mike.

On 08/11/15 17:35, Mike Snitzer wrote:
> On Tue, Aug 11 2015 at  4:07am -0400,
> vaLentin chernoZemski <valentin@siteground.com> wrote:
>
>>> Plese verify your kernel has this commit:
>>> 19fa1a6756e ("dm thin: fix discard support to a previously shared block")
>>>
>>> But it doesn't look like you're using snapshots so this may not matter.
>>
>> The kernel we are using includes the changes listed in that commit.
>>
>>> If you do have the patch I referenced above then something else is going
>>> on.  You should probably run with: lvremove -vvvv to see if lvm is
>>> actually issuing a discard.  Or you could use blktrace to see if the
>>> thin device you're removing is actually receiving a discard.
>>
>> lsblk -D shows DISC-ZERO as 1 for loop dev, thingroup_tmeta and
>> thingroup_tdata
>>
>> However it shows DISC-ZERO as 0 for thingroup-tpool in both tmeta
>> and tdata sections and all its child devices.
>
> DISC-ZERO is discard_zeroes_data.  DM thinp disables that.  It doesn't
> mean discard aren't enabled.  DISC-MAX and DISC-GRAN would need to be 0
> for discards to be disabled.

Understood.

>
>> It appears to me that for some reason device mapper or kernel (not
>> sure which should do that) are not advertising _tpool_ tmeta and
>> tdata devices to support discards (as confirmed by lsblk). That's
>> why during lvremove lvm skips issuing discards on those devices.
>
> Nope, that isn't it.  The pool and thin device are advertising
> discards.  You should verify that the pool is configured to passdown
> discards to the underlying loop device.  Run: dmsetup table
> You should see 'discard_passdown' -- which gets enabled by default if
> the thin-pool's underlying data device supports discards.

passdown was already set in lvm.conf. I forgot to mention that:

Here is a snip:

grep passdown /etc/lvm/lvm.conf
     # Select one of  "ignore", "nopassdown", "passdown"
     thin_pool_discards = "passdown"

dmsetup table yields the following for testgroup:

testgroup-thingroup: 0 20598784 linear 253:35 0
testgroup-thingroup-tpool: 0 20598784 thin-pool 253:33 253:34 128 0 1 
skip_block_zeroing
testgroup-thingroup_tdata: 0 20598784 linear 7:2 24576
testgroup-thingroup_tmeta: 0 20480 linear 7:2 20623360
testgroup-testvol: 0 409600 thin 253:35 1

>> The only references in lvremove -f -vvvv that are stating discards are those
>>
>> #libdm-deptree.c:2681     Suppressed testgroup-thingroup (253:36)
>> identical table reload.
>> #ioctl/libdm-iface.c:1795         dm status   (253:35) ON   [16384] (*1)
>> #libdm-deptree.c:1444         Thin pool transaction id: 3 status: 3
>> 32/2560 1679/160928 - rw discard_passdown.
>> #ioctl/libdm-iface.c:1795         dm message   (253:35) OF  delete 1
>> [16384] (*1)
>> #ioctl/libdm-iface.c:1795         dm message   (253:35) OF
>> set_transaction_id 3 4 [16384] (*1)
>> #ioctl/libdm-iface.c:1795         dm status   (253:35) ON   [16384] (*1)
>> #libdm-deptree.c:1444         Thin pool transaction id: 4 status: 4
>> 18/2560 0/160928 - rw discard_passdown.
>> #activate/dev_manager.c:3127         Creating CLEAN tree for thingroup.
>> #activate/dev_manager.c:1789         Getting device info for
>> testgroup-thingroup [LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool]
>> #ioctl/libdm-iface.c:1795         dm info LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool
>> OF   [16384] (*1)
>> #ioctl/libdm-iface.c:1795         dm deps   (253:36) OF   [16384] (*1)
>> #ioctl/libdm-iface.c:1795         dm deps   (253:35) OF   [16384] (*1)
>> #ioctl/libdm-iface.c:1795         dm deps   (253:34) OF   [16384] (*1)
>> #ioctl/libdm-iface.c:1795         dm deps   (253:33) OF   [16384] (*1)
>
> Looking above it is clear that discard_passdown _is_ enabled.

Got that.

> I'll have to defer to the lvm2 developers, I thought we added explicit
> logging when lvm2 issues discards (as part of lvremove, etc) -- Peter,
> and/or Alasdair?

Once again I want to mention that discard issued by deleting file in a 
mount point reaches blockdev and sparse file shrinks in size just as 
expected.

However the problem is that if I am using lvremove even there are no 
parent snapshots, size of sparse file remains unchanged.

Let me know if there is anything else I can try or if I can pass you 
additional information.

Still did not have the chance to test all latest versions of kernel dm, 
dm libs and lvm.

vaLentin

Re: [linux-lvm] lvremove does not pass discards if volume is part of thin pool

[Peter Rajnoha]

On 08/11/2015 04:35 PM, Mike Snitzer wrote:
> I'll have to defer to the lvm2 developers, I thought we added explicit
> logging when lvm2 issues discards (as part of lvremove, etc) -- Peter,
> and/or Alasdair?
> 

Yes, we log that like (visible in the verbose -vvv output):

  "Discarding X extents offset Y sectors on /dev/Z."

Also, when removing a volume, we issue this message:

# lvremove vg1/lvol0
Do you really want to remove and DISCARD logical volume lvol0? [y/n]: 

For this, the "devices/issue_discards=1" must be set - it's not
by default.

-- 
Peter

Re: [linux-lvm] lvremove does not pass discards if volume is part of thin pool

[Peter Rajnoha]

On 08/12/2015 09:46 AM, Peter Rajnoha wrote:
> On 08/11/2015 04:35 PM, Mike Snitzer wrote:
>> I'll have to defer to the lvm2 developers, I thought we added explicit
>> logging when lvm2 issues discards (as part of lvremove, etc) -- Peter,
>> and/or Alasdair?
>>
> 
> Yes, we log that like (visible in the verbose -vvv output):
> 
>   "Discarding X extents offset Y sectors on /dev/Z."
> 
> Also, when removing a volume, we issue this message:
> 
> # lvremove vg1/lvol0
> Do you really want to remove and DISCARD logical volume lvol0? [y/n]: 
> 
> For this, the "devices/issue_discards=1" must be set - it's not
> by default.
> 

Well, sorry, this applies for usual PVs, not thin pools (I overlooked
the $subject and just saw the last question...)

-- 
Peter

Re: [linux-lvm] lvremove does not pass discards if volume is part of thin pool

[vaLentin chernoZemski]

Hello Mike, Peter,

I just tested my case with LVM2.2.02.128 with the latest kernel 4.1.6 
and it still does not work.

To avoid overloading this thread with logs information was added in the 
following paste:

http://pastebin.com/7qxmUCyb

Let me know if you need anything else that could help us debug this 
issue and fix it.

Thank you in advance.

Sincerely,

vaLentin

On 08/11/15 17:35, Mike Snitzer wrote:
> On Tue, Aug 11 2015 at  4:07am -0400,
> vaLentin chernoZemski <valentin@siteground.com> wrote:
>
>>> Plese verify your kernel has this commit:
>>> 19fa1a6756e ("dm thin: fix discard support to a previously shared block")
>>>
>>> But it doesn't look like you're using snapshots so this may not matter.
>>
>> The kernel we are using includes the changes listed in that commit.
>>
>>> If you do have the patch I referenced above then something else is going
>>> on.  You should probably run with: lvremove -vvvv to see if lvm is
>>> actually issuing a discard.  Or you could use blktrace to see if the
>>> thin device you're removing is actually receiving a discard.
>>
>> lsblk -D shows DISC-ZERO as 1 for loop dev, thingroup_tmeta and
>> thingroup_tdata
>>
>> However it shows DISC-ZERO as 0 for thingroup-tpool in both tmeta
>> and tdata sections and all its child devices.
>
> DISC-ZERO is discard_zeroes_data.  DM thinp disables that.  It doesn't
> mean discard aren't enabled.  DISC-MAX and DISC-GRAN would need to be 0
> for discards to be disabled.
>
>> It appears to me that for some reason device mapper or kernel (not
>> sure which should do that) are not advertising _tpool_ tmeta and
>> tdata devices to support discards (as confirmed by lsblk). That's
>> why during lvremove lvm skips issuing discards on those devices.
>
> Nope, that isn't it.  The pool and thin device are advertising
> discards.  You should verify that the pool is configured to passdown
> discards to the underlying loop device.  Run: dmsetup table
> You should see 'discard_passdown' -- which gets enabled by default if
> the thin-pool's underlying data device supports discards.
>
>> The only references in lvremove -f -vvvv that are stating discards are those
>>
>> #libdm-deptree.c:2681     Suppressed testgroup-thingroup (253:36)
>> identical table reload.
>> #ioctl/libdm-iface.c:1795         dm status   (253:35) ON   [16384] (*1)
>> #libdm-deptree.c:1444         Thin pool transaction id: 3 status: 3
>> 32/2560 1679/160928 - rw discard_passdown.
>> #ioctl/libdm-iface.c:1795         dm message   (253:35) OF  delete 1
>> [16384] (*1)
>> #ioctl/libdm-iface.c:1795         dm message   (253:35) OF
>> set_transaction_id 3 4 [16384] (*1)
>> #ioctl/libdm-iface.c:1795         dm status   (253:35) ON   [16384] (*1)
>> #libdm-deptree.c:1444         Thin pool transaction id: 4 status: 4
>> 18/2560 0/160928 - rw discard_passdown.
>> #activate/dev_manager.c:3127         Creating CLEAN tree for thingroup.
>> #activate/dev_manager.c:1789         Getting device info for
>> testgroup-thingroup [LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool]
>> #ioctl/libdm-iface.c:1795         dm info LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool
>> OF   [16384] (*1)
>> #ioctl/libdm-iface.c:1795         dm deps   (253:36) OF   [16384] (*1)
>> #ioctl/libdm-iface.c:1795         dm deps   (253:35) OF   [16384] (*1)
>> #ioctl/libdm-iface.c:1795         dm deps   (253:34) OF   [16384] (*1)
>> #ioctl/libdm-iface.c:1795         dm deps   (253:33) OF   [16384] (*1)
>
> Looking above it is clear that discard_passdown _is_ enabled.
>
> I'll have to defer to the lvm2 developers, I thought we added explicit
> logging when lvm2 issues discards (as part of lvremove, etc) -- Peter,
> and/or Alasdair?
>

Re: [linux-lvm] lvremove does not pass discards if volume is part of thin pool

[Tomas Janousek]

Hello all,

On Wed, Aug 19, 2015 at 01:40:08PM +0300, vaLentin chernoZemski wrote:
> I just tested my case with LVM2.2.02.128 with the latest kernel 4.1.6 and it
> still does not work.
> 
> [...]
> 
> Let me know if you need anything else that could help us debug this issue
> and fix it.

Has there been any progress here? I'm experiencing the same issue here with
kernel 4.4.0 and lvm:

lvm> version
  LVM version:     2.02.138(2) (2015-12-14)
  Library version: 1.02.114 (2015-12-14)
  Driver version:  4.34.0

Is there perhaps some way to force the discard manually? I thought that maybe
creating a new sparse file/loop device/physical volume and doing pvmove might
help, but what I got instead was a file that wasn't sparse at all, actually
taking more space than the original one. Is this expected?
(The only explanation I can think of is that pvmove doesn't care nor
understand the on-disk format of device mapper thin-pools so it has to copy
everything byte by byte. Is that it?)

Maybe creating a sufficiently large volume in the thin pool, doing a dd
if=/dev/zero of=/path/to/volume/tmp/blanks and then deleting
/path/to/volume/tmp/blanks would help, but could be quite slow. :-(

Regards,
-- 
Tomáš Janoušek, a.k.a. Pivník, a.k.a. Liskni_si, http://work.lisk.in/

Re: [linux-lvm] lvremove does not pass discards if volume is part of thin pool

[Tomas Janousek]

Hello all again,

I will answer one of my questions myself as I just found the answer:

On Fri, Jan 22, 2016 at 06:18:29PM +0100, Tomas Janousek wrote:
> Is there perhaps some way to force the discard manually? I thought that maybe
> creating a new sparse file/loop device/physical volume and doing pvmove might
> help, but what I got instead was a file that wasn't sparse at all, actually
> taking more space than the original one. Is this expected?
> (The only explanation I can think of is that pvmove doesn't care nor
> understand the on-disk format of device mapper thin-pools so it has to copy
> everything byte by byte. Is that it?)
> 
> Maybe creating a sufficiently large volume in the thin pool, doing a dd
> if=/dev/zero of=/path/to/volume/tmp/blanks and then deleting
> /path/to/volume/tmp/blanks would help, but could be quite slow. :-(

This hack with dd if=/dev/zero works indeed, but there's a tool called
blkdiscard in util-linux which works much better. I just call it prior to
lvremove (or lxc-destroy in my case) and the space is freed.

-- 
Tomáš Janoušek, a.k.a. Pivník, a.k.a. Liskni_si, http://work.lisk.in/