pvmove thin volume doesn't move

pvmove thin volume doesn't move

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 1647 bytes --]

Hi.

I've added a new (faster) disk to a VG and now want to move a thinly
provisioned LV to the new PV.  When I try it doesn't find any data to
move though:

# pvmove -n /dev/rootvol_tmp/var /dev/sda7 /dev/sdc
  No data to move for rootvol_tmp.

both disks are in the VG though:

# pvs
  PV         VG            Fmt  Attr PSize    PFree   
  /dev/sda7  rootvol_tmp   lvm2 a--   170.89g    3.62g
  /dev/sdc   rootvol_tmp   lvm2 a--  <279.46g <151.46g

# lvdisplay --maps /dev/rootvol_tmp/var 
  --- Logical volume ---
  LV Path                /dev/rootvol_tmp/var
  LV Name                var
  VG Name                rootvol_tmp
  LV UUID                jurBnO-APSJ-fbp1-DhWZ-8475-vrDh-heccun
  LV Write Access        read/write
  LV Creation host, time localhost, 2019-11-10 11:11:17 -0500
  LV Pool name           pool00
  LV Status              available
  # open                 1
  LV Size                55.27 GiB
  Mapped size            96.94%
  Current LE             14150
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:17
   
  --- Segments ---
  Virtual extents 0 to 14149:
    Type                thin
    Device ID           3

The LV is a thinly provisioned LV:

# lvs
  LV              VG            Attr       LSize    Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert
  pool00          rootvol_tmp   twi-aotz-- <138.88g               80.32  40.28                           
  var             rootvol_tmp   Vwi-aotz--   55.27g pool00        96.94

Any ideas?

Cheers,
b.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 516 bytes --]

Re: pvmove thin volume doesn't move

[Zdenek Kabelac]

Dne 16. 11. 25 v 14:27 Brian J. Murrell napsal(a):
> Hi.
> 
> I've added a new (faster) disk to a VG and now want to move a thinly
> provisioned LV to the new PV.  When I try it doesn't find any data to
> move though:
> 
> # pvmove -n /dev/rootvol_tmp/var /dev/sda7 /dev/sdc
>    No data to move for rootvol_tmp.
> 
> both disks are in the VG though:


Hi

lvm2 cannot move 'individual' thin volume as this volume is 'just virtual' and 
uses chunks from the thin-pool.

So if you want use thinLV from a new faster storage - then whole thin-pool 
must be moved to the faster storage.

And when you will be doing this - move the also _tmeta volume.

(See 'lvs -a'  output for volumes you can move)

Regards

Zdenek

Re: pvmove thin volume doesn't move

[Brian J. Murrell]

Pity.  I don't really want to move all of the LVs in the pool.

Can I create a new pool on the new PV and move just the one LV from one pre-existing pool to the new pool?

Cheers,
b.

Re: pvmove thin volume doesn't move

[Zdenek Kabelac]

Dne 18. 11. 25 v 0:30 Brian J. Murrell napsal(a):
> Pity.  I don't really want to move all of the LVs in the pool.
> 
> Can I create a new pool on the new PV and move just the one LV from one pre-existing pool to the new pool?
> 
> Cheers,
> b.
> 
> 

Hi

You can always create 'new LV' on the PV you prefer to use and then with good 
old 'dd'  command copy deviceA->deviceB - preferably while device your are 
copying is not in use (i.e. if you are copying LV which holds your filesystem 
- the partition should not be mounted) - once all your data are copied you 
just reconfigure fstab to mount different new LV.

lvm2 does not (ATM) support this kind of operation.

Regards

Zdenek

Re: pvmove thin volume doesn't move

[Brian J. Murrell]

On Tue, 2025-11-18 at 00:37 +0100, Zdenek Kabelac wrote:
> 
> 
> Hi

Hi!

> You can always create 'new LV' on the PV you prefer to use and then
> with good 
> old 'dd'  command copy deviceA->deviceB - preferably while device
> your are 
> copying is not in use (i.e. if you are copying LV which holds your
> filesystem 
> - the partition should not be mounted) - once all your data are
> copied you 
> just reconfigure fstab to mount different new LV.

Yes, indeed, of course that is one way.  And I do appreciate the effort
of you taking the time to suggest that.  But not having to do these
kinds of off-line bulk data copy operations is one of the major
attractions of LVM.  Once we have to start reverting back to the more
cumbersome and interrupting methods the added complexity of LVM starts
to lose it's attraction and value.

> lvm2 does not (ATM) support this kind of operation.

Pity that one has to choose between efficient space use and efficient
snapshots or being able to migrate data live but cannot have both.  :-(

Is being able to do this on the roadmap at least or will this always
remain a fundamental limitation?

Cheers,
b.

Re: pvmove thin volume doesn't move

[Zdenek Kabelac]

Dne 19. 11. 25 v 0:14 Brian J. Murrell napsal(a):
> On Tue, 2025-11-18 at 00:37 +0100, Zdenek Kabelac wrote:
>>
>>
>> Hi
> 
> Hi!
> 
>> You can always create 'new LV' on the PV you prefer to use and then
>> with good
>> old 'dd'  command copy deviceA->deviceB - preferably while device
>> your are
>> copying is not in use (i.e. if you are copying LV which holds your
>> filesystem
>> - the partition should not be mounted) - once all your data are
>> copied you
>> just reconfigure fstab to mount different new LV.
> 
> Yes, indeed, of course that is one way.  And I do appreciate the effort
> of you taking the time to suggest that.  But not having to do these
> kinds of off-line bulk data copy operations is one of the major
> attractions of LVM.  Once we have to start reverting back to the more
> cumbersome and interrupting methods the added complexity of LVM starts
> to lose it's attraction and value.
> 
>> lvm2 does not (ATM) support this kind of operation.
> 
> Pity that one has to choose between efficient space use and efficient
> snapshots or being able to migrate data live but cannot have both.  :-(
> 
> Is being able to do this on the roadmap at least or will this always
> remain a fundamental limitation?

Hi

Well AFAIK you are 'lonely' person wanting this.
Usually once user goes with thin-pools - it's decision taken with knowledge 
that the only way out is to take out whole data out of thin-pool and copy them 
to the new place - there is obviously no point on 'picking individual 
thin-chunks out of thin-pool - move them out of thin-pool and reshuffling 
whole thin-pool (and thin-chunks are i.e. 64K while our extents are typical >4M)

So if user wants to 'accelerate' usage of thin-pool (if they still use archaic 
hdd) - they can always cache thin-pool data volume as such on SSD/nvme - 
giving performance boost to the whole thin-pool usage.

So ATM this feature is not really on the roadmap - although there would be 
probably not too complicated path of allowing to 'install' raid on thin LV - 
let the raid do its mirroring and then split raid and eventually drop thinLV 
relaxing the space in thin-pool - I think there was some effort already made 
towards this LV conning support - but it's low on the priority list.


Regards

Zdenek

Re: pvmove thin volume doesn't move

[Matthew Patton]

<quote>
Usually once user goes with thin-pools - it's decision taken with knowledge that the only way out is to take out whole data out of thin-pool and copy them to the new place
</quote>

That is one hell of a wild and unsupported assumption. Practically nobody , even seasoned sysadmins would know of this deficiency. Does the lvm(1) manage splatter this warning prominently?

Pretty sure the HP/ux and IBM implementations support thin movement.

Re: pvmove thin volume doesn't move

[Brian J. Murrell]

On Wed, 2025-11-19 at 09:07 -0500, Matthew Patton wrote:
> 
> That is one hell of a wild and unsupported assumption.

Yeah, I too was going to challenge that statement that I was a "lonely"
outlier in the community of LVM users that would want to be able to
move a thinly-provisioned LV from one PV to another, just as non-
thinly-provisioned LVs can be.

> Practically nobody , even seasoned sysadmins would know of this
> deficiency.

Indeed, I was skeptical that this was commonly understood as well. 
Well by anyone who had not tried it as I did only to find it is not
supported.

It does suck to have to choose between feature sets that are both very
useful.

Cheers,
b.

Re: pvmove thin volume doesn't move

[Zdenek Kabelac]

Dne 19. 11. 25 v 15:07 Matthew Patton napsal(a):
> <quote>
> Usually once user goes with thin-pools - it's decision taken with knowledge that the only way out is to take out whole data out of thin-pool and copy them to the new place
> </quote>
> 
> That is one hell of a wild and unsupported assumption. Practically nobody , even seasoned sysadmins would know of this deficiency. Does the lvm(1) manage splatter this warning prominently?
> 
> Pretty sure the HP/ux and IBM implementations support thin movement.

Well I've no idea which volume managers you are talking about, all I say is - 
lvm2  ATM does not support any sort of 'extraction' of a thinLV out of the 
thin-pool. The main benefit of using 'thin-pool' is the you share storage 
between volumes.

As far as for documentation - I'm pretty sure we never mentioned any support 
of such operation within our lvm2 man pages.

And the initial lines in our 'man lvmthin' explicitly states this:

----
Blocks in a standard LV are allocated (during creation) from the Volume Group 
(VG), but blocks in a thin LV  are  allocated  (during  use)
from a "thin pool".  The thin pool contains blocks of physical storage, and 
thin LV blocks reference blocks in the thin pool.

----

Thin-pool is some sort of a black box from lvm2's POV - it serves volumes - 
and lvm2 knows how to make those volumes available on the system - but the 
mapping of chunks to make a thin LV as fully in hands of thin-pool target.
lvm2 has 'no idea'  which disk space is in-use for any individual thin LV.
(there are tools like 'thin_ls/thin_rmap' for that)


Regards

Zdenek

Re: pvmove thin volume doesn't move

[David Teigland]

On Sun, Nov 16, 2025 at 08:27:30AM -0500, Brian J. Murrell wrote:
> Hi.
> 
> I've added a new (faster) disk to a VG and now want to move a thinly
> provisioned LV to the new PV.  When I try it doesn't find any data to
> move though:
> 
> # pvmove -n /dev/rootvol_tmp/var /dev/sda7 /dev/sdc
>   No data to move for rootvol_tmp.

Thanks for asking, one of the reasons we don't really have a plan for
this, or other things, is that we just don't get users asking for much.
Over the past couple years I've actually noticed a lot of people looking
to copy LVs, and even some external tools that have been written to do it.
There's quite a bit of variation though, and it's often copying over the
network, or recently, copying on a SAN so another host can use the LV.
There has been some work in our group aimed at parts of the problem,
for example https://github.com/device-mapper-utils/blk-archive

So, it's been at the front of mind that LV copying in some fashion is one
of the larger missing features.  What's not clear to me is if we can find
a solution that meets enough of the varied use cases, but I'll try to
raise more discussion about it.

Dave

Re: pvmove thin volume doesn't move

[Zdenek Kabelac]

Dne 19. 11. 25 v 16:46 Brian J. Murrell napsal(a):
> On Wed, 2025-11-19 at 09:07 -0500, Matthew Patton wrote:
>>
>> That is one hell of a wild and unsupported assumption.
> 
> Yeah, I too was going to challenge that statement that I was a "lonely"
> outlier in the community of LVM users that would want to be able to
> move a thinly-provisioned LV from one PV to another, just as non-
> thinly-provisioned LVs can be.
> 
>> Practically nobody , even seasoned sysadmins would know of this
>> deficiency.
> 
> Indeed, I was skeptical that this was commonly understood as well.
> Well by anyone who had not tried it as I did only to find it is not
> supported.
> 
> It does suck to have to choose between feature sets that are both very
> useful.


There is likely a major misunderstanding how thin-provisioning works with lvm2 
(and ATM I'm not sure how we can improve our 'man lvmthin' to make this more 
clear)

It's NOT lvm2 doing 'thin-provisioning' at lvm2 level.

lvm2 just drives a kernel target device - which you can think of like would 
attach some 'magic box' that take  space  X  a gives you back space Y with 
some properties (just like when lvm2 controls  'raidX' target).

lvm2 does not see 'how the magic of provisioning' is made - it just consumes 
it - thus lvm2 itself has no knowledge about the layout of thin-chunks - which 
are often very small compared with lvm2 extent size granularity.

So lvm2 cannot move  i.e. 64KiB thin chunk blocks - when the minimal pvmove 
granularity if 4MiB.

And while I'm not saying the work cannot be done somehow - it's quite a major 
feature request for thin-pool kernel target if that should be an online operation.

To add better example - it's like if you would want to 'extract' the file from 
filesytem and let the rest of 'device' compact.
I guess everyone sticks to 'cp'  command  - which can be seen as equivalent of 
'dd' command in our block device level...



Regards

Zdenek

Re: pvmove thin volume doesn't move

[Brian J. Murrell]

> lvm2  ATM does not support any sort of 'extraction' of a thinLV out
> of the 
> thin-pool.

Maybe that's an explanation that goes over my head with me not being an
LVM2 engineer but rather just a general jack-of-all-trades sysadmin.

But that said, I'm not sure what kind of operation "extraction" is
supposed to be implying but I suppose what I was thinking (hoping for,
now it seems) was that in order to be able to move the blocks of an LV
in a thin-pool from one PV to another one would of course have to
extend the thin-pool to the new device, in the same way that any LV can
span devices.

> The main benefit of using 'thin-pool' is the you share storage 
> between volumes.

And that snapshots of (thinly provisioned) LVs don't degrade
performance.

> As far as for documentation - I'm pretty sure we never mentioned any
> support 
> of such operation within our lvm2 man pages.

pvmove's manpage does describe the ability to move LVs among PVs. 
That, IMO, implies thin-LVs without any kind of disclaimer saying
otherwise.

> Blocks in a standard LV are allocated (during creation) from the
> Volume Group 
> (VG), but blocks in a thin LV  are  allocated  (during  use)
> from a "thin pool".  The thin pool contains blocks of physical
> storage, and 
> thin LV blocks reference blocks in the thin pool.

That sounds like a good description for an LVM2 software engineer
onboarding to thin provisioning.  I don't think that description is
suitable to express the limitations (that are becoming apparent in this
thread) to general jack-of-all-trades system administrators as it
implies more knowledge of LVM2 than such a person could/would generally
have IMO.

> Thin-pool is some sort of a black box from lvm2's POV - it serves
> volumes - 
> and lvm2 knows how to make those volumes available on the system -
> but the 
> mapping of chunks to make a thin LV as fully in hands of thin-pool
> target.
> lvm2 has 'no idea'  which disk space is in-use for any individual
> thin LV.
> (there are tools like 'thin_ls/thin_rmap' for that)

Again, much too technically detailed to express these limitations to a
general sysadmin that has to employ literally dozens of technologies in
one's administrative scope and therefore cannot possibly try understand
all of the underlying design decisions of every one of those dozens of
technologies.

Cheers,
b.

Re: pvmove thin volume doesn't move

[Zdenek Kabelac]

Dne 19. 11. 25 v 17:36 Brian J. Murrell napsal(a):
>> lvm2  ATM does not support any sort of 'extraction' of a thinLV out
>> of the
>> thin-pool.
> 
> Maybe that's an explanation that goes over my head with me not being an
> LVM2 engineer but rather just a general jack-of-all-trades sysadmin.
> 
> But that said, I'm not sure what kind of operation "extraction" is
> supposed to be implying but I suppose what I was thinking (hoping for,
> now it seems) was that in order to be able to move the blocks of an LV
> in a thin-pool from one PV to another one would of course have to
> extend the thin-pool to the new device, in the same way that any LV can
> span devices.

Ok - now I start to see - where we need to enhance our documentation.

So here - 'adding' different space to thin-pool wouldn't not be a good idea.
We will likely need to add at least 'prompt' protection if user would try to 
extend  'hdd'  based LV used for thin-pool data with i.e. non-rotational
(we are in planning phase for new allocation - which will be dealing with this 
- but we should add some protection to existing code base).

thin-pool does it's own decision where it allocates 'chunks' for thin LV and 
for thin-pool itself given space is 'one big device'


> 
>> As far as for documentation - I'm pretty sure we never mentioned any
>> support
>> of such operation within our lvm2 man pages.
> 
> pvmove's manpage does describe the ability to move LVs among PVs.
> That, IMO, implies thin-LVs without any kind of disclaimer saying
> otherwise.
> 

Ok - I'll try to add more details about 'virtual LVs' - which are thinLVs and 
vdoLVs  (and we have even combination of both)

Here we surely can add more disclaimers about the major difference between 
virtual and real LVs.


>> Blocks in a standard LV are allocated (during creation) from the
>> Volume Group
>> (VG), but blocks in a thin LV  are  allocated  (during  use)
>> from a "thin pool".  The thin pool contains blocks of physical
>> storage, and
>> thin LV blocks reference blocks in the thin pool.
> 
> That sounds like a good description for an LVM2 software engineer
> onboarding to thin provisioning.  I don't think that description is
> suitable to express the limitations (that are becoming apparent in this
> thread) to general jack-of-all-trades system administrators as it
> implies more knowledge of LVM2 than such a person could/would generally
> have IMO.

We surely always try to add the best details we can.
But there is also some 'expectancy' user is somewhat knowledgeable.
The performance of thin-pool ain't for free - so there are some drawbacks when 
user settles with thin-pools.

> Again, much too technically detailed to express these limitations to a
> general sysadmin that has to employ literally dozens of technologies in
> one's administrative scope and therefore cannot possibly try understand
> all of the underlying design decisions of every one of those dozens of
> technologies.

Like our error messaging - in case it makes no good explanation to the user 
should be always reported and requested for enhancement.

Regards

Zdenek

Re: pvmove thin volume doesn't move

[matthew patton]

Right,  so LVMThin(1) conveniently omits "Oh BTW standard functionality like pvmove() is not implemented."

lvmthin is a sub-set or sub-class of LVM, therefore any reasonable person would assume all functionality cascades down. Because it pretty much does for everything else.

> lvm2 has 'no idea'  which disk space is in-use for any individual thin LV.
> (there are tools like 'thin_ls/thin_rmap' for that)

And any reasonable person would assume lvmthin would have implemented the necessary hooks so that pvmove() would work by lvmthin handing lvm back a linked-list of all of the pieces in the correct order so they could be written to lvm(thick) or to another lvmthin pool. This is not that big of a cognitive leap.

That Linux LVM thin is missing "obvious" functionality is fine, it just needs to be stated very clearly.

Re: pvmove thin volume doesn't move

[Zdenek Kabelac]

Dne 19. 11. 25 v 18:22 matthew patton napsal(a):
> Right,  so LVMThin(1) conveniently omits "Oh BTW standard functionality like pvmove() is not implemented."
> 
> lvmthin is a sub-set or sub-class of LVM, therefore any reasonable person would assume all functionality cascades down. Because it pretty much does for everything else.
> 
>> lvm2 has 'no idea'  which disk space is in-use for any individual thin LV.
>> (there are tools like 'thin_ls/thin_rmap' for that)
> 
> And any reasonable person would assume lvmthin would have implemented the necessary hooks so that pvmove() would work by lvmthin handing lvm back a linked-list of all of the pieces in the correct order so they could be written to lvm(thick) or to another lvmthin pool. This is not that big of a cognitive leap.
> 
> That Linux LVM thin is missing "obvious" functionality is fine, it just needs to be stated very clearly.


Yep  thanks this is a  legit complain, we will improve our doc to be way more 
clear about this.

I've been also thinking - that adding support of 'dumb' mirroring of thinLV 
into pvmove - isn't a very difficult request - so we will see - although 
obviously copying  10TiB empty thin volume isn't going to be 'a fast 
operation' this way - but it's not any worse then a 'dd'.


Regards

Zdenek

Re: pvmove thin volume doesn't move

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 826 bytes --]

On Wed, 2025-11-19 at 18:31 +0100, Zdenek Kabelac wrote:
> 
> I've been also thinking - that adding support of 'dumb' mirroring of
> thinLV 
> into pvmove - isn't a very difficult request - so we will see -
> although 
> obviously copying  10TiB empty thin volume isn't going to be 'a fast 
> operation' this way - but it's not any worse then a 'dd'.

Not any worse than 'dd' would be more than acceptable.  How can
anything be faster at physically moving data than dd as it is usually
only slowed by the speed of the storage in any case?

Slower, (to a reasonable degree) than dd would even be acceptable if
it's atomic and online like existing pvmove of real LVs since it being
online means nobody is sitting around waiting for it.  So it can be
slower even (again, to a reasonable degree).

Cheers,
b.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: pvmove thin volume doesn't move

[Zdenek Kabelac]

Dne 19. 11. 25 v 18:38 Brian J. Murrell napsal(a):
> On Wed, 2025-11-19 at 18:31 +0100, Zdenek Kabelac wrote:
>>
>> I've been also thinking - that adding support of 'dumb' mirroring of
>> thinLV
>> into pvmove - isn't a very difficult request - so we will see -
>> although
>> obviously copying  10TiB empty thin volume isn't going to be 'a fast
>> operation' this way - but it's not any worse then a 'dd'.
> 
> Not any worse than 'dd' would be more than acceptable.  How can
> anything be faster at physically moving data than dd as it is usually
> only slowed by the speed of the storage in any case?

If the tool would have known 'which areas' are  mapped  (which knows thin-pool 
target internally) then it would need to copy only those blocks.
(this can be just small fraction of the hole volume size)

On a device level - we see 'just' zero areas - but those need to be mirrored - 
as on this level there no difference between 'zero' and 'unwritten' data.

(Btw with this knowledge the Joe's  blk-archive tools is working)

Zdenek

Re: pvmove thin volume doesn't move

[matthew patton]

> If the tool would have known 'which areas' are  mapped  (which knows thin-pool 
> target internally) then it would need to copy only those blocks.

no doubt. but if lvmthin is basically a private implementation that LVM (aka thick) doesn't actually know anything about and is just being used as a pass-thru to thin API, I'm not sure we want to expose thin internals to the caller. I obviously haven't read the code implementing either thick or thin, but if thick does a thin_read(one 4MB extent) then thin should just return a buffer with 4mb populated with all the data converted into a thick+linear representation that Thick is expecting. Then the traditional workflow can resume with the extent written out to its destination PV. In other words you're hydrating a thin representation into a thick representation. Could you take that buffer and thin_write(dest thin pool)? I don't see why not.

Or you could just punt and say pvmove() of a thin is necessarily a hydrate operation and can only be written out into a non-then PV, tough luck. Use offline `dd`if you want more.

I haven't been particularly impressed by LVM caching (it has it's uses don't get me wrong) but I find layering open-cas to be more intuitive and gives me a degree of freedom.

Re: pvmove thin volume doesn't move

[Zdenek Kabelac]

Dne 19. 11. 25 v 20:41 matthew patton napsal(a):
>> If the tool would have known 'which areas' are  mapped  (which knows thin-pool
>> target internally) then it would need to copy only those blocks.
> 
> no doubt. but if lvmthin is basically a private implementation that LVM (aka thick) doesn't actually know anything about and is just being used as a pass-thru to thin API, I'm not sure we want to expose thin internals to the caller. I obviously haven't read the code implementing either thick or thin, but if thick does a thin_read(one 4MB extent) then thin should just return a buffer with 4mb populated with all the data converted into a thick+linear representation that Thick is expecting. Then the traditional workflow can resume with the extent written out to its destination PV. In other words you're hydrating a thin representation into a thick representation. Could you take that buffer and thin_write(dest thin pool)? I don't see why not.
> 


Clearly tiny thin-pool can provide a space to virtual volume of some massive 
size - i.e. you can have 10TiB LV using just couple MiB of the real physical 
storage in a VG.

So if I'd be an 'unaware' admin how this all works - I'd kind of naively 
expect then when I 'pvmove' such thinLV where the original LV takes just those 
'couple' MiB - then I'd expect that copied volume would also take take 
approximately 'same' space and you would be coping couple MiB instead of 
having an operation running for even days.

We can 'kind of script' these things nowadays offline - but making this 
'online' requires a lot of new kernel code...

So while a naive/dumb pvmove may possibly have some minor 'usage' - it has 
many 'weak' points - and plain 'dd' can be likely copy data faster - and as 
said I'm not aware users would ever requested such operation until today.

Typically admins do move the whole storage to the faster hardware - so 
thinpool with its data & metadata is moved to the new drive - which is fully 
supported online.

> Or you could just punt and say pvmove() of a thin is necessarily a hydrate operation and can only be written out into a non-then PV, tough luck. Use offline `dd`if you want more.
> 
> I haven't been particularly impressed by LVM caching (it has it's uses don't get me wrong) but I find layering open-cas to be more intuitive and gives me a degree of freedom.
> 

It's worth to note -  dm-cache target is fully customizable - so anyone can 
come with 'policies' that fits their needs -  somehow this doesn't happen and 
users mostly stick with default 'smq' policy - which is usually 'good enough' 
- but its a hot-spot cache. There is also 'writecache' which is targeted for 
heavy write loads....

If some likes OpenCas more :) obviously we can't change his mind. We've tried 
to make 'caching' simple for lvm2 users aka:
    'lvcreate --type cache -L100G  vg/lv /dev/nvme0p1'
(assuming vg was already vgextended with /dev/nvme0p1)
but surely there are many ways to skin the cat...

Regards

Zdenek