[linux-lvm] Is MD array (containing a PV) reducible?

[linux-lvm] Is MD array (containing a PV) reducible?

[Brian J. Murrell]

[-- Attachment #1.1: Type: text/plain, Size: 752 bytes --]

Hi.

Just hoping to get a sanity check here.  I have a PV on an mdadm raid
array.  the PV's size is:

  PV Size               3.64 TiB / not usable 2.84 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              953829
  Free PE               35813
  Allocated PE          918016

The mdadm array's size is:
        Array Size : 3,906,887,512 (3.64 TiB 4.00 TB)
     Used Dev Size : 3,906,887,512 (3.64 TiB 4.00 TB)

I can shrink this array down to 3,906,886,464 without impacting the PV
(and it's contents) correct?

From my calculations there is 4,022,272 bytes of unused space after the
PV to the end of the array so shrinking it by 1,073,152 shouldn't have
any impact, correct?

Cheers,
b.


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 202 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Is MD array (containing a PV) reducible?

[Phillip Susi]

"Brian J. Murrell" <brian@interlinx.bc.ca> writes:

> I can shrink this array down to 3,906,886,464 without impacting the PV
> (and it's contents) correct?

Looks that way.  That would be how much space is left in the array that
is not quite enough for another 4 MiB PE.

> From my calculations there is 4,022,272 bytes of unused space after the
> PV to the end of the array so shrinking it by 1,073,152 shouldn't have
> any impact, correct?

It shouldn't, but why bother?

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Is MD array (containing a PV) reducible?

[Brian J. Murrell]

[-- Attachment #1.1: Type: text/plain, Size: 2605 bytes --]

On Mon, 2023-10-02 at 14:49 -0400, Phillip Susi wrote:
> 
> Looks that way.  That would be how much space is left in the array
> that
> is not quite enough for another 4 MiB PE.

Good.  Thanks for the sanity check.

> It shouldn't, but why bother?

Because I need to add another member to the array that is just slightly
smaller than the existing array.  It's the same sized disk as what's in
the array currently but is itself an MD array with luks, which is
reserving it's own metadata region from the total disk making it
slightly smaller.

So yes, ultimately I am going to have both an encrypted and unencrypted
member of this raid-1 array.  At least until the encrypted member is
added and then I will remove the unencrypted member and add it to the
luks encrypted array.

The general idea is to (safely -- as in be able to achieve in specific
sepearate steps that have a back-out plan) convert a non-encrypted
raid-1 array into an encrypted raid-1 array.

So what I have right now is:
sdc                                             8:32   0   3.7T  0 disk  
└─md0                                           9:0    0   3.7T  0 raid1 
  ├─backups-backups                           253:13   0   2.5T  0 lvm   /.snapshots
  ├─backups-borg                              253:14   0   300G  0 lvm   
  └─backups-vdo--test                         253:15   0   700G  0 lvm   
    └─vdo-backups                             253:73   0   1.8T  0 vdo   
sdd                                             8:48   0   3.7T  0 disk  
└─md1                                           9:1    0   3.7T  0 raid1 
  └─luks-backup                               253:40   0   3.7T  0 crypt 

I am going to reduce the size of md0 so that it's the same size as md1,
then add md1 to md0 (so yes, I will have a raid-1 array with a raid-1
member).  Then I am going to remove md0 from the md1 array and then add
sdc to md1 so that md0 no longer exists and md1 is the primary raid-1
array.  Ideally, rename md1 to md0 once done but that's aesthetic.

Ultimately I should end up (prior to any attempt to rename) md1 with
sdd and sdc as an encrypted raid-1 array.  The goal is to be able to
remove a disk from the raid array and safely
discard/return/recycle/etc. it knowing that it's content is relatively
secure.

The most problematic scenario of course is a disk dies and you want to
return it under warranty but you cannot access the disk to "scrub" it
and you either send it back with all kinds of sensitive data on it, or
you eat it.

Cheers,
b.


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 202 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Is MD array (containing a PV) reducible?

[grumpy]

[-- Attachment #1: Type: text/plain, Size: 270 bytes --]

> The general idea is to (safely -- as in be able to achieve in specific
> sepearate steps that have a back-out plan) convert a non-encrypted
> raid-1 array into an encrypted raid-1 array.


i have a very similar setup
can you document the steps as you do the conversion

[-- Attachment #2: Type: text/plain, Size: 202 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[-- Attachment #3: Type: text/plain, Size: 202 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Is MD array (containing a PV) reducible?

[Phillip Susi]

"Brian J. Murrell" <brian@interlinx.bc.ca> writes:

> Because I need to add another member to the array that is just slightly
> smaller than the existing array.  It's the same sized disk as what's in
> the array currently but is itself an MD array with luks, which is
> reserving it's own metadata region from the total disk making it
> slightly smaller.

You want to put an MD array inside of another MD array?  Why?

> So yes, ultimately I am going to have both an encrypted and unencrypted
> member of this raid-1 array.  At least until the encrypted member is
> added and then I will remove the unencrypted member and add it to the
> luks encrypted array.
>
> The general idea is to (safely -- as in be able to achieve in specific
> sepearate steps that have a back-out plan) convert a non-encrypted
> raid-1 array into an encrypted raid-1 array.
>
> So what I have right now is:
> sdc                                             8:32   0   3.7T  0 disk  
> └─md0                                           9:0    0   3.7T  0 raid1 
>   ├─backups-backups                           253:13   0   2.5T  0 lvm   /.snapshots
>   ├─backups-borg                              253:14   0   300G  0 lvm   
>   └─backups-vdo--test                         253:15   0   700G  0 lvm   
>     └─vdo-backups                             253:73   0   1.8T  0 vdo   
> sdd                                             8:48   0   3.7T  0 disk  
> └─md1                                           9:1    0   3.7T  0 raid1 
>   └─luks-backup                               253:40   0   3.7T  0 crypt 

I would suggest that rather than

> I am going to reduce the size of md0 so that it's the same size as md1,
> then add md1 to md0 (so yes, I will have a raid-1 array with a raid-1
> member).  Then I am going to remove md0 from the md1 array and then add
> sdc to md1 so that md0 no longer exists and md1 is the primary raid-1
> array.  Ideally, rename md1 to md0 once done but that's aesthetic.

I would suggest instead that you add the luks volume as a new PV to your
VG, then use pvmove to migrate all of your logical volumes from md0 to
md1, then vgreduce md0 from the VG, stop md0, mdadm --zero-superblocks
/dev/sdc, then add sdc to md1.

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Is MD array (containing a PV) reducible?

[Brian J. Murrell]

On Mon, 2023-10-02 at 16:48 -0400, Phillip Susi wrote:
> 
> You want to put an MD array inside of another MD array?  Why?

Just to take advantage of MD's online/background sync to get the data
from md0 array to the md1 array.

> I would suggest instead that you add the luks volume as a new PV to
> your
> VG, then use pvmove to migrate all of your logical volumes from md0
> to
> md1, then vgreduce md0 from the VG, stop md0, mdadm --zero-
> superblocks
> /dev/sdc, then add sdc to md1.

That's an interesting alternative.  Thanks for the suggestion.

Cheers,
b.

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Is MD array (containing a PV) reducible?

[Phillip Susi]

"Brian J. Murrell" <brian@interlinx.bc.ca> writes:

> Just to take advantage of MD's online/background sync to get the data
> from md0 array to the md1 array.

If you weren't using LVM, then that would be a good idea, but LVM can do
that too.

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Is MD array (containing a PV) reducible?

[Brian J. Murrell]

[-- Attachment #1.1: Type: text/plain, Size: 1366 bytes --]

On Mon, 2023-10-02 at 19:37 -0400, Phillip Susi wrote:
> 
> "Brian J. Murrell" <brian@interlinx.bc.ca> writes:
> 
> > Just to take advantage of MD's online/background sync to get the
> > data
> > from md0 array to the md1 array.
> 
> If you weren't using LVM, then that would be a good idea, but LVM can
> do
> that too.

So just to confirm, given:

sdc                                             8:32   0   3.7T  0 disk  
└─md0                                           9:0    0   3.7T  0 raid1 
  ├─backups-backups                           253:13   0   2.5T  0 lvm   /.snapshots
  ├─backups-borg                              253:14   0   300G  0 lvm   
  └─backups-vdo--test                         253:15   0   700G  0 lvm   
    └─vdo-backups                             253:73   0   1.8T  0 vdo   
sdd                                             8:48   0   3.7T  0 disk  
└─md1                                           9:1    0   3.7T  0 raid1 
  └─luks-backup                               253:40   0   3.7T  0 crypt 

I want to do:

# pvcreate /dev/mapper/luks-backup
# vgextend backups /dev/mapper/luks-backup
# pvmove /dev/md0 /dev/luks-backup
# vgreduce backups /dev/md0
# mdadm --stop /dev/md0
# mdadm --zero-superblocks /dev/sdc
# mdadm /dev/md1 --add /dev/sdc

Correct?

Cheers,
b.


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 202 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Is MD array (containing a PV) reducible?

[Phillip Susi]

"Brian J. Murrell" <brian@interlinx.bc.ca> writes:

> So just to confirm, given:
>
> sdc                                             8:32   0   3.7T  0 disk  
> └─md0                                           9:0    0   3.7T  0 raid1 
>   ├─backups-backups                           253:13   0   2.5T  0 lvm   /.snapshots
>   ├─backups-borg                              253:14   0   300G  0 lvm   
>   └─backups-vdo--test                         253:15   0   700G  0 lvm   
>     └─vdo-backups                             253:73   0   1.8T  0 vdo   
> sdd                                             8:48   0   3.7T  0 disk  
> └─md1                                           9:1    0   3.7T  0 raid1 
>   └─luks-backup                               253:40   0   3.7T  0 crypt 
>
> I want to do:
>
> # pvcreate /dev/mapper/luks-backup
> # vgextend backups /dev/mapper/luks-backup
> # pvmove /dev/md0 /dev/luks-backup
> # vgreduce backups /dev/md0
> # mdadm --stop /dev/md0
> # mdadm --zero-superblocks /dev/sdc
> # mdadm /dev/md1 --add /dev/sdc
>
> Correct?

Yep.

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Is MD array (containing a PV) reducible?

[Brian J. Murrell]

[-- Attachment #1.1: Type: text/plain, Size: 1618 bytes --]

On Wed, 2023-10-04 at 09:18 -0400, Phillip Susi wrote:
> 
> Yep.

Much thanks.

But I did just think of the other benefit of using MD to sync the new
device and that's having a recovery path.

That is, if I add luks-backup to /dev/md0 and then wait for it to
finish syncing, I can then shut down the machine, remove /dev/sdc from
the machine and start it back up and ensure that all of the needed
initialization bits are in place to make the luks component come up and
be readable.  If it is, I just remove it from md0 and use it stand-
alone (as it is itself a RAID-1 recall).  Reboot again to make sure
it's all still working and good.

If any of the above goes sideways, I still have the sdc disk as an md0
member and can put it back in the machine to get back to my starting
position and try again, trying to figure out where I went wrong.

With pvmove, once the move is complete there is no going back to my
starting position if something is not right and I cannot access my new
luks device.

So given that, and given that I have stared my pvmove with --atomic and
--interval, do I just SIGINT the pvmove?  Or should I do pvmove --abort
on another terminal?  How can I know when the the abort is complete and
that I can vgreduce luks-backup out of the backups volume group?

Once I have vgreduced, is there anything I should do to
/dev/mapper/luks-backup before I mdadm --manage /dev/md0 --add
/dev/mapper/luks-backup just to wipe any remnants of it being a PV
previously?

I really do appreciate all of the help, advise and patience you have
given/shown me.

Cheers,
b.


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 202 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Is MD array (containing a PV) reducible?

[Phillip Susi]

"Brian J. Murrell" <brian@interlinx.bc.ca> writes:

> But I did just think of the other benefit of using MD to sync the new
> device and that's having a recovery path.
>
> That is, if I add luks-backup to /dev/md0 and then wait for it to
> finish syncing, I can then shut down the machine, remove /dev/sdc from
> the machine and start it back up and ensure that all of the needed
> initialization bits are in place to make the luks component come up and
> be readable.  If it is, I just remove it from md0 and use it stand-
> alone (as it is itself a RAID-1 recall).  Reboot again to make sure
> it's all still working and good.
>
> If any of the above goes sideways, I still have the sdc disk as an md0
> member and can put it back in the machine to get back to my starting
> position and try again, trying to figure out where I went wrong.
>
> With pvmove, once the move is complete there is no going back to my
> starting position if something is not right and I cannot access my new
> luks device.

Instead of pvmove, you can use lvconvert to have LVM mirror across the
drives, then break the mirror, like you would with mdadm.

> So given that, and given that I have stared my pvmove with --atomic and
> --interval, do I just SIGINT the pvmove?  Or should I do pvmove --abort
> on another terminal?  How can I know when the the abort is complete and
> that I can vgreduce luks-backup out of the backups volume group?

lvs or lvdisplay will show if the move is still in progress.

> Once I have vgreduced, is there anything I should do to
> /dev/mapper/luks-backup before I mdadm --manage /dev/md0 --add
> /dev/mapper/luks-backup just to wipe any remnants of it being a PV
> previously?

pvremove will wipe the pv label.

> I really do appreciate all of the help, advise and patience you have
> given/shown me.

You are welcome.

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/