[linux-lvm] Raid1 -> linear conversion = destroyed data

[linux-lvm] Raid1 -> linear conversion = destroyed data

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

I used lvconvert -m 1 --type raid1 to clone my root volume onto a new
drive, and then used --splitmirrors to split off the original volume
as a backup while I worked off the new drive for a few days.  I then
decided the new drive was working well and used lvconvert -m 0 to
convert the volume back to linear.  Instead of dropping the split off
original leg, lvm dropped the NEW copy and redirected all IO to the
OLD data, causing massive filesystem corruption.

This is on Ubuntu 14.10 using lvm 2.02.98 and linux 3.16.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCgAGBQJUgOk5AAoJENRVrw2cjl5RWxIIAIFnwZCd/TSFmJ7x79OFV857
1Uv87aGTHV4kE58qPs0Q7jlhdEev0pw1H9rnBQo7VuGFOZPMNXG/ZbefEaK/I/UZ
1Kga+nIv+gWDmRjLCaWg6sAy+0LzYgOLs0YvNsloPOhPWBEHYZCTczLgvl3cGqGI
IlEJgqbokbwp8Y0it3Cz7xtTDDMwio00FrXBsSi5L5+/Pu5T5beBIssKC/nhQT3Q
pAzRfxPbQr2pw8fOcotXp0qYG+aR1qCEdQnCqSmRphWzY+0XMhP68LMb7W/cPExP
k8oDbhvam+N6jQegNAgzHuqKG1wPzEfuef4o+C1pRigIu3p5NqldXuY5upVwhWw=
=P8No
-----END PGP SIGNATURE-----

Re: [linux-lvm] Raid1 -> linear conversion = destroyed data

[Anatoly Pugachev]

[-- Attachment #1: Type: text/plain, Size: 1046 bytes --]

On Fri, Dec 5, 2014 at 2:07 AM, Phillip Susi <psusi@ubuntu.com> wrote:

>
> I used lvconvert -m 1 --type raid1 to clone my root volume onto a new
> drive, and then used --splitmirrors to split off the original volume
> as a backup while I worked off the new drive for a few days.  I then
> decided the new drive was working well and used lvconvert -m 0 to
> convert the volume back to linear.  Instead of dropping the split off
> original leg, lvm dropped the NEW copy and redirected all IO to the
> OLD data, causing massive filesystem corruption.
>
> This is on Ubuntu 14.10 using lvm 2.02.98 and linux 3.16.
>

Phillip,

can you please paste the commands you've used to convert the volume? I
don't have ubuntu , but rhel7 manual page for lvconvert, example section is
pretty clear on how to convert back to flat (-m0) volume and what physical
device you remove from logical volume, quote:


       Converts mirror logical volume "vg00/lvmirror1" to linear, freeing
physical extents from /dev/sda:

       lvconvert -m0 vg00/lvmirror1 /dev/sda

[-- Attachment #2: Type: text/html, Size: 1467 bytes --]

Re: [linux-lvm] Raid1 -> linear conversion = destroyed data

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 12/05/2014 05:12 AM, Anatoly Pugachev wrote:
> Phillip,
> 
> can you please paste the commands you've used to convert the
> volume? I don't have ubuntu , but rhel7 manual page for lvconvert,
> example section is pretty clear on how to convert back to flat
> (-m0) volume and what physical device you remove from logical
> volume, quote:
> 
> 
> Converts mirror logical volume "vg00/lvmirror1" to linear, freeing 
> physical extents from /dev/sda:
> 
> lvconvert -m0 vg00/lvmirror1 /dev/sda

I simply used lvconvert -m0 vg/lv.  I did not think I would need to
specify which leg should be kept; obviously it should be the one
currently in use rather than the out of date copy.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCgAGBQJUhMWnAAoJENRVrw2cjl5RVKMH/3j5a4RMudXELHbEn5JpGs2r
I9UY+ioN1OZC25FpUkopP1C6agpjX3ZoLduPgLekigHmTQZk6YvnpTsxC81dSQJD
9wK9n4fzuUA0mDu6fBruiAZ0pJFzQ8PE/3uQBDNH0Be61NsBALlNSaoy7QzsdsOe
/v1GlWLVqOMftTIhtmg6p8MIa3f8zegdSsL6YsGr06F9uqZxQgKq8RwZi/YYmbHc
GAeWGl8QUZCjRvnOIFsS/UTt+JcDufTCk/p22QWPB18iWLIwnuuecE5dbA7KgT56
H5ct150LBLlGdjhzBEUT/dK00II1eMDOUAuaucr8j4jOJYM6GEZ1VsdX4hp1M4w=
=F78H
-----END PGP SIGNATURE-----

Re: [linux-lvm] Raid1 -> linear conversion = destroyed data

[Anatoly Pugachev]

On Mon, Dec 8, 2014 at 12:24 AM, Phillip Susi <psusi@ubuntu.com> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> On 12/05/2014 05:12 AM, Anatoly Pugachev wrote:
> > Phillip,
> >
> > can you please paste the commands you've used to convert the
> > volume? I don't have ubuntu , but rhel7 manual page for lvconvert,
> > example section is pretty clear on how to convert back to flat
> > (-m0) volume and what physical device you remove from logical
> > volume, quote:
> >
> >
> > Converts mirror logical volume "vg00/lvmirror1" to linear, freeing
> > physical extents from /dev/sda:
> >
> > lvconvert -m0 vg00/lvmirror1 /dev/sda
>
> I simply used lvconvert -m0 vg/lv.  I did not think I would need to
> specify which leg should be kept; obviously it should be the one
> currently in use rather than the out of date copy.


I wonder how many more people would lost their data before there would
be changes to lvconvert -m0 to require specification of physicalVolume
instead of optional requirement.

Re: [linux-lvm] Raid1 -> linear conversion = destroyed data

[Jason Warr]

On Mon, 08 Dec 2014 02:30:06 -0600, Anatoly Pugachev <matorola@gmail.com>  
wrote:

> On Mon, Dec 8, 2014 at 12:24 AM, Phillip Susi <psusi@ubuntu.com> wrote:
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA512
>>
>> On 12/05/2014 05:12 AM, Anatoly Pugachev wrote:
>> > Phillip,
>> >
>> > can you please paste the commands you've used to convert the
>> > volume? I don't have ubuntu , but rhel7 manual page for lvconvert,
>> > example section is pretty clear on how to convert back to flat
>> > (-m0) volume and what physical device you remove from logical
>> > volume, quote:
>> >
>> >
>> > Converts mirror logical volume "vg00/lvmirror1" to linear, freeing
>> > physical extents from /dev/sda:
>> >
>> > lvconvert -m0 vg00/lvmirror1 /dev/sda
>>
>> I simply used lvconvert -m0 vg/lv.  I did not think I would need to
>> specify which leg should be kept; obviously it should be the one
>> currently in use rather than the out of date copy.
>
>
> I wonder how many more people would lost their data before there would
> be changes to lvconvert -m0 to require specification of physicalVolume
> instead of optional requirement.
>

I may have missed part of the discussion but in order to fully understand  
what is going on I'd like to know what arguments you used when you issued  
the "lvconvert --splitmirrors" command.  There are several options to it  
that can influence what happens to the mirror members.

The best way I know of to get that is from the /etc/lvm/archive/ files for  
the volume group in question.  Those contain the maps for the volume  
group.  A new map version is created in that directory after every command  
that modifies the volume group.  Each map contains a "description" line  
that shows the command that was run making the changes.  The map does not  
include the changes as it is meant to be used as a reference to the  
configuration before the change.

If you only want to send the commands as pulled from the archive maps here  
is an example from the system I have been trying to re-create this failure  
scenario on.

[2:root@hostname /root]# grep Created /etc/lvm/archive/test01VG_0000*
/etc/lvm/archive/test01VG_00000-1787228376.vg:description = "Created  
*before* executing 'vgcreate test01VG /dev/sdar /dev/sdas /dev/sdat  
/dev/sdau /dev/sdav /dev/sdaw /dev/sdax'"
/etc/lvm/archive/test01VG_00001-1025082804.vg:description = "Created  
*before* executing 'lvcreate -L50G -m1 --type raid1 test01VG -n test01LV'"
/etc/lvm/archive/test01VG_00002-178937097.vg:description = "Created  
*before* executing 'lvconvert --splitmirrors 1 --name test01LV-M1  
test01VG/test01LV'"

Re: [linux-lvm] Raid1 -> linear conversion = destroyed data

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 12/08/2014 03:59 PM, Jason Warr wrote:
> I may have missed part of the discussion but in order to fully 
> understand what is going on I'd like to know what arguments you
> used when you issued the "lvconvert --splitmirrors" command.  There
> are several options to it that can influence what happens to the
> mirror members.

The exact sequence of commands was:

lvconvert --type raid1 -m 1 faldara/utopic /dev/sdb1
lvconvert --splitmirrors 1 --trackchanges faldara/utopic --name
utopic-backup /dev/md0
lvconvert --splitmirrors 1 --trackchanges faldara/utopic --name
utopic-backup /dev/md0

The idea was to start using the new drive exclusively while keeping a
backup on the old drive.

Also I thought it was a bit odd that --splitmirrors accepted the
- --name argument, but silently ignored it, instead using an internal
auto generated name.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCgAGBQJUiPAZAAoJENRVrw2cjl5RAS0H/10hFYNUF8n71iXER0iQVAtd
bHauiQBbemml3Q1RgqWHCFj2SYQB0epACFHIdyDUC2kWaNyQ7A9VnbRPsnyEGLzo
Tu2+LjbH2QA1e8J9zHQYq8BO0jHay5n/1S/JZrhUHxW0hyI3vnsVXESNA4GUntqn
tcC7N+wzu8SUIuqjtcm30kEE0ZvzlDv4znADIudJzXJu7JE+TjhIljg5YWY8NBxE
AYbUc1f8tGKzIyzehPETpq6/Hsi4C8C27nw+kugs+HvrmWO29nqhjQczXlbYfvjL
SDNsXLMj68dkjSqJzuuK0jy6HlHgrPxJJtaGfJ92TIcFbm9gVJHWfqClzbd1vao=
=am39
-----END PGP SIGNATURE-----

Re: [linux-lvm] Raid1 -> linear conversion = destroyed data

[Jason Warr]

On Wed, 10 Dec 2014 19:15:05 -0600, Phillip Susi <psusi@ubuntu.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> On 12/08/2014 03:59 PM, Jason Warr wrote:
>> I may have missed part of the discussion but in order to fully
>> understand what is going on I'd like to know what arguments you
>> used when you issued the "lvconvert --splitmirrors" command.  There
>> are several options to it that can influence what happens to the
>> mirror members.
>
> The exact sequence of commands was:
>
> lvconvert --type raid1 -m 1 faldara/utopic /dev/sdb1
> lvconvert --splitmirrors 1 --trackchanges faldara/utopic --name
> utopic-backup /dev/md0

It ignores the name because what it does with "--trackchanges" is rename  
the mirror leg to a visible & read only version of the meta.  Sample:

[5:root@goldmember /etc/sysconfig/network-scripts]# lvs -a -o  
lv_name,lv_attr,lv_dm_path,devices,copy_per
   LV                  Attr       DMPath                                  
Devices
   test01LV            rwi-aor--- /dev/mapper/test01VG-test01LV           
test01LV_rimage_0(0),test01LV_rim
   [test01LV_rimage_0] Iwi-aor--- /dev/mapper/test01VG-test01LV_rimage_0  
/dev/sdas(12802)
   [test01LV_rimage_1] Iwi-aor--- /dev/mapper/test01VG-test01LV_rimage_1  
/dev/sdar(12802)
   [test01LV_rmeta_0]  ewi-aor--- /dev/mapper/test01VG-test01LV_rmeta_0   
/dev/sdau(0)
   [test01LV_rmeta_1]  ewi-aor--- /dev/mapper/test01VG-test01LV_rmeta_1   
/dev/sdat(0)

Would be like so after:

[5:root@goldmember /etc/sysconfig/network-scripts]# lvs -a -o  
lv_name,lv_attr,lv_dm_path,devices,copy_per
   LV                  Attr       DMPath                                  
Devices
   test01LV            rwi-aor--- /dev/mapper/test01VG-test01LV           
test01LV_rimage_0(0),test01LV_rim
   test01LV_rimage_0   Iri-a-r--- /dev/mapper/test01VG-test01LV_rimage_0  
/dev/sdar(1)
   [test01LV_rimage_1] iwi-aor--- /dev/mapper/test01VG-test01LV_rimage_1  
/dev/sdas(12802)
   [test01LV_rmeta_0]  ewi-a-r--- /dev/mapper/test01VG-test01LV_rmeta_0   
/dev/sdar(0)
   [test01LV_rmeta_1]  ewi-aor--- /dev/mapper/test01VG-test01LV_rmeta_1   
/dev/sdas(12801)



> lvconvert --splitmirrors 1 --trackchanges faldara/utopic --name
> utopic-backup /dev/md0
>
> The idea was to start using the new drive exclusively while keeping a
> backup on the old drive.
>
> Also I thought it was a bit odd that --splitmirrors accepted the
> - --name argument, but silently ignored it, instead using an internal
> auto generated name.
>

You can't track changes and rename the "backup" leg at the same time.  If  
you wanted to keep the backup leg after making sure the in use one was  
good you would need to issue a new "--splitmirror"
  command with a new volume name and *no* "--trackchanges" option.

>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iQEcBAEBCgAGBQJUiPAZAAoJENRVrw2cjl5RAS0H/10hFYNUF8n71iXER0iQVAtd
> bHauiQBbemml3Q1RgqWHCFj2SYQB0epACFHIdyDUC2kWaNyQ7A9VnbRPsnyEGLzo
> Tu2+LjbH2QA1e8J9zHQYq8BO0jHay5n/1S/JZrhUHxW0hyI3vnsVXESNA4GUntqn
> tcC7N+wzu8SUIuqjtcm30kEE0ZvzlDv4znADIudJzXJu7JE+TjhIljg5YWY8NBxE
> AYbUc1f8tGKzIyzehPETpq6/Hsi4C8C27nw+kugs+HvrmWO29nqhjQczXlbYfvjL
> SDNsXLMj68dkjSqJzuuK0jy6HlHgrPxJJtaGfJ92TIcFbm9gVJHWfqClzbd1vao=
> =am39
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Raid1 -> linear conversion = destroyed data

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/10/2014 8:50 PM, Jason Warr wrote:
> It ignores the name because what it does with "--trackchanges" is
> rename the mirror leg to a visible & read only version of the meta.
> Sample:

You mean it changes the flags to visible and read only.  I don't see
why it can't rename it as well.

> You can't track changes and rename the "backup" leg at the same
> time. If you wanted to keep the backup leg after making sure the in
> use one was good you would need to issue a new "--splitmirror" 
> command with a new volume name and *no* "--trackchanges" option.

Why not?  A name is just a name; it does not *have* to be called
origin_rimage_0.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)

iQEcBAEBAgAGBQJUiadfAAoJENRVrw2cjl5Rt+kH/2edRv5pygm91ULcpZ9ehRuW
9bB0CpjyrLEp/f8bdqkcoyR2cpcuM2kUXBstvn7YLgyJnBdm8JNt6gZnanJLQ0gr
CXx6Tu2OQfp87NTbUDqJOLku6EaAQ053zQAoBgnIctFaCUGU+hC1L3iwdkYGnVwZ
uTdY0C3/t7ShPDvXqTrIqNugqOj7NFAptpENzxBavrkaIp5w47fvVHpWwFpwO3sU
CheSV+gpqHKlc6T6DfZlDCcCnZHKmMhXzs4EigNZIZFzM8INhn01mJ64tUJKNET7
eLvqIH8labOqAp4PVFido6Kgl6K/BKvGbzGJcqE9IR2V23yvzRHjretAGnFwed4=
=sGDQ
-----END PGP SIGNATURE-----

Re: [linux-lvm] Raid1 -> linear conversion = destroyed data

[Jason Warr]

On Thu, 11 Dec 2014 08:17:03 -0600, Phillip Susi <psusi@ubuntu.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 12/10/2014 8:50 PM, Jason Warr wrote:
>> It ignores the name because what it does with "--trackchanges" is
>> rename the mirror leg to a visible & read only version of the meta.
>> Sample:
>
> You mean it changes the flags to visible and read only.  I don't see
> why it can't rename it as well.
>

Yes, visible and ro is what I meant.

>> You can't track changes and rename the "backup" leg at the same
>> time. If you wanted to keep the backup leg after making sure the in
>> use one was good you would need to issue a new "--splitmirror"
>> command with a new volume name and *no* "--trackchanges" option.
>
> Why not?  A name is just a name; it does not *have* to be called
> origin_rimage_0.

I don't disagree but it is the way the tool works.  My best guess is the  
name is ignored because when you issue "--trackchanges" it is not really  
splitting off the leg, more of off lining it from the perspective of the  
mirror.  It was likely a developer decision for lvconvert to do it that  
way when adding in RAID1 from the md stack.

The documentation for lvconvert could be more clear about it.

>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
>
> iQEcBAEBAgAGBQJUiadfAAoJENRVrw2cjl5Rt+kH/2edRv5pygm91ULcpZ9ehRuW
> 9bB0CpjyrLEp/f8bdqkcoyR2cpcuM2kUXBstvn7YLgyJnBdm8JNt6gZnanJLQ0gr
> CXx6Tu2OQfp87NTbUDqJOLku6EaAQ053zQAoBgnIctFaCUGU+hC1L3iwdkYGnVwZ
> uTdY0C3/t7ShPDvXqTrIqNugqOj7NFAptpENzxBavrkaIp5w47fvVHpWwFpwO3sU
> CheSV+gpqHKlc6T6DfZlDCcCnZHKmMhXzs4EigNZIZFzM8INhn01mJ64tUJKNET7
> eLvqIH8labOqAp4PVFido6Kgl6K/BKvGbzGJcqE9IR2V23yvzRHjretAGnFwed4=
> =sGDQ
> -----END PGP SIGNATURE-----

Re: [linux-lvm] Raid1 -> linear conversion = destroyed data

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/10/2014 8:15 PM, Phillip Susi wrote:
> On 12/08/2014 03:59 PM, Jason Warr wrote:
>> I may have missed part of the discussion but in order to fully 
>> understand what is going on I'd like to know what arguments you 
>> used when you issued the "lvconvert --splitmirrors" command.
>> There are several options to it that can influence what happens
>> to the mirror members.
> 
> The exact sequence of commands was:
> 
> lvconvert --type raid1 -m 1 faldara/utopic /dev/sdb1 lvconvert
> --splitmirrors 1 --trackchanges faldara/utopic --name utopic-backup
> /dev/md0 lvconvert --splitmirrors 1 --trackchanges faldara/utopic
> --name utopic-backup /dev/md0
> 
> The idea was to start using the new drive exclusively while keeping
> a backup on the old drive.

So is this on someone's radar now as a critical bug?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)

iQEcBAEBAgAGBQJUj1h3AAoJENRVrw2cjl5R+DgIAJcR5QNzTboHJBxeKvo1MHoV
Cc+BgMDoyfC0CPiBqMWUC0cFoKUf5nino1uHAlEFUvljmlJqFb4QtoAvRm+DEmcy
h6HnhldPygYPHcjR4el+wq4Jz06Qy0v21OnqcHg9ECoYYmZqqiV3jMYEK1MU4iLU
PczK7Mkdwfn0Uj3sUgmFeXHXHe0jI+zG9D8pQsU3hmImlruT0gy5JOkEI6inRoAO
1ClmWrOSZkoCJO8N22NSNgHn2bgaFBkZubfJ8UL1Frwwh1n9zt/hfs+7qOH1mvDo
Q2CZzuDil+MPVGOHjk6eZZ6hW7zMqRVOoQVwcFrZlF1KQIgMcIGAFb+btNqsTxw=
=0hJ5
-----END PGP SIGNATURE-----

Re: [linux-lvm] Raid1 -> linear conversion = destroyed data

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 12/08/2014 03:30 AM, Anatoly Pugachev wrote:
> I wonder how many more people would lost their data before there
> would be changes to lvconvert -m0 to require specification of
> physicalVolume instead of optional requirement.

It should not be required.  As long as both legs are in sync, it
doesn't matter which one is kept and which one is dropped.  When one
is intentionally out of sync because it has been removed with
- --splitmirrors, then under no circumstances should that one be kept.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCgAGBQJUiPBsAAoJENRVrw2cjl5RDDkH/i6qgKx11L/bb2rVIPGeHTdW
r9yaFegoRvLX0I7+Otrty+VmCjm1giwlDkQ3jBdj8+O+Kija9vyeWFAXwnDkkgQ/
G2cCf3tmPfiua+d+BQD6cej5rVvLBBj6evcl3nu73vVKJLgji8NiPyj/x16vj/og
8z1mTxG1Obrhagk+NLe2WXAX5GWUZOYD0bBS0pV6P/dPZKJrHNhaoVB2fAadj1n7
sW4M3Ew6K9eybahhtMBffLWRG4hVqkbTdOCC6cCgCH9wDdDzhqC/Mm+WORo3t8az
/fhjCNvGIkdOn6Itl6sZMhc91XL748qJUUAZK8JxriFVL6RrfSJ2PmBdDKzyziI=
=GpNN
-----END PGP SIGNATURE-----