* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure @ 2023-04-25 13:49 haaber 2023-04-26 11:10 ` Zdenek Kabelac 2023-04-26 12:06 ` Ming Hung Tsai 0 siblings, 2 replies; 21+ messages in thread From: haaber @ 2023-04-25 13:49 UTC (permalink / raw) To: lvm-devel Dear all, I had a lethally bad hardware failure and to replace the machine. Now I try to get some data back that is not contained in half-year backups ... (I know! but it's too late to be sorry). OK, the old SSD is attached via usb adapter to a brand new machine. I started sudo pvscan sudo vgscan --mknodes sudo vgchange -ay Here is the unexpected output: ?PV /dev/mapper/OLDSSD ? VG vg0 ????? lvm2 [238.27 GiB / <15.79 GiB free] ? Total: 1 [238.27 GiB] / in use: 1 [238.27 GiB] / in no VG: 0 [0?? ] ? Found volume group "vg0" using metadata type lvm2 ? Check of pool vg0/pool00 failed (status:1). Manual repair required! ? 1 logical volume(s) in volume group "vg0" now active then I consulted dr. google for diagnosis, but found only little help. This one https://mellowhost.com/billing/index.php?rp=/knowledgebase/65/How-to-Repair-a-lvm-thin-pool.html suggested to deactivate all sub-volumes so that a repair can work correctly. It happened that only swap was active, so I deactivated it. But repair does still not work: lvconvert --repair vg0/pool00 terminate called after throwing an instance of 'std::runtime_error' ? what():? transaction_manager::new_block() couldn't allocate new block ? Child 21255 exited abnormally ? Repair of thin metadata volume of thin pool vg0/pool00 failed (status:-1). Manual repair required! I would like to find a good soul out there that can give more hints. In particular, could it be a metadata overflow? How to check? I seek not for repair, but a "once only" read access to the pool data .... thank you so much! Bernhard ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-04-25 13:49 Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure haaber @ 2023-04-26 11:10 ` Zdenek Kabelac 2023-04-26 13:12 ` haaber 2023-04-26 12:06 ` Ming Hung Tsai 1 sibling, 1 reply; 21+ messages in thread From: Zdenek Kabelac @ 2023-04-26 11:10 UTC (permalink / raw) To: lvm-devel Dne 25. 04. 23 v 15:49 haaber napsal(a): > Dear all, > > I had a lethally bad hardware failure and to replace the machine.? Now I try > to get some data back that is not contained in half-year backups ... (I know! > but it's too late to be sorry). OK,? the old SSD is attached? via usb adapter > to a brand new machine. I started > > sudo pvscan > sudo vgscan --mknodes > sudo vgchange -ay > > Here is the? unexpected output: > > ?PV /dev/mapper/OLDSSD ? VG?? vg0 ????? lvm2 [238.27 GiB / <15.79 GiB free] > ? Total: 1 [238.27 GiB] / in use: 1 [238.27 GiB] / in no VG: 0 [0?? ] > ? Found volume group "vg0" using metadata type lvm2 > ? Check of pool vg0/pool00 failed (status:1). Manual repair required! > ? 1 logical volume(s) in volume group "vg0" now active > > then I consulted dr. google for diagnosis, but found only little help. This one > > https://mellowhost.com/billing/index.php?rp=/knowledgebase/65/How-to-Repair-a-lvm-thin-pool.html > > suggested to deactivate all sub-volumes so that a repair can work correctly. > It happened that only swap was > active, so I deactivated it. But repair does still not work: > > lvconvert --repair vg0/pool00 > terminate called after throwing an instance of 'std::runtime_error' > ? what():? transaction_manager::new_block() couldn't allocate new block > ? Child 21255 exited abnormally > ? Repair of thin metadata volume of thin pool vg0/pool00 failed > (status:-1). Manual repair required! > > > I would like to find a good soul out there that can give more hints. In > particular, > could it be a metadata overflow? How to check? I seek not for repair, but a > "once only" > read access to the pool data .... > Hi Check 'man lvmthin' "Metadata check and repair" section. If the 'repair' does not work make sure you have 'latest' thin_repair tool (>= v0.9) - as older distros come with ancient less capable version of this tool. Since you likely already tried to repair metadata - you may need to do the manual repair with the use of _meta0 LV (see man lvmthin). If you cannot get 'workable' metadata with 0.9 of thin_repair tool - you will likely need to create a BZ - upload compressed content of your metadata device for futher analysis - whether it's somehow possible to recover bTree. Regards Zdenek ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-04-26 11:10 ` Zdenek Kabelac @ 2023-04-26 13:12 ` haaber 2023-04-27 9:29 ` Zdenek Kabelac 0 siblings, 1 reply; 21+ messages in thread From: haaber @ 2023-04-26 13:12 UTC (permalink / raw) To: lvm-devel Thank you, Ming-Hun and Zdenek for your quick replies. I answer below! On 4/26/23 13:10, Zdenek Kabelac wrote: > Dear all, >> >> I had a lethally bad hardware failure and to replace the machine.? >> Now I try to get some data back that is not contained in half-year >> backups ... (I know! but it's too late to be sorry). OK,? the old SSD >> is attached? via usb adapter to a brand new machine. I started >> >> sudo pvscan >> sudo vgscan --mknodes >> sudo vgchange -ay >> >> Here is the? unexpected output: >> >> ??PV /dev/mapper/OLDSSD ? VG?? vg0 ????? lvm2 [238.27 GiB / <15.79 >> GiB free] >> ?? Total: 1 [238.27 GiB] / in use: 1 [238.27 GiB] / in no VG: 0 [0?? ] >> ?? Found volume group "vg0" using metadata type lvm2 >> ?? Check of pool vg0/pool00 failed (status:1). Manual repair required! >> ?? 1 logical volume(s) in volume group "vg0" now active >> >> then I consulted dr. google for diagnosis, but found only little >> help. This one >> >> https://mellowhost.com/billing/index.php?rp=/knowledgebase/65/How-to-Repair-a-lvm-thin-pool.html >> >> >> suggested to deactivate all sub-volumes so that a repair can work >> correctly. It happened that only swap was >> active, so I deactivated it. But repair does still not work: >> >> lvconvert --repair vg0/pool00 >> terminate called after throwing an instance of 'std::runtime_error' >> ?? what():? transaction_manager::new_block() couldn't allocate new block >> ?? Child 21255 exited abnormally >> ?? Repair of thin metadata volume of thin pool vg0/pool00 failed >> (status:-1). Manual repair required! >> >> >> I would like to find a good soul out there that can give more hints. >> In particular, >> could it be a metadata overflow? How to check? I seek not for repair, >> but a "once only" >> read access to the pool data .... >> > > Hi > > Check? 'man lvmthin'? "Metadata check and repair" section. > If the 'repair' does not work make sure you have 'latest' thin_repair > tool (>= v0.9) - as older distros come with ancient less capable > version of this tool. > > Since you likely already tried to repair metadata - you may need to do > the manual repair with the use? of? _meta0? LV? (see man lvmthin). > > If you cannot get? 'workable' metadata with? 0.9 of thin_repair tool - > you will likely need to create a BZ - upload compressed content of > your metadata device for futher analysis - whether it's somehow > possible to recover bTree. I have thin_repair 0.9 installed. But I first have to dump metadata into a file, so I invoked (after pvscan and vgscan and vgchange -an) ??? root at machine:~#?? thin_dump /dev/mapper/OLDSSD -o thindump.xml -r ??? The following field needs to be provided on the command line due to corruption in the superblock: transaction id Oups. So the? superblock is damaged. What should / could I serve thin_repair as transaction id ?? Since we read only, I tried 0: ??? root at machine:~#? thin_dump /dev/mapper/OLDSSD -o thindump.xml -r --transaction-id 0 ??? The following field needs to be provided on the command line due to corruption in the superblock: data block size Oups. I gave it a try and added --data-block-size 128 just to see. Now it asks for nr of data blocks ... aargh! I cannot guess that one. Could I? " dd " the superblock for inspection into a file? Is there only one superblock? Most fs have several ones, for exactly that reason ... i.e: can I use a copy? I? dd'ed the first 2M of the /dev/mapper/OLDSSD into a file, and gave it a try. After some binary data (less than 1k), follow roughly 1M of json type data like this whatever { id = "bhQocj-EJ6Y-0jXC-oAmr-lxlF-cudL-5ohI1e" status = ["READ", "WRITE", "VISIBLE"] flags = [] creation_time = 1677967121 creation_host = "dom0" segment_count = 1 segment1 { start_extent = 0 extent_count = 512 type = "thin" thin_pool = "pool00" transaction_id = 44995 device_id = 19881 } and then many binary data again. Would this 1M (uncompressed), probably 100K bzipped data be of any help? I could post it somewhere. Again, I do not want to have the thin pool re-usable, but just take a last "clean copy" on a new disc ... best, Bernhard ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-04-26 13:12 ` haaber @ 2023-04-27 9:29 ` Zdenek Kabelac 2023-05-03 16:48 ` haaber 0 siblings, 1 reply; 21+ messages in thread From: Zdenek Kabelac @ 2023-04-27 9:29 UTC (permalink / raw) To: lvm-devel Dne 26. 04. 23 v 15:12 haaber napsal(a): > Thank you, Ming-Hun and Zdenek for your quick replies. I answer below! > > On 4/26/23 13:10, Zdenek Kabelac wrote: >> Dear all, >>> >>> I had a lethally bad hardware failure and to replace the machine. >>> Now I try to get some data back that is not contained in half-year >>> backups ... (I know! but it's too late to be sorry). OK,? the old SSD >>> is attached? via usb adapter to a brand new machine. I started >>> >>> sudo pvscan >>> sudo vgscan --mknodes >>> sudo vgchange -ay >>> >>> Here is the? unexpected output: >>> >>> ??PV /dev/mapper/OLDSSD ? VG?? vg0 ????? lvm2 [238.27 GiB / <15.79 >>> GiB free] >>> ?? Total: 1 [238.27 GiB] / in use: 1 [238.27 GiB] / in no VG: 0 [0?? ] >>> ?? Found volume group "vg0" using metadata type lvm2 >>> ?? Check of pool vg0/pool00 failed (status:1). Manual repair required! >>> ?? 1 logical volume(s) in volume group "vg0" now active >>> >>> then I consulted dr. google for diagnosis, but found only little >>> help. This one >>> >>> https://mellowhost.com/billing/index.php?rp=/knowledgebase/65/How-to-Repair-a-lvm-thin-pool.html >>> >>> >>> suggested to deactivate all sub-volumes so that a repair can work >>> correctly. It happened that only swap was >>> active, so I deactivated it. But repair does still not work: >>> >>> lvconvert --repair vg0/pool00 >>> terminate called after throwing an instance of 'std::runtime_error' >>> ?? what():? transaction_manager::new_block() couldn't allocate new block >>> ?? Child 21255 exited abnormally >>> ?? Repair of thin metadata volume of thin pool vg0/pool00 failed >>> (status:-1). Manual repair required! >>> >>> >>> I would like to find a good soul out there that can give more hints. >>> In particular, >>> could it be a metadata overflow? How to check? I seek not for repair, >>> but a "once only" >>> read access to the pool data .... >>> >> >> Hi >> >> Check? 'man lvmthin'? "Metadata check and repair" section. >> If the 'repair' does not work make sure you have 'latest' thin_repair >> tool (>= v0.9) - as older distros come with ancient less capable >> version of this tool. >> >> Since you likely already tried to repair metadata - you may need to do >> the manual repair with the use? of? _meta0? LV? (see man lvmthin). >> >> If you cannot get? 'workable' metadata with? 0.9 of thin_repair tool - >> you will likely need to create a BZ - upload compressed content of >> your metadata device for futher analysis - whether it's somehow >> possible to recover bTree. > > I have thin_repair 0.9 installed. But I first have to dump metadata into > a file, so I invoked (after pvscan and vgscan and vgchange -an) > > ??? root at machine:~#?? thin_dump /dev/mapper/OLDSSD -o thindump.xml -r > ??? The following field needs to be provided on the command line due to > corruption in the superblock: transaction id > > Oups. So the? superblock is damaged. What should / could I serve > thin_repair as transaction id ?? Since we read only, I tried 0: > > ??? root at machine:~#? thin_dump /dev/mapper/OLDSSD -o thindump.xml -r > --transaction-id 0 > ??? The following field needs to be provided on the command line due to > corruption in the superblock: data block size > > Oups. I gave it a try and added --data-block-size 128 just to see. Now > it asks for nr of data blocks ... aargh! I cannot guess that one. Hi Not sure I'm getting right your process here. There are 2 types of 'metadata' and a different recovery work needed for them. > Could I? " dd " the superblock for inspection into a file? Is there only > one superblock? Most fs have several ones, for exactly that reason ... > i.e: can I use a copy? > > I? dd'ed the first 2M of the /dev/mapper/OLDSSD into a file, and gave it > a try. After some binary data (less than 1k), follow roughly 1M of json > type data like this > > whatever { > id = "bhQocj-EJ6Y-0jXC-oAmr-lxlF-cudL-5ohI1e" > status = ["READ", "WRITE", "VISIBLE"] > flags = [] > creation_time = 1677967121 > creation_host = "dom0" > segment_count = 1 > > segment1 { > start_extent = 0 > extent_count = 512 > > type = "thin" > thin_pool = "pool00" > transaction_id = 44995 > device_id = 19881 > } > > and then many binary data again. Would this 1M (uncompressed), probably > 100K bzipped data be of any help? I could post it somewhere. Again, I do > not want to have the thin pool re-usable, but just take a last "clean > copy" on a new disc ... There are 'lvm2' metadata - which are stored withing the PV disk header (by default this is located in the 1st. 1 MiB of your device). These metadata have absolutely nothing to do with thin-pool metadata! lvm2 just keeps the layout of blocks for your LVs. To get to your thin-pool metadata, you have to activate LV with them. (lvchange -ay vgname/thinpoolmetadata). Once you have your thinpool metadata 'active' (present in DM table), then you can fire 'thin_dump --repair' / 'thin_repair' tool. ATM it's not clear in which state of recovery you are. So do you have you 'lvm2' completely & usable and can you active LV which holds thin-pool metadata ? Can you please provide 'lvs -a' of your volume group ? And if you use 'thin_dump/thin_repair' - your *exact* command line you've been using? Regards Zdenek ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-04-27 9:29 ` Zdenek Kabelac @ 2023-05-03 16:48 ` haaber 2023-05-04 13:17 ` Zdenek Kabelac 0 siblings, 1 reply; 21+ messages in thread From: haaber @ 2023-05-03 16:48 UTC (permalink / raw) To: lvm-devel Dear? Zdenek, I had a forced break, but the subject is still active.. > > To get to your thin-pool metadata, you have to activate LV with them. > (lvchange -ay? vgname/thinpoolmetadata). > my pool is called? qubes_dom0/pool00 (now you know my previous operating system :) So I tried lvchange -ay? qubes_dom0/thinpoolmetadata but that fails: ? Failed to find logical volume "qubes_dom0/thinpoolmetadata" Then I tried lvchange -ay qubes_dom0/pool00_tmeta and that worked (gave a warning). But I do not know how to run thin_repair now :(( > Can you please provide?? 'lvs -a'? of your volume group ? I attached that file.? It's a little mess, I fear.? Thank you for your help, best, Bernhard -------------- next part -------------- LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [lvol0_pmspare] qubes_dom0 ewi------- 108.00m pool00 qubes_dom0 twi---tz-- <214.78g [pool00_tdata] qubes_dom0 Twi------- <214.78g [pool00_tmeta] qubes_dom0 ewi------- 108.00m root qubes_dom0 Vwi---tz-- <214.78g pool00 swap qubes_dom0 -wi-a----- <7.50g vm-Android-private qubes_dom0 Vwi---tz-- 5.00g pool00 vm-Android-private-1633545383-back vm-Android-private-1633545383-back qubes_dom0 Vwi---tz-- 5.00g pool00 vm-Arbeit-private qubes_dom0 Vwi---tz-- 30.00g pool00 vm-Arbeit-private-1678980932-back vm-Arbeit-private-1678980932-back qubes_dom0 Vwi---tz-- 30.00g pool00 vm-Arbeit-private-snap qubes_dom0 Vwi---tz-- 30.00g pool00 vm-Arbeit-private vm-Arbeit-root-snap qubes_dom0 Vwi---tz-- 20.00g pool00 vm-debian-11-root-1679130937-back vm-Arbeit-volatile qubes_dom0 Vwi---tz-- 10.00g pool00 vm-Fernwartung-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-Fernwartung-private-1604748655-back vm-Fernwartung-private-1604748655-back qubes_dom0 Vwi---tz-- 2.00g pool00 vm-GPG-keys-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-GPG-keys-private-snap qubes_dom0 Vwi---tz-- 2.00g pool00 vm-GPG-keys-private vm-GPG-keys-root-snap qubes_dom0 Vwi---tz-- 15.00g pool00 vm-GPG-keys-volatile qubes_dom0 Vwi---tz-- 10.00g pool00 vm-PLM-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-PLM-private-1561430058-back vm-PLM-private-1561430058-back qubes_dom0 Vwi---tz-- 2.00g pool00 vm-Privat-private qubes_dom0 Vwi---tz-- 156.25g pool00 vm-Privat-private-1658835892-back vm-Privat-private-1658835892-back qubes_dom0 Vwi---tz-- 156.25g pool00 vm-Privat-private-snap qubes_dom0 Vwi---tz-- 156.25g pool00 vm-Privat-private vm-Privat-root-snap qubes_dom0 Vwi---tz-- 20.00g pool00 vm-Privat-volatile qubes_dom0 Vwi---tz-- 10.00g pool00 vm-Tresorraum-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-Tresorraum-private-1678980925-back vm-Tresorraum-private-1678980925-back qubes_dom0 Vwi---tz-- 2.00g pool00 vm-Tresorraum-private-snap qubes_dom0 Vwi---tz-- 2.00g pool00 vm-Tresorraum-private vm-Tresorraum-root-snap qubes_dom0 Vwi---tz-- 20.00g pool00 vm-debian-11-root vm-Tresorraum-volatile qubes_dom0 Vwi---tz-- 10.00g pool00 vm-Verwaltung-private qubes_dom0 Vwi---tz-- 5.99g pool00 vm-Verwaltung-private-1677926231-back vm-Verwaltung-private-1677926231-back qubes_dom0 Vwi---tz-- 5.99g pool00 vm-Verwaltung-private-snap qubes_dom0 Vwi---tz-- 5.99g pool00 vm-Verwaltung-private vm-Verwaltung-root-snap qubes_dom0 Vwi---tz-- 20.00g pool00 vm-debian-11-root vm-Verwaltung-volatile qubes_dom0 Vwi---tz-- 10.00g pool00 vm-anon-whonix-private qubes_dom0 Vwi---tz-- 3.00g pool00 vm-anon-whonix-private-1675500726-back vm-anon-whonix-private-1675500726-back qubes_dom0 Vwi---tz-- 3.00g pool00 vm-buster-print-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-buster-print-root qubes_dom0 Vwi---tz-- 10.00g pool00 vm-buster-print-root-1647866494-back vm-buster-print-root-1647866494-back qubes_dom0 Vwi---tz-- 10.00g pool00 vm-debian-11-minimal-firewall-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-debian-11-minimal-firewall-root qubes_dom0 Vwi---tz-- 10.00g pool00 vm-debian-11-minimal-firewall-root-1677968205-back vm-debian-11-minimal-firewall-root-1677968205-back qubes_dom0 Vwi---tz-- 10.00g pool00 vm-debian-11-minimal-net-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-debian-11-minimal-net-root qubes_dom0 Vwi---tz-- 10.00g pool00 vm-debian-11-minimal-net-root-1677968245-back vm-debian-11-minimal-net-root-1677968245-back qubes_dom0 Vwi---tz-- 10.00g pool00 vm-debian-11-minimal-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-debian-11-minimal-root qubes_dom0 Vwi---tz-- 10.00g pool00 vm-debian-11-minimal-root-1640161303-back vm-debian-11-minimal-root-1640161303-back qubes_dom0 Vwi---tz-- 10.00g pool00 vm-debian-11-minimal-usb-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-debian-11-minimal-usb-root qubes_dom0 Vwi---tz-- 10.00g pool00 vm-debian-11-minimal-usb-root-1640161428-back vm-debian-11-minimal-usb-root-1640161428-back qubes_dom0 Vwi---tz-- 10.00g pool00 vm-debian-11-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-debian-11-root qubes_dom0 Vwi---tz-- 20.00g pool00 vm-debian-11-root-1679130937-back vm-debian-11-root-1679130937-back qubes_dom0 Vwi---tz-- 20.00g pool00 vm-debian-dvm-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-default-mgmt-dvm-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-disp383-private-snap qubes_dom0 Vwi---tz-- 2.00g pool00 vm-debian-dvm-private vm-disp383-root-snap qubes_dom0 Vwi---tz-- 20.00g pool00 vm-debian-11-root-1679130937-back vm-disp383-volatile qubes_dom0 Vwi---tz-- 12.00g pool00 vm-dummy-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-dummy-root qubes_dom0 Vwi---tz-- 10.00g pool00 vm-flashing-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-flashing-private-1647856203-back vm-flashing-private-1647856203-back qubes_dom0 Vwi---tz-- 2.00g pool00 vm-mail-privat-private qubes_dom0 Vwi---tz-- 4.00g pool00 vm-mail-privat-private-1678980932-back vm-mail-privat-private-1678980932-back qubes_dom0 Vwi---tz-- 4.00g pool00 vm-mail-privat-private-snap qubes_dom0 Vwi---tz-- 4.00g pool00 vm-mail-privat-private vm-mail-privat-root-snap qubes_dom0 Vwi---tz-- 20.00g pool00 vm-debian-11-root-1679130937-back vm-mail-privat-volatile qubes_dom0 Vwi---tz-- 10.00g pool00 vm-mirage-builder-private qubes_dom0 Vwi---tz-- 20.00g pool00 vm-mirage-builder-private-1652136334-back vm-mirage-builder-private-1652136334-back qubes_dom0 Vwi---tz-- 20.00g pool00 vm-mirage-firewall-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-mirage-firewall-private-1678980930-back vm-mirage-firewall-private-1678980930-back qubes_dom0 Vwi---tz-- 2.00g pool00 vm-mirage-firewall-private-snap qubes_dom0 Vwi---tz-- 2.00g pool00 vm-mirage-firewall-private vm-mirage-firewall-root qubes_dom0 Vwi---tz-- 10.00g pool00 vm-mirage-firewall-root-1678980930-back vm-mirage-firewall-root-1678980930-back qubes_dom0 Vwi---tz-- 10.00g pool00 vm-mirage-firewall-root-snap qubes_dom0 Vwi---tz-- 10.00g pool00 vm-mirage-firewall-root vm-mirage-firewall-volatile qubes_dom0 Vwi---tz-- 10.00g pool00 vm-printing-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-printing-private-1647858008-back vm-printing-private-1647858008-back qubes_dom0 Vwi---tz-- 2.00g pool00 vm-sshkeys-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-sshkeys-private-1623251897-back vm-sshkeys-private-1623251897-back qubes_dom0 Vwi---tz-- 2.00g pool00 vm-sshkeys-private-snap qubes_dom0 Vwi---tz-- 2.00g pool00 vm-sshkeys-private vm-sshkeys-root-snap qubes_dom0 Vwi---tz-- 10.00g pool00 vm-sshkeys-volatile qubes_dom0 Vwi---tz-- 10.00g pool00 vm-sys-firewall-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-sys-firewall-private-1678980932-back vm-sys-firewall-private-1678980932-back qubes_dom0 Vwi---tz-- 2.00g pool00 vm-sys-firewall-private-snap qubes_dom0 Vwi---tz-- 2.00g pool00 vm-sys-firewall-private vm-sys-firewall-root-snap qubes_dom0 Vwi---tz-- 10.00g pool00 vm-debian-11-minimal-firewall-root vm-sys-firewall-volatile qubes_dom0 Vwi---tz-- 10.00g pool00 vm-sys-net-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-sys-net-private-1678980929-back vm-sys-net-private-1678980929-back qubes_dom0 Vwi---tz-- 2.00g pool00 vm-sys-net-private-snap qubes_dom0 Vwi---tz-- 2.00g pool00 vm-sys-net-private vm-sys-net-root-snap qubes_dom0 Vwi---tz-- 10.00g pool00 vm-debian-11-minimal-net-root vm-sys-net-volatile qubes_dom0 Vwi---tz-- 10.00g pool00 vm-sys-usb-private qubes_dom0 Vwi---tz-- 5.00g pool00 vm-sys-usb-private-1679256316-back vm-sys-usb-private-1679256316-back qubes_dom0 Vwi---tz-- 5.00g pool00 vm-sys-whonix-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-sys-whonix-private-1678980932-back vm-sys-whonix-private-1678980932-back qubes_dom0 Vwi---tz-- 2.00g pool00 vm-sys-whonix-private-snap qubes_dom0 Vwi---tz-- 2.00g pool00 vm-sys-whonix-private vm-sys-whonix-root-snap qubes_dom0 Vwi---tz-- 10.00g pool00 vm-whonix-gw-16-root-1679305216-back vm-sys-whonix-volatile qubes_dom0 Vwi---tz-- 10.00g pool00 vm-tribler-private qubes_dom0 Vwi---tz-- 20.00g pool00 vm-tribler-private-1671645867-back vm-tribler-private-1671645867-back qubes_dom0 Vwi---tz-- 20.00g pool00 vm-untrusted-private qubes_dom0 Vwi---tz-- 42.00g pool00 vm-untrusted-private-1678980932-back vm-untrusted-private-1678980932-back qubes_dom0 Vwi---tz-- 42.00g pool00 vm-untrusted-private-snap qubes_dom0 Vwi---tz-- 42.00g pool00 vm-untrusted-private vm-untrusted-root-snap qubes_dom0 Vwi---tz-- 20.00g pool00 vm-debian-11-root-1679130937-back vm-untrusted-volatile qubes_dom0 Vwi---tz-- 10.00g pool00 vm-videokonferenz-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-videokonferenz-private-1621252915-back vm-videokonferenz-private-1621252915-back qubes_dom0 Vwi---tz-- 2.00g pool00 vm-whonix-gw-16-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-whonix-gw-16-root-1679305216-back qubes_dom0 Vwi---tz-- 10.00g pool00 vm-whonix-ws-15-dvm-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-whonix-ws-16-private qubes_dom0 Vwi---tz-- 2.00g pool00 vm-whonix-ws-16-root qubes_dom0 Vwi---tz-- 10.00g pool00 vm-whonix-ws-16-root-1674925902-back vm-whonix-ws-16-root-1674925902-back qubes_dom0 Vwi---tz-- 10.00g pool00 ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-05-03 16:48 ` haaber @ 2023-05-04 13:17 ` Zdenek Kabelac 2023-05-04 16:31 ` haaber 2023-05-04 17:06 ` haaber 0 siblings, 2 replies; 21+ messages in thread From: Zdenek Kabelac @ 2023-05-04 13:17 UTC (permalink / raw) To: lvm-devel Dne 03. 05. 23 v 18:48 haaber napsal(a): > Dear Zdenek, > > I had a forced break, but the subject is still active.. > >> >> To get to your thin-pool metadata, you have to activate LV with them. >> (lvchange -ay? vgname/thinpoolmetadata). >> > my pool is called? qubes_dom0/pool00 (now you know my previous operating > system :) So I tried > > lvchange -ay? qubes_dom0/thinpoolmetadata > > but that fails: ? Failed to find logical volume > "qubes_dom0/thinpoolmetadata" > > Then I tried > > lvchange -ay qubes_dom0/pool00_tmeta Looking at your 'lvs -a' output - you should be able to get this one active. You will need another LV to write fixed metadata into # lvcreate -L128M -n newlv? qubes_dom0 Then you run # thin_repair -i /dev/qubes_dom0/pool00_tmeta -o /dev/qubes_dom0/newlv If you get some repaired set? which you could probably validate with ?# thin_dump?? /dev/qubes_dom0/newlv if you will some some 'good amount' of some data which describe block mapping for many thin volumes. However if you only see couple lines - basically empty thin-pool metadta - you will need to store the content of your original unmodified metadata into a compressed file and upload the file for futher exploration. Let me know what you get from those steps above. Zdenek ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-05-04 13:17 ` Zdenek Kabelac @ 2023-05-04 16:31 ` haaber 2023-05-05 15:14 ` Zdenek Kabelac 2023-05-04 17:06 ` haaber 1 sibling, 1 reply; 21+ messages in thread From: haaber @ 2023-05-04 16:31 UTC (permalink / raw) To: lvm-devel Dear Zdenek On 5/4/23 15:17, Zdenek Kabelac wrote: > >> lvchange -ay qubes_dom0/pool00_tmeta > > > Looking at your 'lvs -a' output - you should be able to get this one > active. > > > You will need another LV to write fixed metadata into > > # lvcreate -L128M -n newlv? qubes_dom0 He was yelling at me: # lvcreate -L128M -n newlv? qubes_dom0 ? WARNING: Sum of all thin volume sizes (<1.62 TiB) exceeds the size of thin pools and the size of whole volume group (238.27 GiB). ? WARNING: You have not turned on protection against thin pools running out of space. ? WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. ? Logical volume "newlv" created. But since he seemed to live with it, I gave it a try and continued. So I activated tmeta by lvchange -ay qubes_dom0/pool00_tmeta > > Then you run > > # thin_repair -i /dev/qubes_dom0/pool00_tmeta -o /dev/qubes_dom0/newlv > # thin_repair?? -i /dev/qubes_dom0/pool00_tmeta? -o /dev/qubes_dom0/newlv terminate called after throwing an instance of 'std::runtime_error' ? what():? transaction_manager::new_block() couldn't allocate new block Aborted > > Let me know what you get from those steps above. > and that is where I am stuck now. By the way: # thin_dump?? /dev/qubes_dom0/newlv bad checksum in superblock, wanted 1490015127 thank you for your help & time,? Bernhard ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-05-04 16:31 ` haaber @ 2023-05-05 15:14 ` Zdenek Kabelac 0 siblings, 0 replies; 21+ messages in thread From: Zdenek Kabelac @ 2023-05-05 15:14 UTC (permalink / raw) To: lvm-devel Dne 04. 05. 23 v 18:31 haaber napsal(a): > Dear Zdenek > > On 5/4/23 15:17, Zdenek Kabelac wrote: >> >>> lvchange -ay qubes_dom0/pool00_tmeta >> >> >> Looking at your 'lvs -a' output - you should be able to get this one >> active. >> >> >> You will need another LV to write fixed metadata into >> >> # lvcreate -L128M -n newlv? qubes_dom0 > > He was yelling at me: > > # lvcreate -L128M -n newlv? qubes_dom0 > > ? WARNING: Sum of all thin volume sizes (<1.62 TiB) exceeds the size of > thin pools and the size of whole volume group (238.27 GiB). > ? WARNING: You have not turned on protection against thin pools running > out of space. > ? WARNING: Set activation/thin_pool_autoextend_threshold below 100 to > trigger automatic extension of thin pools before they get full. > ? Logical volume "newlv" created. > > But since he seemed to live with it, I gave it a try and continued. So I > activated tmeta by > > lvchange -ay qubes_dom0/pool00_tmeta > >> >> Then you run >> >> # thin_repair -i /dev/qubes_dom0/pool00_tmeta -o /dev/qubes_dom0/newlv >> > # thin_repair?? -i /dev/qubes_dom0/pool00_tmeta? -o /dev/qubes_dom0/newlv > terminate called after throwing an instance of 'std::runtime_error' > ? what():? transaction_manager::new_block() couldn't allocate new block > Aborted Yeah this will need? Ming's investigation with raw binary date from your metadata volume (see my 2nd. mail) Regards Zdenek ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-05-04 13:17 ` Zdenek Kabelac 2023-05-04 16:31 ` haaber @ 2023-05-04 17:06 ` haaber 2023-05-05 9:42 ` Ming Hung Tsai 2023-05-05 15:07 ` Zdenek Kabelac 1 sibling, 2 replies; 21+ messages in thread From: haaber @ 2023-05-04 17:06 UTC (permalink / raw) To: lvm-devel Dear Zdenek, here https://we.tl/t-41PiPG2V1G???? is the output of #thin_dump?? /dev/qubes_dom0/pool00_tmeta? > pool00_tmeta metadata contains errors (run thin_check for details). perhaps you wanted to run with --repairmetadata contains errors (run thin_check for details). perhaps you wanted to run with --repair best, Bernhard ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-05-04 17:06 ` haaber @ 2023-05-05 9:42 ` Ming Hung Tsai 2023-05-05 15:07 ` Zdenek Kabelac 1 sibling, 0 replies; 21+ messages in thread From: Ming Hung Tsai @ 2023-05-05 9:42 UTC (permalink / raw) To: lvm-devel Hi, The thin_dump output looks fine, so I would like to know why there's error messages emitted. Could you help provide the raw metadata dump please? Just 'dd' the entire '/dev/qubes_dom0/pool00_tmeta' into a file, then upload the compressed file. Thanks, Ming-Hung Tsai On Fri, May 5, 2023 at 1:07?AM haaber <haaber@web.de> wrote: > Dear Zdenek, > > here https://we.tl/t-41PiPG2V1G is the output of > > #thin_dump /dev/qubes_dom0/pool00_tmeta > pool00_tmeta > > metadata contains errors (run thin_check for details). > > perhaps you wanted to run with --repairmetadata contains errors (run > thin_check for details). > > perhaps you wanted to run with --repair > > > best, Bernhard > > > > > > -- > lvm-devel mailing list > lvm-devel at redhat.com > https://listman.redhat.com/mailman/listinfo/lvm-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20230505/c1dec7d8/attachment.htm> ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-05-04 17:06 ` haaber 2023-05-05 9:42 ` Ming Hung Tsai @ 2023-05-05 15:07 ` Zdenek Kabelac 2023-05-05 16:25 ` Ming Hung Tsai 2023-05-11 7:39 ` haaber 1 sibling, 2 replies; 21+ messages in thread From: Zdenek Kabelac @ 2023-05-05 15:07 UTC (permalink / raw) To: lvm-devel Dne 04. 05. 23 v 19:06 haaber napsal(a): > Dear Zdenek, > > here https://we.tl/t-41PiPG2V1G???? is the output of > > #thin_dump?? /dev/qubes_dom0/pool00_tmeta? > pool00_tmeta > Hi We need the exact binary copy of _tmeta LV - thus just use dd if=/dev/qubes_dom0/pool00_tmeta /tmp/tmeta_copy bs=512K bzip2 /tmp/tmeta_copy With this data - also provide full lvm2 metadata for this VG (should be as a file in /etc/lvm/backup - or you could run just vgcfgbackup) thin_dump is already 'processed' result - so not usable for futher investigation. (Although it might interesting idea to add some 'option' to actually provide the above 'binary backup' is built-in feature of this tool for bug reporting...) Regards Zdenek ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-05-05 15:07 ` Zdenek Kabelac @ 2023-05-05 16:25 ` Ming Hung Tsai 2023-05-11 7:39 ` haaber 1 sibling, 0 replies; 21+ messages in thread From: Ming Hung Tsai @ 2023-05-05 16:25 UTC (permalink / raw) To: lvm-devel RHEL & Fedora RPMs already have the thin_metadata_pack/unpack tools integrated. Other distros may offer the tools if they had updated the package to v1.0.x On Fri, May 5, 2023 at 11:08?PM Zdenek Kabelac > Hi > > We need the exact binary copy of _tmeta LV - thus just use > > dd if=/dev/qubes_dom0/pool00_tmeta /tmp/tmeta_copy bs=512K > bzip2 /tmp/tmeta_copy > > With this data - also provide full lvm2 metadata for this VG > (should be as a file in /etc/lvm/backup - or you could run > just vgcfgbackup) > > thin_dump is already 'processed' result - so not usable for futher > investigation. > > (Although it might interesting idea to add some 'option' to actually > provide > the above 'binary backup' is built-in feature of this tool for bug > reporting...) > > > Regards > > Zdenek > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20230506/fd287b37/attachment.htm> ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-05-05 15:07 ` Zdenek Kabelac 2023-05-05 16:25 ` Ming Hung Tsai @ 2023-05-11 7:39 ` haaber 2023-05-12 3:29 ` Ming Hung Tsai 2023-05-17 15:17 ` Ming Hung Tsai 1 sibling, 2 replies; 21+ messages in thread From: haaber @ 2023-05-11 7:39 UTC (permalink / raw) To: lvm-devel Dear all, We need the exact binary copy of _tmeta? LV? - thus just use > > dd if=/dev/qubes_dom0/pool00_tmeta? /tmp/tmeta_copy? bs=512K > bzip2 /tmp/tmeta_copy > the output is here: https://we.tl/t-AEmlc5CYeH > With this data - also provide full lvm2 metadata for this VG > (should be as a file in? /etc/lvm/backup? - or you could run > just vgcfgbackup) > I attached that one directly. Thank you very much! best, Bernhard -------------- next part -------------- A non-text attachment was scrubbed... Name: qubes_dom0.bz2 Type: application/octet-stream Size: 8156 bytes Desc: not available URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20230511/aab57f85/attachment-0001.obj> ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-05-11 7:39 ` haaber @ 2023-05-12 3:29 ` Ming Hung Tsai 2023-05-12 18:05 ` haaber 2023-05-17 15:17 ` Ming Hung Tsai 1 sibling, 1 reply; 21+ messages in thread From: Ming Hung Tsai @ 2023-05-12 3:29 UTC (permalink / raw) To: lvm-devel Hi, There's one corrupted leaf node in device #20081, which might possibly be caused by the hardware failure and that stops thin_dump or lvconvert from working. Running thin_check would show you the details (I'm using the v1.0.5): TRANSACTION_ID=45452 METADATA_FREE_BLOCKS=11827 1 nodes in data mapping tree contain errors 0 io errors, 1 checksum errors Thin device 20081 has 1 error nodes and is missing 22664 mappings, while expected 296263 Check of mappings failed The issue is repairable by rolling back to the previous transaction. I'm going to patch the program to make it easier to use. It should be ready next week, and you can try to learn how to build the Rust version for now. 1. Install the Rust toolchain via the rustup script (https://rustup.rs/) 2. Clone the thin-provisioning-tools.git repo, then build it (cargo build --release) 3. Try the built pdata_tools binary (placed under ./target/release/) Ming-Hung Tsai On Thu, May 11, 2023 at 3:39?PM haaber <haaber@web.de> wrote: > Dear all, > > We need the exact binary copy of _tmeta LV - thus just use > > > > dd if=/dev/qubes_dom0/pool00_tmeta /tmp/tmeta_copy bs=512K > > bzip2 /tmp/tmeta_copy > > > the output is here: https://we.tl/t-AEmlc5CYeH > > > With this data - also provide full lvm2 metadata for this VG > > (should be as a file in /etc/lvm/backup - or you could run > > just vgcfgbackup) > > > I attached that one directly. Thank you very much! > > best, Bernhard > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20230512/8865d41b/attachment.htm> ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-05-12 3:29 ` Ming Hung Tsai @ 2023-05-12 18:05 ` haaber 2023-05-13 3:20 ` Ming Hung Tsai 0 siblings, 1 reply; 21+ messages in thread From: haaber @ 2023-05-12 18:05 UTC (permalink / raw) To: lvm-devel Dear Ming-Hung, > > There's one corrupted leaf node in device #20081, which might possibly > be caused by the hardware failure and that stops thin_dump or > lvconvert from working. Running thin_check would show you the details > (I'm using the v1.0.5): > > TRANSACTION_ID=45452 > METADATA_FREE_BLOCKS=11827 > 1 nodes in data mapping tree contain errors > 0 io errors, 1 checksum errors > Thin device 20081 has 1 error nodes and is missing 22664 mappings, > while expected 296263 > Check of mappings failed > > The issue is repairable by rolling back to the previous transaction. > I'm going to patch the program to make it easier to use. It should be > ready next week, and you can try to learn how to build the Rust > version for now. > > 1. Install the Rust toolchain via the rustup script (https://rustup.rs/) > 2. Clone the thin-provisioning-tools.git repo, then build it (cargo > build --release) > 3. Try the built pdata_tools binary (placed under ./target/release/) > thank you for this inspection! ? I now have hope again to recover my data :) Silly question: I? cloned https://github.com/jthornber/thin-provisioning-tools and installed that way successfully 1.0.4 but that is not the 1.0.5 branch you talked about. Could you point the right 1.0.5 git, please? thank you, Bernhard ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-05-12 18:05 ` haaber @ 2023-05-13 3:20 ` Ming Hung Tsai 0 siblings, 0 replies; 21+ messages in thread From: Ming Hung Tsai @ 2023-05-13 3:20 UTC (permalink / raw) To: lvm-devel You're right, that's a typo, and now it's on 1.0.4 On Sat, May 13, 2023 at 2:10?AM haaber <haaber@web.de> wrote: > Dear Ming-Hung, > > thank you for this inspection! I now have hope again to recover my data > :) > > Silly question: I cloned > https://github.com/jthornber/thin-provisioning-tools and installed that > way successfully 1.0.4 but that is not the 1.0.5 branch you talked > about. Could you point the right 1.0.5 git, please? > > thank you, Bernhard > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20230513/4b14d25d/attachment.htm> ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-05-11 7:39 ` haaber 2023-05-12 3:29 ` Ming Hung Tsai @ 2023-05-17 15:17 ` Ming Hung Tsai 2023-05-20 20:34 ` haaber 1 sibling, 1 reply; 21+ messages in thread From: Ming Hung Tsai @ 2023-05-17 15:17 UTC (permalink / raw) To: lvm-devel Hi, I've pushed the changes upstream. Now you should be able to repair the pool via "lvconvert --repair" after installation. On Thu, May 11, 2023 at 3:39?PM haaber <haaber@web.de> wrote: > Dear all, > > We need the exact binary copy of _tmeta LV - thus just use > > > > dd if=/dev/qubes_dom0/pool00_tmeta /tmp/tmeta_copy bs=512K > > bzip2 /tmp/tmeta_copy > > > the output is here: https://we.tl/t-AEmlc5CYeH > > > With this data - also provide full lvm2 metadata for this VG > > (should be as a file in /etc/lvm/backup - or you could run > > just vgcfgbackup) > > > I attached that one directly. Thank you very much! > > best, Bernhard > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20230517/3aabf4cc/attachment.htm> ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-05-17 15:17 ` Ming Hung Tsai @ 2023-05-20 20:34 ` haaber 2023-05-22 7:40 ` Ming Hung Tsai 0 siblings, 1 reply; 21+ messages in thread From: haaber @ 2023-05-20 20:34 UTC (permalink / raw) To: lvm-devel Hi Ming thank you so much. I compiled it, and did make install, but lvconvert is still the old one. It should be possible to do the job with ? pdata_tools? directly, right !?? I had formerly created a "newlv" inside my qubes_dom0 pool, so I did?? run ./pdata_tools? thin_repair -i /dev/qubes_dom0/pool00_tmeta?? -o /dev/qubes_dom0/newlv that command worked 5 seconds, and came back without any notice, which usually is good sign. I did not run it with a verbose flag, stupid me. Can I now run some command that activates the pool with "newlv" as metadata ? Or backup the old metadata file, and then copy "newlv" metadata into the pool00_tmeta ? I am, of course,? afraid of destroying it all, so close to? the end, so I better ask once more? :-) best, Bernhard On 5/17/23 17:17, Ming Hung Tsai wrote: > Hi, > > I've pushed the changes upstream. Now you should be able to repair the > pool via "lvconvert --repair" after installation. > > On Thu, May 11, 2023 at 3:39?PM haaber <haaber@web.de> wrote: > > Dear all, > > We need the exact binary copy of _tmeta? LV? - thus just use > > > > dd if=/dev/qubes_dom0/pool00_tmeta? /tmp/tmeta_copy bs=512K > > bzip2 /tmp/tmeta_copy > > > the output is here: https://we.tl/t-AEmlc5CYeH > > > With this data - also provide full lvm2 metadata for this VG > > (should be as a file in? /etc/lvm/backup? - or you could run > > just vgcfgbackup) > > > I attached that one directly. Thank you very much! > > best, Bernhard > > > -- > lvm-devel mailing list > lvm-devel at redhat.com > https://listman.redhat.com/mailman/listinfo/lvm-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20230520/47f3e0ff/attachment.htm> ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-05-20 20:34 ` haaber @ 2023-05-22 7:40 ` Ming Hung Tsai 2023-05-23 15:24 ` [SOLVED] " haaber 0 siblings, 1 reply; 21+ messages in thread From: Ming Hung Tsai @ 2023-05-22 7:40 UTC (permalink / raw) To: lvm-devel Hi, There's a debug option `-v` for thin_dump/thin_repair to show verbose logs including repairing details. In your case, you should see one compatible root pair found in your metadata: ``` compatible roots (1): (1150, 7643) ``` Once you had thin_repair'ed the metadata, you could swap the repaired one into the pool by using lvconvert: `lvconvert qubes_dom0/pool00 --swapmetadata --poolmetadata qubes_dom0/newlv` The two volumes "pool00_tmeta" and "newlv" then will have their names swapped, i.e., "newlv" becomes "pool00_tmeta" and the original "pool00_tmeta" becomes "newlv", so you have the backup. On Mon, May 22, 2023 at 2:57?PM haaber <haaber@web.de> wrote: > Hi Ming > > thank you so much. I compiled it, and did make install, but lvconvert is > still the old one. It should be possible to do the job with pdata_tools > directly, right !? I had formerly created a > "newlv" inside my qubes_dom0 pool, so I did run > > ./pdata_tools thin_repair -i /dev/qubes_dom0/pool00_tmeta -o > /dev/qubes_dom0/newlv > > that command worked 5 seconds, and came back without any notice, which > usually is good sign. I did not run it with a verbose flag, stupid me. > > Can I now run some command that activates the pool with "newlv" as > metadata ? Or backup the old metadata file, and then copy "newlv" metadata > into the pool00_tmeta ? I am, of course, afraid of destroying it all, so > close to the end, so I better ask once more :-) > > best, Bernhard > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20230522/3f4a66ed/attachment.htm> ^ permalink raw reply [flat|nested] 21+ messages in thread
* [SOLVED] Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-05-22 7:40 ` Ming Hung Tsai @ 2023-05-23 15:24 ` haaber 0 siblings, 0 replies; 21+ messages in thread From: haaber @ 2023-05-23 15:24 UTC (permalink / raw) To: lvm-devel Dear Ming, Zdenek and others, On 5/22/23 09:40, Ming Hung Tsai wrote: > lvconvert qubes_dom0/pool00 --swapmetadata --poolmetadata qubes_dom0/newlv issue solved, thank you SO MUCH! Just copying all data to a new drive :) best, Bernhard ^ permalink raw reply [flat|nested] 21+ messages in thread
* Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure 2023-04-25 13:49 Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure haaber 2023-04-26 11:10 ` Zdenek Kabelac @ 2023-04-26 12:06 ` Ming Hung Tsai 1 sibling, 0 replies; 21+ messages in thread From: Ming Hung Tsai @ 2023-04-26 12:06 UTC (permalink / raw) To: lvm-devel Hi, On Tue, Apr 25, 2023 at 9:54?PM haaber <haaber@web.de> wrote: > > Dear all, > > I had a lethally bad hardware failure and to replace the machine. Now I try to get some data back that is not contained in half-year backups ... (I know! but it's too late to be sorry). OK, the old SSD is attached via usb adapter to a brand new machine. I started > > sudo pvscan > sudo vgscan --mknodes > sudo vgchange -ay > > Here is the unexpected output: > > PV /dev/mapper/OLDSSD VG vg0 lvm2 [238.27 GiB / <15.79 GiB free] > Total: 1 [238.27 GiB] / in use: 1 [238.27 GiB] / in no VG: 0 [0 ] > Found volume group "vg0" using metadata type lvm2 > Check of pool vg0/pool00 failed (status:1). Manual repair required! > 1 logical volume(s) in volume group "vg0" now active > > then I consulted dr. google for diagnosis, but found only little help. This one > > https://mellowhost.com/billing/index.php?rp=/knowledgebase/65/How-to-Repair-a-lvm-thin-pool.html > > suggested to deactivate all sub-volumes so that a repair can work correctly. It happened that only swap was > active, so I deactivated it. But repair does still not work: > > lvconvert --repair vg0/pool00 > terminate called after throwing an instance of 'std::runtime_error' > what(): transaction_manager::new_block() couldn't allocate new block > Child 21255 exited abnormally > Repair of thin metadata volume of thin pool vg0/pool00 failed > (status:-1). Manual repair required! > > > I would like to find a good soul out there that can give more hints. In particular, > could it be a metadata overflow? How to check? I seek not for repair, but a "once only" > read access to the pool data .... That should be the 'metadata overflow' you're referring to, i.e., running out of metadata space. By default, lvconvert allocates a new metadata volume of the same size, and that might not be sufficient for restoring a bulk of snapshots. The new version of thin-provisioning-tools (1.0.x) has addressed this issue, so you could give it a try. Alternatively, you might have to run thin_repair manually on a larger metadata volume if you want to stick with the current version. Ming-Hung Tsai ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2023-05-23 15:24 UTC | newest] Thread overview: 21+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-04-25 13:49 Data recovery -- thin provisioned LVM metadata (?) problem after hardware failure haaber 2023-04-26 11:10 ` Zdenek Kabelac 2023-04-26 13:12 ` haaber 2023-04-27 9:29 ` Zdenek Kabelac 2023-05-03 16:48 ` haaber 2023-05-04 13:17 ` Zdenek Kabelac 2023-05-04 16:31 ` haaber 2023-05-05 15:14 ` Zdenek Kabelac 2023-05-04 17:06 ` haaber 2023-05-05 9:42 ` Ming Hung Tsai 2023-05-05 15:07 ` Zdenek Kabelac 2023-05-05 16:25 ` Ming Hung Tsai 2023-05-11 7:39 ` haaber 2023-05-12 3:29 ` Ming Hung Tsai 2023-05-12 18:05 ` haaber 2023-05-13 3:20 ` Ming Hung Tsai 2023-05-17 15:17 ` Ming Hung Tsai 2023-05-20 20:34 ` haaber 2023-05-22 7:40 ` Ming Hung Tsai 2023-05-23 15:24 ` [SOLVED] " haaber 2023-04-26 12:06 ` Ming Hung Tsai
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.