From: "Robert Schöftner" <rmu@unfoo.net>
To: linux-lvm@redhat.com
Subject: [linux-lvm] harddisk dies while pvmove is in progress
Date: Mon, 12 Sep 2011 22:30:22 +0200 [thread overview]
Message-ID: <4E6E6BDE.1040301@unfoo.net> (raw)
hi!
i recently noticed some read errors off of one of my "storage"
harddisks. layout was something like 1*2tb + 1*1,5tb in a linear
mapping, 1 VG (storage_1), 2 LVs (shares, homevideo). the 1,5gig
harddisk failed.
so i ordered some disks, built a raid5 out of 4 2tb disks, and added it
to storage_1. I pvmoved the 2tb disk to the new raid. then i tried to
move the failed 1,5tb disk to the new raid. this process stopped when
pvmove was at about 50%, when the drive went completely dead. server was
rebooted.
it seems pvmove was able to move eveything of LV shares to the raid.
situation as of now:
* LVs fail to activate (needed lvchange -a y --partial)
* after activation, homevideo is completely dead, shares seems to work
without any problems (though it generates lots of device-mapper: raid1:
Unable to read primary mirror during recovery messages)
* percentage-counter counting, unsure if actual disk activity happens
* pvmove complains:
# pvmove
/dev/dm-4: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
/dev/dm-4: read failed after 0 of 4096 at 971224580096:
Eingabe-/Ausgabefehler
/dev/dm-4: read failed after 0 of 4096 at 971224637440:
Eingabe-/Ausgabefehler
/dev/dm-4: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
/dev/dm-5: read failed after 0 of 4096 at 1500298280960:
Eingabe-/Ausgabefehler
/dev/dm-5: read failed after 0 of 4096 at 1500298338304:
Eingabe-/Ausgabefehler
Couldn't find device with uuid u1F6AW-pvCR-gM1c-O1c5-oWa0-s1Nd-oKTkf5.
Cannot change VG storage_1 while PVs are missing.
Consider vgreduce --removemissing.
# lvs -a -o +devices storage_1
/dev/dm-4: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
/dev/dm-4: read failed after 0 of 4096 at 971224580096:
Eingabe-/Ausgabefehler
/dev/dm-4: read failed after 0 of 4096 at 971224637440:
Eingabe-/Ausgabefehler
/dev/dm-4: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
/dev/dm-5: read failed after 0 of 4096 at 1500298280960:
Eingabe-/Ausgabefehler
/dev/dm-5: read failed after 0 of 4096 at 1500298338304:
Eingabe-/Ausgabefehler
Couldn't find device with uuid u1F6AW-pvCR-gM1c-O1c5-oWa0-s1Nd-oKTkf5.
LV VG Attr LSize Origin Snap% Move Log
Copy% Convert Devices
homevideo storage_1 -wI-a-
1,08t pvmove0(126141)
homevideo storage_1 -wI-a-
1,08t /dev/md127(553732)
[pvmove0] storage_1 p-C-ao 1,36t unknown device
38,07 unknown device(0),/dev/md127(604932)
[pvmove0] storage_1 p-C-ao 1,36t unknown device
38,07 unknown device(126141),/dev/md127(731073)
shares storage_1 -wI-a-
2,59t /dev/md127(128000)
shares storage_1 -wI-a-
2,59t pvmove0(0)
shares storage_1 -wI-a-
2,59t /dev/md127(485699)
shares storage_1 -wI-a-
2,59t /dev/md127(0)
# dmsetup table
storage_1-pvmove0-missing_1_0: 0 1896923136 error
storage_1-shares: 0 2930270208 linear 9:127 1048579072
storage_1-shares: 2930270208 1033347072 linear 252:5 0
storage_1-shares: 3963617280 557326336 linear 9:127 3978849280
storage_1-shares: 4520943616 1048576000 linear 9:127 3072
storage_1-homevideo: 0 1896923136 linear 252:5 1033347072
storage_1-homevideo: 1896923136 419430400 linear 9:127 4536175616
# dmsetup info -c
storage_1-pvmove0 252 5 L--w 2 2 0
LVM-ONKgAr1yc19fIeDyMClPFjDv92vxDEAFyOZtSj8iJexF2g6PMNRff3rYytIZ5RYJ
storage_1-pvmove0-missing_1_0 252 4 L--w 1 1 0
LVM-ONKgAr1yc19fIeDyMClPFjDv92vxDEAFyOZtSj8iJexF2g6PMNRff3rYytIZ5RYJ-missing_1_0
storage_1-shares 252 6 L--w 0 4 0
LVM-ONKgAr1yc19fIeDyMClPFjDv92vxDEAFvbEPv7JJKoctUb2kNtGqjFrfsrHiS0v3
storage_1-homevideo 252 7 L--w 0 2 0
LVM-ONKgAr1yc19fIeDyMClPFjDv92vxDEAFwFm52QuLABa7OKlHdYp3yEe8wxp6K07I
# dmsetup status
storage_1-pvmove0: 0 1033347072 linear
storage_1-pvmove0: 1033347072 1896923136 mirror 2 252:4 9:127
89145/1852464 1 SA 1 core
storage_1-pvmove0-missing_1_0: 0 1896923136 error
storage_1-shares: 0 2930270208 linear
storage_1-shares: 2930270208 1033347072 linear
storage_1-shares: 3963617280 557326336 linear
storage_1-shares: 4520943616 1048576000 linear
storage_1-homevideo: 0 1896923136 linear
storage_1-homevideo: 1896923136 419430400 linear
[irrelevant VGs removed from output]
So, LV homevideo is definitely lost, but shares seems to be OK, modulo
the "running"/interrupted pvmove. Is there a way to clean up this mess
without creating another LV and copying the contents of shares over?
thanx
Robert
next reply other threads:[~2011-09-12 20:30 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-09-12 20:30 Robert Schöftner [this message]
2011-09-12 21:08 ` [linux-lvm] harddisk dies while pvmove is in progress Stuart D. Gathman
2011-09-12 22:15 ` Robert Schöftner
2011-09-14 7:23 ` Robert Schöftner
2011-09-14 16:55 ` Ray Morris
2011-09-14 17:25 ` Stuart D. Gathman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4E6E6BDE.1040301@unfoo.net \
--to=rmu@unfoo.net \
--cc=linux-lvm@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).