From: David T-G <davidtg-robot@justpickone.org>
To: Linux RAID list <linux-raid@vger.kernel.org>
Subject: how do i fix these RAID5 arrays?
Date: Wed, 23 Nov 2022 22:07:36 +0000 [thread overview]
Message-ID: <20221123220736.GD19721@jpo> (raw)
Hi, all --
TL;DR : I'm providing lots of detail to try to not leave anything
unexplained, but in the end I need to remove "removed" devices from
RAID5 arrays and add them back to rebuild.
I have 3ea 10T (in round numbers, of course :-) drives
diskfarm:~ # fdisk -l /dev/sd[bcd]
Disk /dev/sdb: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: TOSHIBA HDWR11A
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: EDF3B089-018E-454F-BD3F-6161A0A0FBFB
Device Start End Sectors Size Type
/dev/sdb51 2048 3254781951 3254779904 1.5T Linux LVM
/dev/sdb52 3254781952 6509561855 3254779904 1.5T Linux LVM
/dev/sdb53 6509561856 9764341759 3254779904 1.5T Linux LVM
/dev/sdb54 9764341760 13019121663 3254779904 1.5T Linux LVM
/dev/sdb55 13019121664 16273901567 3254779904 1.5T Linux LVM
/dev/sdb56 16273901568 19528681471 3254779904 1.5T Linux LVM
/dev/sdb128 19528681472 19532873694 4192223 2G Linux filesystem
Disk /dev/sdc: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: TOSHIBA HDWR11A
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 1AD8FC0A-5ADD-49E6-9BB2-6161A0BEFBFB
Device Start End Sectors Size Type
/dev/sdc51 2048 3254781951 3254779904 1.5T Linux LVM
/dev/sdc52 3254781952 6509561855 3254779904 1.5T Linux LVM
/dev/sdc53 6509561856 9764341759 3254779904 1.5T Linux LVM
/dev/sdc54 9764341760 13019121663 3254779904 1.5T Linux LVM
/dev/sdc55 13019121664 16273901567 3254779904 1.5T Linux LVM
/dev/sdc56 16273901568 19528681471 3254779904 1.5T Linux LVM
/dev/sdc128 19528681472 19532873694 4192223 2G Linux filesystem
Disk /dev/sdd: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: TOSHIBA HDWR11A
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: EDF3B089-018E-454F-BD3F-6161A0A0FBFB
Device Start End Sectors Size Type
/dev/sdd51 2048 3254781951 3254779904 1.5T Linux LVM
/dev/sdd52 3254781952 6509561855 3254779904 1.5T Linux LVM
/dev/sdd53 6509561856 9764341759 3254779904 1.5T Linux LVM
/dev/sdd54 9764341760 13019121663 3254779904 1.5T Linux LVM
/dev/sdd55 13019121664 16273901567 3254779904 1.5T Linux LVM
/dev/sdd56 16273901568 19528681471 3254779904 1.5T Linux LVM
/dev/sdd128 19528681472 19532873694 4192223 2G Linux filesystem
that I've sliced, RAID5-ed
diskfarm:~ # mdadm -D /dev/md51
/dev/md51:
Version : 1.2
Creation Time : Thu Nov 4 00:46:28 2021
Raid Level : raid5
Array Size : 4881773568 (4.55 TiB 5.00 TB)
Used Dev Size : 1627257856 (1551.87 GiB 1666.31 GB)
Raid Devices : 4
Total Devices : 3
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Nov 23 02:53:35 2022
State : clean, degraded
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : bitmap
Name : diskfarm:51 (local to host diskfarm)
UUID : 9330e44f:35baf039:7e971a8e:da983e31
Events : 37727
Number Major Minor RaidDevice State
0 259 9 0 active sync /dev/sdb51
1 259 2 1 active sync /dev/sdc51
3 259 16 2 active sync /dev/sdd51
- 0 0 3 removed
diskfarm:~ # mdadm -E /dev/md51
/dev/md51:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : cccbe073:d92c6ecd:77ba5c46:5db6b3f0
Name : diskfarm:10T (local to host diskfarm)
Creation Time : Thu Nov 4 00:56:36 2021
Raid Level : raid0
Raid Devices : 6
Avail Dev Size : 6508767232 sectors (3.03 TiB 3.33 TB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=3254515712 sectors
State : clean
Device UUID : 4eb64186:15de3406:50925d42:54df22e1
Update Time : Thu Nov 4 00:56:36 2021
Bad Block Log : 512 entries available at offset 8 sectors
Checksum : 45a70eae - correct
Events : 0
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
into arrays (ignore the "degraded" for the moment), and striped
diskfarm:~ # mdadm -D /dev/md50
/dev/md50:
Version : 1.2
Creation Time : Thu Nov 4 00:56:36 2021
Raid Level : raid0
Array Size : 19526301696 (18.19 TiB 19.99 TB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent
Update Time : Thu Nov 4 00:56:36 2021
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : -unknown-
Chunk Size : 512K
Consistency Policy : none
Name : diskfarm:10T (local to host diskfarm)
UUID : cccbe073:d92c6ecd:77ba5c46:5db6b3f0
Events : 0
Number Major Minor RaidDevice State
0 9 51 0 active sync /dev/md/51
1 9 52 1 active sync /dev/md/52
2 9 53 2 active sync /dev/md/53
3 9 54 3 active sync /dev/md/54
4 9 55 4 active sync /dev/md/55
5 9 56 5 active sync /dev/md/56
diskfarm:~ # mdadm -E /dev/md50
/dev/md50:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
into a 20T array, the idea being that each piece of which should take
less time to rebuild if something fails. That was all great, and then I
wanted to add another disk
diskfarm:~ # fdisk -l /dev/sdk
Disk /dev/sdk: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: TOSHIBA HDWR11A
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: FAB535F8-F57B-4BA4-8DEB-B0DEB49496C1
Device Start End Sectors Size Type
/dev/sdk51 2048 3254781951 3254779904 1.5T Linux LVM
/dev/sdk52 3254781952 6509561855 3254779904 1.5T Linux LVM
/dev/sdk53 6509561856 9764341759 3254779904 1.5T Linux LVM
/dev/sdk54 9764341760 13019121663 3254779904 1.5T Linux LVM
/dev/sdk55 13019121664 16273901567 3254779904 1.5T Linux LVM
/dev/sdk56 16273901568 19528681471 3254779904 1.5T Linux LVM
/dev/sdk128 19528681472 19532873694 4192223 2G Linux filesystem
to it to give me 30T usable.
I sliced up the new drive as above, added each slice to each RAID5 array,
and then grow-ed each array to take advantage of it. And, sure enough,
for 52 it worked:
diskfarm:~ # mdadm -D /dev/md52
/dev/md52:
Version : 1.2
Creation Time : Thu Nov 4 00:47:09 2021
Raid Level : raid5
Array Size : 4881773568 (4.55 TiB 5.00 TB)
Used Dev Size : 1627257856 (1551.87 GiB 1666.31 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Nov 23 02:52:00 2022
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : bitmap
Name : diskfarm:52 (local to host diskfarm)
UUID : d9eada18:29478a43:37654ef5:d34df19c
Events : 10996
Number Major Minor RaidDevice State
0 259 10 0 active sync /dev/sdb52
1 259 3 1 active sync /dev/sdc52
3 259 17 2 active sync /dev/sdd52
4 259 24 3 active sync /dev/sdk52
diskfarm:~ # mdadm -E /dev/md52
/dev/md52:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : cccbe073:d92c6ecd:77ba5c46:5db6b3f0
Name : diskfarm:10T (local to host diskfarm)
Creation Time : Thu Nov 4 00:56:36 2021
Raid Level : raid0
Raid Devices : 6
Avail Dev Size : 6508767232 sectors (3.03 TiB 3.33 TB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=3254515712 sectors
State : clean
Device UUID : 74ab812f:7e1695ec:360638b6:0c73d8b0
Update Time : Thu Nov 4 00:56:36 2021
Bad Block Log : 512 entries available at offset 8 sectors
Checksum : 18d743dd - correct
Events : 0
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
THAT is what really confuses me. I ran (sorry; they're gone) the same
commands for each device; they should work the same way! But, obviously,
something ain't right.
On the 5 broken ones, we have one each removed device
diskfarm:~ # mdadm -D /dev/md5[13456] | egrep '^/dev|active|removed'
/dev/md51:
0 259 9 0 active sync /dev/sdb51
1 259 2 1 active sync /dev/sdc51
3 259 16 2 active sync /dev/sdd51
- 0 0 3 removed
/dev/md53:
0 259 11 0 active sync /dev/sdb53
1 259 4 1 active sync /dev/sdc53
3 259 18 2 active sync /dev/sdd53
- 0 0 3 removed
/dev/md54:
0 259 12 0 active sync /dev/sdb54
1 259 5 1 active sync /dev/sdc54
3 259 19 2 active sync /dev/sdd54
- 0 0 3 removed
/dev/md55:
0 259 13 0 active sync /dev/sdb55
1 259 6 1 active sync /dev/sdc55
3 259 20 2 active sync /dev/sdd55
- 0 0 3 removed
/dev/md56:
0 259 14 0 active sync /dev/sdb56
1 259 7 1 active sync /dev/sdc56
3 259 21 2 active sync /dev/sdd56
- 0 0 3 removed
that are obviously the sdk (new disk) slice. If md52 were also broken,
I'd figure that the disk was somehow unplugged, but I don't think I can
plug in one sixth of a disk and leave the rest unhooked :-) So ... In
addition to wondering how I got here, how do I remove the "removed" ones
and then re-add them to build and grow and finalize this?
TIA
:-D
--
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt
next reply other threads:[~2022-11-23 22:07 UTC|newest]
Thread overview: 62+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-11-23 22:07 David T-G [this message]
2022-11-23 22:28 ` how do i fix these RAID5 arrays? Roman Mamedov
2022-11-24 0:01 ` Roger Heflin
2022-11-24 21:20 ` David T-G
2022-11-24 21:49 ` Wol
2022-11-25 13:36 ` and dm-integrity, too (was "Re: how do i fix these RAID5 arrays?") David T-G
2022-11-24 21:10 ` how do i fix these RAID5 arrays? David T-G
2022-11-24 21:33 ` Wol
2022-11-25 1:16 ` Roger Heflin
2022-11-25 13:22 ` David T-G
[not found] ` <CAAMCDed1-4zFgHMS760dO1pThtkrn8K+FMuG-QQ+9W-FE0iq9Q@mail.gmail.com>
2022-11-25 19:49 ` David T-G
2022-11-28 14:24 ` md RAID0 can be grown (was "Re: how do i fix these RAID5 arrays?") David T-G
2022-11-29 21:17 ` Jani Partanen
2022-11-29 22:22 ` Roman Mamedov
2022-12-03 5:41 ` md vs LVM and VMs and ... (was "Re: md RAID0 can be grown (was ...") David T-G
2022-12-03 12:06 ` Wols Lists
2022-12-03 18:04 ` batches and serial numbers (was "Re: md vs LVM and VMs and ...") David T-G
2022-12-03 20:07 ` Wols Lists
2022-12-04 2:47 ` batches and serial numbers David T-G
2022-12-04 13:54 ` Wols Lists
2022-12-04 13:04 ` batches and serial numbers (was "Re: md vs LVM and VMs and ...") Reindl Harald
2022-12-03 5:41 ` md RAID0 can be grown David T-G
2022-11-25 13:30 ` about linear and about RAID10 (was "Re: how do i fix these RAID5 arrays?") David T-G
2022-11-25 14:23 ` Wols Lists
2022-11-25 19:50 ` about linear and about RAID10 David T-G
2022-11-25 18:00 ` about linear and about RAID10 (was "Re: how do i fix these RAID5 arrays?") Roger Heflin
2022-11-28 14:46 ` about linear and about RAID10 David T-G
2022-11-28 15:32 ` Reindl Harald
[not found] ` <CAAMCDecXkcmUe=ZFnJ_NndND0C2=D5qSoj1Hohsrty8y1uqdfw@mail.gmail.com>
2022-11-28 17:03 ` Reindl Harald
2022-11-28 20:45 ` John Stoffel
2022-12-03 5:58 ` David T-G
2022-12-03 12:16 ` Wols Lists
2022-12-03 18:27 ` David T-G
2022-12-03 23:26 ` Wol
2022-12-04 2:53 ` David T-G
2022-12-04 13:13 ` Reindl Harald
2022-12-04 13:08 ` Reindl Harald
2022-12-03 5:45 ` David T-G
2022-12-03 12:20 ` Reindl Harald
[not found] ` <CAAMCDee_YrhXo+5hp31YXgUHkyuUr-zTXOqi0-HUjMrHpYMkTQ@mail.gmail.com>
2022-12-03 5:52 ` stripe size checking (was "Re: about linear and about RAID10") David T-G
2022-11-25 14:49 ` how do i fix these RAID5 arrays? Wols Lists
2022-11-26 20:02 ` John Stoffel
2022-11-27 9:33 ` Wols Lists
2022-11-27 11:46 ` Reindl Harald
2022-11-27 11:52 ` Wols Lists
2022-11-27 12:06 ` Reindl Harald
2022-11-27 14:33 ` Wol
2022-11-27 18:08 ` Roman Mamedov
2022-11-27 19:21 ` Wol
2022-11-28 1:26 ` Reindl Harald
2022-11-27 18:23 ` Reindl Harald
2022-11-27 19:30 ` Wol
2022-11-27 19:51 ` Reindl Harald
2022-11-27 14:10 ` piergiorgio.sartor
2022-11-27 18:21 ` Reindl Harald
2022-11-27 19:37 ` Piergiorgio Sartor
2022-11-27 19:52 ` Reindl Harald
2022-11-27 22:05 ` Wol
2022-11-27 22:08 ` Reindl Harald
2022-11-27 22:11 ` Reindl Harald
2022-11-27 22:17 ` Roman Mamedov
2022-11-27 14:58 ` John Stoffel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20221123220736.GD19721@jpo \
--to=davidtg-robot@justpickone.org \
--cc=linux-raid@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).