* Re: RAID5 Performance
From: Brad Campbell @ 2016-07-27 3:15 UTC (permalink / raw)
To: Adam Goryachev, linux-raid@vger.kernel.org
In-Reply-To: <7b7d730f-2951-ba5f-7f6b-33624b59a02d@websitemanagers.com.au>
On 27/07/16 10:24, Adam Goryachev wrote:
> Hi all,
>
> I know, age old question, but I have the chance to change things up a
> bit, and I wanted to collect some thoughts/ideas.
>
> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top,
> DRBD on top, and finally iSCSI on top (and then used as VM raw disks for
> mostly windows VM's).
Wow. More layers than a wedding cake.
>
> My suspicion is that the actual load is made up of rather small random
> read/write, because that is the scenario that produced the worst
> performance results when I was initially setting this up, and seems to
> be what we are getting in practice.
>
> The last option is, what if I moved to RAID10? Would that provide a
> significant performance boost (completely removes the need to worry
> about chunk/stripe size because we always just write the exact data we
> want, no need to read/compute/write)?
> OR, is that read/compute overhead negligible since I'm using SSD and
> read performance is so quick?
I'll only comment from personal experience with anecdotal evidence.
I have a RAID10 comprised of 6 256GB SSD (3 Intel & 3 Samsung) used as
the backing for multiple VMs (raw files on Ext4).
Initially I played with a number of RAID types when setting up the array
(back in 2012) and found RAID10 offered the best compromise for my use
case. This was based on CPU usage (raid 5 & 6 parity calculations on
*every write*), the need for RMW cycles for small writes and trying to
balance block sizes. None of these things are an issue with RAID10 and
in general I found a measurable reduction in overhead and consequential
performance boost in the VM's.
This is a single machine with a relatively underpowered AMD FX-8350 CPU.
I found that with multiple VM's hitting the disks I was losing enough
CPU time to RAID overhead that it made things noticeably less responsive.
My only issue is half the disks return a deterministic value after
discard and the other half don't, so any raid check operations return a
gazillion mismatches. Not an operational issue, but one worth mentioning
if you were to use different model drives.
Regards,
Brad
^ permalink raw reply
* RAID5 Performance
From: Adam Goryachev @ 2016-07-27 2:24 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Hi all,
I know, age old question, but I have the chance to change things up a
bit, and I wanted to collect some thoughts/ideas.
Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top,
DRBD on top, and finally iSCSI on top (and then used as VM raw disks for
mostly windows VM's).
My current array looks like this:
/dev/md1:
Version : 1.2
Creation Time : Wed Aug 22 00:47:03 2012
Raid Level : raid5
Array Size : 3281935552 (3129.90 GiB 3360.70 GB)
Used Dev Size : 468847936 (447.13 GiB 480.10 GB)
Raid Devices : 8
Total Devices : 8
Persistence : Superblock is persistent
Update Time : Wed Jul 27 11:32:00 2016
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : san1:1 (local to host san1)
UUID : 707957c0:b7195438:06da5bc4:485d301c
Events : 2185221
Number Major Minor RaidDevice State
7 8 65 0 active sync /dev/sde1
13 8 1 1 active sync /dev/sda1
8 8 81 2 active sync /dev/sdf1
5 8 113 3 active sync /dev/sdh1
9 8 97 4 active sync /dev/sdg1
12 8 17 5 active sync /dev/sdb1
10 8 49 6 active sync /dev/sdd1
11 8 33 7 active sync /dev/sdc1
I've configured the following non-standard options:
echo 4096 > /sys/block/md1/md/stripe_cache_size
The following apply to all SSD's installed:
echo noop > $disk/queue/scheduler
echo 128 > ${disk}/queue/nr_requests
What I can measure (at peak periods) with iostat:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
sdi 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 78.00 59.00 79.00 86.00 0.74 0.52
15.55 0.02 0.15 0.20 0.09 0.15 2.40
sdg 35.00 48.00 68.00 79.00 0.52 0.44
13.39 0.02 0.14 0.24 0.05 0.11 1.60
sdf 46.00 65.00 86.00 98.00 0.76 0.58
14.96 0.03 0.17 0.09 0.24 0.09 1.60
sdh 97.00 45.00 70.00 141.00 0.66 0.68
12.96 0.08 0.36 0.29 0.40 0.34 7.20
sde 101.00 75.00 87.00 94.00 0.79 0.61
15.76 0.08 0.42 0.32 0.51 0.29 5.20
sdb 85.00 54.00 94.00 102.00 0.84 0.56
14.62 0.01 0.04 0.09 0.00 0.04 0.80
sdc 85.00 74.00 98.00 106.00 0.79 0.66
14.53 0.01 0.06 0.04 0.08 0.04 0.80
sdd 230.00 199.00 266.00 353.00 2.19 2.11
14.24 0.18 0.28 0.23 0.32 0.16 9.60
drbd0 0.00 0.00 0.00 2.00 0.00 0.00
4.50 0.08 38.00 0.00 38.00 20.00 4.00
drbd12 0.00 0.00 1.00 1.00 0.00 0.00
7.50 0.03 14.00 4.00 24.00 14.00 2.80
drbd1 0.00 0.00 0.00 2.00 0.00 0.03
32.00 0.09 44.00 0.00 44.00 22.00 4.40
drbd9 0.00 0.00 2.00 0.00 0.01 0.00
8.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd11 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd3 0.00 0.00 4.00 197.00 0.02 1.01
10.47 7.92 41.03 0.00 41.87 4.98 100.00
drbd4 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd17 0.00 0.00 1.00 0.00 0.00 0.00
8.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd5 0.00 0.00 0.00 7.00 0.00 0.03
8.00 0.22 30.29 0.00 30.29 28.57 20.00
drbd19 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd6 0.00 0.00 2.00 0.00 0.01 0.00
8.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd7 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd8 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd13 0.00 0.00 90.00 44.00 1.74 0.38
32.35 1.72 13.46 0.40 40.18 4.27 57.20
drbd15 0.00 0.00 2.00 33.00 0.02 0.29
17.86 1.40 40.91 0.00 43.39 28.34 99.20
drbd18 0.00 0.00 1.00 3.00 0.00 0.03
16.00 0.08 21.00 0.00 28.00 21.00 8.40
drbd14 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd10 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
As you can see, the DRBD devices are busy, and slowing down the VM's,
looking at the drives on the second server we can see why:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
sdf 67.00 76.00 64.00 113.00 0.52 0.62
13.17 0.26 1.47 0.06 2.27 1.45 25.60
sdg 39.00 61.00 50.00 114.00 0.35 0.56
11.38 0.45 2.76 0.08 3.93 2.71 44.40
sdd 49.00 67.00 50.00 109.00 0.39 0.57
12.40 0.75 4.73 0.00 6.90 4.70 74.80
sdh 55.00 54.00 52.00 104.00 0.42 0.51
12.12 0.81 5.21 0.23 7.69 5.13 80.00
sde 67.00 67.00 75.00 129.00 0.56 0.65
12.13 0.94 4.59 0.69 6.85 4.24 86.40
sda 64.00 76.00 58.00 109.00 0.48 0.61
13.29 0.84 5.03 0.21 7.60 4.89 81.60
sdb 35.00 72.00 57.00 104.00 0.36 0.57
11.84 0.69 4.27 0.14 6.54 4.22 68.00
sdc 118.00 144.00 228.00 269.00 1.39 1.50
11.92 1.21 2.43 1.88 2.90 1.50 74.40
md1 0.00 0.00 0.00 260.00 0.00 1.70
13.38 0.00 0.00 0.00 0.00 0.00 0.00
I've confirmed that the problem is that we have mixed two models of SSD
(520 series and 530 series), and that the 530 series drives perform
significantly worse (under load) in comparison. Above, the two 520
series are sdf and sdg while the other drives are 530 series. So, we
will be replacing all of the drives across both systems with 545s series
1000GB SSD's (which I've confirmed will operate same or better than the
520 series, sdc on the first machine above is one of these already).
Over the years, I've learned a lot about RAID and optimisation,
originally I configured things to optimise for super fast streaming
reads and streaming writes, but in practice, the actual work-load is
small random read/write, with the writes causing the biggest load.
Looking at this:
http://serverfault.com/questions/384273/optimizing-raid-5-for-backuppc-use-small-random-reads
>
> *
>
> Enhance the queue depth. Standard kernel queue depth is OK for old
> single drives with small caches, but not for modern drives or RAID
> arrays:
>
> echo 512 > /sys/block/sda/queue/nr_requests
>
So my question is should I increase the configured nr_requests above the
current 128?
If the chunk size is 64k, and there are 8 drives in total, then the
stripe size is currently 64k*7 = 448k, is this too big? My reading of
the mdadm man page suggests the minimum chunk size is 4k ("In any case
it must be a multiple of 4KB"). If I set the chunk size to 4k, then the
stripe size becomes 28k, which means for a random 4k write, we only need
to write 28k instead of 448k ?
The drives report a sector size of 512k, which I guess means the
smallest meaningful write that the drive can do is 512k, so should I
increase the chunk size to 512k to match? Or does that make it even worse?
Finally, the drive reports Host_Writes_32MiB in SMART, does that mean
that the drive needs to replace a entire 32MB chunk in order to
overwrite a sector? I'm guessing a chunk size of 32M is just crazy though...
Is there a better way to actually measure the different sizes and
quantity of read/writes being issued, so that I can make a more accurate
decision on chunk size/stripe size/etc... iostat seems to show an
average numbers, but not the number of 1k read/write, 4k read/write, 16k
read/write etc...
My suspicion is that the actual load is made up of rather small random
read/write, because that is the scenario that produced the worst
performance results when I was initially setting this up, and seems to
be what we are getting in practice.
The last option is, what if I moved to RAID10? Would that provide a
significant performance boost (completely removes the need to worry
about chunk/stripe size because we always just write the exact data we
want, no need to read/compute/write)?
OR, is that read/compute overhead negligible since I'm using SSD and
read performance is so quick?
For completeness, PV information:
PV Name /dev/md1
VG Name vg0
PV Size 3.06 TiB / not usable 2.94 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 801253
Free PE 33281
Allocated PE 767972
PV UUID c0PIEb-tUka-zBk3-lcGM-H89s-ayde-hcMUBZ
Any advice or assistance would be greatly appreciated.
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
* Kernel deadlock during mdadm reshape
From: Michael Shaver @ 2016-07-27 2:18 UTC (permalink / raw)
To: mdraid
I am experiencing the exact same problem reported in this thread:
http://www.spinics.net/lists/raid/msg52235.html
Also reported here:
https://forums.gentoo.org/viewtopic-t-1043706.html
And here:
https://bbs.archlinux.org/viewtopic.php?id=212108
I have a raid5 array of 2TB disks currently stuck at 94% of a mdadm
reshape squeal to a grow operation from 4 disks to 5. In my case, I did
have a drive drop out of the array during the reshape.
The PC has been rebooted many times now in an attempt to restart the
process, but no matter what I do, the array immediately locks up upon
assembly. The md127_raid5 kernel process immediately spikes to near
100% cpu, and md127_reshape immediately deadlocks, followed by udev
shortly after. At this point, any attempt to mount or interact with the
array will cause processes to hang.
Been trying to recover for about three weeks now, starting to run out of
ideas of what to try next.
What I have tried thus far:
1. Disabled all manner of intrusive security enforcement (selinux)
2. Attempted to 'freeze-reshape' but to no effect
3. Attempted to assemble with 'invalid-backup' but to no effect
4. Changed min and max through put values for array reshape but to no effect
5. Ran extended SMART tests against all drives (all pass, the faulty
drive has issues with going to sleep)
6. Booted live recovery CDs from a variety of kernel versions (as far
back as 3.6.10 and as far forward as 4.6.3)
7. Compiled latest mdadm
8. Disabled udev
9. Tried killing the md127_raid5 process before it could spike but to no
effect
10. Tried killing the md127_reshape process before it could deadlock but
to no effect
11. Swapped out drives to a different physical PC
Nothing I do seems to have any effect. The issue reproduces exactly the
same under all scenarios.
> mdadm --add /dev/md127 /dev/sdf1
> mdadm --grow /dev/md127 --raid-devices=5
--backup-file=/home/user/grow_md127.bak
> cat /prod/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdd1[1] sde1[5] sda1[4] sdf1[2]
5860147200 blocks super 1.2 level 5, 128k chunk, algorithm 2
[5/4] [_UUUU]
[==================>..] reshape = 94.3% (1842696832/1953382400)
finish=99999.99min speed=0K/sec
bitmap: 2/15 pages [8KB], 65536KB chunk
unused devices: <none>
> ps aux | grep md127
root 3568 98.4 0.0 0 0 ? R 21:35 1:16
[md127_raid5]
root 3569 0.0 0.0 0 0 ? D 21:35 0:00
[md127_reshape]
> ps aux | grep md | grep D
root 3569 0.0 0.0 0 0 ? D 21:35 0:00
[md127_reshape]
root 3570 0.0 0.0 0 0 ? D 21:35 0:00
[systemd-udevd]
> cat /proc/3569/stack
[<ffffffffc066af50>] raid5_get_active_stripe+0x310/0x6f0 [raid456]
[<ffffffffc066f87b>] reshape_request+0x2fb/0x940 [raid456]
[<ffffffffc06701e6>] raid5_sync_request+0x326/0x3a0 [raid456]
[<ffffffff8164136c>] md_do_sync+0x88c/0xe50
[<ffffffff8163dde9>] md_thread+0x139/0x150
[<ffffffff810c6c98>] kthread+0xd8/0xf0
[<ffffffff817da5c2>] ret_from_fork+0x22/0x40
[<ffffffffffffffff>] 0xffffffffffffffff
> cat /proc/3570/stack
[<ffffffff811b64d8>] __lock_page+0xc8/0xe0
[<ffffffff811cb8dd>] truncate_inode_pages_range+0x46d/0x880
[<ffffffff811cbd05>] truncate_inode_pages+0x15/0x20
[<ffffffff81281d8f>] kill_bdev+0x2f/0x40
[<ffffffff812832e5>] __blkdev_put+0x85/0x290
[<ffffffff8128399c>] blkdev_put+0x4c/0x110
[<ffffffff81283a85>] blkdev_close+0x25/0x30
[<ffffffff81249abf>] __fput+0xdf/0x1f0
[<ffffffff81249c0e>] ____fput+0xe/0x10
[<ffffffff810c514f>] task_work_run+0x7f/0xa0
[<ffffffff810ab0a8>] do_exit+0x2d8/0xb60
[<ffffffff810ab9b7>] do_group_exit+0x47/0xb0
[<ffffffff810b6cd1>] get_signal+0x291/0x610
[<ffffffff8102e137>] do_signal+0x37/0x710
[<ffffffff8100320c>] exit_to_mode_loop+0x8c/0xd0
[<ffffffff81003d21>] syscall_return_slowpath+0xa1/0xb0
[<ffffffff817da43a>] entry_SYSCALL_64_fastpath+0xa2/0xa4
[<ffffffffffffffff>] 0xffffffffffffffff
> cat /proc/3568/stack
[<ffffffffffffffff>] 0xffffffffffffffff
> mdadm -S /dev/md127 hangs
> reboot
> mdadm --assemble /dev/md127 /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sde1
/dev/sdf1 --verbose --backup-file=/home/user/grow_md127.bak
mdadm: /dev/sda1 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdc1 is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdd1 is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sde1 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sdf1 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/md127 has an active reshape - checking if critical section
needs to be restored
mdadm: No backup metadata on /home/user/grow_md127.bak
mdadm: too-old timestamp on backup-metadata on device-4
mdadm: If you think it is should be safe, try 'export
MDADM_GROW_ALLOW_OLD=1'
mdadm: added /dev/sdc1 to /dev/md127 as 0 (possibly out of date)
mdadm: added /dev/sdf1 to /dev/md127 as 2
mdadm: added /dev/sda1 to /dev/md127 as 3
mdadm: added /dev/sde1 to /dev/md127 as 4
mdadm: added /dev/sdd1 to /dev/md127 as 1
mdadm: /dev/md127 has been started with 4 drives (out of 5).
> cat /prod/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdd1[1] sde1[5] sda1[4] sdf1[2]
5860147200 blocks super 1.2 level 5, 128k chunk, algorithm 2
[5/4] [_UUUU]
[==================>..] reshape = 94.3% (1842696832/1953382400)
finish=99999.99min speed=0K/sec
bitmap: 2/15 pages [8KB], 65536KB chunk
unused devices: <none>
> mdadm -S /dev/md127 hangs
> reboot
> export MDADM_GROW_ALLOW_OLD=1
> mdadm --assemble /dev/md127 /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sde1
/dev/sdf1 --verbose --backup-file=/home/user/grow_md127.bak
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda1 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdc1 is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdd1 is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sde1 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sdf1 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/md127 has an active reshape - checking if critical section
needs to be restored
mdadm: No backup metadata on /home/user/grow_md127.bak
mdadm: accepting backup with timestamp 1467397557 for array with
timestamp 1469583355
mdadm: backup-metadata found on device-4 but is not needed
mdadm: added /dev/sdc1 to /dev/md127 as 0 (possibly out of date)
mdadm: added /dev/sdf1 to /dev/md127 as 2
mdadm: added /dev/sda1 to /dev/md127 as 3
mdadm: added /dev/sde1 to /dev/md127 as 4
mdadm: added /dev/sdd1 to /dev/md127 as 1
mdadm: /dev/md127 has been started with 4 drives (out of 5).
> cat /prod/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdd1[1] sde1[5] sda1[4] sdf1[2]
5860147200 blocks super 1.2 level 5, 128k chunk, algorithm 2
[5/4] [_UUUU]
[==================>..] reshape = 94.3% (1842696832/1953382400)
finish=99999.99min speed=0K/sec
bitmap: 2/15 pages [8KB], 65536KB chunk
unused devices: <none>
> mdadm -D /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Sun May 18 16:54:52 2014
Raid Level : raid5
Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
Raid Devices : 5
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Tue Jul 26 21:53:57 2016
State : clean, degraded, reshaping
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 128K
Reshape Status : 94% complete
Delta Devices : 1, (4->5)
Name : rza.eth0.net:0 (local to host rza.eth0.net)
UUID : 9d5d1606:414b51f8:b5173999:7239c63f
Events : 345137
Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 49 1 active sync /dev/sdd1
2 8 81 2 active sync /dev/sdf1
4 8 1 3 active sync /dev/sda1
5 8 65 4 active sync /dev/sde1
Looking for pointers on where to look next, if anyone has suggestions.
I am starting to step through code and debugging the kernel, but this is
out of my depth.
A couple of specific questions:
1. Am I correct in my understanding that the code for the md127_raid5
and md127_reshape processes are effectively in kernel space? My
understanding is that mdadm manages those kernel space processes? If I
want to debug the deadlock, I should be looking in the kernel portion of
linux raid?
2. Does md_reshape require md_raid5 to be running and vise-versa?
Would it be possible to force mdadm to only start one process or the other?
thanks for any tips or suggestions!
Michael
^ permalink raw reply
* Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB
From: bobzer @ 2016-07-27 2:16 UTC (permalink / raw)
To: Wols Lists, Adam Goryachev, Peter Grandi, Linux RAID
In-Reply-To: <5797C5CE.6020300@youngman.org.uk>
Thank you very much for your help
I realized that there's much better idea than my first idea....
I can plug all my drives the sata plug are not a problem :-)
I use around 4.5TB on my current raid 5
I will continue to think on this problem and in the meanwhile try to
find another disk
i didn't think about :
> > In general an in-place migration is a very dangerous operation
> > because it stresses existing hardware a lot plus it uses code
> > that is rarely used and is quite complex. Given that your
> > situation is already compromised.
>
> But copying everything off will stress it just as much, surely? The
> alternatives imho are worse ...
at first i thought about using rsync to do my copy (anyway i don't have a GUI)
at second i thought that an in-place migration would be a nice and
safe operation
now i don't what to think ?
the 3 4TB disk are new but the 2 2TB are 4 year old and consumer
grade (the 4TB are seagate NAS HDD)
so i'm asking myself should I do a copy (with rsync or another tool)
or an in-place migration ? which one is the safest ?
thanks again
Cheers,
Math
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: David C. Rankin @ 2016-07-27 0:19 UTC (permalink / raw)
To: mdraid
In-Reply-To: <CAJCQCtQQ=bEuN9f+4NX5VmVgtt41opTSddM7GQ8G6HMcb+VXmA@mail.gmail.com>
On 07/26/2016 06:23 PM, Chris Murphy wrote:
> Yeah fine, or type the count= first so that if you accidentally hit
> return after you've completed typing if= and of= but before count= you
> aren't zeroing your drive very slowly but still too fast for a cancel
> to help save you.
>
> Just don't use bs=1 when you go to zero the backup GPT, because the
> sector values are predicated on 512 byte sectors, which happens to be
> dd's default bs= size. So if you use bs=1 without altering the seek
> value, you'll break something again. :-D
>
Heh, heh... Yes, my proclivity for an errant key is notorious ;-)
All and all, thank you and Phil and the rest. This exercise has really helped
separate how the distinction between the array itself how it relates to the
filesystem and how that interacts with the underlying partitioning (the fact
that whole disk raid1 is fine). Where the initial confusion hit was my lack of
understanding the mdadm will in fact use the entire disk if you tell it to --
even if you intended to create the array out of partitions.
Once that occurred and I happily created the ext4 filesystem, I just blindly
thought it was within the sdc1/sdd1 partitions, and it never occurred to me that
it wasn't until this fiasco occurred. Even with the 50 times I've looked at the
mdstat info, nothing clicked regarding the missing number. The rest has been a
good learning experience. Stopping, restarting in degraded mode, etc.. when that
is something you rarely do (I think my last post involving working though a
drive failure was 2013...)
So thanks to all, In 165 more minutes I should be up and running again:
Personalities : [raid1]
md4 : active raid1 sdd[2] sdc[0]
2930135488 blocks super 1.2 [2/1] [U_]
[======>..............] recovery = 34.5% (1013009664/2930135488)
finish=165.7min speed=192823K/sec
bitmap: 2/22 pages [8KB], 65536KB chunk
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: Chris Murphy @ 2016-07-26 23:23 UTC (permalink / raw)
To: David C. Rankin; +Cc: mdraid
In-Reply-To: <5797EB6D.2000104@suddenlinkmail.com>
On Tue, Jul 26, 2016 at 4:59 PM, David C. Rankin
<drankinatty@suddenlinkmail.com> wrote:
> On 07/26/2016 05:10 PM, Phil Turmel wrote:
>> You won't be able to re-add sdd because as Chris said, "fixing" the
>> primary GPT broke mdadm's superblock. After zeroing the beginning 4k of
>> sdd, --add it to your array and let it rebuild.
>>
>> Phil
>
> Thanks Phil, what's the quickest way to zero the 4k, something like
>
> # dd of=/dev/sdd if=/dev/zero bs=1 count=4096
Yeah fine, or type the count= first so that if you accidentally hit
return after you've completed typing if= and of= but before count= you
aren't zeroing your drive very slowly but still too fast for a cancel
to help save you.
Just don't use bs=1 when you go to zero the backup GPT, because the
sector values are predicated on 512 byte sectors, which happens to be
dd's default bs= size. So if you use bs=1 without altering the seek
value, you'll break something again. :-D
--
Chris Murphy
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: Chris Murphy @ 2016-07-26 23:18 UTC (permalink / raw)
To: David C. Rankin; +Cc: mdraid
In-Reply-To: <5797E869.8080403@suddenlinkmail.com>
On Tue, Jul 26, 2016 at 4:47 PM, David C. Rankin
<drankinatty@suddenlinkmail.com> wrote:
> On 07/26/2016 03:47 PM, Chris Murphy wrote:
>>> So basically I just need to fix the partition table on sdc,
>> No just remove the GPT signatures, "45 46 49 20 50 41 52 54" and the
>> PMBR signature "55 aa" from the two drives.
>>
>> Restoring the primary GPT on sdd overwrote part of the mdadm metadata.
>> I'm not sure if --readd alone will fix that, or if one of the
>> --update= options is necessary as well, and if so which one.
>
> OK,
>
> Here is where I need a bit more help. Would I use 'dd' to write the zeros at
> some offset?, or was your mention of wipefs earlier intended as the approach to
> take (e.g., "wipefs -b -t or -o to remove the GPT signatures, while avoiding the
> mdadm and file system signatures.")
wipefs with -b is safer because it only erases the signature, which is
tiny, and easy to replace if you get the command wrong because it's
static information, and backs all of it up to the local directory.
You can use wipefs -a -b on this /dev/sdd because you do in fact want
all the signatures gone before you --add it back to the array and let
it rebuild. But you do not want to use -a on sdc because that'll find
and remove the signatures for mdadm and ext4 unless you use -t instead
of -a to limit what wipefs is going to wipe.
You can certainly use dd, you just have to make sure you get the
command exactly right, and seeing as this whole thread started out
because a command wasn't exactly right :-) I'm helping you err on the
side of caution.
So if you use dd, you're going to zero the first 2 512 byte sectors,
i.e. count=2. That will clobber the PMBR and the primary GPT header.
You don't have to hit anything more than that, but it doesn't hurt
anything to wipe the first 4096 bytes.
To get rid of the backup GPT you'll zero the last two sectors of the
drive. So first get the total number of sectors from something like
gdisk -l which gets you this information (in part):
Disk /dev/sda: 1953525168 sectors, 931.5 GiB
And do
dd if=/dev/zero of=/dev/sda seek=1953525167
That'll erase ..67 and ..68, but the header is in ..67, one sector
before the last one. Nothing should be in the last sector anyway but
I'd check first! I don't know if ext4 put something there. And do not
use the "last usable sector" because that's full 34 sectors from the
end and there very well may be ext4 metadata in there that you do not
want to step on with /dev/sdc.
> The real question for me is what is the effect of having /dev/sdc1 and
> /dev/sdd1 as unused partitions on the drive while I'm using the whole drive. Is
> that something that can bite me later?
It already bit you. All you have to do is forget again that you're not
using this partition table for anything, and then try to repair it and
you're back in this same situation. You or someone else who ends up
managing the drive. So yeah, it's not an in-use valid structure so I'd
invalidate it so that libblkid unambiguously tells you the only
signatures that matter onthe drive -> drives are not partitioned, they
are completely under the control of mdadm, and the logical array from
those members is ext4 or whatever.
Right now I understand I have a couple of
> options:
>
> Option 1: attempt a re-add of /dev/sdd to the md4 array currently running in
> degraded mode.
Just --add as Phil says. That'll add the proper metadata to sdd. First
get rid of the PMBR and GPT signatures.
>
> Do I need to delete sdd1 now while the disk is not being used before attempting
> a re-add sdd to the md4 array?
Yes.
>Does it matter?
Yes.
> Then if that can be successfully
> readded/synced, do I care about the fact that sdc has sdc1 on it and should I
> then --fail --remove sdc, fix the GPT header, delete sdc1 and then readd sdc to
> the md4 array? (or just leave as and ignore the GPT header issue reported by gdisk?)
You do not need to rebuild that drive, there's nothing wrong with it
other than the misleading, and currently unused, GPT and PMBR. Feel
free to just deal with /dev/sdd first, including its rebuild to
completion, before messing with /dev/sdc. And once you do move on to
/dev/sdc, I would umount the file system, stop the array, and then
overwrite the proper sectors as described with dd or wipefs -t, and
then either reboot or run partprobe to make sure the kernel's idea of
the drive's state is up to date. And then you can restart the array.
>
> Option 2: shrink the filesystem on sdc
Oh no don't do that... that's a PITA and totally not necessary.
> Option 3: If it all fails, and I start from scratch, what is the best way to
> wipe both drives completely to make sure there is no lingering trace of a
> superblock, etc. before recreating array?
Well you have to do a breakdown to really get it right, starting from the top.
First you wipefs -a /dev/md4 so you get rid of the ext4 signature.
Then you wipefs -a the member drives to get rid of GPT, PMBR, and
mdadm signatures.
>
> # mdadm -S /dev/md4
> # mdadm --zero-superblock /dev/sdc
> # mdadm --zero-superblock /dev/sdd
> # gdisk to 'fix' /dev/sdc
> # mdadm --create --verbose /dev/md4 --level=1 --metadata=1.2 \
> --raid-devices=2 /dev/sdc1 /dev/sdd1
> # mkfs.ext4 -v -L data -m 0.005 -b 4096 -E stride=16,stripe-width=32 /dev/md4
> # update mdadm.conf
> # (recopy data)
If you really care about having it partitioned, yes
>
> So it looks like it boils down to:
>
> (a) do I need to worry about removing unused sdc1/sdd1?
Worry is a strong word. It hasn't been a problem up until you got a
complaint from gdisk, didn't remember what you did when you built this
storage stack, and then fixed something that was actually not being
used anyway, which then broke something you were using.
So I think it's better to remove things you aren't using.
Then do I need to use
> 'dd' or 'wipefs' to fix the GPT and PMBR signatures on sdc (and I assume do
> nothing to sdd if I don't need to delete sdd1)
>
> (b) nuke it all and start over (if so what is the plan above OK?)
You do not need to nuke it all.
> I'll try the re-add of sdd to a and report back after your response.
After wiping sdd, you will use --add, not --re-add.
--
Chris Murphy
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: David C. Rankin @ 2016-07-26 22:59 UTC (permalink / raw)
To: mdraid
In-Reply-To: <767a0ad7-bb45-b0b8-5d8b-08f65d346fa9@turmel.org>
On 07/26/2016 05:10 PM, Phil Turmel wrote:
> You won't be able to re-add sdd because as Chris said, "fixing" the
> primary GPT broke mdadm's superblock. After zeroing the beginning 4k of
> sdd, --add it to your array and let it rebuild.
>
> Phil
Thanks Phil, what's the quickest way to zero the 4k, something like
# dd of=/dev/sdd if=/dev/zero bs=1 count=4096
??
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: David C. Rankin @ 2016-07-26 22:47 UTC (permalink / raw)
To: mdraid
In-Reply-To: <CAJCQCtSkE+-C5U9OEm0-zazLW2KHCMUqduDTA=UHsMxmf4DVrQ@mail.gmail.com>
On 07/26/2016 03:47 PM, Chris Murphy wrote:
>> So basically I just need to fix the partition table on sdc,
> No just remove the GPT signatures, "45 46 49 20 50 41 52 54" and the
> PMBR signature "55 aa" from the two drives.
>
> Restoring the primary GPT on sdd overwrote part of the mdadm metadata.
> I'm not sure if --readd alone will fix that, or if one of the
> --update= options is necessary as well, and if so which one.
OK,
Here is where I need a bit more help. Would I use 'dd' to write the zeros at
some offset?, or was your mention of wipefs earlier intended as the approach to
take (e.g., "wipefs -b -t or -o to remove the GPT signatures, while avoiding the
mdadm and file system signatures.")
The real question for me is what is the effect of having /dev/sdc1 and
/dev/sdd1 as unused partitions on the drive while I'm using the whole drive. Is
that something that can bite me later? Right now I understand I have a couple of
options:
Option 1: attempt a re-add of /dev/sdd to the md4 array currently running in
degraded mode.
Do I need to delete sdd1 now while the disk is not being used before attempting
a re-add sdd to the md4 array? Does it matter? Then if that can be successfully
readded/synced, do I care about the fact that sdc has sdc1 on it and should I
then --fail --remove sdc, fix the GPT header, delete sdc1 and then readd sdc to
the md4 array? (or just leave as and ignore the GPT header issue reported by gdisk?)
Option 2: shrink the filesystem on sdc so it will fit in sdc1 and move the
filesystem to the sdc1 partition before re-adding. (this I don't understand as
well -- how to move the shrunken filesystem from sdc to sdc1?) If I understand,
moving to sdc1 doesn't buy me anything and isn't necessary here. So we can
strike option 2 if this is correct.
Option 3: If it all fails, and I start from scratch, what is the best way to
wipe both drives completely to make sure there is no lingering trace of a
superblock, etc. before recreating array?
# mdadm -S /dev/md4
# mdadm --zero-superblock /dev/sdc
# mdadm --zero-superblock /dev/sdd
# gdisk to 'fix' /dev/sdc
# mdadm --create --verbose /dev/md4 --level=1 --metadata=1.2 \
--raid-devices=2 /dev/sdc1 /dev/sdd1
# mkfs.ext4 -v -L data -m 0.005 -b 4096 -E stride=16,stripe-width=32 /dev/md4
# update mdadm.conf
# (recopy data)
So it looks like it boils down to:
(a) do I need to worry about removing unused sdc1/sdd1? Then do I need to use
'dd' or 'wipefs' to fix the GPT and PMBR signatures on sdc (and I assume do
nothing to sdd if I don't need to delete sdd1)
(b) nuke it all and start over (if so what is the plan above OK?)
I'll try the re-add of sdd to a and report back after your response.
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: Phil Turmel @ 2016-07-26 22:10 UTC (permalink / raw)
To: David C. Rankin, mdraid
In-Reply-To: <5797D23A.7090606@suddenlinkmail.com>
On 07/26/2016 05:12 PM, David C. Rankin wrote:
> Let me know if you want me to copy any of the records for you with dd, etc..
> if they would have any value to your testing. I'm happy to do it. Otherwise,
> I'll scan and responses to the other posts in this thread to see if I can
> understand what my best way out of this mess is. Thanks!
Just zero the first 4k of each drive, wiping the broken GPT partition
table. Continue using the whole device in your array. No reformat or
other data movement required.
There's nothing wrong with using entire disks in your array.
You won't be able to re-add sdd because as Chris said, "fixing" the
primary GPT broke mdadm's superblock. After zeroing the beginning 4k of
sdd, --add it to your array and let it rebuild.
Phil
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: David C. Rankin @ 2016-07-26 21:12 UTC (permalink / raw)
To: mdraid
In-Reply-To: <CAJCQCtTUO9nLRDUVQ+nkykiQ2ebc-SO8Oh9VP3_WMVKO2fs7dw@mail.gmail.com>
On 07/26/2016 10:55 AM, Chris Murphy wrote:
> However, it does appear that in the OP's case that the array was
> created on the whole disk device, not on the partition. If true, I
> would remove the signatures on the GPT primary and secondary headers
> to make sure they're invalidated. Otherwise it's ambiguous what these
> drives are all about, are they single partition drives that are empty?
> Or are they whole device md members?
>
> I'd look at using wipefs -b -t or -o to remove the GPT signatures,
> while avoiding the mdadm and file system signatures.
Chris,
The sequence of events (stupidity - from the actual bash_history) that led to
this issue (I think) was partitioning sdc and sdd with sdc1 and sdd1, but then
creating the arrays with the whole disks by omitting the '1' during the create,
e.g.
# mdadm --create --verbose /dev/md4 --level=1 --metadata=1.2 \
--raid-devices=2 /dev/sdc /dev/sdd
then creating the filesystem on the array:
# mkfs.ext4 -v -L data -m 0.005 -b 4096 -E stride=16,stripe-width=32 /dev/md4
The gdisk warning regarding the primary GPT header is due to the unused
/dev/sdc1 and /dev/sdd1 partition header being overwritten during the raid
creation process mistakenly on /dev/sdc and /dev/sdd (whole drives). There is
nothing else on this server that would have attempted a read/write to the drives.
This is a backup box sits idle and configured to take over in case of a
problem with the primary server. There is only one user 'me' and, or course
'root', and the only logins are the weekly ssh for 'pacman -Syu' to update the
Archlinux install. The box simply boots to the multi-user.target and idles. That
is why I am confident it wasn't some other utility that caused the corruption.
The only thing possibility is the case where the Gigabyte virtual.bios file was
written to the beginning of the array (which seems unlikely that it would write
to an array instead of a single drive)
Since the header corruption was identical for sdc and sdd, it looks like a
side effect of whole-disk raid creation on top of two disks that were
partitioned and intended for the raid to exist on sdc1 and sdd1. My guess is
gdisk is complaining about the unused sdc1/sdd1 GPT header. My drive,
filesystem, array foo runs out before being able to look at the actual record on
the drive as you did in your test case above.
Let me know if you want me to copy any of the records for you with dd, etc..
if they would have any value to your testing. I'm happy to do it. Otherwise,
I'll scan and responses to the other posts in this thread to see if I can
understand what my best way out of this mess is. Thanks!
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: Chris Murphy @ 2016-07-26 20:47 UTC (permalink / raw)
To: David C. Rankin; +Cc: mdraid
In-Reply-To: <5797C415.8010206@suddenlinkmail.com>
On Tue, Jul 26, 2016 at 2:12 PM, David C. Rankin
<drankinatty@suddenlinkmail.com> wrote:
> On 07/26/2016 04:52 AM, Adam Goryachev wrote:
>> No, I'm saying that is an excellent idea, and it is exactly what I always do.
>> The problem is that you created the single large primary partition, and then
>> used the raw drive for the raid array instead of using the partition.
>
> Damn, that is exactly what I did, even though I intended to use sdc1 and sdd1.
> Checking bash_history for root, I find:
>
> mdadm --create --verbose --level=1 --metadata=1.2 --raid-devices=2 /dev/md4
> /dev/sdc /dev/sdd
When I try this, mdadm complains:
[root@f24s ~]# !994
mdadm -C /dev/md0 -n 2 -l raid1 /dev/VG/1 /dev/VG/2
mdadm: /dev/VG/1 appears to be part of a raid array:
level=raid0 devices=0 ctime=Wed Dec 31 17:00:00 1969
mdadm: partition table exists on /dev/VG/1 but will be lost or
meaningless after creating array
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
Continue creating array? n
mdadm: create aborted.
[root@f24s ~]# rpm -q mdadm
mdadm-3.3.4-4.fc24.x86_64
OK and now I re-read the original post, and the table is also damaged.
What I think is happening is mdadm v1.2 metadata at 4K offset is
inside the area for table entries 5-128. So even though there are no
such entries, gdisk is seeing this unexpected data in areas that
should be zero'd. But nothing else has actually been stepped on.
But it doesn't matter because you're not using the partitions you've
created anyway. It's still a good idea to remove the signature from
the GPT and the PMBR (three signatures).
> So basically I just need to fix the partition table on sdc,
No just remove the GPT signatures, "45 46 49 20 50 41 52 54" and the
PMBR signature "55 aa" from the two drives.
Restoring the primary GPT on sdd overwrote part of the mdadm metadata.
I'm not sure if --readd alone will fix that, or if one of the
--update= options is necessary as well, and if so which one.
--
Chris Murphy
^ permalink raw reply
* GPT Table broken on a Raid1 (necroposting)
From: Luis Panadero Guardeño @ 2016-07-26 20:34 UTC (permalink / raw)
To: linux-raid
http://linux-raid.vger.kernel.narkive.com/nhUT84sK/gpt-table-broken-on-a-raid1#post21
I just, had nearly the same problem but on reverse. I get a GPT error
on the Raid 1 volume, not on the raid hard disks. On my case, is on a
server that have an Intel Matrix Storage fake raid, and have only a
Ubuntu server 14.04.4 x64 installed it.
Also, I got, at same time, other more serious problem. The server,
have four hard disks. Two small of ~400MiB (before was of ~250MiB) and
tow big of 1 TiB. On the bigger, I had set a Raid 1 where the data and
some third party software had been installed (Jenkins, GitLab, Nexus),
and on the smaller, I setup another Raid 1 where the OS is installed.
A few days ago, one of the small hard disk begun to fail, and we
procced to replace it for a new unit. Well, the Raid proceed to dump
the empty fresh hard disk over the hard disk with the OS!!! Lucky, the
old hard disk, can been read and I can recover the OS image from it,
but I would like to know what we did wrong (apart of using a
FakeRaid). The hardware guy decided to poweroff the server to add the
new hard disk (yeah, I thought that not have much sense having a Raid
if you need to stop the server to replace a hard disk...), and add the
new hard disk to the raid using the Intel tool on the BIOS. I think
that this was what whipped out our OS sane hard disk. Could be ?
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: David C. Rankin @ 2016-07-26 20:34 UTC (permalink / raw)
To: mdraid
In-Reply-To: <CAJCQCtQ73Tp_ps57qgDm=S_qdFMHy8J0Z=kB+_TfQD2ugj_6ow@mail.gmail.com>
On 07/26/2016 10:19 AM, Chris Murphy wrote:
> It'd be interesting to see mdadm -E for sdc and sdd.
>
> GPT uses LBA 0-3. And mdadm metadata 1.2 is 4K from the start. These
> do not overlap. So I'm unconvinced that mdadm -C applied to sdc and
> sdd instead of sdc1 and sdd1 is the source of the problem.
>
> Further, gdisk specifically said the GPT header was corrupt. The PMBR,
> LBA 0, is intact, and the table data (LBA 2) is intact. Only LBA 2 was
> stepped on by something?
>
> What do you get for gdisk -l /dev/sdc? Another warning or is it OK?
>
> Also for what it's worth: primary, extended, logical are terms that do
> not apply to GPT partitioned disks. There is only one kind of
> partition with GPT disks, no distinctions.
>
>
Chris, sdc is now my degraded array (in the original drive state), sdd is the
one I fixed the gdisk error on. /dev/sdc still reports the main GPT header
corruption. The mdadm -E on both sdc/sdd show:
# mdadm -E /dev/sdc
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 6e520607:f152d8b9:dd2a3bec:5f9dc875
Name : valkyrie:4 (local to host valkyrie)
Creation Time : Mon Mar 21 02:27:21 2016
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 5860271024 (2794.39 GiB 3000.46 GB)
Array Size : 2930135488 (2794.39 GiB 3000.46 GB)
Used Dev Size : 5860270976 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=48 sectors
State : clean
Device UUID : e15f0ea7:7e973d0c:f7ae51a1:9ee4b3a4
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Jul 26 15:15:50 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : ff210f72 - correct
Events : 4248
Device Role : Active device 0
Array State : A. ('A' == active, '.' == missing, 'R' == replacing)
# mdadm -E /dev/sdd
/dev/sdd:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
The GPT header error shown on sdc is:
# gdisk -l /dev/sdc
GPT fdisk (gdisk) version 1.0.1
Caution: invalid main GPT header, but valid backup; regenerating main header
from backup!
Caution! After loading partitions, the CRC doesn't check out!
Warning! Main partition table CRC mismatch! Loaded backup partition table
instead of main partition table!
Warning! One or more CRCs don't match. You should repair the disk!
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: damaged
****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
Disk /dev/sdc: 5860533168 sectors, 2.7 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 3F835DD0-AA89-4F86-86BF-181F53FA1847
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 2048-sector boundaries
Total free space is 212958 sectors (104.0 MiB)
Number Start (sector) End (sector) Size Code Name
1 8192 5860328334 2.7 TiB FD00 Linux RAI
The sdd drive is just waiting in limbo for me to decide how best to tackle
putting all the pieces together again. Either (1) blow away md4 and lose the
array, recreate the array using sdc1/sdd1, create a new filesystem, and then
recopy the data (few hours of watching it format and copy....) or (2) save the
array and add sdd back and see if it will be come part of the array and just
live with my initial screwup of using sdc/sdd instead of sdc1/sdd1 and ignore
the gdisk warning (that is complaining about a partition table that isn't being
used anyway?) gdisk on sdd is now happy (of course nothing is using sdd at the
moment).
# gdisk -l /dev/sdd
GPT fdisk (gdisk) version 1.0.1
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Disk /dev/sdd: 5860533168 sectors, 2.7 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 3F835DD0-AA89-4F86-86BF-181F53FA1847
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 2048-sector boundaries
Total free space is 212958 sectors (104.0 MiB)
Number Start (sector) End (sector) Size Code Name
1 8192 5860328334 2.7 TiB FD00 Linux RAID
What do you think my best option is? Or, is it a take your pick type scenario?
Thanks for your help.
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: David C. Rankin @ 2016-07-26 20:24 UTC (permalink / raw)
To: mdraid
In-Reply-To: <54ca746c-36eb-c1cb-1975-9ecc619fb1b2@turmel.org>
On 07/26/2016 12:14 PM, Phil Turmel wrote:
> The drive probably came with a single large partition in a GPT table.
> mdadm was created using the whole disk, which neither needs nor expects
> there to be a partition table. (I do not use partitions for my big
> arrays -- whole disks only.)
>
> When mdadm created the array, it probably wiped out the starting table,
> but not the GPT backup. Which may or may not be outside the used data
> area of the array. So you allowed a partition tool to "fix" a partition
> table that isn't being used. The simplest solution is to zero the first
> 4k of the disk and zero the GPT backup partition table location.
>
> I suggest re-adding /dev/sdd to the array (not /dev/sdd1, and not
> --add), whether you delete the table or not. A --re-add will probably
> be nearly instant, since there is a bitmap.
Well, no, that was my stupidity in forgetting to add the '1' at the end of
sdc/sdd during mdadm --create. So you are saying it would be OK to add sdd back
into the array and just use the whole drive, even though there is an
empty/unformatted sdd1 (as there was all along) and that the whole-drive array
is OK?
Will adding it back into the array overwrite the primary PT again, or was that
just a result of initial array creation due to the fact I was using the whole
drive _and_ sdd1 existed as well.
(Well that looks to be the gdisk complaint, the sdd1 table was overwritten, but
that didn't effect the array operation because it was using the whole drive. --
so I could have ignored the gdisk warning and gone about my merry way without
this day of learning and angst? Ah, but what's the fun in that...?)
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB
From: Wols Lists @ 2016-07-26 20:19 UTC (permalink / raw)
To: Peter Grandi, Linux RAID
In-Reply-To: <22423.44287.974770.114127@tree.ty.sabi.co.uk>
On 26/07/16 19:33, Peter Grandi wrote:
> [ ... ]
>> migrate all my data from a raid 5 of 4x2TB (actually 3 because
>> i'm degraded right now) to a raid 6 of 4x4TB so exactly i got :
>> - raid 5, should be 4 disk of 2T but got problem and so right
>> now it's just 2x2T and a 2Tb disk image in a 4T disk (the
>> raid crashed and is not start right now but is clean)
>> - 2 disk of 4TB
>> - the raid 5 use lvm2 on top of mdadm
>> I want a raid 6 [ ... ]
>
> In general an in-place migration is a very dangerous operation
> because it stresses existing hardware a lot plus it uses code
> that is rarely used and is quite complex. Given that your
> situation is already compromised.
But copying everything off will stress it just as much, surely? The
alternatives imho are worse ...
>
> Plus your goal does not make a lot of sense: from a RAID5 of 4
> drives to a RAID6 of 4 drives. Very strange.
>
From a 6TB raid5 to an 8TB raid6? What's strange about that?
> Also I note that you currently don't have 4 drives of 4TB, but
> 3, so there you get your wish 2 drives of 2TB as the fourth
> member as in «2 2TB as a raid 0»
>
> The good thing about your plan is to use the larger drives to
> make a *copy* of your data, so you don't quite do an in-place
> migration.
>
So beg or borrow a 2TB (or larger) drive, provided you have enough slots
for 6 drives. Borrow or buy (they're cheap enough) an add-in SATA card
if you need to.
Now he can add the second 4TB drive to complete his raid5.
Add in the third 4TB to replace the 2TB partition and free up the partly
used 4TB drive.
Use the now-free first 4TB drive to replace one of the 2TB drives. Shame
he can't reshape the resulting array into a 3 x 4TB raid5 (at least, I
don't think he can).
Use the borrowed drive to replace the last 2TB drive.
Combine the 2 2TB drives into a 4TB raid0 array, and use that to replace
the borrowed drive.
We now have a 4 x 4TB raid5 array which he can convert to raid6.
If the worst comes to the worst and he can't get that spare drive, he
could always fail the last 2TB drive (taking it back to degraded raid5
:-( then combine the 2 x 2TB and add them back to get a fully working
array again.
Unfortunately, I can't think of any way to get to the planned final
config without either borrowing an extra drive, or having (at some
point) a degraded array.
Cheers,
Wol
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: David C. Rankin @ 2016-07-26 20:12 UTC (permalink / raw)
To: mdraid
In-Reply-To: <ff6872cd-98eb-aad2-a71a-7af52050ff30@websitemanagers.com.au>
On 07/26/2016 04:52 AM, Adam Goryachev wrote:
> No, I'm saying that is an excellent idea, and it is exactly what I always do.
> The problem is that you created the single large primary partition, and then
> used the raw drive for the raid array instead of using the partition.
Damn, that is exactly what I did, even though I intended to use sdc1 and sdd1.
Checking bash_history for root, I find:
mdadm --create --verbose --level=1 --metadata=1.2 --raid-devices=2 /dev/md4
/dev/sdc /dev/sdd
then
mkfs.ext4 -v -L data -m 0.005 -b 4096 -E stride=16,stripe-width=32 /dev/md4
So basically I just need to fix the partition table on sdc, that will leave both
sdc and sdd with health partition tables and a primary partition of sdc1 and
sdd1. I'll blow away the current degraded md4 and recreate md4 with
mdadm --create --verbose --level=1 --metadata=1.2 --raid-devices=2 /dev/md4
/dev/sdc1 /dev/sdd1
mkfs.ext4 -v -L data -m 0.005 -b 4096 -E stride=16,stripe-width=32 /dev/md4
Then its just a matter of re-copying the data, uncommenting fstab and updating
mdadm.conf?
This even solved the mystery of where the original corruption came from. That's
a hole in one.
As for either shrinking the filesystem on sdc to fit on sdc1 and sdd1, is that
worth attempting, or is it probably better to just blow away the existing array
and then recreate it all as indicated above? If I can save the filesystem, I
save a few hours of formatting, but I worry about the reliability of shrinking
the filesystem. (there is plenty of room, I have 258G used out of 2.7T) What is
the consensus? Is shrinking reliable, or is it something to consider as a worse
case scenario (as in the hypothetical replacing of a disk that is slightly
smaller than the failed one)?
Thank you again for your help. Glad to see Murphy's law is still well in force
and effect. (an entire PT mystery caused by some dummy that forgot to append the
'1' during the mdadm --create -- who would have imagined :)
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB
From: Peter Grandi @ 2016-07-26 18:33 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <CADzS=aqYQEoqxHbp7keeuYqN2dLur1O=RfctfuFsraggxiWTKg@mail.gmail.com>
[ ... ]
> migrate all my data from a raid 5 of 4x2TB (actually 3 because
> i'm degraded right now) to a raid 6 of 4x4TB so exactly i got :
> - raid 5, should be 4 disk of 2T but got problem and so right
> now it's just 2x2T and a 2Tb disk image in a 4T disk (the
> raid crashed and is not start right now but is clean)
> - 2 disk of 4TB
> - the raid 5 use lvm2 on top of mdadm
> I want a raid 6 [ ... ]
In general an in-place migration is a very dangerous operation
because it stresses existing hardware a lot plus it uses code
that is rarely used and is quite complex. Given that your
situation is already compromised.
Plus your goal does not make a lot of sense: from a RAID5 of 4
drives to a RAID6 of 4 drives. Very strange.
Also I note that you currently don't have 4 drives of 4TB, but
3, so there you get your wish 2 drives of 2TB as the fourth
member as in «2 2TB as a raid 0»
The good thing about your plan is to use the larger drives to
make a *copy* of your data, so you don't quite do an in-place
migration.
So, the questions really are how much data you have on your
existing 4x (degraded) 2TB RAID5, which will be at most 6TB, and
how many drives you can connect *at the same time*. You seem
sure to be able to connect all 5 drives: the 3x 4TB and the 2x
2TB. I really hope none of them is USB.
So in total you have 2x 2TB drives and 3x 4TB drives, of which
currently the 2TB drive are full and 1x 4TB drive is half full,
thus you have 6TB of data without redundancy and 10TB of free
space. In a very ideal world you would get an extra 4TB disk,
but you seem unable to do so...
Also, given that you are are seriously considering all of this,
I must assume that you are a very experienced RAID and storage
guru knowing all the little details that matter to success, and
to recovery in case of problems arising.
Let's call A and B the 2x 2TB disks, C the 4TB disk with the 2TB
disk image, and D and E the empty 4TB disks.
The least scary option might be:
* Split into two partitions D and E.
* Block copy the image on C to D1 and E2.
* Re-partition C also in two.
* Block copy B to C1.
* Block copy A to E2.
* Now we have greatly increased redundancy, as we have the 3x
2TB data slices on two separate sets, A, C1, D1 and D2, B,
E2.
* Add E1 as a spare to the A, C1, D1 RAID5 set, and start it so
you end up with a full RAID5 set on A, C1, D1, E1 after
resync end.
* Now that the first RAID5 set is not degraded, you can erase
the copies on D2, B, E2, and create a second RAID5 set on
B, C2, D2, E2, which will be empty.
A slight improvement to your scheme:
* Add a second 2TB disk image to C.
* Add the second 2TB disk image to the existing RAID5 and wait
until resync to end, creating a full RAID5 set; having two
members on the same disk is far from ideal, but better than
nothing, and temporary.
* Make D and E as a degraded RAID6 or RAID10.
* Copy the data to the newly created RAID6 or RAID10.
* Reset A, B, C.
* Make A and B into a RAID0.
* Add A+B and C to the RAID6/RAID10 as spares and wait for the
resync to end.
A scheme that relies on in-place conversion from RAID5 to RAID6:
* Copy the image on C to E.
* Copy B to D.
* Copy A to C.
* Create an empty RAID0 of A+B.
* Add A+B as spare to to C, D, E, and wait for the resync to
end, creating a full RAID5.
* In-place expand the RAID5 to 4TB members, that should take
no time.
* In-place convert the RAID5 to RAID6.
But all these seem to me at the limit of plausibility.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: Phil Turmel @ 2016-07-26 17:14 UTC (permalink / raw)
To: Adam Goryachev, David C. Rankin, mdraid
In-Reply-To: <ff6872cd-98eb-aad2-a71a-7af52050ff30@websitemanagers.com.au>
Hi David, Adam,
On 07/26/2016 05:52 AM, Adam Goryachev wrote:
> On 26/07/2016 18:20, David C. Rankin wrote:
>>>> # cat /proc/mdstat
>>> Personalities : [raid1]
>>> md4 : active raid1 sdc[0]
>>> 2930135488 blocks super 1.2 [2/1] [U_]
>>> bitmap: 0/22 pages [0KB], 65536KB chunk
> No, I'm saying that is an excellent idea, and it is exactly what I
> always do. The problem is that you created the single large primary
> partition, and then used the raw drive for the raid array instead of
> using the partition.
The drive probably came with a single large partition in a GPT table.
mdadm was created using the whole disk, which neither needs nor expects
there to be a partition table. (I do not use partitions for my big
arrays -- whole disks only.)
When mdadm created the array, it probably wiped out the starting table,
but not the GPT backup. Which may or may not be outside the used data
area of the array. So you allowed a partition tool to "fix" a partition
table that isn't being used. The simplest solution is to zero the first
4k of the disk and zero the GPT backup partition table location.
I suggest re-adding /dev/sdd to the array (not /dev/sdd1, and not
--add), whether you delete the table or not. A --re-add will probably
be nearly instant, since there is a bitmap.
Phil
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: Chris Murphy @ 2016-07-26 15:55 UTC (permalink / raw)
To: Chris Murphy; +Cc: David C. Rankin, mdraid
In-Reply-To: <CAJCQCtQ73Tp_ps57qgDm=S_qdFMHy8J0Z=kB+_TfQD2ugj_6ow@mail.gmail.com>
# tr '\0' '\377' < /dev/zero > /dev/VG/1
## then used gdisk to create new GPT and a single partition
# hexdump -C /dev/VG/1
[root@f24s ~]# hexdump -C /dev/VG/1
00000000 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
*
000001b0 ff ff ff ff ff ff ff ff 00 00 00 00 00 00 00 fe |................|
000001c0 ff ff ee ff ff ff 01 00 00 00 ff ff 9f 00 00 00 |................|
000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00 |EFI PART....\...|
00000210 6d c0 57 bd 00 00 00 00 01 00 00 00 00 00 00 00 |m.W.............|
00000220 ff ff 9f 00 00 00 00 00 22 00 00 00 00 00 00 00 |........".......|
00000230 de ff 9f 00 00 00 00 00 dc 74 3e 4a 76 e2 b6 44 |.........t>Jv..D|
00000240 8d 30 bd d8 07 62 45 f8 02 00 00 00 00 00 00 00 |.0...bE.........|
00000250 80 00 00 00 80 00 00 00 3f d0 39 d5 00 00 00 00 |........?.9.....|
00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000400 0f 88 9d a1 fc 05 3b 4d a0 06 74 3f 0f 84 91 1e |......;M..t?....|
00000410 b9 3b b1 50 5f 9f 80 47 a7 82 44 cc 2b 52 56 98 |.;.P_..G..D.+RV.|
00000420 00 08 00 00 00 00 00 00 de ff 9f 00 00 00 00 00 |................|
00000430 00 00 00 00 00 00 00 00 4c 00 69 00 6e 00 75 00 |........L.i.n.u.|
00000440 78 00 20 00 52 00 41 00 49 00 44 00 00 00 00 00 |x. .R.A.I.D.....|
00000450 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00004400 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
*
^C
And then wiping with 1's again, mdadm -C with default v1.2 metadata on
the whole device.
00000000 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
*
00001000 fc 4e 2b a9 01 00 00 00 00 00 00 00 00 00 00 00 |.N+.............|
00001010 f9 b7 39 86 d6 37 9b c4 04 b8 f2 a8 91 ef 8b 8b |..9..7..........|
00001020 66 32 34 73 3a 30 00 00 00 00 00 00 00 00 00 00 |f24s:0..........|
00001030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00001040 3a 85 97 57 00 00 00 00 01 00 00 00 00 00 00 00 |:..W............|
00001050 00 e0 9f 00 00 00 00 00 00 00 00 00 02 00 00 00 |................|
00001060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001080 00 20 00 00 00 00 00 00 00 e0 9f 00 00 00 00 00 |. ..............|
00001090 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000010a0 00 00 00 00 00 00 00 00 8d 69 45 42 e9 a4 ad c2 |.........iEB....|
000010b0 c1 b1 f8 8c d5 8d 7e 22 00 00 08 00 48 00 00 00 |......~"....H...|
000010c0 43 85 97 57 00 00 00 00 04 00 00 00 00 00 00 00 |C..W............|
000010d0 00 4b 07 00 00 00 00 00 28 cd 5d 00 80 00 00 00 |.K......(.].....|
000010e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001100 00 00 01 00 fe ff fe ff fe ff fe ff fe ff fe ff |................|
00001110 fe ff fe ff fe ff fe ff fe ff fe ff fe ff fe ff |................|
*
00001200 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
*
^C
So yeah, if gdisk is used first, then mdadm second, and mdadm is
pointed to the whole block device rather than a partition, mdadm does
not step on any part of the GPT. Therefore something else hit LBA 1 in
the OP's case (I previously said LBA 2, that's a typo, the header is
on LBA 1 at least on 512 byte logical sector drives). Maybe it's a
rare case of silent data corruption on that sector?
However, it does appear that in the OP's case that the array was
created on the whole disk device, not on the partition. If true, I
would remove the signatures on the GPT primary and secondary headers
to make sure they're invalidated. Otherwise it's ambiguous what these
drives are all about, are they single partition drives that are empty?
Or are they whole device md members?
I'd look at using wipefs -b -t or -o to remove the GPT signatures,
while avoiding the mdadm and file system signatures.
Chris Murphy
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: Chris Murphy @ 2016-07-26 15:19 UTC (permalink / raw)
To: David C. Rankin; +Cc: mdraid
In-Reply-To: <57971D3C.8040602@suddenlinkmail.com>
It'd be interesting to see mdadm -E for sdc and sdd.
GPT uses LBA 0-3. And mdadm metadata 1.2 is 4K from the start. These
do not overlap. So I'm unconvinced that mdadm -C applied to sdc and
sdd instead of sdc1 and sdd1 is the source of the problem.
Further, gdisk specifically said the GPT header was corrupt. The PMBR,
LBA 0, is intact, and the table data (LBA 2) is intact. Only LBA 2 was
stepped on by something?
What do you get for gdisk -l /dev/sdc? Another warning or is it OK?
Also for what it's worth: primary, extended, logical are terms that do
not apply to GPT partitioned disks. There is only one kind of
partition with GPT disks, no distinctions.
--
Chris Murphy
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: Adam Goryachev @ 2016-07-26 9:52 UTC (permalink / raw)
To: David C. Rankin, mdraid
In-Reply-To: <57971D3C.8040602@suddenlinkmail.com>
On 26/07/2016 18:20, David C. Rankin wrote:
> On 07/26/2016 12:28 AM, David C. Rankin wrote:
>> On 07/25/2016 11:18 PM, Adam Goryachev wrote:
>>> It sounds/looks like you partitioned the two drives with GPT, and then used the
>>> entire drive for the RAID, which probably overwrote at least one of the GPT
>>> entries. Now gparted has overwritten part of the disk where mdadm keeps it's data.
>>>
>>> So, good news, assuming you really haven't touched sdc, then it should still be
>>> fine. Try the following:
>>> mdadm --manage --stop /dev/md4
>>>
>>> Check it has stopped cat /proc/mdstat and md4 should not appear at all.
>>>
>>> Now re-assemble with only the one working member:
>>> mdadm --assemble --force /dev/md4 /dev/sdc
>>>
>>> If you are lucky, you will then be able to mount /dev/md4 as needed.
>>>
>>> If not, please provide:
>>> Output of the above mdadm --assemble
>>> Logs from syslog/dmesg in relation to the assembly attempt
>>> mdadm --query /dev/sdc
>>> mdadm --query /dev/sdc1
>>> mdadm --query /dev/sdd
>>> mdadm --query /dev/sdd1
>>> mdadm --detail /dev/md4 (after the assemble above).
>>>
>>> Being RAID1, it shouldn't be too hard to recover your data, just need to get
>>> some more information about the current state.
>>>
>>> Once you have the array started, your next step is to avoid the problem in
>>> future. So send through the above details, and then additional advice can be
>>> provided. Generally I've seen most people create the partition and then use the
>>> partition for RAID, that way the partition is marked as in-use by the array. The
>>> alternative is to wipe the beginning and end of the drive (/dev/zero) and then
>>> re-add to the array. Once synced, you can repeat with the other drive. The
>>> problem is if something (eg your BIOS) decides to "initialise" the drive for
>>> you, then it will overwrite your data/mdadm data.
>>>
>>> Hope the above helps.
>>>
>>> Regards,
>>> Adam
>> Adam,
>>
>> Thank you! There are a lot of things in life I'm good at, speaking mdadm
>> fluently, when I deal with it once every 2 years -- isn't one of them.
>>
>> /dev/sdc was still OK and did assemble in degraded mode just fine:
>>
>> # mdadm --manage --stop /dev/md4
>> mdadm: stopped /dev/md4
>>
>> # cat /proc/mdstat
>> Personalities : [raid1]
>> md1 : active raid1 sdb6[1] sda6[0]
>> 52396032 blocks super 1.2 [2/2] [UU]
>>
>> md0 : active raid1 sdb5[1] sda5[0]
>> 511680 blocks super 1.2 [2/2] [UU]
>>
>> md3 : active raid1 sdb8[1] sda8[0]
>> 2115584 blocks super 1.2 [2/2] [UU]
>>
>> md2 : active raid1 sdb7[1] sda7[0]
>> 921030656 blocks super 1.2 [2/2] [UU]
>> bitmap: 0/7 pages [0KB], 65536KB chunk
>>
>> # mdadm --assemble --force /dev/md4 /dev/sdc
>> mdadm: /dev/md4 has been started with 1 drive (out of 2).
>>
>> # cat /proc/mdstat
>> Personalities : [raid1]
>> md4 : active raid1 sdc[0]
>> 2930135488 blocks super 1.2 [2/1] [U_]
>> bitmap: 0/22 pages [0KB], 65536KB chunk
>>
>> Up and running, mounted with all data in tact (well, at least until I hit the
>> address in the partition table where the mdadm data overwrote part of the
>> partition table -- I see a Segmentation Fault coming)
I don't think you will have any problem here. Please let us know if you do.
>>
>> So I take it having one large raid1 filesystem created out of a primary
>> partition on a disk is a bad idea? My goal in doing so was to create the largest
>> block of storage out of the two drives I could (saving 100M unpartitioned at the
>> end in case of drive failure and disk size variance)
No, I'm saying that is an excellent idea, and it is exactly what I
always do. The problem is that you created the single large primary
partition, and then used the raw drive for the raid array instead of
using the partition.
>> I take it adding sdd back into md4 is not a good idea at this point.
Remember the difference between the drive and the partition. sdd is the
entire drive, that is what you will create partitions on. sdd1 is the
first partition that you will use for RAID. So, you can add sdd1 to md4
once you are sure the partitions are configured correctly. You might
also zero out the beginning/end of the partition just to ensure there is
no old mdadm data there.
>> Do I implement a new partition scheme on sdd, and then "create" a new single
>> disk raid1 array (say md5), mount it on some temporary mount point, copy the
>> data, then stop both, assemble what was sdd/md5 as md4 then nuke the partitions
>> on sdc, repartition sdc (as I did sdd) and then add sdc to the new array with
>> sdd? (or I could dump the data to some temp location, nuke both sdc and sdd,
>> repartition, recreate, assemble and then copy back to the new fully functional
>> array -- that sounds better)
>>
>> What are your thoughts on the partition scheme and the approach outlined above?
>> And thank you again for steering me straight and saving the data.
The main problem you will have is that the size of sdd1 will be smaller
than sdc, because the partition table (GPT) will take up some space. So
you may indeed need to either reduce the size of the FS, reduce the size
of md4, before you can add sdd1 as the other half of the mirror.
The other option is to do as you say, create a new RAID1 mirror between
sdd1 and missing, format, mount, copy data, then stop md4, clear mdadm
data from sdc, partition sdc properly, add sdc1 to md5, wait for resync.
When done, if you want you can umount, stop md5, assemble as md4, and
then mount.
Remember to update mdadm.conf afterwards.
>>
>>
> Adam,
>
> Here is the detail on md4, if is makes any difference on your words of wisdom.
>
> # mdadm --query /dev/md4
> /dev/md4: 2794.39GiB raid1 2 devices, 0 spares. Use mdadm --detail for more detail.
>
> # mdadm --detail /dev/md4
> /dev/md4:
> Version : 1.2
> Creation Time : Mon Mar 21 02:27:21 2016
> Raid Level : raid1
> Array Size : 2930135488 (2794.39 GiB 3000.46 GB)
> Used Dev Size : 2930135488 (2794.39 GiB 3000.46 GB)
> Raid Devices : 2
> Total Devices : 1
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Tue Jul 26 01:12:27 2016
> State : clean, degraded
> Active Devices : 1
> Working Devices : 1
> Failed Devices : 0
> Spare Devices : 0
>
> Name : valkyrie:4 (local to host valkyrie)
> UUID : 6e520607:f152d8b9:dd2a3bec:5f9dc875
> Events : 4240
>
> Number Major Minor RaidDevice State
> 0 8 32 0 active sync /dev/sdc
> - 0 0 1 removed
>
> And the last entry in mdadm.conf assembling/activating the array:
>
> # tail -n 2 /etc/mdadm.conf
> ARRAY /dev/md3 metadata=1.2 name=archiso:3 UUID=8b37af66:b34403aa:fa4ce6f1:5eb4b7c8
> ARRAY /dev/md4 metadata=1.2 name=valkyrie:4 UUID=6e520607:f152d8b9:dd2a3bec:5f9dc875
>
> Thanks again!
>
>
All that just confirms that you used the entire drive instead of the
partition. So one more time, please be careful to notice the difference,
sdc is the full drive, sdc1 is the first partition on the drive, sdc2
would be the second, etc...
Regards,
Adam
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: David C. Rankin @ 2016-07-26 8:20 UTC (permalink / raw)
To: mdraid
In-Reply-To: <5796F4E0.9070206@suddenlinkmail.com>
On 07/26/2016 12:28 AM, David C. Rankin wrote:
> On 07/25/2016 11:18 PM, Adam Goryachev wrote:
>> It sounds/looks like you partitioned the two drives with GPT, and then used the
>> entire drive for the RAID, which probably overwrote at least one of the GPT
>> entries. Now gparted has overwritten part of the disk where mdadm keeps it's data.
>>
>> So, good news, assuming you really haven't touched sdc, then it should still be
>> fine. Try the following:
>> mdadm --manage --stop /dev/md4
>>
>> Check it has stopped cat /proc/mdstat and md4 should not appear at all.
>>
>> Now re-assemble with only the one working member:
>> mdadm --assemble --force /dev/md4 /dev/sdc
>>
>> If you are lucky, you will then be able to mount /dev/md4 as needed.
>>
>> If not, please provide:
>> Output of the above mdadm --assemble
>> Logs from syslog/dmesg in relation to the assembly attempt
>> mdadm --query /dev/sdc
>> mdadm --query /dev/sdc1
>> mdadm --query /dev/sdd
>> mdadm --query /dev/sdd1
>> mdadm --detail /dev/md4 (after the assemble above).
>>
>> Being RAID1, it shouldn't be too hard to recover your data, just need to get
>> some more information about the current state.
>>
>> Once you have the array started, your next step is to avoid the problem in
>> future. So send through the above details, and then additional advice can be
>> provided. Generally I've seen most people create the partition and then use the
>> partition for RAID, that way the partition is marked as in-use by the array. The
>> alternative is to wipe the beginning and end of the drive (/dev/zero) and then
>> re-add to the array. Once synced, you can repeat with the other drive. The
>> problem is if something (eg your BIOS) decides to "initialise" the drive for
>> you, then it will overwrite your data/mdadm data.
>>
>> Hope the above helps.
>>
>> Regards,
>> Adam
>
> Adam,
>
> Thank you! There are a lot of things in life I'm good at, speaking mdadm
> fluently, when I deal with it once every 2 years -- isn't one of them.
>
> /dev/sdc was still OK and did assemble in degraded mode just fine:
>
> # mdadm --manage --stop /dev/md4
> mdadm: stopped /dev/md4
>
> # cat /proc/mdstat
> Personalities : [raid1]
> md1 : active raid1 sdb6[1] sda6[0]
> 52396032 blocks super 1.2 [2/2] [UU]
>
> md0 : active raid1 sdb5[1] sda5[0]
> 511680 blocks super 1.2 [2/2] [UU]
>
> md3 : active raid1 sdb8[1] sda8[0]
> 2115584 blocks super 1.2 [2/2] [UU]
>
> md2 : active raid1 sdb7[1] sda7[0]
> 921030656 blocks super 1.2 [2/2] [UU]
> bitmap: 0/7 pages [0KB], 65536KB chunk
>
> # mdadm --assemble --force /dev/md4 /dev/sdc
> mdadm: /dev/md4 has been started with 1 drive (out of 2).
>
> # cat /proc/mdstat
> Personalities : [raid1]
> md4 : active raid1 sdc[0]
> 2930135488 blocks super 1.2 [2/1] [U_]
> bitmap: 0/22 pages [0KB], 65536KB chunk
>
> Up and running, mounted with all data in tact (well, at least until I hit the
> address in the partition table where the mdadm data overwrote part of the
> partition table -- I see a Segmentation Fault coming)
>
> So I take it having one large raid1 filesystem created out of a primary
> partition on a disk is a bad idea? My goal in doing so was to create the largest
> block of storage out of the two drives I could (saving 100M unpartitioned at the
> end in case of drive failure and disk size variance)
>
> How should I proceed if I want to create a large raid1 array out of the two
> disks? Should I create a logical/extended partition setup and then create the
> array out of the extended partition? (that is the setup I have for all other
> raid1 disks that also hold /boot, /, /home, etc....
>
> I take it adding sdd back into md4 is not a good idea at this point.
>
> Do I implement a new partition scheme on sdd, and then "create" a new single
> disk raid1 array (say md5), mount it on some temporary mount point, copy the
> data, then stop both, assemble what was sdd/md5 as md4 then nuke the partitions
> on sdc, repartition sdc (as I did sdd) and then add sdc to the new array with
> sdd? (or I could dump the data to some temp location, nuke both sdc and sdd,
> repartition, recreate, assemble and then copy back to the new fully functional
> array -- that sounds better)
>
> What are your thoughts on the partition scheme and the approach outlined above?
> And thank you again for steering me straight and saving the data.
>
>
>
Adam,
Here is the detail on md4, if is makes any difference on your words of wisdom.
# mdadm --query /dev/md4
/dev/md4: 2794.39GiB raid1 2 devices, 0 spares. Use mdadm --detail for more detail.
# mdadm --detail /dev/md4
/dev/md4:
Version : 1.2
Creation Time : Mon Mar 21 02:27:21 2016
Raid Level : raid1
Array Size : 2930135488 (2794.39 GiB 3000.46 GB)
Used Dev Size : 2930135488 (2794.39 GiB 3000.46 GB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Tue Jul 26 01:12:27 2016
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Name : valkyrie:4 (local to host valkyrie)
UUID : 6e520607:f152d8b9:dd2a3bec:5f9dc875
Events : 4240
Number Major Minor RaidDevice State
0 8 32 0 active sync /dev/sdc
- 0 0 1 removed
And the last entry in mdadm.conf assembling/activating the array:
# tail -n 2 /etc/mdadm.conf
ARRAY /dev/md3 metadata=1.2 name=archiso:3 UUID=8b37af66:b34403aa:fa4ce6f1:5eb4b7c8
ARRAY /dev/md4 metadata=1.2 name=valkyrie:4 UUID=6e520607:f152d8b9:dd2a3bec:5f9dc875
Thanks again!
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: David C. Rankin @ 2016-07-26 5:28 UTC (permalink / raw)
To: mdraid
In-Reply-To: <413b1150-ca38-1907-ea5d-b68ad2f75710@websitemanagers.com.au>
On 07/25/2016 11:18 PM, Adam Goryachev wrote:
> It sounds/looks like you partitioned the two drives with GPT, and then used the
> entire drive for the RAID, which probably overwrote at least one of the GPT
> entries. Now gparted has overwritten part of the disk where mdadm keeps it's data.
>
> So, good news, assuming you really haven't touched sdc, then it should still be
> fine. Try the following:
> mdadm --manage --stop /dev/md4
>
> Check it has stopped cat /proc/mdstat and md4 should not appear at all.
>
> Now re-assemble with only the one working member:
> mdadm --assemble --force /dev/md4 /dev/sdc
>
> If you are lucky, you will then be able to mount /dev/md4 as needed.
>
> If not, please provide:
> Output of the above mdadm --assemble
> Logs from syslog/dmesg in relation to the assembly attempt
> mdadm --query /dev/sdc
> mdadm --query /dev/sdc1
> mdadm --query /dev/sdd
> mdadm --query /dev/sdd1
> mdadm --detail /dev/md4 (after the assemble above).
>
> Being RAID1, it shouldn't be too hard to recover your data, just need to get
> some more information about the current state.
>
> Once you have the array started, your next step is to avoid the problem in
> future. So send through the above details, and then additional advice can be
> provided. Generally I've seen most people create the partition and then use the
> partition for RAID, that way the partition is marked as in-use by the array. The
> alternative is to wipe the beginning and end of the drive (/dev/zero) and then
> re-add to the array. Once synced, you can repeat with the other drive. The
> problem is if something (eg your BIOS) decides to "initialise" the drive for
> you, then it will overwrite your data/mdadm data.
>
> Hope the above helps.
>
> Regards,
> Adam
Adam,
Thank you! There are a lot of things in life I'm good at, speaking mdadm
fluently, when I deal with it once every 2 years -- isn't one of them.
/dev/sdc was still OK and did assemble in degraded mode just fine:
# mdadm --manage --stop /dev/md4
mdadm: stopped /dev/md4
# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb6[1] sda6[0]
52396032 blocks super 1.2 [2/2] [UU]
md0 : active raid1 sdb5[1] sda5[0]
511680 blocks super 1.2 [2/2] [UU]
md3 : active raid1 sdb8[1] sda8[0]
2115584 blocks super 1.2 [2/2] [UU]
md2 : active raid1 sdb7[1] sda7[0]
921030656 blocks super 1.2 [2/2] [UU]
bitmap: 0/7 pages [0KB], 65536KB chunk
# mdadm --assemble --force /dev/md4 /dev/sdc
mdadm: /dev/md4 has been started with 1 drive (out of 2).
# cat /proc/mdstat
Personalities : [raid1]
md4 : active raid1 sdc[0]
2930135488 blocks super 1.2 [2/1] [U_]
bitmap: 0/22 pages [0KB], 65536KB chunk
Up and running, mounted with all data in tact (well, at least until I hit the
address in the partition table where the mdadm data overwrote part of the
partition table -- I see a Segmentation Fault coming)
So I take it having one large raid1 filesystem created out of a primary
partition on a disk is a bad idea? My goal in doing so was to create the largest
block of storage out of the two drives I could (saving 100M unpartitioned at the
end in case of drive failure and disk size variance)
How should I proceed if I want to create a large raid1 array out of the two
disks? Should I create a logical/extended partition setup and then create the
array out of the extended partition? (that is the setup I have for all other
raid1 disks that also hold /boot, /, /home, etc....
I take it adding sdd back into md4 is not a good idea at this point.
Do I implement a new partition scheme on sdd, and then "create" a new single
disk raid1 array (say md5), mount it on some temporary mount point, copy the
data, then stop both, assemble what was sdd/md5 as md4 then nuke the partitions
on sdc, repartition sdc (as I did sdd) and then add sdc to the new array with
sdd? (or I could dump the data to some temp location, nuke both sdc and sdd,
repartition, recreate, assemble and then copy back to the new fully functional
array -- that sounds better)
What are your thoughts on the partition scheme and the approach outlined above?
And thank you again for steering me straight and saving the data.
--
David C. Rankin, J.D.,P.E.
^ permalink raw reply
* Re: GPT corruption on Primary Header, backup OK, fixing primary nuked array -- help?
From: Adam Goryachev @ 2016-07-26 4:18 UTC (permalink / raw)
To: David C. Rankin, mdraid
In-Reply-To: <5796B46B.6060905@suddenlinkmail.com>
On 26/07/16 10:52, David C. Rankin wrote:
> Neil, all,
>
> I really stepped in it this time. I have had a 3T raid1 array with 2 disks
> sdc/sdd that has worked fine since the new disks were partitioned and the arrays
> were created in August of last year. (simple 2-disk, raid1, ext4 - no
> encryption) Current kernel info on Archlinux is:
>
> # uname -a
> Linux valkyrie 4.6.4-1-ARCH #1 SMP PREEMPT Mon Jul 11 19:12:32 CEST 2016 x86_64
> GNU/Linux
>
> When the disks were partitioned originally and the arrays created, listing the
> partitioning showed no partition table problems. Today, a simple check of the
> partitioning by listing the partitions on sdc with 'gdisk -l /dev/sdc' brought
> up a curious error:
>
> # gdisk -l /dev/sdc
> GPT fdisk (gdisk) version 1.0.1
>
> Caution: invalid main GPT header, but valid backup; regenerating main header
> from backup!
>
> Caution! After loading partitions, the CRC doesn't check out!
> Warning! Main partition table CRC mismatch! Loaded backup partition table
> instead of main partition table!
>
> Warning! One or more CRCs don't match. You should repair the disk!
>
> Partition table scan:
> MBR: protective
> BSD: not present
> APM: not present
> GPT: damaged
>
> ****************************************************************************
> Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
> verification and recovery are STRONGLY recommended.
> ****************************************************************************
> Disk /dev/sdc: 5860533168 sectors, 2.7 TiB
> Logical sector size: 512 bytes
> Disk identifier (GUID): 3F835DD0-AA89-4F86-86BF-181F53FA1847
> Partition table holds up to 128 entries
> First usable sector is 34, last usable sector is 5860533134
> Partitions will be aligned on 2048-sector boundaries
> Total free space is 212958 sectors (104.0 MiB)
>
> Number Start (sector) End (sector) Size Code Name
> 1 8192 5860328334 2.7 TiB FD00 Linux RAID
>
> (sdd showed the same - it was probably fine all along and just the result of
> creating the arrays, but that would be par for my day...)
>
> Huh? All was functioning fine, even with the error -- until I tried to "fix" it.
> First, I searched for possible reasons on how the primary GPT table became
> corrupt. The reasons range from some non-GPT aware app tried to access the table
> (not anything I can think of here) or perhaps the Gigabyte "virtual bios" wrote
> a copy of the bios within the larger GPT table causing the issue, see:
> https://francisfisher.me.uk/problem/2014/warning-about-large-hard-discs-gpt-and-gigabyte-motherboards-such-as-ga-p35-ds4/)
> That sounds flaky, but I do have a Gigabyte GA-990FXA-UD3 Rev. 4 board.
>
> So after reading the posts, and reading the unix.stackexchange, superuser, etc.
> posts on the subject:
>
> http://www.rodsbooks.com/gdisk/repairing.html
> http://askubuntu.com/questions/465510/gpt-talbe-corrupt-after-raid1-setup
> https://ubuntuforums.org/showthread.php?t=1956173
> ...
>
> and various parted bugs about the opposite:
>
> https://lists.gnu.org/archive/html/bug-parted/2015-07/msg00003.html
>
> I came up with a plan to:
>
> - boot the Archlinux recovery cd 20160301 release CD
> - use gdisk /dev/sdd; r; v; c; w; to correct the table
> - --fail and --remove the disk from the array, and
> - readd the new disk, let it sync, then do the same for /dev/sdc
>
> (steps 1 & 2 went fine, but that's where I screwed up...).
>
> Now I'm left with an array (/dev/md4) in an inactive and probably
> un-salvageable. The data on the disks is backed up, so if there is no way to
> assemble and recover the data, I'm only out the time to recopy it. If I can save
> that, fine, but it isn't pressing. The current array state is:
>
> # cat /proc/mdstat
> Personalities : [raid1]
> md1 : active raid1 sdb6[1] sda6[0]
> 52396032 blocks super 1.2 [2/2] [UU]
>
> md0 : active raid1 sdb5[1] sda5[0]
> 511680 blocks super 1.2 [2/2] [UU]
>
> md3 : active raid1 sdb8[1] sda8[0]
> 2115584 blocks super 1.2 [2/2] [UU]
>
> md2 : active raid1 sdb7[1] sda7[0]
> 921030656 blocks super 1.2 [2/2] [UU]
> bitmap: 0/7 pages [0KB], 65536KB chunk
>
> md4 : inactive sdc[0](S)
> 2930135512 blocks super 1.2
>
> unused devices: <none>
>
> This is where I'm stuck. I've got the primary partition table issue on sdd
> fixed, I have not touched sdc (it is in the same state it was, when it was
> functioning with the complaint about the primary gpt partition table. I have
> tried activating the array with sdd1 "missing", but no joy. After correcting the
> partition table on sdd, it still contains the original partition, but I cannot
> get it (or sdc) to assemble in degraded or raid mode.
>
> I need help. Is there anything I can try to salvage the array? (at least one
> disk of the array?) If not, is there a way I can activate (or at least mount
> either sdc or sdd? -- it would be easier to dump the data rather than copying
> from multiple sources. It's ~258G, not huge, but not small)
>
> I know worst case is to wipe both disks (gdisk /dev/sd[cd] x; z; yes; yes) and
> start over, but with one disk of md4 that I haven't touched, it seems like I
> should be able to recover something?
>
> If the answer is just no, no, ..., then what is the best approach? zap with
> gdisk, wipe the superblocks and start over?
>
> If you need any other information that I haven't included, just let me know. I
> have the binary dumps of partition tables from sdc and sdd (from gdisk written
> to disk before any changes to sdd). Anyway, if there is anything else, just let
> me know and I'll post it.
>
> The server on which this array resides is running (this was just a data array,
> the boot, root, and home arrays are fine (they are mbr). I've just commented the
> mdadm.conf and fstab entries for the effected array.
>
> Last, but less important, any idea where this primary GPT corruption originated?
> (or was it fine all along and the error just a result of them being members of
> the array?) There are numerous posts over the last year related to:
>
> "invalid main GPT header, but valid backup"
>
> (and relating to raid1)
>
> but not many answers as to why. (if this was just a normal gdisk response from a
> raided disk, then there is a lot of 'bad' info out there. What is my best
> approach for attempting recovery from this self-created mess? Thanks.
>
>
It sounds/looks like you partitioned the two drives with GPT, and then
used the entire drive for the RAID, which probably overwrote at least
one of the GPT entries. Now gparted has overwritten part of the disk
where mdadm keeps it's data.
So, good news, assuming you really haven't touched sdc, then it should
still be fine. Try the following:
mdadm --manage --stop /dev/md4
Check it has stopped cat /proc/mdstat and md4 should not appear at all.
Now re-assemble with only the one working member:
mdadm --assemble --force /dev/md4 /dev/sdc
If you are lucky, you will then be able to mount /dev/md4 as needed.
If not, please provide:
Output of the above mdadm --assemble
Logs from syslog/dmesg in relation to the assembly attempt
mdadm --query /dev/sdc
mdadm --query /dev/sdc1
mdadm --query /dev/sdd
mdadm --query /dev/sdd1
mdadm --detail /dev/md4 (after the assemble above).
Being RAID1, it shouldn't be too hard to recover your data, just need to
get some more information about the current state.
Once you have the array started, your next step is to avoid the problem
in future. So send through the above details, and then additional advice
can be provided. Generally I've seen most people create the partition
and then use the partition for RAID, that way the partition is marked as
in-use by the array. The alternative is to wipe the beginning and end of
the drive (/dev/zero) and then re-add to the array. Once synced, you can
repeat with the other drive. The problem is if something (eg your BIOS)
decides to "initialise" the drive for you, then it will overwrite your
data/mdadm data.
Hope the above helps.
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox