* Re: RAID0 & diskstats.
From: NeilBrown @ 2015-03-04 21:31 UTC (permalink / raw)
To: Dave Jones; +Cc: Linux Kernel, linux RAID
In-Reply-To: <20150304210904.GA26981@codemonkey.org.uk>
[-- Attachment #1: Type: text/plain, Size: 1886 bytes --]
On Wed, 4 Mar 2015 16:09:04 -0500 Dave Jones <davej@codemonkey.org.uk> wrote:
> Hi Neil,
> According to Documentation/iostats.txt, the 9th column of
> /proc/diskstats (and its modern replacement in sysfs) should go to 0
> as IO completes.
>
> I assembled a RAID0 stripe using two SSD's, and saw this..
>
> # mdadm --assemble /dev/md0
> mdadm: /dev/md0 has been started with 2 drives.
> # cat /sys/block/md0/stat
> 167 0 5656 0 5 0 4096 0 172 3408 582825
> # cat /sys/block/md0/stat
> 167 0 5656 0 5 0 4096 0 172 231469 39809317
>
> The 10th & 11th fields constantly increase, as field 9 remains non-zero.
> If I mount and umount a filesystem on that volume, it works as expected,
> but the 9th 'IOs inflight' field continues to rise and never decreases even
> though the IO has obviously completed.
>
> # umount /mnt/ssd
> # cat /sys/block/md0/stat
> 167 0 5656 0 9 0 4225 0 176 571384 98278615
>
> The underlying disks have their respective stats entries behaving as
> expected, it only seems to affect the upper md layer.
>
> Some missing accounting somewhere in md ?
>
> (Only tested on 4.0rc2 so far, and only on RAID0)
>
> Dave
blockdev stats often aren't really a good match for md/raid...
"in_flight" assumes a queue, and raid0 doesn't have one. It just redirects
each request to the relevant device and lets the device handle it.
The only useful thing we could do here is make that value always zero.
I guess I need to add a "generic_end_io_acct()" call to md.c somewhere.
For raid1/5 there is probably somewhere sensible to put it.
For raid0/linear, it probably goes immediately after generic_start_io_acct().
Patches welcome :-)
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Last working drive in RAID1
From: NeilBrown @ 2015-03-04 21:46 UTC (permalink / raw)
To: Eric Mei; +Cc: linux-raid
In-Reply-To: <54F7633F.3020503@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1699 bytes --]
On Wed, 04 Mar 2015 12:55:43 -0700 Eric Mei <meijia@gmail.com> wrote:
> Hi,
>
> It is interesting to notice that RAID1 won't mark the last working drive
> as Faulty no matter what. The responsible code seems here:
>
> static void error(struct mddev *mddev, struct md_rdev *rdev)
> {
> ...
> /*
> * If it is not operational, then we have already marked it as dead
> * else if it is the last working disks, ignore the error, let the
> * next level up know.
> * else mark the drive as failed
> */
> if (test_bit(In_sync, &rdev->flags)
> && (conf->raid_disks - mddev->degraded) == 1) {
> /*
> * Don't fail the drive, act as though we were just a
> * normal single drive.
> * However don't try a recovery from this drive as
> * it is very likely to fail.
> */
> conf->recovery_disabled = mddev->recovery_disabled;
> return;
> }
> ...
> }
>
> The end result is that even if all the drives are physically gone, there
> still one drive remains in array forever, and mdadm continues to report
> the array is degraded instead of failed. RAID10 also has similar behavior.
>
> Is there any reason we absolutely don't want to fail the last drive of
> RAID1?
>
When a RAID1 only has one drive remaining, then it should act as much as
possible like a single plain ordinary drive.
How does /dev/sda behave when you physically remove the device? md0 (as a
raid1 with one drive) should do the same.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Hung RAID5 array with discard
From: Terry Hardie @ 2015-03-04 21:47 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CANHB05vDxWqfST41hXgfApthHuxdSuyJYCKqS4jkOa7OZ0kpsA@mail.gmail.com>
Well, I'm dissapointed no one responded to this. This basically means
linux RAID 4/5/6 and discard is fundamentally broken, and no one wants
to acknowledge it.
I hope someone finds this post while I still have my lab available and
I can help them troubleshoot this issue.
I tried this again today on 3.13.0-44-generic (Ubuntu) and was easily
able to reproduce it.
On Wed, Dec 17, 2014 at 7:08 PM, Terry Hardie <thardie@instartlogic.com> wrote:
> Hi,
>
> I am testing 3 SSDs (1TB Crucial M550 with DRZAT, and I tested they do
> return zeros after discard) with RAID5 and discard. I create the array
> with a 64k chunk size, and it starts to sync. During it's initial
> reconstruction, I do a mkfs.ext4, which starts to do the "Discarding
> device blocks". After a short period (I believe when the mkfs reaches
> the point where the reconstruction is at, all IO to the disks freezes,
> and mkfs does not advance. iostat shows 2 of the 3 drives at 100%
> utilization with no data read or written. After 2 minutes, I get the
> hung task dump. Most CPUs are idle, and here are a few which are not,
> which look like a deadlock to me:
>
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154399] INFO:
> rcu_sched detected stalls on CPUs/tasks: { 4 5} (detected by 3,
> t=285032 jiffies, g=1160, c=1159, q=0)
>
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154902] NMI
> backtrace for cpu 4
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154904] CPU: 4 PID:
> 2146 Comm: md3_raid5 Tainted: G W IOX 3.13.0-43-generic
> #72~precise1
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154905] Hardware
> name: Supermicro SYS-2028TP-HC1R/X10DRT-P, BIOS 1.0a 08/28/2014
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154906] task:
> ffff88202594c800 ti: ffff8810245a0000 task.ti: ffff8810245a0000
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154907] RIP:
> 0010:[<ffffffff817644c1>] [<ffffffff817644c1>]
> _raw_spin_lock_irqsave+0x41/0x60
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154910] RSP:
> 0018:ffff8810245a1cc8 EFLAGS: 00000006
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154911] RAX:
> 0000000000002ec5 RBX: ffff882028a6ec00 RCX: 0000000000007b78
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154912] RDX:
> 0000000000000202 RSI: 0000000000007b78 RDI: ffff882028a6ec10
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154913] RBP:
> ffff8810245a1cc8 R08: 0000000000007b76 R09: ffff882023629170
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154914] R10:
> 0000000000000000 R11: ffff882028a6ec00 R12: ffff882028a6ee68
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154915] R13:
> 0000000000000003 R14: 0000000000000002 R15: ffff882028a6ec10
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154916] FS:
> 0000000000000000(0000) GS:ffff88103fc80000(0000)
> knlGS:0000000000000000
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154917] CS: 0010
> DS: 0000 ES: 0000 CR0: 0000000080050033
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154918] CR2:
> 00007f208c2d0000 CR3: 0000000001c0d000 CR4: 00000000001407e0
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154919] Stack:
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154920]
> ffff8810245a1d18 ffffffffa0149890 0000000000000002 ffff882028a6ee88
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154923]
> ffff882028a6ee68 ffff882028a6ec00 0000000000000008 ffff882028a6ee68
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154926]
> 0000000000000000 ffff882028a6ee50 ffff8810245a1d98 ffffffffa015212f
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154929] Call Trace:
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154933]
> [<ffffffffa0149890>] release_inactive_stripe_list+0x50/0x160 [raid456]
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154937]
> [<ffffffffa015212f>] handle_active_stripes.isra.38+0x7f/0x190
> [raid456]
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154940]
> [<ffffffffa0152758>] raid5d+0x198/0x2f0 [raid456]
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154942]
> [<ffffffff815d30a7>] md_thread+0x117/0x150
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154945]
> [<ffffffff810affe0>] ? __wake_up_sync+0x20/0x20
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154947]
> [<ffffffff815d2f90>] ? md_rdev_init+0x110/0x110
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154949]
> [<ffffffff8108fb59>] kthread+0xc9/0xe0
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154952]
> [<ffffffff8108fa90>] ? flush_kthread_worker+0xb0/0xb0
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154954]
> [<ffffffff8176d5bc>] ret_from_fork+0x7c/0xb0
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154956]
> [<ffffffff8108fa90>] ? flush_kthread_worker+0xb0/0xb0
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154957] Code: 1f 44
> 00 00 b8 00 00 02 00 f0 0f c1 07 89 c1 c1 e9 10 66 39 c1 75 05 48 89
> d0 5d c3 83 e1 fe 0f b7 f1 b8 00 80 00 00 44 0f b7 07 <66> 44 39 c1 74
> e6 f3 90 83 e8 01 75 ef 0f 1f 80 00 00 00 00 eb
>
>
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155001] NMI
> backtrace for cpu 5
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155003] CPU: 5 PID:
> 2147 Comm: md3_resync Tainted: G W IOX 3.13.0-43-generic
> #72~precise1
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155004] Hardware
> name: Supermicro SYS-2028TP-HC1R/X10DRT-P, BIOS 1.0a 08/28/2014
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155005] task:
> ffff88202594b000 ti: ffff8810274a0000 task.ti: ffff8810274a0000
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155006] RIP:
> 0010:[<ffffffffa01483b7>] [<ffffffffa01483b7>]
> __find_stripe+0x57/0xa0 [raid456]
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155010] RSP:
> 0018:ffff8810274a1b68 EFLAGS: 00000006
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155011] RAX:
> ffff882027092da0 RBX: 0000000000a30c10 RCX: 0000000000000001
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155012] RDX:
> 0000000000000c10 RSI: 0000000000a30c10 RDI: ffff882028a6ec00
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155013] RBP:
> ffff8810274a1b88 R08: 0000000000000000 R09: 0000000000000000
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155014] R10:
> 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155015] R13:
> ffff882028a6ec00 R14: 0000000000000000 R15: ffff882028a6eda8
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155016] FS:
> 0000000000000000(0000) GS:ffff88103fca0000(0000)
> knlGS:0000000000000000
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155017] CS: 0010
> DS: 0000 ES: 0000 CR0: 0000000080050033
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155018] CR2:
> 00000000006e1dc8 CR3: 0000000001c0d000 CR4: 00000000001407e0
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155019] Stack:
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155020]
> ffff8810274a1ba8 ffff882028a6ec00 000000007b767b00 ffff882028a6ec10
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155023]
> ffff8810274a1c28 ffffffffa0150555 ffff882023773b50 ffff882028a6eda8
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155026]
> 0000000200000001 ffff882028a6ec08 0000000000000000 0000000000a30c10
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155029] Call Trace:
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155033]
> [<ffffffffa0150555>] get_active_stripe+0x115/0x3e0 [raid456]
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155036]
> [<ffffffffa014aea8>] ? release_stripe+0x68/0x100 [raid456]
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155040]
> [<ffffffffa0154f3b>] sync_request+0x11b/0x2a0 [raid456]
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155042]
> [<ffffffff815d5ccf>] md_do_sync+0x84f/0xdb0
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155046]
> [<ffffffff810affe0>] ? __wake_up_sync+0x20/0x20
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155048]
> [<ffffffff815d30a7>] md_thread+0x117/0x150
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155050]
> [<ffffffff815d2f90>] ? md_rdev_init+0x110/0x110
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155052]
> [<ffffffff8108fb59>] kthread+0xc9/0xe0
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155054]
> [<ffffffff8108fa90>] ? flush_kthread_worker+0xb0/0xb0
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155057]
> [<ffffffff8176d5bc>] ret_from_fork+0x7c/0xb0
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155059]
> [<ffffffff8108fa90>] ? flush_kthread_worker+0xb0/0xb0
> Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155060] Code: e2 f8
> 0f 00 00 48 8b 04 02 48 85 c0 75 25 f6 05 29 25 01 00 04 75 3e 31 c0
> 48 83 c4 08 5b 41 5c 41 5d 5d c3 66 44 39 60 30 74 ee <48> 8b 00 48 85
> c0 74 db 48 39 58 38 75 f2 eb e9 48 89 f2 48 c7
>
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670082] INFO: task
> mkfs.ext4:2235 blocked for more than 120 seconds.
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670109]
> Tainted: G W IOX 3.13.0-43-generic #72~precise1
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670130] "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670155] mkfs.ext4
> D ffff881024fe39e0 0 2235 2080 0x00000000
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670158]
> ffff882026eafa68 0000000000000082 ffff88103fc73480 ffff882026eaffd8
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670162]
> 0000000000013480 0000000000013480 ffff8820293e8000 ffff88202208b000
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670165]
> ffff882026eafa78 ffff882028a6ec00 ffff882028a6ed98 ffff882028a6ec0c
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670169] Call Trace:
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670175]
> [<ffffffff81760ae9>] schedule+0x29/0x70
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670181]
> [<ffffffffa01506e3>] get_active_stripe+0x2a3/0x3e0 [raid456]
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670185]
> [<ffffffff8134c152>] ? blk_check_plugged+0x72/0xb0
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670189]
> [<ffffffff810affe0>] ? __wake_up_sync+0x20/0x20
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670193]
> [<ffffffffa0155e44>] make_discard_request+0x108/0x12c4 [raid456]
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670196]
> [<ffffffff810affe0>] ? __wake_up_sync+0x20/0x20
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670201]
> [<ffffffffa0155c91>] make_request+0x581/0x590 [raid456]
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670204]
> [<ffffffff8109cfd6>] ? ttwu_do_activate.constprop.82+0x66/0x70
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670207]
> [<ffffffff8109d097>] ? ttwu_queue+0xb7/0xd0
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670210]
> [<ffffffff8109f950>] ? try_to_wake_up+0x190/0x210
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670212]
> [<ffffffff815d2c53>] md_make_request+0xd3/0x230
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670216]
> [<ffffffff8115b085>] ? mempool_alloc_slab+0x15/0x20
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670219]
> [<ffffffff8134ceb7>] generic_make_request.part.62+0x77/0xb0
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670222]
> [<ffffffff8134d428>] generic_make_request+0x68/0x70
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670225]
> [<ffffffff8134d4a8>] submit_bio+0x78/0x160
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670228]
> [<ffffffff81202f80>] ? bio_alloc_bioset+0xa0/0x1d0
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670232]
> [<ffffffff813578c0>] blkdev_issue_discard+0x1f0/0x2a0
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670235]
> [<ffffffff8135c1f4>] blkdev_ioctl+0x354/0x810
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670238]
> [<ffffffff8101361d>] ? __switch_to+0x16d/0x4d0
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670241]
> [<ffffffff81204370>] block_ioctl+0x40/0x50
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670244]
> [<ffffffff811dd5c5>] do_vfs_ioctl+0x75/0x2c0
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670247]
> [<ffffffff817606be>] ? __schedule+0x38e/0x700
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670249]
> [<ffffffff811dd8a1>] SyS_ioctl+0x91/0xb0
> Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670252]
> [<ffffffff8176d66d>] system_call_fastpath+0x1a/0x1f
>
>
>
>
> If I do the mkfs.ext4 after the initial reconstruction is done, is
> gets all the way through. I don't want to put this system into
> production, since this could mean this condition could show up in the
> future if the array needs to reconstruct again at a future point while
> in service.
>
> This is a test system in a lab, so I'd be happy to try some tests.
>
> Terry
^ permalink raw reply
* Re: Last working drive in RAID1
From: Eric Mei @ 2015-03-04 22:48 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150305084634.2d590fe4@notabene.brown>
Hi Neil,
I see, that does make sense. Thank you.
But it impose a problem for HA. We have 2 nodes as active-standby pair,
if HW on node 1 have problem (e.g. SAS cable get pulled, thus all access
to physical drives are gone), we hope the array failover to node 2. But
with lingering drive reference, mdadm will report array is still alive
thus failover won't happen.
I guess it depends on what kind of error on the drive. If it's just a
media error we should keep it online as much as possible. But if the
drive is really bad or physically gone, keeping the stale reference
won't help anything. Back to your comparison with single drive /dev/sda,
I think MD as an array should do the same as /dev/sda, not the
individual drive inside MD, for them we should just let it go. How do
you think?
Eric
On 2015-03-04 2:46 PM, NeilBrown wrote:
> On Wed, 04 Mar 2015 12:55:43 -0700 Eric Mei <meijia@gmail.com> wrote:
>
>> Hi,
>>
>> It is interesting to notice that RAID1 won't mark the last working drive
>> as Faulty no matter what. The responsible code seems here:
>>
>> static void error(struct mddev *mddev, struct md_rdev *rdev)
>> {
>> ...
>> /*
>> * If it is not operational, then we have already marked it as dead
>> * else if it is the last working disks, ignore the error, let the
>> * next level up know.
>> * else mark the drive as failed
>> */
>> if (test_bit(In_sync, &rdev->flags)
>> && (conf->raid_disks - mddev->degraded) == 1) {
>> /*
>> * Don't fail the drive, act as though we were just a
>> * normal single drive.
>> * However don't try a recovery from this drive as
>> * it is very likely to fail.
>> */
>> conf->recovery_disabled = mddev->recovery_disabled;
>> return;
>> }
>> ...
>> }
>>
>> The end result is that even if all the drives are physically gone, there
>> still one drive remains in array forever, and mdadm continues to report
>> the array is degraded instead of failed. RAID10 also has similar behavior.
>>
>> Is there any reason we absolutely don't want to fail the last drive of
>> RAID1?
>>
> When a RAID1 only has one drive remaining, then it should act as much as
> possible like a single plain ordinary drive.
>
> How does /dev/sda behave when you physically remove the device? md0 (as a
> raid1 with one drive) should do the same.
>
> NeilBrown
^ permalink raw reply
* Re: Last working drive in RAID1
From: NeilBrown @ 2015-03-04 23:26 UTC (permalink / raw)
To: Eric Mei; +Cc: linux-raid
In-Reply-To: <54F78BD9.403@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1812 bytes --]
On Wed, 04 Mar 2015 15:48:57 -0700 Eric Mei <meijia@gmail.com> wrote:
> Hi Neil,
>
> I see, that does make sense. Thank you.
>
> But it impose a problem for HA. We have 2 nodes as active-standby pair,
> if HW on node 1 have problem (e.g. SAS cable get pulled, thus all access
> to physical drives are gone), we hope the array failover to node 2. But
> with lingering drive reference, mdadm will report array is still alive
> thus failover won't happen.
>
> I guess it depends on what kind of error on the drive. If it's just a
> media error we should keep it online as much as possible. But if the
> drive is really bad or physically gone, keeping the stale reference
> won't help anything. Back to your comparison with single drive /dev/sda,
> I think MD as an array should do the same as /dev/sda, not the
> individual drive inside MD, for them we should just let it go. How do
> you think?
If there were some what that md could be told that the device really was gone
and just just returning errors, then I would be OK with it being marked as
faulty and being removed from the array.
I don't think there is any mechanism in the kernel to allow that. It would
be easiest to capture a "REMOVE" event via udev, and have udev run "mdadm" to
tell the md array that the device was gone.
Currently there is no way to do that ... I guess we could change raid1 so
that a 'fail' event that came from user-space would always cause the device
to be marked failed, even when an IO error would not...
To preserve current behaviour, it should require something like "faulty-force"
to be written to the "state" file. We would need to check that raid1 copes
with having zero working drives - currently it might always assume there is
at least one device.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Help with raid 10 issue
From: Stefan Lamby @ 2015-03-05 8:18 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Hi.
Is it OK to ask for support here to this list or is it the wrong place?
If so, could you please give a hint where to go?
Thanks.
Stefan
^ permalink raw reply
* Re: Help with raid 10 issue
From: Phil Turmel @ 2015-03-05 13:12 UTC (permalink / raw)
To: Stefan Lamby, linux-raid@vger.kernel.org
In-Reply-To: <493130695.186182.1425543528690.JavaMail.open-xchange@app10.ox.hosteurope.de>
On 03/05/2015 03:18 AM, Stefan Lamby wrote:
>
> Hi.
> Is it OK to ask for support here to this list or is it the wrong place?
> If so, could you please give a hint where to go?
This is the right place. Post your array details (mdadm -D), member
details (mdadm -E) and a summary of what happened. Include excerpts
from your dmesg and/or syslogs that look like they might be relevant.
These should all be text files, so pasting them directly into your email
would be preferred.
Regards,
Phil Turmel
^ permalink raw reply
* Re: [PATCH 0/1] RFC: Use /dev/md/X as default name
From: Jes Sorensen @ 2015-03-05 14:56 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <wrfjoao84uy4.fsf@ultrasam.lan.trained-monkey.org>
Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> NeilBrown <neilb@suse.de> writes:
>> On Mon, 02 Mar 2015 11:32:16 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
>>> Neil,
>>>
>>> Finally had time to go back and look at this - I don't quite understand
>>> your request here.
>>>
>>> As far as I can see, the above code isn't run at all during device
>>> creation, so I don't get how modifying it as you suggest will make
>>> /dev/md/111 appear if I create /dev/md111?
>>>
>>> Cheers,
>>> Jes
>>
>> Wheels within wheels of deception and indirection....
>>
>> mdadm doesn't create the devices directly, udev does that.
>>
>> IMPORT{program}="BINDIR/mdadm --detail --export $devnode"
>> ENV{DEVTYPE}=="disk", ENV{MD_DEVNAME}=="?*", SYMLINK+="md/$env{MD_DEVNAME}"
>>
>>
>> So the symlink from /dev/md gets created iff "mdadm --detail --export"
>> reports an MD_DEVNAME.
>>
>> Hence the change suggested.
>>
>> mdadm *can* create the devices itself, but only if udev isn't running, or
>> mdamd has been explicitly told to ignore udev. Normally that doesn't happen.
>
> Ahhh, I didn't see anything happening on my system, but it may be the
> RHEL/Fedora udev rules file is missing something. I'll have to take a
> closer look there!
Actually I remember why I didn't get that far - I added some debug code
to the place you suggested I modify, and it never triggered during
Create.
Cheers,
Jes
^ permalink raw reply
* Re: Last working drive in RAID1
From: Wols Lists @ 2015-03-05 15:55 UTC (permalink / raw)
To: NeilBrown, Eric Mei; +Cc: linux-raid
In-Reply-To: <20150305102622.016ec792@notabene.brown>
On 04/03/15 23:26, NeilBrown wrote:
> On Wed, 04 Mar 2015 15:48:57 -0700 Eric Mei <meijia@gmail.com>
> wrote:
>
>> Hi Neil,
>>
>> I see, that does make sense. Thank you.
>>
>> But it impose a problem for HA. We have 2 nodes as active-standby
>> pair, if HW on node 1 have problem (e.g. SAS cable get pulled,
>> thus all access to physical drives are gone), we hope the array
>> failover to node 2. But with lingering drive reference, mdadm
>> will report array is still alive thus failover won't happen.
>>
>> I guess it depends on what kind of error on the drive. If it's
>> just a media error we should keep it online as much as possible.
>> But if the drive is really bad or physically gone, keeping the
>> stale reference won't help anything. Back to your comparison with
>> single drive /dev/sda, I think MD as an array should do the same
>> as /dev/sda, not the individual drive inside MD, for them we
>> should just let it go. How do you think?
>
> If there were some what that md could be told that the device
> really was gone and just just returning errors, then I would be OK
> with it being marked as faulty and being removed from the array.
>
> I don't think there is any mechanism in the kernel to allow that.
> It would be easiest to capture a "REMOVE" event via udev, and have
> udev run "mdadm" to tell the md array that the device was gone.
>
> Currently there is no way to do that ... I guess we could change
> raid1 so that a 'fail' event that came from user-space would
> always cause the device to be marked failed, even when an IO error
> would not... To preserve current behaviour, it should require
> something like "faulty-force" to be written to the "state" file.
> We would need to check that raid1 copes with having zero working
> drives - currently it might always assume there is at least one
> device.
>
Sorry to butt in, but I'm finding this conversation a bit surreal ...
take everything I say with a pinch of salt. But the really weird bit
was "what does linux do if /dev/sda disappears?"
In the old days, with /dev/hd*, the * had a hard mapping to the
hardware. hda was the ide0 primary, hdd was the ide1 secondary, etc
etc. I think I ran several systems with just hdb and hdd. Not a good
idea, but.
Nowadays, with sd*, the letter is assigned in order of finding the
drive. So if sda is removed, linux moves all the other drives and what
was sdb becomes sda. Which is why you're advised now to always refer
to drives by their BLKDEV or whatever, as linux provides no guarantees
whatsoever about sd*. The blockdev may only be a symlink to whatever
the sd*n code of the disk is, but it makes sure you get the disk you
want when the sd*n changes under you.
Equally surreal is the comment about "what does raid1 do with no
working devices?". Surely it will do nothing, if there's no spinning
rust or whatever underneath it? You can't corrupt it if there's
nothing there to corrupt?
Sorry again if this is inappropriate, but you're coming over as so
buried in the trees that you can't see the wood.
Cheers,
Wol
^ permalink raw reply
* Raid 10 Issue
From: Stefan Lamby @ 2015-03-05 17:56 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Hello List.
I was setting up a new machine using ubuntu 14.04.02 lts using its installer,
configuring a raid 10 with 2 disks and lvm on top of it. I was using 2 disks and
now I like to add 2 more disks to the array so i want to end up with 4 disks, no
spare.
Searching the internet I found that I am not able to --grow the array with the
mdadm version this ubuntu is using (v3.2.5).
Is that right?
So I decided to build a new array that way and try to move my data afterwards,
which failed:
(Is it OK to do it that way or do you recommend another?)
root@kvm15:~# mdadm --verbose --create --level=10 --raid-devices=4 /dev/md10
missing missing /dev/sdc1 /dev/sdd1
mdadm: layout defaults to n2
mdadm: layout defaults to n2
mdadm: chunk size defaults to 512K
mdadm: /dev/sdc1 appears to be part of a raid array:
level=raid10 devices=4 ctime=Fri Feb 27 15:49:14 2015
mdadm: /dev/sdd1 appears to be part of a raid array:
level=raid10 devices=4 ctime=Fri Feb 27 15:49:14 2015
mdadm: size set to 1904165376K
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: RUN_ARRAY failed: Input/output error
<<<<<<<<<<<<<<<<<<<<<<<<<<<
root@kvm15:~#
root@kvm15:~#
root@kvm15:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid10] [raid6] [raid5]
[raid4]
md0 : active raid10 sdb1[1] sda1[0]
1904165376 blocks super 1.2 2 near-copies [2/2] [UU]
unused devices: <none>
md0 btw. is the current (running) array.
I did a few tries to get this running. This is must be the reason why mdadm
detects already existing raid config.
The partitions for sdc and sdd are created using fdisk, they do have the same
layout like disk sdb, which looks like this:
(parted) print
Modell: ATA WDC WD20PURX-64P (scsi)
Festplatte /dev/sdc: 2000GB
Sektorgröße (logisch/physisch): 512B/4096B
Partitionstabelle: msdos
Nummer Anfang Ende Größe Typ Dateisystem Flags
1 50.4GB 2000GB 1950GB primary RAID
Any help is very welcome.
Thanks.
Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* smartctl-timeouts_v1.01 scripts (scterc timeout fixes)
From: Chris @ 2015-03-05 18:21 UTC (permalink / raw)
To: smartmontools-support, linux-raid
In-Reply-To: <20150227144951.648ba949@smtp.arcor.de>
[-- Attachment #1: Type: text/plain, Size: 1721 bytes --]
Hello list,
could readers please respond with a short yes/no, if they would like
smartctl scripts that fix mismatching scterc/controller timeouts to
prevent data loss?
(Testing them only requires to unzip the file into /etc/udev/rules.d/)
And dear maintainers, would you consider shipping the
scripts?
(Fixing the timeouts seems important for all systems without a
default smartctl -l scterc timeout. And all redundancy controlling
packages (mdadm, lvm, btrfs, ...) may want to suggest or depend on the
smartctl package and set shorter timeouts, i.e. they call the
appropriate scripts depending on redundancy states. Thus smartctl seems
the right package to ship the scripts and default udev rule, and other
packages ship their device specific udev rules.)
Cheers,
Chris
PS: v1.01 has smartctl-default_test.rules for udev and is working well
here with drives that have scterc timeout default to "Disabled".
Am Fri, 27 Feb 2015 14:49:51 +0100
schrieb Chris <email.bug@arcor.de>:
>
> Hi there,
>
> here are some "smartctl-timeouts" scripts that fix commonly
> mismatching defaults with drives that have no error recovery timeout
> configured, which has often lead to data loss. (See the README for
> further information.)
>
> The scripts are the result of discussions on the linux-raid
> mailinglist, but it may be best to ship the scripts with smartctl
> already (possibly together with a default "non-redundant-disk" udev
> rule to ensure proper default timeouts), and let redundancy
> controlling packages like mdadm, lvm, btrfs etc. ship their own udev
> rules to call the scripts appropriately.
>
> Please respond whether you'd like to ship the scripts together with
> smartctl.
>
> Cheers,
> Chris
[-- Attachment #2: smartctl-timeouts_v1.01.zip --]
[-- Type: application/zip, Size: 11266 bytes --]
^ permalink raw reply
* LAON AANBIEDING OP 1,5% rente
From: DIAMOND SWISS LOAN COMPANY @ 2015-03-05 19:38 UTC (permalink / raw)
groeten
We zijn Diamond Zwitserse lening bedrijf het geven van leningen per post advertentie, Ter ondersteuning van banken zeer ernstig en georganiseerd. Als je benteen industriële, aannemer, met pensioen persoon of alle anderen willen krediet aan zijn of haar bedrijf te herstarten of het opzetten van een bedrijf
etc.
we geven leningen. Op 1,5% rente?
Neem vandaag nog contact met de volgende informatie hieronder.
naam:
adres:
leeftijd:
Geslacht:
Contact-Phone:
beroep:
Maandelijks inkomen:
Het benodigde bedrag van de lening:
Duur van de lening:
Het doel van de lening:
land:
Postcode:
Alle informatie moet worden gericht aan ons op: diamondswissloanscompanys@gmail.com
"Wij tonen u een betere manier om uw financiële vrijheid"
Met vriendelijke groet,
De heer Diamond Peters (Managing Director).
De heer Bill Anthony (Marketing Manager).
=============================================================================================================================
Greetings
We are DIAMOND SWISS LOAN COMPANY giving out loans by mail advert, In support of banks very serious and organized. If you are
an industrial, contractor, retire person or all others wanting credit to restart his or her business or to set up a company
etc.
we give out loans. At 1.5% interest rate?
Contact us today with the following information below.
Name:
Address:
Age:
Gender:
Contact-Phone:
Occupation:
Monthly income:
The required amount of loan:
Duration of loan:
The purpose of the loan:
Country:
ZIP code:
All information should be sent to us at: diamondswissloanscompanys@gmail.com
"We show you a better way to your financial freedom"
With kind regards,
Mr. Diamond Peters (Managing Director).
Mr. Bill Anthony (Marketing Manager).
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Last working drive in RAID1
From: Eric Mei @ 2015-03-05 19:54 UTC (permalink / raw)
To: Wols Lists, NeilBrown; +Cc: linux-raid
In-Reply-To: <54F87C7C.8020501@youngman.org.uk>
On 2015-03-05 8:55 AM, Wols Lists wrote:
> On 04/03/15 23:26, NeilBrown wrote:
>> On Wed, 04 Mar 2015 15:48:57 -0700 Eric Mei <meijia@gmail.com>
>> wrote:
>>
>>> Hi Neil,
>>>
>>> I see, that does make sense. Thank you.
>>>
>>> But it impose a problem for HA. We have 2 nodes as active-standby
>>> pair, if HW on node 1 have problem (e.g. SAS cable get pulled,
>>> thus all access to physical drives are gone), we hope the array
>>> failover to node 2. But with lingering drive reference, mdadm
>>> will report array is still alive thus failover won't happen.
>>>
>>> I guess it depends on what kind of error on the drive. If it's
>>> just a media error we should keep it online as much as possible.
>>> But if the drive is really bad or physically gone, keeping the
>>> stale reference won't help anything. Back to your comparison with
>>> single drive /dev/sda, I think MD as an array should do the same
>>> as /dev/sda, not the individual drive inside MD, for them we
>>> should just let it go. How do you think?
>> If there were some what that md could be told that the device
>> really was gone and just just returning errors, then I would be OK
>> with it being marked as faulty and being removed from the array.
>>
>> I don't think there is any mechanism in the kernel to allow that.
>> It would be easiest to capture a "REMOVE" event via udev, and have
>> udev run "mdadm" to tell the md array that the device was gone.
>>
>> Currently there is no way to do that ... I guess we could change
>> raid1 so that a 'fail' event that came from user-space would
>> always cause the device to be marked failed, even when an IO error
>> would not... To preserve current behaviour, it should require
>> something like "faulty-force" to be written to the "state" file.
>> We would need to check that raid1 copes with having zero working
>> drives - currently it might always assume there is at least one
>> device.
>>
> Sorry to butt in, but I'm finding this conversation a bit surreal ...
> take everything I say with a pinch of salt. But the really weird bit
> was "what does linux do if /dev/sda disappears?"
>
> In the old days, with /dev/hd*, the * had a hard mapping to the
> hardware. hda was the ide0 primary, hdd was the ide1 secondary, etc
> etc. I think I ran several systems with just hdb and hdd. Not a good
> idea, but.
>
> Nowadays, with sd*, the letter is assigned in order of finding the
> drive. So if sda is removed, linux moves all the other drives and what
> was sdb becomes sda. Which is why you're advised now to always refer
> to drives by their BLKDEV or whatever, as linux provides no guarantees
> whatsoever about sd*. The blockdev may only be a symlink to whatever
> the sd*n code of the disk is, but it makes sure you get the disk you
> want when the sd*n changes under you.
>
> Equally surreal is the comment about "what does raid1 do with no
> working devices?". Surely it will do nothing, if there's no spinning
> rust or whatever underneath it? You can't corrupt it if there's
> nothing there to corrupt?
>
> Sorry again if this is inappropriate, but you're coming over as so
> buried in the trees that you can't see the wood.
>
> Cheers,
> Wol
Hi Wol,
I think Neil's intention regarding "/dev/sda" waslike this: For a single
drive if it's physically gone, its user will still keep the reference to
it; and RAID1 with single drive should behave the same, i.e. without
knowledge of what exactly happened on this drive, MD is not comfortable
to make decision for application about the status of the last drive,
thus refuse to mark it as failed. The whole thing has not much to do
with FS arrangement under /dev, which is usually managed by udev.
Eric
^ permalink raw reply
* Re: Last working drive in RAID1
From: Phil Turmel @ 2015-03-05 20:00 UTC (permalink / raw)
To: Wols Lists, NeilBrown, Eric Mei; +Cc: linux-raid
In-Reply-To: <54F87C7C.8020501@youngman.org.uk>
On 03/05/2015 10:55 AM, Wols Lists wrote:
> Sorry to butt in, but I'm finding this conversation a bit surreal ...
> take everything I say with a pinch of salt. But the really weird bit
> was "what does linux do if /dev/sda disappears?"
>
> In the old days, with /dev/hd*, the * had a hard mapping to the
> hardware. hda was the ide0 primary, hdd was the ide1 secondary, etc
> etc. I think I ran several systems with just hdb and hdd. Not a good
> idea, but.
>
> Nowadays, with sd*, the letter is assigned in order of finding the
> drive. So if sda is removed, linux moves all the other drives and what
> was sdb becomes sda.
On reboot, yes. Not live. If /dev/sda has anything using it when
unplugged, it stays there and gives errors. Existing devices retain
their names, and new devices are added to the end. Only if all users
are completely unhooked will a kernel name get re-used live.
> Which is why you're advised now to always refer
> to drives by their BLKDEV or whatever, as linux provides no guarantees
> whatsoever about sd*. The blockdev may only be a symlink to whatever
> the sd*n code of the disk is, but it makes sure you get the disk you
> want when the sd*n changes under you.
Correct, but once assembled or mounted, the kernel names are locked.
> Equally surreal is the comment about "what does raid1 do with no
> working devices?". Surely it will do nothing, if there's no spinning
> rust or whatever underneath it? You can't corrupt it if there's
> nothing there to corrupt?
It has to stay there to give errors to the upper layers that are still
hooked to it. When they are administratively "unhooked", aka unmounted
or disassociated with mdadm --remove.
Or, quite possibly, the device is plugged back in, at which point the
device name is there for it (as long as you use the same port, of
course). In which case the filesystem may very well resume successfully.
> Sorry again if this is inappropriate, but you're coming over as so
> buried in the trees that you can't see the wood.
Sometimes the expanded view is appropriate :-)
Regards,
Phil Turmel
^ permalink raw reply
* Re: Raid 10 Issue
From: Phil Turmel @ 2015-03-05 20:07 UTC (permalink / raw)
To: Stefan Lamby, linux-raid@vger.kernel.org
In-Reply-To: <1888475554.226351.1425578160090.JavaMail.open-xchange@app09.ox.hosteurope.de>
On 03/05/2015 12:56 PM, Stefan Lamby wrote:
> Hello List.
>
> I was setting up a new machine using ubuntu 14.04.02 lts using its installer,
> configuring a raid 10 with 2 disks and lvm on top of it. I was using 2 disks and
> now I like to add 2 more disks to the array so i want to end up with 4 disks, no
> spare.
>
> Searching the internet I found that I am not able to --grow the array with the
> mdadm version this ubuntu is using (v3.2.5).
> Is that right?
>
> So I decided to build a new array that way and try to move my data afterwards,
> which failed:
> (Is it OK to do it that way or do you recommend another?)
No, you should be able to do this. Probably without any shutdown.
Please show the full layout of your drives, partitions, and lvm.
I suggest lsdrv[1] for working layouts. If your email is set to use
utf8, just paste the result in a reply.
Regards,
Phil Turmel
[1] https://github.com/pturmel/lsdrv
^ permalink raw reply
* Re: Last working drive in RAID1
From: Eric Mei @ 2015-03-05 20:23 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150305102622.016ec792@notabene.brown>
On 2015-03-04 4:26 PM, NeilBrown wrote:
> On Wed, 04 Mar 2015 15:48:57 -0700 Eric Mei <meijia@gmail.com> wrote:
>
>> Hi Neil,
>>
>> I see, that does make sense. Thank you.
>>
>> But it impose a problem for HA. We have 2 nodes as active-standby pair,
>> if HW on node 1 have problem (e.g. SAS cable get pulled, thus all access
>> to physical drives are gone), we hope the array failover to node 2. But
>> with lingering drive reference, mdadm will report array is still alive
>> thus failover won't happen.
>>
>> I guess it depends on what kind of error on the drive. If it's just a
>> media error we should keep it online as much as possible. But if the
>> drive is really bad or physically gone, keeping the stale reference
>> won't help anything. Back to your comparison with single drive /dev/sda,
>> I think MD as an array should do the same as /dev/sda, not the
>> individual drive inside MD, for them we should just let it go. How do
>> you think?
> If there were some what that md could be told that the device really was gone
> and just just returning errors, then I would be OK with it being marked as
> faulty and being removed from the array.
>
> I don't think there is any mechanism in the kernel to allow that. It would
> be easiest to capture a "REMOVE" event via udev, and have udev run "mdadm" to
> tell the md array that the device was gone.
>
> Currently there is no way to do that ... I guess we could change raid1 so
> that a 'fail' event that came from user-space would always cause the device
> to be marked failed, even when an IO error would not...
> To preserve current behaviour, it should require something like "faulty-force"
> to be written to the "state" file. We would need to check that raid1 copes
> with having zero working drives - currently it might always assume there is
> at least one device.
I guess we don't need to know exactly what happened physically, it
should be good enough to know "drive stopped working". If a drive
stopped working, keeping it doesn't add much value anyway. And I think
serious error detected in MD (e.g. superblock write error, bad block
table write error) might be a good criteria to make that judgement.
But as you said current code may assume at least one drive present, need
a more careful review.
Eric
^ permalink raw reply
* Re: Last working drive in RAID1
From: NeilBrown @ 2015-03-05 21:52 UTC (permalink / raw)
To: Phil Turmel; +Cc: Wols Lists, Eric Mei, linux-raid
In-Reply-To: <54F8B5D2.8070304@turmel.org>
[-- Attachment #1: Type: text/plain, Size: 2692 bytes --]
On Thu, 05 Mar 2015 15:00:18 -0500 Phil Turmel <philip@turmel.org> wrote:
> On 03/05/2015 10:55 AM, Wols Lists wrote:
> > Sorry to butt in, but I'm finding this conversation a bit surreal ...
> > take everything I say with a pinch of salt. But the really weird bit
> > was "what does linux do if /dev/sda disappears?"
> >
> > In the old days, with /dev/hd*, the * had a hard mapping to the
> > hardware. hda was the ide0 primary, hdd was the ide1 secondary, etc
> > etc. I think I ran several systems with just hdb and hdd. Not a good
> > idea, but.
> >
> > Nowadays, with sd*, the letter is assigned in order of finding the
> > drive. So if sda is removed, linux moves all the other drives and what
> > was sdb becomes sda.
>
> On reboot, yes. Not live. If /dev/sda has anything using it when
> unplugged, it stays there and gives errors. Existing devices retain
> their names, and new devices are added to the end. Only if all users
> are completely unhooked will a kernel name get re-used live.
>
> > Which is why you're advised now to always refer
> > to drives by their BLKDEV or whatever, as linux provides no guarantees
> > whatsoever about sd*. The blockdev may only be a symlink to whatever
> > the sd*n code of the disk is, but it makes sure you get the disk you
> > want when the sd*n changes under you.
>
> Correct, but once assembled or mounted, the kernel names are locked.
>
> > Equally surreal is the comment about "what does raid1 do with no
> > working devices?". Surely it will do nothing, if there's no spinning
> > rust or whatever underneath it? You can't corrupt it if there's
> > nothing there to corrupt?
>
> It has to stay there to give errors to the upper layers that are still
> hooked to it. When they are administratively "unhooked", aka unmounted
> or disassociated with mdadm --remove.
>
> Or, quite possibly, the device is plugged back in, at which point the
> device name is there for it (as long as you use the same port, of
> course). In which case the filesystem may very well resume successfully.
I was with you right up to this last point.
When a device is unplugged and then plugged back in, it will always get a new
name. Detecting "is the same device" is far from fool-proof, particularly as
the device could have been plugged into some other machine and has 'fsck' etc
run.
Once a mounted device is unplugged, that mount is permanently unusable.
NeilBrown
>
> > Sorry again if this is inappropriate, but you're coming over as so
> > buried in the trees that you can't see the wood.
>
> Sometimes the expanded view is appropriate :-)
>
> Regards,
>
> Phil Turmel
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Last working drive in RAID1
From: Chris @ 2015-03-05 21:54 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <54F8B5D2.8070304@turmel.org>
Am Thu, 05 Mar 2015 15:00:18 -0500
schrieb Phil Turmel <philip@turmel.org>:
> It has to stay there to give errors to the upper layers that are still
> hooked to it. When they are administratively "unhooked", aka
> unmounted or disassociated with mdadm --remove.
>
> Or, quite possibly, the device is plugged back in, at which point the
> device name is there for it (as long as you use the same port, of
> course). In which case the filesystem may very well resume
> successfully.
From reading this it makes sense that the md device stays there, just
as the the physical device nodes. (to give errors, and to recover)
However, as I understood this thread, md does not seem to inform upper
layers or the user (even not through its own --monitor?) properly.
To me, marking the last disk within an array as failed (*within* the
array) just seems to make more sense, so /proc/mdstat actually
iforms about the md error state (and the md device returning errors on
access).
Regards,
Chris
^ permalink raw reply
* Re: Raid 10 Issue
From: Robin Hill @ 2015-03-06 8:54 UTC (permalink / raw)
To: Stefan Lamby; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <1888475554.226351.1425578160090.JavaMail.open-xchange@app09.ox.hosteurope.de>
[-- Attachment #1: Type: text/plain, Size: 1860 bytes --]
On Thu Mar 05, 2015 at 06:56:00PM +0100, Stefan Lamby wrote:
> Hello List.
>
> I was setting up a new machine using ubuntu 14.04.02 lts using its installer,
> configuring a raid 10 with 2 disks and lvm on top of it. I was using 2 disks and
> now I like to add 2 more disks to the array so i want to end up with 4 disks, no
> spare.
>
> Searching the internet I found that I am not able to --grow the array with the
> mdadm version this ubuntu is using (v3.2.5).
> Is that right?
>
> So I decided to build a new array that way and try to move my data afterwards,
> which failed:
> (Is it OK to do it that way or do you recommend another?)
>
> root@kvm15:~# mdadm --verbose --create --level=10 --raid-devices=4 /dev/md10
> missing missing /dev/sdc1 /dev/sdd1
> mdadm: layout defaults to n2
> mdadm: layout defaults to n2
> mdadm: chunk size defaults to 512K
> mdadm: /dev/sdc1 appears to be part of a raid array:
> level=raid10 devices=4 ctime=Fri Feb 27 15:49:14 2015
> mdadm: /dev/sdd1 appears to be part of a raid array:
> level=raid10 devices=4 ctime=Fri Feb 27 15:49:14 2015
> mdadm: size set to 1904165376K
> Continue creating array? y
> mdadm: Defaulting to version 1.2 metadata
> mdadm: RUN_ARRAY failed: Input/output error
> <<<<<<<<<<<<<<<<<<<<<<<<<<<
> root@kvm15:~#
>
IIRC, in a RAID10 setup, the redundant pair is held on adjacent drives.
You've specified two adjacent drives as missing, so the array cannot be
run. Try doing:
mdadm --verbose --create --level=10 --raid-devices=4 /dev/md10 \
missing /dev/sdc1 missing /dev/sdd1
HTH,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: Last working drive in RAID1
From: Chris @ 2015-03-06 9:21 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20150306085222.19dd30ce@notabene.brown>
Am Fri, 6 Mar 2015 08:52:22 +1100
schrieb NeilBrown <neilb@suse.de>:
> > Or, quite possibly, the device is plugged back in, at which point
> > the device name is there for it (as long as you use the same port,
> > of course). In which case the filesystem may very well resume
> > successfully.
>
> I was with you right up to this last point.
> When a device is unplugged and then plugged back in, it will always
> get a new name.
Right, that is what I see when "rotating" disks or connecting to a
docking station. And I did also see the last disk of an external
storage raid not going away (fail) when unplugged.
The times I had devices really break though, I don't think they
triggered an unplug event. They still seemed fully connected but had
motor-start, bus errors or something. In some occasions the failure was
only intermittent, and the device node continued to work again (within
the controller timeout or without a permanent error remaining after a
reset).
So if the last working drive could be marked as failed, when it
actually failed, that would also provide the proper information for
system failover on replicated hosts, in the many cases when a
controller/bus/drive fails without an unplug event. Cases that the udev
rule only idea does not seem to cover.
Regards,
Chris
^ permalink raw reply
* Re: Raid 10 Issue [SOLVED]
From: Stefan Lamby @ 2015-03-06 9:32 UTC (permalink / raw)
To: Robin Hill; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <20150306085407.GA32079@cthulhu.home.robinhill.me.uk>
> Robin Hill <robin@robinhill.me.uk> hat am 6. März 2015 um 09:54 geschrieben:
>
>
> On Thu Mar 05, 2015 at 06:56:00PM +0100, Stefan Lamby wrote:
>
> > Hello List.
> >
> > I was setting up a new machine using ubuntu 14.04.02 lts using its
> > installer,
> > configuring a raid 10 with 2 disks and lvm on top of it. I was using 2 disks
> > and
> > now I like to add 2 more disks to the array so i want to end up with 4
> > disks, no
> > spare.
> >
> > Searching the internet I found that I am not able to --grow the array with
> > the
> > mdadm version this ubuntu is using (v3.2.5).
> > Is that right?
> >
> > So I decided to build a new array that way and try to move my data
> > afterwards,
> > which failed:
> > (Is it OK to do it that way or do you recommend another?)
> >
> > root@kvm15:~# mdadm --verbose --create --level=10 --raid-devices=4 /dev/md10
> > missing missing /dev/sdc1 /dev/sdd1
> > mdadm: layout defaults to n2
> > mdadm: layout defaults to n2
> > mdadm: chunk size defaults to 512K
> > mdadm: /dev/sdc1 appears to be part of a raid array:
> > level=raid10 devices=4 ctime=Fri Feb 27 15:49:14 2015
> > mdadm: /dev/sdd1 appears to be part of a raid array:
> > level=raid10 devices=4 ctime=Fri Feb 27 15:49:14 2015
> > mdadm: size set to 1904165376K
> > Continue creating array? y
> > mdadm: Defaulting to version 1.2 metadata
> > mdadm: RUN_ARRAY failed: Input/output error
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<
> > root@kvm15:~#
> >
> IIRC, in a RAID10 setup, the redundant pair is held on adjacent drives.
> You've specified two adjacent drives as missing, so the array cannot be
> run. Try doing:
> mdadm --verbose --create --level=10 --raid-devices=4 /dev/md10 \
> missing /dev/sdc1 missing /dev/sdd1
>
Dang!
One shot one catch, thank you so much!
Dear Maintainer,
please take a one-to-one copy of Robins explaination and print it if another
dumb user like me tries to do the same and is caught also. This could help avoid
a lot of frustration. There are posts in the internet which claim the order of
the given devices is critical.
Thank you all for taking the time to help.
Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Raid 10 Issue - Swapping Data from Array to Array
From: Stefan Lamby @ 2015-03-06 10:09 UTC (permalink / raw)
To: Phil Turmel, linux-raid@vger.kernel.org
In-Reply-To: <54F8B792.9060206@turmel.org>
> Phil Turmel <philip@turmel.org> hat am 5. März 2015 um 21:07 geschrieben:
>
>
> On 03/05/2015 12:56 PM, Stefan Lamby wrote:
> > Hello List.
> >
> > I was setting up a new machine using ubuntu 14.04.02 lts using its
> > installer,
> > configuring a raid 10 with 2 disks and lvm on top of it. I was using 2 disks
> > and
> > now I like to add 2 more disks to the array so i want to end up with 4
> > disks, no
> > spare.
> >
> > Searching the internet I found that I am not able to --grow the array with
> > the
> > mdadm version this ubuntu is using (v3.2.5).
> > Is that right?
> >
> > So I decided to build a new array that way and try to move my data
> > afterwards,
> > which failed:
> > (Is it OK to do it that way or do you recommend another?)
>
> No, you should be able to do this. Probably without any shutdown.
> Please show the full layout of your drives, partitions, and lvm.
>
> I suggest lsdrv[1] for working layouts. If your email is set to use
> utf8, just paste the result in a reply.
>
> Regards,
>
> Phil Turmel
>
> [1] https://github.com/pturmel/lsdrv
>
Hi Phil.
I like your suggestion using lsdrv. Pretty nice. Here is the output (including
the newly created array):
root@kvm15:~/lsdrv/lsdrv# ./lsdrv
PCI [ata_piix] 00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2
port SATA IDE Controller #2
├scsi 0:0:0:0 HL-DT-ST DVD-RAM GH60L {K1XA5SF1137}
│└sr0 3.68g [11:0] udf 'UDF_Volume'
└scsi 1:x:x:x [Empty]
PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family)
SATA AHCI Controller
├scsi 2:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M1LPT1AE}
│└sda 1.82t [8:0] Partitioned (dos)
│ └sda1 1.77t [8:1] MD raid10,near2 (0/2) (w/ sdb1) in_sync 'kvm15:0'
{75079a2f-acb8-c475-85f8-ca430ad85c4c}
│ └md0 1.77t [9:0] MD v1.2 raid10,near2 (2) clean, 512k Chunk
{75079a2f:acb8c475:85f8ca43:0ad85c4c}
│ │ PV LVM2_member 1.01t used, 780.45g free
{2hsby0-0FOT-PPbC-il1r-ux9J-lUd2-nPHj7T}
│ └VG vg_raid10 1.77t 780.45g free {HbjouC-RgUe-YYNB-z2ns-4kzK-RwJH-RHWSWq}
│ ├dm-0 479.39g [252:0] LV home ext4 {2d67d9cc-0378-4669-9d72-7b7c7071dea8}
│ │└Mounted as /dev/mapper/vg_raid10-home @ /home
│ ├dm-1 93.13g [252:1] LV root ext4 {c14e4524-e95c-45c2-bfa0-75d529ed48fe}
│ │└Mounted as /dev/mapper/vg_raid10-root @ /
│ ├dm-4 23.28g [252:4] LV swap swap {9e1a582f-1c88-44a2-be90-aafcb96805c7}
│ ├dm-3 46.56g [252:3] LV tmp ext4 {ac67d0d9-049c-4cf2-9a0e-591cdb6a3559}
│ │└Mounted as /dev/mapper/vg_raid10-tmp @ /tmp
│ └dm-2 393.13g [252:2] LV var ext4 {ff71c558-c1f8-4410-8e2a-dc9c77c27a03}
│ └Mounted as /dev/mapper/vg_raid10-var @ /var
├scsi 3:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M5LAR62D}
│└sdb 1.82t [8:16] Partitioned (dos)
│ └sdb1 1.77t [8:17] MD raid10,near2 (1/2) (w/ sda1) in_sync 'kvm15:0'
{75079a2f-acb8-c475-85f8-ca430ad85c4c}
│ └md0 1.77t [9:0] MD v1.2 raid10,near2 (2) clean, 512k Chunk
{75079a2f:acb8c475:85f8ca43:0ad85c4c}
│ PV LVM2_member 1.01t used, 780.45g free
{2hsby0-0FOT-PPbC-il1r-ux9J-lUd2-nPHj7T}
├scsi 4:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M7YA1ANR}
│└sdc 1.82t [8:32] Partitioned (dos)
│ └sdc1 1.77t [8:33] MD raid10,near2 (1/4) (w/ sdd1) in_sync 'kvm15:10'
{c4540426-9c66-8fe2-4795-13f242d233b4}
│ └md10 3.55t [9:10] MD v1.2 raid10,near2 (4) clean DEGRADEDx2, 512k Chunk
{c4540426:9c668fe2:479513f2:42d233b4}
│ Empty/Unknown
└scsi 5:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M5AFRYVP}
└sdd 1.82t [8:48] Partitioned (dos)
└sdd1 1.77t [8:49] MD raid10,near2 (3/4) (w/ sdc1) in_sync 'kvm15:10'
{c4540426-9c66-8fe2-4795-13f242d233b4}
└md10 3.55t [9:10] MD v1.2 raid10,near2 (4) clean DEGRADEDx2, 512k Chunk
{c4540426:9c668fe2:479513f2:42d233b4}
Empty/Unknown
This is what I got right now.
What do you recommend to do?
Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Raid 10 Issue - Swapping Data from Array to Array
From: Phil Turmel @ 2015-03-06 12:57 UTC (permalink / raw)
To: Stefan Lamby, linux-raid@vger.kernel.org
In-Reply-To: <1315687872.8638.1425636591443.JavaMail.open-xchange@app04.ox.hosteurope.de>
On 03/06/2015 05:09 AM, Stefan Lamby wrote:
>
>
> This is what I got right now.
> What do you recommend to do?
1) pvcreate on the new array
2) vgextend to add the new array to the volume group
3) pvmove to get the data into the new array
4) vgreduce to disconnect the old array from lvm
5) pvremove to wipe the old array's lvm meta
6) stop the old array
7) mdadm --zero-superblock to clear the old members
8) mdadm --add to put those members into the new array
Regards,
Phil Turmel
^ permalink raw reply
* Re: Raid 10 Issue - Booting in case raid failed
From: Stefan Lamby @ 2015-03-06 19:06 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
In-Reply-To: <1315687872.8638.1425636591443.JavaMail.open-xchange@app04.ox.hosteurope.de>
Hi list.
If everything will work out OK, I will end up with an raid 10 array with 4
devices.
My partition design and layout structure will be found at the end, if needed.
There are a few questions left for me in case I have to boot with a failed disk:
1) As you might have seen from the partition design, only partition sda1 has the
boot flag set. As far as I guess, the ubuntu installer was using grub-install
only for sda. I am kind of afraid what will happen, in case sda will fail in the
future. Will it be a good idea to grub-install to all the other devices also?
2) What about the boot flag, if I need to grub-install the other devices also?
Should it be O or 1? Do I have to leave it set to false and in case things go
wrong boot from a live cd and set it to on to boot from another device?
What do you recommend?
Thanks
Stefan
Here is my layout - please do not care
root@kvm15:~# fdisk -l /dev/sda
Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 Köpfe, 63 Sektoren/Spur, 243201 Zylinder, zusammen 3907029168 Sektoren
Einheiten = Sektoren von 1 × 512 = 512 Bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Festplattenidentifikation: 0x00071c2b
Gerät boot. Anfang Ende Blöcke Id System
/dev/sda1 * 98435072 3907028991 1904296960 fd Linux raid autodetect
Here as an example for all other disks
Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 Köpfe, 63 Sektoren/Spur, 243201 Zylinder, zusammen 3907029168 Sektoren
Einheiten = Sektoren von 1 × 512 = 512 Bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Festplattenidentifikation: 0x0008624b
Gerät boot. Anfang Ende Blöcke Id System
/dev/sdb1 98435072 3907028991 1904296960 fd Linux raid autodetect
PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family)
SATA AHCI Controller
├scsi 2:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M1LPT1AE}
│└sda 1.82t [8:0] Partitioned (dos)
│ └sda1 1.77t [8:1] MD raid10,near2 (0/2) (w/ sdb1) in_sync 'kvm15:0'
{75079a2f-acb8-c475-85f8-ca430ad85c4c}
│ └md0 1.77t [9:0] MD v1.2 raid10,near2 (2) clean, 512k Chunk
{75079a2f:acb8c475:85f8ca43:0ad85c4c}
│ │ PV LVM2_member 1.01t used, 780.45g free
{2hsby0-0FOT-PPbC-il1r-ux9J-lUd2-nPHj7T}
│ └VG vg_raid10 5.32t (w/ md10) 3.30t free
{HbjouC-RgUe-YYNB-z2ns-4kzK-RwJH-RHWSWq}
│ ├dm-0 479.39g [252:0] LV home ext4 {2d67d9cc-0378-4669-9d72-7b7c7071dea8}
│ │└Mounted as /dev/mapper/vg_raid10-home @ /home
│ ├dm-1 93.13g [252:1] LV root ext4 {c14e4524-e95c-45c2-bfa0-75d529ed48fe}
│ │└Mounted as /dev/mapper/vg_raid10-root @ /
│ ├dm-4 23.28g [252:4] LV swap swap {9e1a582f-1c88-44a2-be90-aafcb96805c7}
│ ├dm-3 46.56g [252:3] LV tmp ext4 {ac67d0d9-049c-4cf2-9a0e-591cdb6a3559}
│ │└Mounted as /dev/mapper/vg_raid10-tmp @ /tmp
│ └dm-2 393.13g [252:2] LV var ext4 {ff71c558-c1f8-4410-8e2a-dc9c77c27a03}
│ └Mounted as /dev/mapper/vg_raid10-var @ /var
├scsi 3:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M5LAR62D}
│└sdb 1.82t [8:16] Partitioned (dos)
│ └sdb1 1.77t [8:17] MD raid10,near2 (1/2) (w/ sda1) in_sync 'kvm15:0'
{75079a2f-acb8-c475-85f8-ca430ad85c4c}
│ └md0 1.77t [9:0] MD v1.2 raid10,near2 (2) clean, 512k Chunk
{75079a2f:acb8c475:85f8ca43:0ad85c4c}
│ PV LVM2_member 1.01t used, 780.45g free
{2hsby0-0FOT-PPbC-il1r-ux9J-lUd2-nPHj7T}
├scsi 4:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M7YA1ANR}
│└sdc 1.82t [8:32] Partitioned (dos)
│ └sdc1 1.77t [8:33] MD raid10,near2 (1/4) (w/ sdd1) in_sync 'kvm15:10'
{c4540426-9c66-8fe2-4795-13f242d233b4}
│ └md10 3.55t [9:10] MD v1.2 raid10,near2 (4) active DEGRADEDx2, 512k Chunk
{c4540426:9c668fe2:479513f2:42d233b4}
│ │ PV LVM2_member 1.01t used, 2.54t free
{wYV8fH-uOp2-E88P-EIp5-6U33-chYg-S94vvy}
│ └VG vg_raid10 5.32t (w/ md0) 3.30t free
{HbjouC-RgUe-YYNB-z2ns-4kzK-RwJH-RHWSWq}
└scsi 5:0:0:0 ATA WDC WD20PURX-64P {WD-WCC4M5AFRYVP}
└sdd 1.82t [8:48] Partitioned (dos)
└sdd1 1.77t [8:49] MD raid10,near2 (3/4) (w/ sdc1) in_sync 'kvm15:10'
{c4540426-9c66-8fe2-4795-13f242d233b4}
└md10 3.55t [9:10] MD v1.2 raid10,near2 (4) active DEGRADEDx2, 512k Chunk
{c4540426:9c668fe2:479513f2:42d233b4}
PV LVM2_member 1.01t used, 2.54t free
{wYV8fH-uOp2-E88P-EIp5-6U33-chYg-S94vvy}
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: raid 10 - Recovering raid - what stats are most helpfull?
From: Stefan Lamby @ 2015-03-06 19:23 UTC (permalink / raw)
To: Phil Turmel, linux-raid@vger.kernel.org
In-Reply-To: <54F85645.9000603@turmel.org>
> Phil Turmel <philip@turmel.org> hat am 5. März 2015 um 14:12 geschrieben:
>
>
> On 03/05/2015 03:18 AM, Stefan Lamby wrote:
> >
> > Hi.
> > Is it OK to ask for support here to this list or is it the wrong place?
> > If so, could you please give a hint where to go?
>
> This is the right place. Post your array details (mdadm -D), member
> details (mdadm -E) and a summary of what happened. Include excerpts
> from your dmesg and/or syslogs that look like they might be relevant.
>
> These should all be text files, so pasting them directly into your email
> would be preferred.
>
> Regards,
>
> Phil Turmel
>
>
Hi Phil,
hello List.
This would be also very interesting - just in case:
Hopefully it will never happen, but if the raid 10 array crashes, what
information will be the most usefull for later recovery? Is this the output of
cat /proc/mdstat or as mentioned by Phil the output of mdadm --detail or what
ever?
I would like to be prepared and write a small shell script which will write this
to a file to be included in the backups.
What information about the raid would you suggest to collect and to save
somewhere to have it handy in case it is needed?
Thank you very much for your help.
Stefan
P.S.: Phil, thank you so much for the hints regarding moving the data. It looks
very promising. I'll know more in detail tomorrow. pvmove is taking all night, I
guess.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox