* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Chris @ 2015-02-18 11:04 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAJCQCtTv-HMX77GWqA+_1rfkWodsTETp_Y3w5En-N4nGCymWsA@mail.gmail.com>
>
Hello all,
the discussion about SCTERC boils down to letting the drive attempt ERC a
little more or less. For any given disk experience seems to tell the slight
difference is, that if ERC is allowed longer you may see the first
unrecoverable erros (UREs) just a little (maybe only a month) later.
UREs are inevitable. Thus, if I run a filesystem on just a single drive it
will get corrupted at some point, nothing to do about it.
Wait, except..., use a redundant raid! And here it makes a lot of a
difference that the drive's ERC actually terminates before the controller
timeout, to not loose all your redundacy again and be in hight risk of UREs
showing up during the re-sync.
So for a proper comparison we need to look at the difference it makes in the
usage scenarios (error delay vs. loosing redundant error resilence + URE
triggering), not at the single recoverable/unrecoverable error incidence. It
looks to me, that it makes a lot of a differnce to redundant raids and no
qualitative difference to single disk filesystems.
And we need to keep in mind that single disk filesystems do also depend on
the disk to stop grinding away with ERC attempts before the controller
timout. Otherwise disk reset may make the system clear buffers and loose
open files? Without prolonging the linux default controller timout, SCTERC
can prevent that where supported.
> in any case the proper place to change the default kernel command
> timer value is in the kernel, not with a udev rule.
Right. And as you write increasing the controller timout has clear downsides.
Noteing as well, as long as the proposed script (a temporary safety measure)
maximizes the controller timeout to remedy for disks that don's support
SCTERC, this would even fix the timout mismatch for single disk filesystems.
(Letting the controller wait until the disk finally succeeds or fails its
recovery attempts.)
So the proposed script actually provides a case that brings benefit for
raid0 setups as well (as long as the linux default is not adaptive to the
disk parameters), but increasing the controller timout in all cases would
introduce long and unreported i/o blocking into all redundant setups.
> I don't know if a udev rule can say "If the drive exclusively uses md,
> lvm, btrfs, zfs raid1, 4+ or nested of those, and if the drive does
> not support configurable SCT ERC, then change the kernel command timer
> for those devices to ~120 seconds" then that might be a plausible
> solution to use consumer drives the manufacturer rather explicitly
> proscribes from use in raid...
The script called by the udev rule could do that, but can be kept as simple
as proposed, and can set SCTERC regardles, because setting SCTERC below the
controller timout makes a qualitative difference in running the redundant
arrays and a marginal difference in running non-redundant filesystems. (And
nevertheless, set long controller timout for devices that don's support SCTERC.)
After all, this looks like a quite simple change is appropriate:
In udev-md-raid-assembly.rules, below LABEL="md_inc" (only handling all md
suppported devices) add one rule:
# fix timouts for redundant raids, if possible
TEST="/usr/sbin/smartctl", ENV{MD_LEVEL}=="raid[1-9]*",
RUN+="/usr/bin/mdadm-erc-timout-fix"
And in a new /usr/bin/mdadm-erc-timout-fix file implement:
if smartctl -l scterc ${HDD_DEV} returns "Disabled"
/usr/sbin/smartctl -l scterc,70,70 ${HDD_DEV}
else
if smartctl -l scterc ${HDD_DEV} does not return "seconds"
echo 180 >/sys/block/${HDD_DEV}/device/timeout
Regards,
Chris
^ permalink raw reply
* mdadm monitoring for RAID-1 - no events after pulling out all disks.
From: Iakiv Shchelkunov @ 2015-02-18 11:42 UTC (permalink / raw)
To: linux-raid
Hello everyone. Please, help me with one mdadm monitoring issue.
I use mdadm 3.2.5 on Debian 7.6. Besides system HDD, I have 2 SSDs
with the same partition table.
I have an array RAID1 over those SSDs with the same partitions in it.
As I need to monitor mdadm errors, I have a tool for monitoring.
So, I do next:
mdadm –assemble /dev/md1 /dev/sda1 /dev/sdb1
//run the md which was created earlier
mdadm --monitor --daemonize --scan --syslog
--program=/usr/local/mdmonitor //monitoring
The md /dev/md1 is running, everything is okey.
When I pull out the first SSD and try to write anything on active
array, mdadm calls my program with parameters “Fail” (or “FailSpare”),
the md device name /dev/md1 and partition name, as it’s supposed to
do.
But when I pull out second SSD and there is no one SSD is present,
then try to write data on the md, mdadm doesn’t run my program to
inform about fail.
Also /proc/mdstat isn’t changed – the array is still ‘active’, even
without any physically present SSD. Of course, I can’t do any write
operations, I’ve got an error trying to do that, but I have no event
from mdadm monitoring.
Has anybody faced with similar troubles? Is there any solution?
Thanks in advance for your time, Iakiv.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: RAID1 might_sleep() warning on 3.19-rc7
From: Peter Zijlstra @ 2015-02-18 13:47 UTC (permalink / raw)
To: NeilBrown; +Cc: Tony Battersby, linux-raid, lkml, axboe, Linus Torvalds
In-Reply-To: <20150218120936.3c7beed2@notabene.brown>
On Wed, Feb 18, 2015 at 12:09:36PM +1100, NeilBrown wrote:
> On Fri, 13 Feb 2015 15:48:03 +0100 Peter Zijlstra <peterz@infradead.org>
> wrote:
>
> > On Fri, Feb 13, 2015 at 11:27:46AM +0100, Peter Zijlstra wrote:
> >
> > > > I've moved blk_flush_plug to the beginning of the function.
> > >
> > > > I wondered if it really make sense to call blk_flush_plug with nr_iowait
> > > > elevated and delayacct_blkio active. blk_flush_plug() could call schedule()
> > > > for non-"io" reasons and maybe that could upset stuff???
> > >
> > > Yeah, good question that. Lemme ponder that a bit.
> >
> > Yes, I thikn your version makes most sense as, you say, even regular
> > schedule() call nested in my version would go towards blk delayacct --
> > and I doubt that was the intent; even though the current kernel works
> > that way.
> >
> > I'll move the now rudimentary io_schedule() into sched.h as an inline.
>
> Thanks.
>
> Are you OK with this going to -stable for 3.19?
OK, lets do that.
^ permalink raw reply
* help with the little script (erc timout fix)
From: Chris @ 2015-02-18 15:04 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20150217104906.62d36c62@notabene.brown>
Hello,
by adapting what I could find, I compiled the following short snippet now.
Could list members please look at this novice code and suggest a way to
determine the containing disk device $HDD_DEV from the parition/disk,
before I dare to test this.
In udev-md-raid-assembly.rules, below LABEL="md_inc" (section only handling
all md suppported devices) add:
# fix timouts for redundant raids, if possible
IMPORT{program}="BINDIR/mdadm --examine --export $tempnode"
TEST="/usr/sbin/smartctl", ENV{MD_LEVEL}=="raid[1-9]*",
RUN+="BINDIR/mdadm-erc-timout-fix.sh $tempnode"
And in a new mdadm-erc-timout-fix.sh file implement:
#! /bin/sh
HDD_DEV= $1 somehow stipping off the tailing numbers?
if smartctl -l scterc ${HDD_DEV} | grep -q Disabled ; then
/usr/sbin/smartctl -l scterc,70,70 ${HDD_DEV}
else
if ! smartctl -l scterc ${HDD_DEV} | grep -q seconds ; then
echo 180 >/sys/block/${HDD_DEV}/device/timeout
fi
fi
Correct execution during boot would seem to require that distro
package managers hook smartctl and the script into the initramfs
generation.
Regards,
Chris
^ permalink raw reply
* Potential Bug in "echo 0 > /dev/md0"
From: Alireza Haghdoost @ 2015-02-18 16:11 UTC (permalink / raw)
To: Linux RAID; +Cc: Neil Brown
I understand this is not the right way to talk with md device but my
understanding is that if some one run this command by mistake (or
vandalism) , it should not results a kernel crash:
[root] [ /home/arh ]
# echo 0 > /dev/md0
[root] [ /home/arh ]
# dmesg
[1463111.320277] BUG: soft lockup - CPU#4 stuck for 22s! [whoopsie:1829]
[1463111.320284] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
pps_core
[1463111.320326] CPU: 4 PID: 1829 Comm: whoopsie Tainted: G D W
3.13.0Write-Hole-Monitor #47
[1463111.320328] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
2.1.3 05/21/2014
[1463111.320329] task: ffff8801a425c7d0 ti: ffff8801a3e04000 task.ti:
ffff8801a3e04000
[1463111.320331] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
smp_call_function_single+0xc6/0x190
[1463111.320335] RSP: 0000:ffff8801a3e05a60 EFLAGS: 00000202
[1463111.320336] RAX: 0000000000000001 RBX: ffffffff813624f4 RCX:
0000000000000000
[1463111.320338] RDX: ffff8801a3e05ad8 RSI: ffff8801a9a54e80 RDI:
0000000000000001
[1463111.320339] RBP: ffff8801a3e05ac8 R08: ffff8801a3e05b70 R09:
0000000000000000
[1463111.320341] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff88031e6bec80
[1463111.320342] R13: ffff880035d70700 R14: ffff8801a9a54400 R15:
ffff88031e6bec80
[1463111.320344] FS: 00007f9ae4924840(0000) GS:ffff8801a9a40000(0000)
knlGS:0000000000000000
[1463111.320346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1463111.320347] CR2: 00007f9ae43bc280 CR3: 00000001a3adc000 CR4:
00000000000407e0
[1463111.320349] Stack:
[1463111.320350] ffff8801a3e05af8 ffffffff8178f5ae 0000000000000000
0000000000000000
[1463111.320353] 0000000000000000 0000000000000000 0000000000000000
ffff8801a9a54480
[1463111.320357] 0000000000000002 0000000000000001 ffff8801a3e05b70
0000000000000004
[1463111.320360] Call Trace:
[1463111.320364] [<ffffffff8178f5ae>] ?
schedule_hrtimeout_range_clock+0xce/0x170
[1463111.320367] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
[1463111.320369] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
[1463111.320372] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
[1463111.320375] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
[1463111.320378] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
[1463111.320381] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
[1463111.320384] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
[1463111.320386] [<ffffffff8109f8ef>] task_numa_fault+0x26f/0x890
[1463111.320390] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
[1463111.320394] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
[1463111.320396] [<ffffffff8117bc30>] ? change_protection+0x690/0x720
[1463111.320399] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
[1463111.320403] [<ffffffff81191b0b>] ? change_prot_numa+0x1b/0x40
[1463111.320405] [<ffffffff8109c806>] ? task_numa_work+0x266/0x300
[1463111.320408] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
[1463111.320412] [<ffffffff81012e67>] ? do_notify_resume+0x97/0xb0
[1463111.320414] [<ffffffff81794348>] page_fault+0x28/0x30
[1463111.320415] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
89 76
[1463111.344253] BUG: soft lockup - CPU#6 stuck for 22s! [Xorg:1842]
[1463111.344254] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
pps_core
[1463111.344298] CPU: 6 PID: 1842 Comm: Xorg Tainted: G D W
3.13.0Write-Hole-Monitor #47
[1463111.344299] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
2.1.3 05/21/2014
[1463111.344302] task: ffff8800367b8000 ti: ffff880035f9e000 task.ti:
ffff880035f9e000
[1463111.344303] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
smp_call_function_single+0xc6/0x190
[1463111.344306] RSP: 0000:ffff880035f9fa60 EFLAGS: 00003202
[1463111.344308] RAX: 0000000000000001 RBX: 0000000000400000 RCX:
0000000000000000
[1463111.344309] RDX: ffff880035f9fad8 RSI: ffff8801a9a74e80 RDI:
0000000000000003
[1463111.344311] RBP: ffff880035f9fac8 R08: ffff880035f9fb70 R09:
0000000000000000
[1463111.344312] R10: 0000000000000000 R11: 0000000000000000 R12:
00000000003fc02a
[1463111.344313] R13: ffffffff811ca5fc R14: ffff880035f9f9f0 R15:
ffffffff81150fa3
[1463111.344315] FS: 00007f2f338019c0(0000) GS:ffff8801a9a60000(0000)
knlGS:0000000000000000
[1463111.344317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1463111.344318] CR2: 00007f2f344fdedc CR3: 000000031dd2b000 CR4:
00000000000407e0
[1463111.344320] Stack:
[1463111.344321] ffff880035f9fdf8 ffff880035f9fdf0 0000000000000000
0000000000000000
[1463111.344324] 0000000000000000 0000000000000000 0000000000000000
ffff8801a9a74480
[1463111.344327] 0000000000000002 0000000000000003 ffff880035f9fb70
0000000000000006
[1463111.344330] Call Trace:
[1463111.344334] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
[1463111.344336] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
[1463111.344339] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
[1463111.344342] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
[1463111.344345] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
[1463111.344347] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
[1463111.344351] [<ffffffff811ca500>] ? poll_select_copy_remaining+0x130/0x130
[1463111.344354] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
[1463111.344356] [<ffffffff8109fd00>] task_numa_fault+0x680/0x890
[1463111.344360] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
[1463111.344363] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
[1463111.344366] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
[1463111.344368] [<ffffffff811b7579>] ? do_readv_writev+0x169/0x220
[1463111.344371] [<ffffffff813624f4>] ? timerqueue_del+0x24/0x70
[1463111.344374] [<ffffffff8108baa6>] ? __remove_hrtimer+0x46/0xa0
[1463111.344377] [<ffffffff8108bec8>] ? hrtimer_try_to_cancel+0x48/0xe0
[1463111.344380] [<ffffffff81068b53>] ? do_setitimer+0xe3/0x2a0
[1463111.344382] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
[1463111.344385] [<ffffffff81794348>] page_fault+0x28/0x30
[1463111.344386] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
89 76
[root] [ /home/arh ]
^ permalink raw reply
* Re: Potential Bug in "echo 0 > /dev/md0"
From: Jes Sorensen @ 2015-02-18 16:19 UTC (permalink / raw)
To: Alireza Haghdoost; +Cc: Linux RAID, Neil Brown
In-Reply-To: <CAB-428mj-P-+M0MnKhmojRe9ZQQzsSTBfCSS91ZK5KRy5p1mSg@mail.gmail.com>
Alireza Haghdoost <alireza@cs.umn.edu> writes:
> I understand this is not the right way to talk with md device but my
> understanding is that if some one run this command by mistake (or
> vandalism) , it should not results a kernel crash:
>
> [root] [ /home/arh ]
> # echo 0 > /dev/md0
It shouldn't, however before anyone can debug this, you need to provide
a proper bug report with information about the kernel version you are
running, the configuration of /dev/md0 etc.
Please include 'cat /proc/mdstat' output with it.
JEs
>
> [root] [ /home/arh ]
> # dmesg
> [1463111.320277] BUG: soft lockup - CPU#4 stuck for 22s! [whoopsie:1829]
> [1463111.320284] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
> pps_core
> [1463111.320326] CPU: 4 PID: 1829 Comm: whoopsie Tainted: G D W
> 3.13.0Write-Hole-Monitor #47
> [1463111.320328] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
> 2.1.3 05/21/2014
> [1463111.320329] task: ffff8801a425c7d0 ti: ffff8801a3e04000 task.ti:
> ffff8801a3e04000
> [1463111.320331] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
> smp_call_function_single+0xc6/0x190
> [1463111.320335] RSP: 0000:ffff8801a3e05a60 EFLAGS: 00000202
> [1463111.320336] RAX: 0000000000000001 RBX: ffffffff813624f4 RCX:
> 0000000000000000
> [1463111.320338] RDX: ffff8801a3e05ad8 RSI: ffff8801a9a54e80 RDI:
> 0000000000000001
> [1463111.320339] RBP: ffff8801a3e05ac8 R08: ffff8801a3e05b70 R09:
> 0000000000000000
> [1463111.320341] R10: 0000000000000000 R11: 0000000000000000 R12:
> ffff88031e6bec80
> [1463111.320342] R13: ffff880035d70700 R14: ffff8801a9a54400 R15:
> ffff88031e6bec80
> [1463111.320344] FS: 00007f9ae4924840(0000) GS:ffff8801a9a40000(0000)
> knlGS:0000000000000000
> [1463111.320346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [1463111.320347] CR2: 00007f9ae43bc280 CR3: 00000001a3adc000 CR4:
> 00000000000407e0
> [1463111.320349] Stack:
> [1463111.320350] ffff8801a3e05af8 ffffffff8178f5ae 0000000000000000
> 0000000000000000
> [1463111.320353] 0000000000000000 0000000000000000 0000000000000000
> ffff8801a9a54480
> [1463111.320357] 0000000000000002 0000000000000001 ffff8801a3e05b70
> 0000000000000004
> [1463111.320360] Call Trace:
> [1463111.320364] [<ffffffff8178f5ae>] ?
> schedule_hrtimeout_range_clock+0xce/0x170
> [1463111.320367] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
> [1463111.320369] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.320372] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.320375] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
> [1463111.320378] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
> [1463111.320381] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
> [1463111.320384] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
> [1463111.320386] [<ffffffff8109f8ef>] task_numa_fault+0x26f/0x890
> [1463111.320390] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
> [1463111.320394] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
> [1463111.320396] [<ffffffff8117bc30>] ? change_protection+0x690/0x720
> [1463111.320399] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
> [1463111.320403] [<ffffffff81191b0b>] ? change_prot_numa+0x1b/0x40
> [1463111.320405] [<ffffffff8109c806>] ? task_numa_work+0x266/0x300
> [1463111.320408] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
> [1463111.320412] [<ffffffff81012e67>] ? do_notify_resume+0x97/0xb0
> [1463111.320414] [<ffffffff81794348>] page_fault+0x28/0x30
> [1463111.320415] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
> 89 76
> [1463111.344253] BUG: soft lockup - CPU#6 stuck for 22s! [Xorg:1842]
> [1463111.344254] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
> pps_core
> [1463111.344298] CPU: 6 PID: 1842 Comm: Xorg Tainted: G D W
> 3.13.0Write-Hole-Monitor #47
> [1463111.344299] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
> 2.1.3 05/21/2014
> [1463111.344302] task: ffff8800367b8000 ti: ffff880035f9e000 task.ti:
> ffff880035f9e000
> [1463111.344303] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
> smp_call_function_single+0xc6/0x190
> [1463111.344306] RSP: 0000:ffff880035f9fa60 EFLAGS: 00003202
> [1463111.344308] RAX: 0000000000000001 RBX: 0000000000400000 RCX:
> 0000000000000000
> [1463111.344309] RDX: ffff880035f9fad8 RSI: ffff8801a9a74e80 RDI:
> 0000000000000003
> [1463111.344311] RBP: ffff880035f9fac8 R08: ffff880035f9fb70 R09:
> 0000000000000000
> [1463111.344312] R10: 0000000000000000 R11: 0000000000000000 R12:
> 00000000003fc02a
> [1463111.344313] R13: ffffffff811ca5fc R14: ffff880035f9f9f0 R15:
> ffffffff81150fa3
> [1463111.344315] FS: 00007f2f338019c0(0000) GS:ffff8801a9a60000(0000)
> knlGS:0000000000000000
> [1463111.344317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [1463111.344318] CR2: 00007f2f344fdedc CR3: 000000031dd2b000 CR4:
> 00000000000407e0
> [1463111.344320] Stack:
> [1463111.344321] ffff880035f9fdf8 ffff880035f9fdf0 0000000000000000
> 0000000000000000
> [1463111.344324] 0000000000000000 0000000000000000 0000000000000000
> ffff8801a9a74480
> [1463111.344327] 0000000000000002 0000000000000003 ffff880035f9fb70
> 0000000000000006
> [1463111.344330] Call Trace:
> [1463111.344334] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
> [1463111.344336] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.344339] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.344342] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
> [1463111.344345] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
> [1463111.344347] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
> [1463111.344351] [<ffffffff811ca500>] ? poll_select_copy_remaining+0x130/0x130
> [1463111.344354] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
> [1463111.344356] [<ffffffff8109fd00>] task_numa_fault+0x680/0x890
> [1463111.344360] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
> [1463111.344363] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
> [1463111.344366] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
> [1463111.344368] [<ffffffff811b7579>] ? do_readv_writev+0x169/0x220
> [1463111.344371] [<ffffffff813624f4>] ? timerqueue_del+0x24/0x70
> [1463111.344374] [<ffffffff8108baa6>] ? __remove_hrtimer+0x46/0xa0
> [1463111.344377] [<ffffffff8108bec8>] ? hrtimer_try_to_cancel+0x48/0xe0
> [1463111.344380] [<ffffffff81068b53>] ? do_setitimer+0xe3/0x2a0
> [1463111.344382] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
> [1463111.344385] [<ffffffff81794348>] page_fault+0x28/0x30
> [1463111.344386] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
> 89 76
> [root] [ /home/arh ]
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Potential Bug in "echo 0 > /dev/md0"
From: Alireza Haghdoost @ 2015-02-18 16:25 UTC (permalink / raw)
To: Jes Sorensen; +Cc: Linux RAID, Neil Brown
In-Reply-To: <wrfjtwyjb3b1.fsf@redhat.com>
Here you are:
[root] [ /home/arh ]
# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5]
[raid4] [multipath] [faulty]
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
314374656 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
[root] [ /home/arh ]
# uname -a
Linux mist01-umh 3.13.0 #47 SMP Sun Feb 1 10:27:24 CST 2015 x86_64
x86_64 x86_64 GNU/Linux
[root] [ /home/arh ]
# cat /etc/issue
Ubuntu 14.04.1 LTS \n \l
[root] [ /home/arh ]
# mdadm --version
mdadm - v3.2.5 - 18th May 2012
[root] [ /home/arh ]
# mdadm --detail /dev/md0
<No output, Freezes since kernel is crashed !>
On Wed, Feb 18, 2015 at 10:19 AM, Jes Sorensen <Jes.Sorensen@redhat.com> wrote:
> Alireza Haghdoost <alireza@cs.umn.edu> writes:
>> I understand this is not the right way to talk with md device but my
>> understanding is that if some one run this command by mistake (or
>> vandalism) , it should not results a kernel crash:
>>
>> [root] [ /home/arh ]
>> # echo 0 > /dev/md0
>
> It shouldn't, however before anyone can debug this, you need to provide
> a proper bug report with information about the kernel version you are
> running, the configuration of /dev/md0 etc.
>
> Please include 'cat /proc/mdstat' output with it.
>
> JEs
>
>>
>> [root] [ /home/arh ]
>> # dmesg
>> [1463111.320277] BUG: soft lockup - CPU#4 stuck for 22s! [whoopsie:1829]
>> [1463111.320284] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
>> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
>> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
>> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
>> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
>> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
>> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
>> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
>> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
>> pps_core
>> [1463111.320326] CPU: 4 PID: 1829 Comm: whoopsie Tainted: G D W
>> 3.13.0Write-Hole-Monitor #47
>> [1463111.320328] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
>> 2.1.3 05/21/2014
>> [1463111.320329] task: ffff8801a425c7d0 ti: ffff8801a3e04000 task.ti:
>> ffff8801a3e04000
>> [1463111.320331] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
>> smp_call_function_single+0xc6/0x190
>> [1463111.320335] RSP: 0000:ffff8801a3e05a60 EFLAGS: 00000202
>> [1463111.320336] RAX: 0000000000000001 RBX: ffffffff813624f4 RCX:
>> 0000000000000000
>> [1463111.320338] RDX: ffff8801a3e05ad8 RSI: ffff8801a9a54e80 RDI:
>> 0000000000000001
>> [1463111.320339] RBP: ffff8801a3e05ac8 R08: ffff8801a3e05b70 R09:
>> 0000000000000000
>> [1463111.320341] R10: 0000000000000000 R11: 0000000000000000 R12:
>> ffff88031e6bec80
>> [1463111.320342] R13: ffff880035d70700 R14: ffff8801a9a54400 R15:
>> ffff88031e6bec80
>> [1463111.320344] FS: 00007f9ae4924840(0000) GS:ffff8801a9a40000(0000)
>> knlGS:0000000000000000
>> [1463111.320346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [1463111.320347] CR2: 00007f9ae43bc280 CR3: 00000001a3adc000 CR4:
>> 00000000000407e0
>> [1463111.320349] Stack:
>> [1463111.320350] ffff8801a3e05af8 ffffffff8178f5ae 0000000000000000
>> 0000000000000000
>> [1463111.320353] 0000000000000000 0000000000000000 0000000000000000
>> ffff8801a9a54480
>> [1463111.320357] 0000000000000002 0000000000000001 ffff8801a3e05b70
>> 0000000000000004
>> [1463111.320360] Call Trace:
>> [1463111.320364] [<ffffffff8178f5ae>] ?
>> schedule_hrtimeout_range_clock+0xce/0x170
>> [1463111.320367] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
>> [1463111.320369] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>> [1463111.320372] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>> [1463111.320375] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
>> [1463111.320378] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
>> [1463111.320381] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
>> [1463111.320384] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
>> [1463111.320386] [<ffffffff8109f8ef>] task_numa_fault+0x26f/0x890
>> [1463111.320390] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
>> [1463111.320394] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
>> [1463111.320396] [<ffffffff8117bc30>] ? change_protection+0x690/0x720
>> [1463111.320399] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
>> [1463111.320403] [<ffffffff81191b0b>] ? change_prot_numa+0x1b/0x40
>> [1463111.320405] [<ffffffff8109c806>] ? task_numa_work+0x266/0x300
>> [1463111.320408] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
>> [1463111.320412] [<ffffffff81012e67>] ? do_notify_resume+0x97/0xb0
>> [1463111.320414] [<ffffffff81794348>] page_fault+0x28/0x30
>> [1463111.320415] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
>> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
>> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
>> 89 76
>> [1463111.344253] BUG: soft lockup - CPU#6 stuck for 22s! [Xorg:1842]
>> [1463111.344254] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
>> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
>> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
>> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
>> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
>> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
>> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
>> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
>> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
>> pps_core
>> [1463111.344298] CPU: 6 PID: 1842 Comm: Xorg Tainted: G D W
>> 3.13.0Write-Hole-Monitor #47
>> [1463111.344299] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
>> 2.1.3 05/21/2014
>> [1463111.344302] task: ffff8800367b8000 ti: ffff880035f9e000 task.ti:
>> ffff880035f9e000
>> [1463111.344303] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
>> smp_call_function_single+0xc6/0x190
>> [1463111.344306] RSP: 0000:ffff880035f9fa60 EFLAGS: 00003202
>> [1463111.344308] RAX: 0000000000000001 RBX: 0000000000400000 RCX:
>> 0000000000000000
>> [1463111.344309] RDX: ffff880035f9fad8 RSI: ffff8801a9a74e80 RDI:
>> 0000000000000003
>> [1463111.344311] RBP: ffff880035f9fac8 R08: ffff880035f9fb70 R09:
>> 0000000000000000
>> [1463111.344312] R10: 0000000000000000 R11: 0000000000000000 R12:
>> 00000000003fc02a
>> [1463111.344313] R13: ffffffff811ca5fc R14: ffff880035f9f9f0 R15:
>> ffffffff81150fa3
>> [1463111.344315] FS: 00007f2f338019c0(0000) GS:ffff8801a9a60000(0000)
>> knlGS:0000000000000000
>> [1463111.344317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [1463111.344318] CR2: 00007f2f344fdedc CR3: 000000031dd2b000 CR4:
>> 00000000000407e0
>> [1463111.344320] Stack:
>> [1463111.344321] ffff880035f9fdf8 ffff880035f9fdf0 0000000000000000
>> 0000000000000000
>> [1463111.344324] 0000000000000000 0000000000000000 0000000000000000
>> ffff8801a9a74480
>> [1463111.344327] 0000000000000002 0000000000000003 ffff880035f9fb70
>> 0000000000000006
>> [1463111.344330] Call Trace:
>> [1463111.344334] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
>> [1463111.344336] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>> [1463111.344339] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>> [1463111.344342] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
>> [1463111.344345] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
>> [1463111.344347] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
>> [1463111.344351] [<ffffffff811ca500>] ? poll_select_copy_remaining+0x130/0x130
>> [1463111.344354] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
>> [1463111.344356] [<ffffffff8109fd00>] task_numa_fault+0x680/0x890
>> [1463111.344360] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
>> [1463111.344363] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
>> [1463111.344366] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
>> [1463111.344368] [<ffffffff811b7579>] ? do_readv_writev+0x169/0x220
>> [1463111.344371] [<ffffffff813624f4>] ? timerqueue_del+0x24/0x70
>> [1463111.344374] [<ffffffff8108baa6>] ? __remove_hrtimer+0x46/0xa0
>> [1463111.344377] [<ffffffff8108bec8>] ? hrtimer_try_to_cancel+0x48/0xe0
>> [1463111.344380] [<ffffffff81068b53>] ? do_setitimer+0xe3/0x2a0
>> [1463111.344382] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
>> [1463111.344385] [<ffffffff81794348>] page_fault+0x28/0x30
>> [1463111.344386] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
>> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
>> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
>> 89 76
>> [root] [ /home/arh ]
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 0/1] RFC: Use /dev/md/X as default name
From: Jes Sorensen @ 2015-02-18 18:12 UTC (permalink / raw)
To: neilb; +Cc: linux-raid
In-Reply-To: <1423684389-10977-1-git-send-email-Jes.Sorensen@redhat.com>
Jes.Sorensen@redhat.com writes:
> From: Jes Sorensen <Jes.Sorensen@redhat.com>
>
> I have received some issues for when creating an array using a
> /dev/mdX name, the matching symlink in /dev/md/X isn't
> created. Whereas if you create /dev/md/X, /dev/mdX is created
> automatically.
>
> I was trying to see if there was a better way of dealing with this,
> but I couldn't find one. If you have suggestions for a better solution
> I am all ears.
>
> Thoughts?
Hi Neil,
Any thoughts on this one?
Cheers,
Jes
>
> Jes
>
> Jes Sorensen (1):
> mddev_create(): choose /dev/md/ name over /dev/mdX
>
> mdopen.c | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
^ permalink raw reply
* (unknown),
From: DeadManMoving @ 2015-02-18 19:42 UTC (permalink / raw)
To: linux-raid
unsubscribe linux-raid
^ permalink raw reply
* Re: Potential Bug in "echo 0 > /dev/md0"
From: Jes Sorensen @ 2015-02-18 20:20 UTC (permalink / raw)
To: Alireza Haghdoost; +Cc: Linux RAID, Neil Brown
In-Reply-To: <CAB-428k=MXBZRbwTp6h=KfZQ_o4vP5TyfPqutsmoFW6to2Ft9A@mail.gmail.com>
Alireza Haghdoost <alireza@cs.umn.edu> writes:
> Here you are:
>
> [root] [ /home/arh ]
> # cat /proc/mdstat
> Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5]
> [raid4] [multipath] [faulty]
> md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
> 314374656 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
>
>
> [root] [ /home/arh ]
> # uname -a
> Linux mist01-umh 3.13.0 #47 SMP Sun Feb 1 10:27:24 CST 2015 x86_64
> x86_64 x86_64 GNU/Linux
>
> [root] [ /home/arh ]
> # cat /etc/issue
> Ubuntu 14.04.1 LTS \n \l
I am unfamiliar with Ubuntu's kernels, is this a distro kernel or a self
compiled kernel? If this is a distro kernel, please start out by
reporting the bug to your distribution through their bug tracking
system.
3.13 is very old, so it would be help if you tried against a recent
kernel.
Jes
>
> [root] [ /home/arh ]
> # mdadm --version
> mdadm - v3.2.5 - 18th May 2012
>
> [root] [ /home/arh ]
> # mdadm --detail /dev/md0
> <No output, Freezes since kernel is crashed !>
>
> On Wed, Feb 18, 2015 at 10:19 AM, Jes Sorensen <Jes.Sorensen@redhat.com> wrote:
>> Alireza Haghdoost <alireza@cs.umn.edu> writes:
>>> I understand this is not the right way to talk with md device but my
>>> understanding is that if some one run this command by mistake (or
>>> vandalism) , it should not results a kernel crash:
>>>
>>> [root] [ /home/arh ]
>>> # echo 0 > /dev/md0
>>
>> It shouldn't, however before anyone can debug this, you need to provide
>> a proper bug report with information about the kernel version you are
>> running, the configuration of /dev/md0 etc.
>>
>> Please include 'cat /proc/mdstat' output with it.
>>
>> JEs
>>
>>>
>>> [root] [ /home/arh ]
>>> # dmesg
>>> [1463111.320277] BUG: soft lockup - CPU#4 stuck for 22s! [whoopsie:1829]
>>> [1463111.320284] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
>>> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
>>> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
>>> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
>>> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
>>> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
>>> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
>>> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
>>> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
>>> pps_core
>>> [1463111.320326] CPU: 4 PID: 1829 Comm: whoopsie Tainted: G D W
>>> 3.13.0Write-Hole-Monitor #47
>>> [1463111.320328] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
>>> 2.1.3 05/21/2014
>>> [1463111.320329] task: ffff8801a425c7d0 ti: ffff8801a3e04000 task.ti:
>>> ffff8801a3e04000
>>> [1463111.320331] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
>>> smp_call_function_single+0xc6/0x190
>>> [1463111.320335] RSP: 0000:ffff8801a3e05a60 EFLAGS: 00000202
>>> [1463111.320336] RAX: 0000000000000001 RBX: ffffffff813624f4 RCX:
>>> 0000000000000000
>>> [1463111.320338] RDX: ffff8801a3e05ad8 RSI: ffff8801a9a54e80 RDI:
>>> 0000000000000001
>>> [1463111.320339] RBP: ffff8801a3e05ac8 R08: ffff8801a3e05b70 R09:
>>> 0000000000000000
>>> [1463111.320341] R10: 0000000000000000 R11: 0000000000000000 R12:
>>> ffff88031e6bec80
>>> [1463111.320342] R13: ffff880035d70700 R14: ffff8801a9a54400 R15:
>>> ffff88031e6bec80
>>> [1463111.320344] FS: 00007f9ae4924840(0000) GS:ffff8801a9a40000(0000)
>>> knlGS:0000000000000000
>>> [1463111.320346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [1463111.320347] CR2: 00007f9ae43bc280 CR3: 00000001a3adc000 CR4:
>>> 00000000000407e0
>>> [1463111.320349] Stack:
>>> [1463111.320350] ffff8801a3e05af8 ffffffff8178f5ae 0000000000000000
>>> 0000000000000000
>>> [1463111.320353] 0000000000000000 0000000000000000 0000000000000000
>>> ffff8801a9a54480
>>> [1463111.320357] 0000000000000002 0000000000000001 ffff8801a3e05b70
>>> 0000000000000004
>>> [1463111.320360] Call Trace:
>>> [1463111.320364] [<ffffffff8178f5ae>] ?
>>> schedule_hrtimeout_range_clock+0xce/0x170
>>> [1463111.320367] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
>>> [1463111.320369] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>>> [1463111.320372] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>>> [1463111.320375] [<ffffffff81095e80>] ?
>>> __migrate_swap_task.part.68+0x80/0x80
>>> [1463111.320378] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
>>> [1463111.320381] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
>>> [1463111.320384] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
>>> [1463111.320386] [<ffffffff8109f8ef>] task_numa_fault+0x26f/0x890
>>> [1463111.320390] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
>>> [1463111.320394] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
>>> [1463111.320396] [<ffffffff8117bc30>] ? change_protection+0x690/0x720
>>> [1463111.320399] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
>>> [1463111.320403] [<ffffffff81191b0b>] ? change_prot_numa+0x1b/0x40
>>> [1463111.320405] [<ffffffff8109c806>] ? task_numa_work+0x266/0x300
>>> [1463111.320408] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
>>> [1463111.320412] [<ffffffff81012e67>] ? do_notify_resume+0x97/0xb0
>>> [1463111.320414] [<ffffffff81794348>] page_fault+0x28/0x30
>>> [1463111.320415] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
>>> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
>>> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
>>> 89 76
>>> [1463111.344253] BUG: soft lockup - CPU#6 stuck for 22s! [Xorg:1842]
>>> [1463111.344254] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
>>> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
>>> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
>>> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
>>> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
>>> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
>>> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
>>> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
>>> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
>>> pps_core
>>> [1463111.344298] CPU: 6 PID: 1842 Comm: Xorg Tainted: G D W
>>> 3.13.0Write-Hole-Monitor #47
>>> [1463111.344299] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
>>> 2.1.3 05/21/2014
>>> [1463111.344302] task: ffff8800367b8000 ti: ffff880035f9e000 task.ti:
>>> ffff880035f9e000
>>> [1463111.344303] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
>>> smp_call_function_single+0xc6/0x190
>>> [1463111.344306] RSP: 0000:ffff880035f9fa60 EFLAGS: 00003202
>>> [1463111.344308] RAX: 0000000000000001 RBX: 0000000000400000 RCX:
>>> 0000000000000000
>>> [1463111.344309] RDX: ffff880035f9fad8 RSI: ffff8801a9a74e80 RDI:
>>> 0000000000000003
>>> [1463111.344311] RBP: ffff880035f9fac8 R08: ffff880035f9fb70 R09:
>>> 0000000000000000
>>> [1463111.344312] R10: 0000000000000000 R11: 0000000000000000 R12:
>>> 00000000003fc02a
>>> [1463111.344313] R13: ffffffff811ca5fc R14: ffff880035f9f9f0 R15:
>>> ffffffff81150fa3
>>> [1463111.344315] FS: 00007f2f338019c0(0000) GS:ffff8801a9a60000(0000)
>>> knlGS:0000000000000000
>>> [1463111.344317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [1463111.344318] CR2: 00007f2f344fdedc CR3: 000000031dd2b000 CR4:
>>> 00000000000407e0
>>> [1463111.344320] Stack:
>>> [1463111.344321] ffff880035f9fdf8 ffff880035f9fdf0 0000000000000000
>>> 0000000000000000
>>> [1463111.344324] 0000000000000000 0000000000000000 0000000000000000
>>> ffff8801a9a74480
>>> [1463111.344327] 0000000000000002 0000000000000003 ffff880035f9fb70
>>> 0000000000000006
>>> [1463111.344330] Call Trace:
>>> [1463111.344334] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
>>> [1463111.344336] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>>> [1463111.344339] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>>> [1463111.344342] [<ffffffff81095e80>] ?
>>> __migrate_swap_task.part.68+0x80/0x80
>>> [1463111.344345] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
>>> [1463111.344347] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
>>> [1463111.344351] [<ffffffff811ca500>] ?
>>> poll_select_copy_remaining+0x130/0x130
>>> [1463111.344354] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
>>> [1463111.344356] [<ffffffff8109fd00>] task_numa_fault+0x680/0x890
>>> [1463111.344360] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
>>> [1463111.344363] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
>>> [1463111.344366] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
>>> [1463111.344368] [<ffffffff811b7579>] ? do_readv_writev+0x169/0x220
>>> [1463111.344371] [<ffffffff813624f4>] ? timerqueue_del+0x24/0x70
>>> [1463111.344374] [<ffffffff8108baa6>] ? __remove_hrtimer+0x46/0xa0
>>> [1463111.344377] [<ffffffff8108bec8>] ? hrtimer_try_to_cancel+0x48/0xe0
>>> [1463111.344380] [<ffffffff81068b53>] ? do_setitimer+0xe3/0x2a0
>>> [1463111.344382] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
>>> [1463111.344385] [<ffffffff81794348>] page_fault+0x28/0x30
>>> [1463111.344386] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
>>> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
>>> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
>>> 89 76
>>> [root] [ /home/arh ]
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: help with the little script (erc timout fix)
From: NeilBrown @ 2015-02-18 21:25 UTC (permalink / raw)
To: Chris; +Cc: linux-raid
In-Reply-To: <loom.20150218T155053-576@post.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 2459 bytes --]
On Wed, 18 Feb 2015 15:04:53 +0000 (UTC) Chris <email.bug@arcor.de> wrote:
>
> Hello,
>
> by adapting what I could find, I compiled the following short snippet now.
>
> Could list members please look at this novice code and suggest a way to
> determine the containing disk device $HDD_DEV from the parition/disk,
> before I dare to test this.
>
>
>
> In udev-md-raid-assembly.rules, below LABEL="md_inc" (section only handling
> all md suppported devices) add:
>
> # fix timouts for redundant raids, if possible
> IMPORT{program}="BINDIR/mdadm --examine --export $tempnode"
> TEST="/usr/sbin/smartctl", ENV{MD_LEVEL}=="raid[1-9]*",
> RUN+="BINDIR/mdadm-erc-timout-fix.sh $tempnode"
It might make sense to have 2 rules, one for partitions and one for disks
(based on ENV{DEVTYPE}). Then use $parent to get the device from the
partition, and $devnode to get the device of the disk.
>
> And in a new mdadm-erc-timout-fix.sh file implement:
>
> #! /bin/sh
>
> HDD_DEV= $1 somehow stipping off the tailing numbers?
>
> if smartctl -l scterc ${HDD_DEV} | grep -q Disabled ; then
> /usr/sbin/smartctl -l scterc,70,70 ${HDD_DEV}
> else
> if ! smartctl -l scterc ${HDD_DEV} | grep -q seconds ; then
> echo 180 >/sys/block/${HDD_DEV}/device/timeout
> fi
> fi
You should be consistent and use /usr/sbin/smartctl everywhere, or explicitly
set $PATH and just use smartctl everywhere.
>
> Correct execution during boot would seem to require that distro
> package managers hook smartctl and the script into the initramfs
> generation.
>
> Regards,
> Chris
One problem with this approach is that it assumes circumstances don't change.
If you have a working RAID1, then limiting the timeout on both devices makes
sense. If you have a degraded RAID1 with only one device left then you
really want the drive to try as hard as it can to get the data.
There is a "FAILFAST" mechanism in the kernel which allows the filesystem to
md etc to indicate that it wants accesses to "fail fast", which presumably
means to use a smaller timeout.
I would rather md used this flag where appropriate, and for the device to
respond to it by using suitable timeouts.
The problem is that FAILFAST isn't documented usefully and it is very hard to
figure out what exactly (if anything) it does.
But until that is resolved, a fix like this is probably a good idea.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Potential Bug in "echo 0 > /dev/md0"
From: NeilBrown @ 2015-02-18 21:32 UTC (permalink / raw)
To: Alireza Haghdoost; +Cc: Linux RAID
In-Reply-To: <CAB-428mj-P-+M0MnKhmojRe9ZQQzsSTBfCSS91ZK5KRy5p1mSg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 8509 bytes --]
On Wed, 18 Feb 2015 10:11:20 -0600 Alireza Haghdoost <alireza@cs.umn.edu>
wrote:
> I understand this is not the right way to talk with md device but my
> understanding is that if some one run this command by mistake (or
> vandalism) , it should not results a kernel crash:
>
> [root] [ /home/arh ]
> # echo 0 > /dev/md0
>
> [root] [ /home/arh ]
> # dmesg
> [1463111.320277] BUG: soft lockup - CPU#4 stuck for 22s! [whoopsie:1829]
> [1463111.320284] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
> pps_core
> [1463111.320326] CPU: 4 PID: 1829 Comm: whoopsie Tainted: G D W
^^^^^^^^
What is "whoopsie" ???
> 3.13.0Write-Hole-Monitor #47
^^^^^^^^^^^^^^^^^^^
What is "Write-Hole-Monitor".
There is no evidence that this is related to RAID, except that it presumably
happens at about the same time that you write to /dev/md0.
It certainly isn't running and md/raid code when it reports a soft-lockup.
NeilBrown
> [1463111.320328] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
> 2.1.3 05/21/2014
> [1463111.320329] task: ffff8801a425c7d0 ti: ffff8801a3e04000 task.ti:
> ffff8801a3e04000
> [1463111.320331] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
> smp_call_function_single+0xc6/0x190
> [1463111.320335] RSP: 0000:ffff8801a3e05a60 EFLAGS: 00000202
> [1463111.320336] RAX: 0000000000000001 RBX: ffffffff813624f4 RCX:
> 0000000000000000
> [1463111.320338] RDX: ffff8801a3e05ad8 RSI: ffff8801a9a54e80 RDI:
> 0000000000000001
> [1463111.320339] RBP: ffff8801a3e05ac8 R08: ffff8801a3e05b70 R09:
> 0000000000000000
> [1463111.320341] R10: 0000000000000000 R11: 0000000000000000 R12:
> ffff88031e6bec80
> [1463111.320342] R13: ffff880035d70700 R14: ffff8801a9a54400 R15:
> ffff88031e6bec80
> [1463111.320344] FS: 00007f9ae4924840(0000) GS:ffff8801a9a40000(0000)
> knlGS:0000000000000000
> [1463111.320346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [1463111.320347] CR2: 00007f9ae43bc280 CR3: 00000001a3adc000 CR4:
> 00000000000407e0
> [1463111.320349] Stack:
> [1463111.320350] ffff8801a3e05af8 ffffffff8178f5ae 0000000000000000
> 0000000000000000
> [1463111.320353] 0000000000000000 0000000000000000 0000000000000000
> ffff8801a9a54480
> [1463111.320357] 0000000000000002 0000000000000001 ffff8801a3e05b70
> 0000000000000004
> [1463111.320360] Call Trace:
> [1463111.320364] [<ffffffff8178f5ae>] ?
> schedule_hrtimeout_range_clock+0xce/0x170
> [1463111.320367] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
> [1463111.320369] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.320372] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.320375] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
> [1463111.320378] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
> [1463111.320381] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
> [1463111.320384] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
> [1463111.320386] [<ffffffff8109f8ef>] task_numa_fault+0x26f/0x890
> [1463111.320390] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
> [1463111.320394] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
> [1463111.320396] [<ffffffff8117bc30>] ? change_protection+0x690/0x720
> [1463111.320399] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
> [1463111.320403] [<ffffffff81191b0b>] ? change_prot_numa+0x1b/0x40
> [1463111.320405] [<ffffffff8109c806>] ? task_numa_work+0x266/0x300
> [1463111.320408] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
> [1463111.320412] [<ffffffff81012e67>] ? do_notify_resume+0x97/0xb0
> [1463111.320414] [<ffffffff81794348>] page_fault+0x28/0x30
> [1463111.320415] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
> 89 76
> [1463111.344253] BUG: soft lockup - CPU#6 stuck for 22s! [Xorg:1842]
> [1463111.344254] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
> pps_core
> [1463111.344298] CPU: 6 PID: 1842 Comm: Xorg Tainted: G D W
> 3.13.0Write-Hole-Monitor #47
> [1463111.344299] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
> 2.1.3 05/21/2014
> [1463111.344302] task: ffff8800367b8000 ti: ffff880035f9e000 task.ti:
> ffff880035f9e000
> [1463111.344303] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
> smp_call_function_single+0xc6/0x190
> [1463111.344306] RSP: 0000:ffff880035f9fa60 EFLAGS: 00003202
> [1463111.344308] RAX: 0000000000000001 RBX: 0000000000400000 RCX:
> 0000000000000000
> [1463111.344309] RDX: ffff880035f9fad8 RSI: ffff8801a9a74e80 RDI:
> 0000000000000003
> [1463111.344311] RBP: ffff880035f9fac8 R08: ffff880035f9fb70 R09:
> 0000000000000000
> [1463111.344312] R10: 0000000000000000 R11: 0000000000000000 R12:
> 00000000003fc02a
> [1463111.344313] R13: ffffffff811ca5fc R14: ffff880035f9f9f0 R15:
> ffffffff81150fa3
> [1463111.344315] FS: 00007f2f338019c0(0000) GS:ffff8801a9a60000(0000)
> knlGS:0000000000000000
> [1463111.344317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [1463111.344318] CR2: 00007f2f344fdedc CR3: 000000031dd2b000 CR4:
> 00000000000407e0
> [1463111.344320] Stack:
> [1463111.344321] ffff880035f9fdf8 ffff880035f9fdf0 0000000000000000
> 0000000000000000
> [1463111.344324] 0000000000000000 0000000000000000 0000000000000000
> ffff8801a9a74480
> [1463111.344327] 0000000000000002 0000000000000003 ffff880035f9fb70
> 0000000000000006
> [1463111.344330] Call Trace:
> [1463111.344334] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
> [1463111.344336] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.344339] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.344342] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
> [1463111.344345] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
> [1463111.344347] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
> [1463111.344351] [<ffffffff811ca500>] ? poll_select_copy_remaining+0x130/0x130
> [1463111.344354] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
> [1463111.344356] [<ffffffff8109fd00>] task_numa_fault+0x680/0x890
> [1463111.344360] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
> [1463111.344363] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
> [1463111.344366] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
> [1463111.344368] [<ffffffff811b7579>] ? do_readv_writev+0x169/0x220
> [1463111.344371] [<ffffffff813624f4>] ? timerqueue_del+0x24/0x70
> [1463111.344374] [<ffffffff8108baa6>] ? __remove_hrtimer+0x46/0xa0
> [1463111.344377] [<ffffffff8108bec8>] ? hrtimer_try_to_cancel+0x48/0xe0
> [1463111.344380] [<ffffffff81068b53>] ? do_setitimer+0xe3/0x2a0
> [1463111.344382] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
> [1463111.344385] [<ffffffff81794348>] page_fault+0x28/0x30
> [1463111.344386] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
> 89 76
> [root] [ /home/arh ]
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 0/1] RFC: Use /dev/md/X as default name
From: NeilBrown @ 2015-02-18 21:37 UTC (permalink / raw)
To: Jes Sorensen; +Cc: linux-raid
In-Reply-To: <wrfjsie36qdi.fsf@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1512 bytes --]
On Wed, 18 Feb 2015 13:12:09 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
wrote:
> Jes.Sorensen@redhat.com writes:
> > From: Jes Sorensen <Jes.Sorensen@redhat.com>
> >
> > I have received some issues for when creating an array using a
> > /dev/mdX name, the matching symlink in /dev/md/X isn't
> > created. Whereas if you create /dev/md/X, /dev/mdX is created
> > automatically.
> >
> > I was trying to see if there was a better way of dealing with this,
> > but I couldn't find one. If you have suggestions for a better solution
> > I am all ears.
> >
> > Thoughts?
>
> Hi Neil,
>
> Any thoughts on this one?
Thanks for the reminder....
I'm not sure that I really see the problem.
"I ask it to create /dev/mdX and it doesn't create /dev/md/X".
Well ... no. You didn't ask it to. If you want it to create /dev/md/X,
then ... ask it to.
/dev/mdX is the canonical name. It always gets created.
/dev/md/X is a convenient alias. It gets created if requested.
Is there really a problem here worth solving?
Maybe I missed something.
NeilBrown
>
> Cheers,
> Jes
>
> >
> > Jes
> >
> > Jes Sorensen (1):
> > mddev_create(): choose /dev/md/ name over /dev/mdX
> >
> > mdopen.c | 6 +++++-
> > 1 file changed, 5 insertions(+), 1 deletion(-)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 0/1] RFC: Use /dev/md/X as default name
From: Jes Sorensen @ 2015-02-18 22:11 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150219083708.2ab46006@notabene.brown>
NeilBrown <neilb@suse.de> writes:
> On Wed, 18 Feb 2015 13:12:09 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
> wrote:
>
>> Jes.Sorensen@redhat.com writes:
>> > From: Jes Sorensen <Jes.Sorensen@redhat.com>
>> >
>> > I have received some issues for when creating an array using a
>> > /dev/mdX name, the matching symlink in /dev/md/X isn't
>> > created. Whereas if you create /dev/md/X, /dev/mdX is created
>> > automatically.
>> >
>> > I was trying to see if there was a better way of dealing with this,
>> > but I couldn't find one. If you have suggestions for a better solution
>> > I am all ears.
>> >
>> > Thoughts?
>>
>> Hi Neil,
>>
>> Any thoughts on this one?
>
> Thanks for the reminder....
>
> I'm not sure that I really see the problem.
>
> "I ask it to create /dev/mdX and it doesn't create /dev/md/X".
>
> Well ... no. You didn't ask it to. If you want it to create /dev/md/X,
> then ... ask it to.
>
> /dev/mdX is the canonical name. It always gets created.
> /dev/md/X is a convenient alias. It gets created if requested.
>
> Is there really a problem here worth solving?
>
> Maybe I missed something.
I have had complaints in Fedora from the installer people that they rely
on the /dev/md/ name being created when they create a new device. It is
also inconsistent because /dev/md/<X> will be created if you run
'mdadm -As' later on.
I don't see it as a major issue, but I can see why it is frustrating for
some and I think there is something to be said for being consistent in
behavior.
Cheers,
Jes
^ permalink raw reply
* Re: [PATCH 0/1] RFC: Use /dev/md/X as default name
From: NeilBrown @ 2015-02-18 22:32 UTC (permalink / raw)
To: Jes Sorensen; +Cc: linux-raid
In-Reply-To: <wrfj61ay6fai.fsf@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 2738 bytes --]
On Wed, 18 Feb 2015 17:11:33 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
wrote:
> NeilBrown <neilb@suse.de> writes:
> > On Wed, 18 Feb 2015 13:12:09 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
> > wrote:
> >
> >> Jes.Sorensen@redhat.com writes:
> >> > From: Jes Sorensen <Jes.Sorensen@redhat.com>
> >> >
> >> > I have received some issues for when creating an array using a
> >> > /dev/mdX name, the matching symlink in /dev/md/X isn't
> >> > created. Whereas if you create /dev/md/X, /dev/mdX is created
> >> > automatically.
> >> >
> >> > I was trying to see if there was a better way of dealing with this,
> >> > but I couldn't find one. If you have suggestions for a better solution
> >> > I am all ears.
> >> >
> >> > Thoughts?
> >>
> >> Hi Neil,
> >>
> >> Any thoughts on this one?
> >
> > Thanks for the reminder....
> >
> > I'm not sure that I really see the problem.
> >
> > "I ask it to create /dev/mdX and it doesn't create /dev/md/X".
> >
> > Well ... no. You didn't ask it to. If you want it to create /dev/md/X,
> > then ... ask it to.
> >
> > /dev/mdX is the canonical name. It always gets created.
> > /dev/md/X is a convenient alias. It gets created if requested.
> >
> > Is there really a problem here worth solving?
> >
> > Maybe I missed something.
>
> I have had complaints in Fedora from the installer people that they rely
> on the /dev/md/ name being created when they create a new device. It is
> also inconsistent because /dev/md/<X> will be created if you run
> 'mdadm -As' later on.
If they rely on the /dev/md/ name being created, then surely they should ask
for it to be created.
Is it really harder to run "mdadm -C /dev/md/0" than "mdadm -C /dev/md0" ??
If you create an array as "/dev/md0", then after subsequent "mdadm
-As" /dev/md0 will exist.
If you create an array as "/dev/md/0", then after subsequent mdadm
-As", /dev/md/0 will exist.
The fact that something unasked for also exists is a bonus.
>
> I don't see it as a major issue, but I can see why it is frustrating for
> some and I think there is something to be said for being consistent in
> behavior.
I think the behaviour is perfectly consistent. It just follows rules that
are slightly less trivial that some people appear to want.
However....
If you changed
mp = map_by_uuid(&map, info->uuid);
if (mp && mp->path &&
strncmp(mp->path, "/dev/md/", 8) == 0) {
printf("MD_DEVNAME=");
print_escape(mp->path+8);
putchar('\n');
}
in Detail.c so that when mp->path were "/dev/md0", MD_DEVNAME became "0",
then you should get the result that you are after, and I probably wouldn't
object to the patch.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* [PATCH/RFC/RFT] md: allow resync to go faster when there is competing IO.
From: NeilBrown @ 2015-02-19 6:04 UTC (permalink / raw)
To: linux RAID
[-- Attachment #1: Type: text/plain, Size: 3303 bytes --]
Hi all,
as you probably know, when md is doing resync and notices other IO it
throttles the resync to a configured "minimum", which defaults to
1MB/sec/device.
On a lot of modern devices, that is extremely slow.
I don't want to change the default (not all drives are the same) so I
wanted to come up with something that it a little bit dynamic.
After a bit of pondering and a bit of trial and error, I have the following.
It sometimes does what I want. I don't think it is ever really bad.
I'd appreciate it if people could test it on different hardware, different
configs, different loads.
What I have been doing is running
while :; do cat /sys/block/md0/md/sync_speed; sleep 5;
done > /root/some-file
while a resync is happening and a load is being imposed.
I do this with the old kernel and with this patch applied, then use
gnuplot to look at the sync_speed graphs.
I'd like to see that the new code is never slower than the old, and rarely more
than 20% of the available throughput when there is significant load.
Any test results or other observations most welcome,
Thanks,
NeilBrown
When md notices non-sync IO happening while it is trying
to resync (or reshape or recover) it slows down to the
set minimum.
The default minimum might have made sense many years ago
but the drives have become faster. Changing the default
to match the times isn't really a long term solution.
This patch changes the code so that instead of waiting until the speed
has dropped to the target, it just waits until pending requests
have completed, and then waits about as long again.
This means that the delay inserted is a function of the speed
of the devices.
Test show that:
- for some loads, the resync speed is unchanged. For those loads
increasing the minimum doesn't change the speed either.
So this is a good result. To increase resync speed under such
loads we would probably need to increase the resync window
size.
- for other loads, resync speed does increase to a reasonable
fraction (e.g. 20%) of maximum possible, and throughput of
the load only drops a little bit (e.g. 10%)
- for other loads, throughput of the non-sync load drops quite a bit
more. These seem to be latency-sensitive loads.
So it isn't a perfect solution, but it is mostly an improvement.
Signed-off-by: NeilBrown <neilb@suse.de>
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 94741ee6ae69..ce6624b3cc1b 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -7669,11 +7669,20 @@ void md_do_sync(struct md_thread *thread)
/((jiffies-mddev->resync_mark)/HZ +1) +1;
if (currspeed > speed_min(mddev)) {
- if ((currspeed > speed_max(mddev)) ||
- !is_mddev_idle(mddev, 0)) {
+ if (currspeed > speed_max(mddev)) {
msleep(500);
goto repeat;
}
+ if (!is_mddev_idle(mddev, 0)) {
+ /*
+ * Give other IO more of a chance.
+ * The faster the devices, the less we wait.
+ */
+ unsigned long start = jiffies;
+ wait_event(mddev->recovery_wait,
+ !atomic_read(&mddev->recovery_active));
+ schedule_timeout_uninterruptible(jiffies-start);
+ }
}
}
printk(KERN_INFO "md: %s: %s %s.\n",mdname(mddev), desc,
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Chris Murphy @ 2015-02-19 6:12 UTC (permalink / raw)
Cc: linux-raid
In-Reply-To: <loom.20150218T102011-486@post.gmane.org>
On Wed, Feb 18, 2015 at 4:04 AM, Chris <email.bug@arcor.de> wrote:
>>
>
> Hello all,
>
> the discussion about SCTERC boils down to letting the drive attempt ERC a
> little more or less. For any given disk experience seems to tell the slight
> difference is, that if ERC is allowed longer you may see the first
> unrecoverable erros (UREs) just a little (maybe only a month) later.
>
> UREs are inevitable. Thus, if I run a filesystem on just a single drive it
> will get corrupted at some point, nothing to do about it.
On a single randomly selective drive, I disagree. In aggregate, that's
true, eventually it will happen, you just won't know which drive or
when it'll happen. I have a number of 5+ year old drives that have
never reported a URE. Meanwhile another drive has so many bad sectors
I only keep it around for abusive purposes.
>
> Wait, except..., use a redundant raid! And here it makes a lot of a
> difference that the drive's ERC actually terminates before the controller
> timeout, to not loose all your redundacy again and be in hight risk of UREs
> showing up during the re-sync.
>
> So for a proper comparison we need to look at the difference it makes in the
> usage scenarios (error delay vs. loosing redundant error resilence + URE
> triggering), not at the single recoverable/unrecoverable error incidence. It
> looks to me, that it makes a lot of a differnce to redundant raids and no
> qualitative difference to single disk filesystems.
>
> And we need to keep in mind that single disk filesystems do also depend on
> the disk to stop grinding away with ERC attempts before the controller
> timout. Otherwise disk reset may make the system clear buffers and loose
> open files? Without prolonging the linux default controller timout, SCTERC
> can prevent that where supported.
To get to one size fits all, where SCT ERC is disabled (consumer
drive), and the kernel command timer is increased accordingly, we
still need the delay reportable to user space. You can't have a by
default 2-3 minute showstopper without an explanation so that the user
can tune this back to 30 seconds or get rid of the drive or some other
mitigation. Otherwise this is a 2-3 minute silent failure. I know a
huge number of users who would assume this is a crash and force power
off the system.
The option where SCT ERC is configurable, you could also do this one
size fits all by setting this to say 50-70 deciseconds, and for read
failures to cause recovery if raid1+ is used, or cause a read retry
if it's single, raid0, or linear. In other words, control the retries
in software for these drives.
>> I don't know if a udev rule can say "If the drive exclusively uses md,
>> lvm, btrfs, zfs raid1, 4+ or nested of those, and if the drive does
>> not support configurable SCT ERC, then change the kernel command timer
>> for those devices to ~120 seconds" then that might be a plausible
>> solution to use consumer drives the manufacturer rather explicitly
>> proscribes from use in raid...
>
> The script called by the udev rule could do that, but can be kept as simple
> as proposed, and can set SCTERC regardles, because setting SCTERC below the
> controller timout makes a qualitative difference in running the redundant
> arrays and a marginal difference in running non-redundant filesystems. (And
> nevertheless, set long controller timout for devices that don's support SCTERC.)
I can't agree at all, lacking facts, that this change is marginal for
non-redundant configurations. I've seen no data how common long
recovery incidents are, or how much more common data loss would be if
long recovery were prevented.
The mere fact they exist suggests they're necessary. It may very well
be that the ECC code or hardware used is so slow that it really does
take so unbelievably long (really 30 seconds is an eternity, and a
minute seems outrageous, and 2-3 minutes seems wholly ridiculous as in
worthy of brutal unrelenting ridicule); but that doesn't even matter
even if it is true, that's the behavior of the ECC whether we like it
or not, we can't just willy nilly turn these things off without
understanding the consequences. Just saying it's marginal doesn't make
it true.
So if SCT ERC is short, now you have to have a mitigation for the
possibly higher number of URE's this will result in, in the form of
kernel instigated read retries on read fail. And in fact, this may be
false. The retries the drive does internally might be completely
different than the kernel doing another read. The way data is encoded
on the drive these days bears no resemblance to discreet 1's and 0's.
And you also need a reliable opt out for SSD's. Their failures seem
rather different.
--
Chris Murphy
^ permalink raw reply
* mdadm raid 5 one disk overwritten file system failed
From: John Andre Taule @ 2015-02-19 7:38 UTC (permalink / raw)
To: linux-raid
Hi!
Case: mdadm Raid 5 4 2TB disks. ext4 formatted spanning the raid.
Attack: dd if=/dev/zero of=/dev/sdb bs=1M
Expected result would be a raid that could be recovered without data loss.
Result was that the file system failed and not possible to recover.
As I understand it if this was a "hardware type fake" raid controller, the
outcome would be uncertain. However I'm a bit confused as to why the raid
(or more specifically the file system) would fail so horrible when losing
one disk. Is there perhaps critical information written "outside" the raid
on the physical disk, and this where overwritten in the attack?
It would be nice to have an exact idea as to why it failed so hard, and how
obvious it should be that this attack would have more consequence then a
degraded raid.
//Regards
^ permalink raw reply
* Re: mdadm raid 5 one disk overwritten file system failed
From: Mikael Abrahamsson @ 2015-02-19 11:20 UTC (permalink / raw)
To: John Andre Taule; +Cc: linux-raid
In-Reply-To: <020601d04c17$08c10290$1a4307b0$@johnandre.net>
On Thu, 19 Feb 2015, John Andre Taule wrote:
> Hi!
>
> Case: mdadm Raid 5 4 2TB disks. ext4 formatted spanning the raid.
> Attack: dd if=/dev/zero of=/dev/sdb bs=1M
>
> Expected result would be a raid that could be recovered without data loss.
>
> Result was that the file system failed and not possible to recover.
>
> As I understand it if this was a "hardware type fake" raid controller, the
> outcome would be uncertain. However I'm a bit confused as to why the raid
> (or more specifically the file system) would fail so horrible when losing
> one disk. Is there perhaps critical information written "outside" the raid
> on the physical disk, and this where overwritten in the attack?
Did you stop the array before you did the dd command, or you just did it?
If you just did it, most likely you overwrote the superblock on the drive
(located near the beginning of the drive by recent default), plus part of
the file system.
> It would be nice to have an exact idea as to why it failed so hard, and
> how obvious it should be that this attack would have more consequence
> then a degraded raid.
Because if the drive was active then the operating system most likely
didn't notice that you overwrote part of the data on the disk and the
drive wasn't failed.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* SV: mdadm raid 5 one disk overwritten file system failed
From: John Andre Taule @ 2015-02-19 14:00 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <alpine.DEB.2.02.1502191218140.4007@uplift.swm.pp.se>
The array was not stopped before dd was running. The "hacker" logged on,
left the command running and logged of. It was discovered the next morning
about 5 hours later, and there was very high load on the server, I think
this is why the command where discovered at all. This is how that raid have
performed earlier when a drive have failed.
I'm a bit surprised that overwriting anything on the physical disk should
corrupt the file system on the raid. I would think that would be similar to
a disk crashing or failing in other ways.
What you say that Linux might not have seen the disk as failing is
interesting. This could explain why the file system got corrupted.
-----Opprinnelig melding-----
Fra: Mikael Abrahamsson [mailto:swmike@swm.pp.se]
Sendt: 19. februar 2015 12:20
Til: John Andre Taule
Kopi: linux-raid@vger.kernel.org
Emne: Re: mdadm raid 5 one disk overwritten file system failed
On Thu, 19 Feb 2015, John Andre Taule wrote:
> Hi!
>
> Case: mdadm Raid 5 4 2TB disks. ext4 formatted spanning the raid.
> Attack: dd if=/dev/zero of=/dev/sdb bs=1M
>
> Expected result would be a raid that could be recovered without data loss.
>
> Result was that the file system failed and not possible to recover.
>
> As I understand it if this was a "hardware type fake" raid controller,
> the outcome would be uncertain. However I'm a bit confused as to why
> the raid (or more specifically the file system) would fail so horrible
> when losing one disk. Is there perhaps critical information written
> "outside" the raid on the physical disk, and this where overwritten in the
attack?
Did you stop the array before you did the dd command, or you just did it?
If you just did it, most likely you overwrote the superblock on the drive
(located near the beginning of the drive by recent default), plus part of
the file system.
> It would be nice to have an exact idea as to why it failed so hard,
> and how obvious it should be that this attack would have more
> consequence then a degraded raid.
Because if the drive was active then the operating system most likely didn't
notice that you overwrote part of the data on the disk and the drive wasn't
failed.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: mdadm raid 5 one disk overwritten file system failed
From: Mikael Abrahamsson @ 2015-02-19 14:23 UTC (permalink / raw)
To: John Andre Taule; +Cc: linux-raid
In-Reply-To: <021f01d04c4c$5def1740$19cd45c0$@johnandre.net>
On Thu, 19 Feb 2015, John Andre Taule wrote:
> I'm a bit surprised that overwriting anything on the physical disk
> should corrupt the file system on the raid. I would think that would be
> similar to a disk crashing or failing in other ways.
Errr, in raid5 you have data blocks and parity blocks. WHen you overwrite
one of the component drives with zeroes, you're effectively doing the same
as writing 0:es to a non-raid drive every 3 $stripesize. You're zero:ing a
lot of the filesystem information.
> What you say that Linux might not have seen the disk as failing is
> interesting. This could explain why the file system got corrupted.
Correct. There is no mechanism that periodically checks the contents of
the superblock and fails the drive if it's not there anymore. So the drive
is never failed.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: mdadm raid 5 one disk overwritten file system failed
From: Adam Goryachev @ 2015-02-19 14:39 UTC (permalink / raw)
To: John Andre Taule; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1502191521450.4007@uplift.swm.pp.se>
On 20/02/2015 01:23, Mikael Abrahamsson wrote:
> On Thu, 19 Feb 2015, John Andre Taule wrote:
>
>> I'm a bit surprised that overwriting anything on the physical disk
>> should corrupt the file system on the raid. I would think that would
>> be similar to a disk crashing or failing in other ways.
>
> Errr, in raid5 you have data blocks and parity blocks. WHen you
> overwrite one of the component drives with zeroes, you're effectively
> doing the same as writing 0:es to a non-raid drive every 3
> $stripesize. You're zero:ing a lot of the filesystem information.
>
>> What you say that Linux might not have seen the disk as failing is
>> interesting. This could explain why the file system got corrupted.
>
> Correct. There is no mechanism that periodically checks the contents
> of the superblock and fails the drive if it's not there anymore. So
> the drive is never failed.
>
In addition, there is no checking of the data when read to confirm that
the data on the first 4 disks = the checksum on the 5th disk (assuming a
5 disk raid5). This applies equally to all raid levels as currently
working from linux md raid. While there are some use cases where it
would be nice to confirm that the data read is correct, this has not yet
been implemented (for live operation, you can schedule a check at
periodic intervals).
Even if MD noticed that the value of the first 4 disks did not equal the
checksum on the 5th disk, it has no method to determine which disk
contained the wrong value (could be any of the data stripes, or the
parity stripe). raid6 begins to allow for this type of check, and I
remember a lot of work being done on this, however, I think that was
still an offline tool more useful for data recovery from partially
failed multiple drives.
From memory, there are filesystems which will do what you are asking
(check that the data received from disk is correct, use multiple 'disks'
and ensure protection from x failed drives, etc. I am certain zfs and
btrfs both support this. (I've never used either due to stability
concerns, but I read about them every now and then....)
Regards,
Adam
^ permalink raw reply
* SV: mdadm raid 5 one disk overwritten file system failed
From: John Andre Taule @ 2015-02-19 16:21 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <54E5F5A7.3090609@websitemanagers.com.au>
How common would this knowledge be?
Personally I would never do something like this on a live system just
because there is too many unknown variables in play. I know what the
different raids do. I am not working on this full time, my day to day work
is toward the user end of the application stack. Usually we use Areca
hardware raids, but this particular raid used mdadm, well because that's
what was available at the time. Its been stable enough, I think its survived
2 or 3 failed drives, of course not at the same time.
I would like to thank the list for confirming my suspicion that there was
something else at play here that made the /dev/zero do more damage then the
"hacker" believed it would do.
//Regards
-----Opprinnelig melding-----
Fra: Adam Goryachev [mailto:mailinglists@websitemanagers.com.au]
Sendt: 19. februar 2015 15:40
Til: John Andre Taule
Kopi: linux-raid@vger.kernel.org
Emne: Re: mdadm raid 5 one disk overwritten file system failed
On 20/02/2015 01:23, Mikael Abrahamsson wrote:
> On Thu, 19 Feb 2015, John Andre Taule wrote:
>
>> I'm a bit surprised that overwriting anything on the physical disk
>> should corrupt the file system on the raid. I would think that would
>> be similar to a disk crashing or failing in other ways.
>
> Errr, in raid5 you have data blocks and parity blocks. WHen you
> overwrite one of the component drives with zeroes, you're effectively
> doing the same as writing 0:es to a non-raid drive every 3
> $stripesize. You're zero:ing a lot of the filesystem information.
>
>> What you say that Linux might not have seen the disk as failing is
>> interesting. This could explain why the file system got corrupted.
>
> Correct. There is no mechanism that periodically checks the contents
> of the superblock and fails the drive if it's not there anymore. So
> the drive is never failed.
>
In addition, there is no checking of the data when read to confirm that the
data on the first 4 disks = the checksum on the 5th disk (assuming a
5 disk raid5). This applies equally to all raid levels as currently working
from linux md raid. While there are some use cases where it would be nice to
confirm that the data read is correct, this has not yet been implemented
(for live operation, you can schedule a check at periodic intervals).
Even if MD noticed that the value of the first 4 disks did not equal the
checksum on the 5th disk, it has no method to determine which disk contained
the wrong value (could be any of the data stripes, or the parity stripe).
raid6 begins to allow for this type of check, and I remember a lot of work
being done on this, however, I think that was still an offline tool more
useful for data recovery from partially failed multiple drives.
From memory, there are filesystems which will do what you are asking (check
that the data received from disk is correct, use multiple 'disks'
and ensure protection from x failed drives, etc. I am certain zfs and btrfs
both support this. (I've never used either due to stability concerns, but I
read about them every now and then....)
Regards,
Adam
^ permalink raw reply
* Re: mdadm raid 5 one disk overwritten file system failed
From: Piergiorgio Sartor @ 2015-02-19 17:15 UTC (permalink / raw)
To: John Andre Taule; +Cc: linux-raid
In-Reply-To: <020601d04c17$08c10290$1a4307b0$@johnandre.net>
On Thu, Feb 19, 2015 at 08:38:19AM +0100, John Andre Taule wrote:
> Hi!
>
> Case: mdadm Raid 5 4 2TB disks. ext4 formatted spanning the raid.
> Attack: dd if=/dev/zero of=/dev/sdb bs=1M
>
> Expected result would be a raid that could be recovered without data loss.
>
> Result was that the file system failed and not possible to recover.
>
> As I understand it if this was a "hardware type fake" raid controller, the
> outcome would be uncertain. However I'm a bit confused as to why the raid
> (or more specifically the file system) would fail so horrible when losing
> one disk. Is there perhaps critical information written "outside" the raid
> on the physical disk, and this where overwritten in the attack?
>
> It would be nice to have an exact idea as to why it failed so hard, and how
> obvious it should be that this attack would have more consequence then a
> degraded raid.
In this situation, there is no HDD failure.
The kernel, the md driver, the sata driver and so on,
cannot detect any failure, because there is none.
The HDD is alive and kicking and well writing.
Just to be clear and avoid confusion, the (redundant)
RAID does *not* check, at each read operation, that
the data is consistent. It does only use redundancy
in order to re-generate missing data *after* a failure
is detected.
So, writing to a RAID component does not trigger any
error, hence no failure, hence no reconstruction, but
a corrupted filesystem.
bye,
pg
>
> //Regards
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
piergiorgio
^ permalink raw reply
* Re: [PATCH] [md] raid5: check faulty flag for array status during recovery.
From: NeilBrown @ 2015-02-19 21:51 UTC (permalink / raw)
To: Eric Mei; +Cc: linux-raid, eric.mei
In-Reply-To: <94BC57C5-6223-435B-96FD-7DA2F6B4E561@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 3665 bytes --]
On Tue, 6 Jan 2015 15:24:24 -0700 Eric Mei <meijia@gmail.com> wrote:
> Hi Neil,
>
> In a MDRAID derived work we found and fixed a data corruption bug. We think this also affect vanilla MDRAID, but we didn’t directly prove that by constructing a test to show the corruption. Following is the theoretical analysis, please kindly review and see if I missed something.
>
> To rebuild a stripe, MD checks whether array will be optimal after rebuild complete, if that’s true, we’ll mark the WIB bit to be cleared, the purpose is to enable “incremental rebuild”. The code section is like this:
>
> /* Need to check if array will still be degraded after recovery/resync
> * We don't need to check the 'failed' flag as when that gets set,
> * recovery aborts.
> */
> for (i = 0; i < conf->raid_disks; i++)
> if (conf->disks[i].rdev == NULL)
> still_degraded = 1;
>
> The problem is that only checking rdev == NULL might not be enough. Suppose both 2 drives D0 and D1 failed and marked as Faulty; We immediately removed D0 from array, but because some lingering IO on D1, it remains in array with Faulty flags on. A new drive pulled in, rebuild against D0 starts. Now because no rdev is NULL, MD thinks array will be optimal. If some writes happened before rebuild reaches the region, their dirty bits in WIB will be cleared. When later add D1 back into array, we’ll skip rebuilding those stripes, thus data corruption.
>
> The attached patch (against 3.18.0-rc6) is supposed to fix this issue.
>
> Thanks
> Eric
>
Hi Eric,
sorry for the delay, and thanks for the reminder...
The issue you described could only affect RAID6 as it requires the array to
continue with two failed drives.
However in the RAID6 case I think you are correct - there is a chance of
corruption if there is a double failure and a delay in removing one device.
Your patch isn't quite safe as conf->disks[i].rdev can become NULL at any
moment, so it could become NULL between testing and de-referencing.
So I've modified it as follows.
Thanks,
NeilBrown
Author: Eric Mei <eric.mei@seagate.com>
Date: Tue Jan 6 09:35:02 2015 -0800
raid5: check faulty flag for array status during recovery.
When we have more than 1 drive failure, it's possible we start
rebuild one drive while leaving another faulty drive in array.
To determine whether array will be optimal after building, current
code only check whether a drive is missing, which could potentially
lead to data corruption. This patch is to add checking Faulty flag.
Signed-off-by: NeilBrown <neilb@suse.de>
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index bc6d7595ad76..022a0d99e110 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -5120,12 +5120,17 @@ static inline sector_t sync_request(struct mddev *mddev, sector_t sector_nr, int
schedule_timeout_uninterruptible(1);
}
/* Need to check if array will still be degraded after recovery/resync
- * We don't need to check the 'failed' flag as when that gets set,
- * recovery aborts.
+ * Note in case of > 1 drive failures it's possible we're rebuilding
+ * one drive while leaving another faulty drive in array.
*/
- for (i = 0; i < conf->raid_disks; i++)
- if (conf->disks[i].rdev == NULL)
+ rcu_read_lock();
+ for (i = 0; i < conf->raid_disks; i++) {
+ struct md_rdev *rdev = ACCESS_ONCE(conf->disks[i].rdev);
+
+ if (rdev == NULL || test_bit(Faulty, &rdev->flags))
still_degraded = 1;
+ }
+ rcu_read_unlock();
bitmap_start_sync(mddev->bitmap, sector_nr, &sync_blocks, still_degraded);
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox