* Re: [PATCH 0/1] RFC: Use /dev/md/X as default name
From: Jes Sorensen @ 2015-02-18 22:11 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150219083708.2ab46006@notabene.brown>
NeilBrown <neilb@suse.de> writes:
> On Wed, 18 Feb 2015 13:12:09 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
> wrote:
>
>> Jes.Sorensen@redhat.com writes:
>> > From: Jes Sorensen <Jes.Sorensen@redhat.com>
>> >
>> > I have received some issues for when creating an array using a
>> > /dev/mdX name, the matching symlink in /dev/md/X isn't
>> > created. Whereas if you create /dev/md/X, /dev/mdX is created
>> > automatically.
>> >
>> > I was trying to see if there was a better way of dealing with this,
>> > but I couldn't find one. If you have suggestions for a better solution
>> > I am all ears.
>> >
>> > Thoughts?
>>
>> Hi Neil,
>>
>> Any thoughts on this one?
>
> Thanks for the reminder....
>
> I'm not sure that I really see the problem.
>
> "I ask it to create /dev/mdX and it doesn't create /dev/md/X".
>
> Well ... no. You didn't ask it to. If you want it to create /dev/md/X,
> then ... ask it to.
>
> /dev/mdX is the canonical name. It always gets created.
> /dev/md/X is a convenient alias. It gets created if requested.
>
> Is there really a problem here worth solving?
>
> Maybe I missed something.
I have had complaints in Fedora from the installer people that they rely
on the /dev/md/ name being created when they create a new device. It is
also inconsistent because /dev/md/<X> will be created if you run
'mdadm -As' later on.
I don't see it as a major issue, but I can see why it is frustrating for
some and I think there is something to be said for being consistent in
behavior.
Cheers,
Jes
^ permalink raw reply
* Re: [PATCH 0/1] RFC: Use /dev/md/X as default name
From: NeilBrown @ 2015-02-18 21:37 UTC (permalink / raw)
To: Jes Sorensen; +Cc: linux-raid
In-Reply-To: <wrfjsie36qdi.fsf@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1512 bytes --]
On Wed, 18 Feb 2015 13:12:09 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
wrote:
> Jes.Sorensen@redhat.com writes:
> > From: Jes Sorensen <Jes.Sorensen@redhat.com>
> >
> > I have received some issues for when creating an array using a
> > /dev/mdX name, the matching symlink in /dev/md/X isn't
> > created. Whereas if you create /dev/md/X, /dev/mdX is created
> > automatically.
> >
> > I was trying to see if there was a better way of dealing with this,
> > but I couldn't find one. If you have suggestions for a better solution
> > I am all ears.
> >
> > Thoughts?
>
> Hi Neil,
>
> Any thoughts on this one?
Thanks for the reminder....
I'm not sure that I really see the problem.
"I ask it to create /dev/mdX and it doesn't create /dev/md/X".
Well ... no. You didn't ask it to. If you want it to create /dev/md/X,
then ... ask it to.
/dev/mdX is the canonical name. It always gets created.
/dev/md/X is a convenient alias. It gets created if requested.
Is there really a problem here worth solving?
Maybe I missed something.
NeilBrown
>
> Cheers,
> Jes
>
> >
> > Jes
> >
> > Jes Sorensen (1):
> > mddev_create(): choose /dev/md/ name over /dev/mdX
> >
> > mdopen.c | 6 +++++-
> > 1 file changed, 5 insertions(+), 1 deletion(-)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Potential Bug in "echo 0 > /dev/md0"
From: NeilBrown @ 2015-02-18 21:32 UTC (permalink / raw)
To: Alireza Haghdoost; +Cc: Linux RAID
In-Reply-To: <CAB-428mj-P-+M0MnKhmojRe9ZQQzsSTBfCSS91ZK5KRy5p1mSg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 8509 bytes --]
On Wed, 18 Feb 2015 10:11:20 -0600 Alireza Haghdoost <alireza@cs.umn.edu>
wrote:
> I understand this is not the right way to talk with md device but my
> understanding is that if some one run this command by mistake (or
> vandalism) , it should not results a kernel crash:
>
> [root] [ /home/arh ]
> # echo 0 > /dev/md0
>
> [root] [ /home/arh ]
> # dmesg
> [1463111.320277] BUG: soft lockup - CPU#4 stuck for 22s! [whoopsie:1829]
> [1463111.320284] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
> pps_core
> [1463111.320326] CPU: 4 PID: 1829 Comm: whoopsie Tainted: G D W
^^^^^^^^
What is "whoopsie" ???
> 3.13.0Write-Hole-Monitor #47
^^^^^^^^^^^^^^^^^^^
What is "Write-Hole-Monitor".
There is no evidence that this is related to RAID, except that it presumably
happens at about the same time that you write to /dev/md0.
It certainly isn't running and md/raid code when it reports a soft-lockup.
NeilBrown
> [1463111.320328] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
> 2.1.3 05/21/2014
> [1463111.320329] task: ffff8801a425c7d0 ti: ffff8801a3e04000 task.ti:
> ffff8801a3e04000
> [1463111.320331] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
> smp_call_function_single+0xc6/0x190
> [1463111.320335] RSP: 0000:ffff8801a3e05a60 EFLAGS: 00000202
> [1463111.320336] RAX: 0000000000000001 RBX: ffffffff813624f4 RCX:
> 0000000000000000
> [1463111.320338] RDX: ffff8801a3e05ad8 RSI: ffff8801a9a54e80 RDI:
> 0000000000000001
> [1463111.320339] RBP: ffff8801a3e05ac8 R08: ffff8801a3e05b70 R09:
> 0000000000000000
> [1463111.320341] R10: 0000000000000000 R11: 0000000000000000 R12:
> ffff88031e6bec80
> [1463111.320342] R13: ffff880035d70700 R14: ffff8801a9a54400 R15:
> ffff88031e6bec80
> [1463111.320344] FS: 00007f9ae4924840(0000) GS:ffff8801a9a40000(0000)
> knlGS:0000000000000000
> [1463111.320346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [1463111.320347] CR2: 00007f9ae43bc280 CR3: 00000001a3adc000 CR4:
> 00000000000407e0
> [1463111.320349] Stack:
> [1463111.320350] ffff8801a3e05af8 ffffffff8178f5ae 0000000000000000
> 0000000000000000
> [1463111.320353] 0000000000000000 0000000000000000 0000000000000000
> ffff8801a9a54480
> [1463111.320357] 0000000000000002 0000000000000001 ffff8801a3e05b70
> 0000000000000004
> [1463111.320360] Call Trace:
> [1463111.320364] [<ffffffff8178f5ae>] ?
> schedule_hrtimeout_range_clock+0xce/0x170
> [1463111.320367] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
> [1463111.320369] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.320372] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.320375] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
> [1463111.320378] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
> [1463111.320381] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
> [1463111.320384] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
> [1463111.320386] [<ffffffff8109f8ef>] task_numa_fault+0x26f/0x890
> [1463111.320390] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
> [1463111.320394] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
> [1463111.320396] [<ffffffff8117bc30>] ? change_protection+0x690/0x720
> [1463111.320399] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
> [1463111.320403] [<ffffffff81191b0b>] ? change_prot_numa+0x1b/0x40
> [1463111.320405] [<ffffffff8109c806>] ? task_numa_work+0x266/0x300
> [1463111.320408] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
> [1463111.320412] [<ffffffff81012e67>] ? do_notify_resume+0x97/0xb0
> [1463111.320414] [<ffffffff81794348>] page_fault+0x28/0x30
> [1463111.320415] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
> 89 76
> [1463111.344253] BUG: soft lockup - CPU#6 stuck for 22s! [Xorg:1842]
> [1463111.344254] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
> pps_core
> [1463111.344298] CPU: 6 PID: 1842 Comm: Xorg Tainted: G D W
> 3.13.0Write-Hole-Monitor #47
> [1463111.344299] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
> 2.1.3 05/21/2014
> [1463111.344302] task: ffff8800367b8000 ti: ffff880035f9e000 task.ti:
> ffff880035f9e000
> [1463111.344303] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
> smp_call_function_single+0xc6/0x190
> [1463111.344306] RSP: 0000:ffff880035f9fa60 EFLAGS: 00003202
> [1463111.344308] RAX: 0000000000000001 RBX: 0000000000400000 RCX:
> 0000000000000000
> [1463111.344309] RDX: ffff880035f9fad8 RSI: ffff8801a9a74e80 RDI:
> 0000000000000003
> [1463111.344311] RBP: ffff880035f9fac8 R08: ffff880035f9fb70 R09:
> 0000000000000000
> [1463111.344312] R10: 0000000000000000 R11: 0000000000000000 R12:
> 00000000003fc02a
> [1463111.344313] R13: ffffffff811ca5fc R14: ffff880035f9f9f0 R15:
> ffffffff81150fa3
> [1463111.344315] FS: 00007f2f338019c0(0000) GS:ffff8801a9a60000(0000)
> knlGS:0000000000000000
> [1463111.344317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [1463111.344318] CR2: 00007f2f344fdedc CR3: 000000031dd2b000 CR4:
> 00000000000407e0
> [1463111.344320] Stack:
> [1463111.344321] ffff880035f9fdf8 ffff880035f9fdf0 0000000000000000
> 0000000000000000
> [1463111.344324] 0000000000000000 0000000000000000 0000000000000000
> ffff8801a9a74480
> [1463111.344327] 0000000000000002 0000000000000003 ffff880035f9fb70
> 0000000000000006
> [1463111.344330] Call Trace:
> [1463111.344334] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
> [1463111.344336] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.344339] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.344342] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
> [1463111.344345] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
> [1463111.344347] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
> [1463111.344351] [<ffffffff811ca500>] ? poll_select_copy_remaining+0x130/0x130
> [1463111.344354] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
> [1463111.344356] [<ffffffff8109fd00>] task_numa_fault+0x680/0x890
> [1463111.344360] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
> [1463111.344363] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
> [1463111.344366] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
> [1463111.344368] [<ffffffff811b7579>] ? do_readv_writev+0x169/0x220
> [1463111.344371] [<ffffffff813624f4>] ? timerqueue_del+0x24/0x70
> [1463111.344374] [<ffffffff8108baa6>] ? __remove_hrtimer+0x46/0xa0
> [1463111.344377] [<ffffffff8108bec8>] ? hrtimer_try_to_cancel+0x48/0xe0
> [1463111.344380] [<ffffffff81068b53>] ? do_setitimer+0xe3/0x2a0
> [1463111.344382] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
> [1463111.344385] [<ffffffff81794348>] page_fault+0x28/0x30
> [1463111.344386] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
> 89 76
> [root] [ /home/arh ]
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: help with the little script (erc timout fix)
From: NeilBrown @ 2015-02-18 21:25 UTC (permalink / raw)
To: Chris; +Cc: linux-raid
In-Reply-To: <loom.20150218T155053-576@post.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 2459 bytes --]
On Wed, 18 Feb 2015 15:04:53 +0000 (UTC) Chris <email.bug@arcor.de> wrote:
>
> Hello,
>
> by adapting what I could find, I compiled the following short snippet now.
>
> Could list members please look at this novice code and suggest a way to
> determine the containing disk device $HDD_DEV from the parition/disk,
> before I dare to test this.
>
>
>
> In udev-md-raid-assembly.rules, below LABEL="md_inc" (section only handling
> all md suppported devices) add:
>
> # fix timouts for redundant raids, if possible
> IMPORT{program}="BINDIR/mdadm --examine --export $tempnode"
> TEST="/usr/sbin/smartctl", ENV{MD_LEVEL}=="raid[1-9]*",
> RUN+="BINDIR/mdadm-erc-timout-fix.sh $tempnode"
It might make sense to have 2 rules, one for partitions and one for disks
(based on ENV{DEVTYPE}). Then use $parent to get the device from the
partition, and $devnode to get the device of the disk.
>
> And in a new mdadm-erc-timout-fix.sh file implement:
>
> #! /bin/sh
>
> HDD_DEV= $1 somehow stipping off the tailing numbers?
>
> if smartctl -l scterc ${HDD_DEV} | grep -q Disabled ; then
> /usr/sbin/smartctl -l scterc,70,70 ${HDD_DEV}
> else
> if ! smartctl -l scterc ${HDD_DEV} | grep -q seconds ; then
> echo 180 >/sys/block/${HDD_DEV}/device/timeout
> fi
> fi
You should be consistent and use /usr/sbin/smartctl everywhere, or explicitly
set $PATH and just use smartctl everywhere.
>
> Correct execution during boot would seem to require that distro
> package managers hook smartctl and the script into the initramfs
> generation.
>
> Regards,
> Chris
One problem with this approach is that it assumes circumstances don't change.
If you have a working RAID1, then limiting the timeout on both devices makes
sense. If you have a degraded RAID1 with only one device left then you
really want the drive to try as hard as it can to get the data.
There is a "FAILFAST" mechanism in the kernel which allows the filesystem to
md etc to indicate that it wants accesses to "fail fast", which presumably
means to use a smaller timeout.
I would rather md used this flag where appropriate, and for the device to
respond to it by using suitable timeouts.
The problem is that FAILFAST isn't documented usefully and it is very hard to
figure out what exactly (if anything) it does.
But until that is resolved, a fix like this is probably a good idea.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Potential Bug in "echo 0 > /dev/md0"
From: Jes Sorensen @ 2015-02-18 20:20 UTC (permalink / raw)
To: Alireza Haghdoost; +Cc: Linux RAID, Neil Brown
In-Reply-To: <CAB-428k=MXBZRbwTp6h=KfZQ_o4vP5TyfPqutsmoFW6to2Ft9A@mail.gmail.com>
Alireza Haghdoost <alireza@cs.umn.edu> writes:
> Here you are:
>
> [root] [ /home/arh ]
> # cat /proc/mdstat
> Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5]
> [raid4] [multipath] [faulty]
> md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
> 314374656 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
>
>
> [root] [ /home/arh ]
> # uname -a
> Linux mist01-umh 3.13.0 #47 SMP Sun Feb 1 10:27:24 CST 2015 x86_64
> x86_64 x86_64 GNU/Linux
>
> [root] [ /home/arh ]
> # cat /etc/issue
> Ubuntu 14.04.1 LTS \n \l
I am unfamiliar with Ubuntu's kernels, is this a distro kernel or a self
compiled kernel? If this is a distro kernel, please start out by
reporting the bug to your distribution through their bug tracking
system.
3.13 is very old, so it would be help if you tried against a recent
kernel.
Jes
>
> [root] [ /home/arh ]
> # mdadm --version
> mdadm - v3.2.5 - 18th May 2012
>
> [root] [ /home/arh ]
> # mdadm --detail /dev/md0
> <No output, Freezes since kernel is crashed !>
>
> On Wed, Feb 18, 2015 at 10:19 AM, Jes Sorensen <Jes.Sorensen@redhat.com> wrote:
>> Alireza Haghdoost <alireza@cs.umn.edu> writes:
>>> I understand this is not the right way to talk with md device but my
>>> understanding is that if some one run this command by mistake (or
>>> vandalism) , it should not results a kernel crash:
>>>
>>> [root] [ /home/arh ]
>>> # echo 0 > /dev/md0
>>
>> It shouldn't, however before anyone can debug this, you need to provide
>> a proper bug report with information about the kernel version you are
>> running, the configuration of /dev/md0 etc.
>>
>> Please include 'cat /proc/mdstat' output with it.
>>
>> JEs
>>
>>>
>>> [root] [ /home/arh ]
>>> # dmesg
>>> [1463111.320277] BUG: soft lockup - CPU#4 stuck for 22s! [whoopsie:1829]
>>> [1463111.320284] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
>>> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
>>> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
>>> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
>>> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
>>> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
>>> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
>>> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
>>> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
>>> pps_core
>>> [1463111.320326] CPU: 4 PID: 1829 Comm: whoopsie Tainted: G D W
>>> 3.13.0Write-Hole-Monitor #47
>>> [1463111.320328] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
>>> 2.1.3 05/21/2014
>>> [1463111.320329] task: ffff8801a425c7d0 ti: ffff8801a3e04000 task.ti:
>>> ffff8801a3e04000
>>> [1463111.320331] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
>>> smp_call_function_single+0xc6/0x190
>>> [1463111.320335] RSP: 0000:ffff8801a3e05a60 EFLAGS: 00000202
>>> [1463111.320336] RAX: 0000000000000001 RBX: ffffffff813624f4 RCX:
>>> 0000000000000000
>>> [1463111.320338] RDX: ffff8801a3e05ad8 RSI: ffff8801a9a54e80 RDI:
>>> 0000000000000001
>>> [1463111.320339] RBP: ffff8801a3e05ac8 R08: ffff8801a3e05b70 R09:
>>> 0000000000000000
>>> [1463111.320341] R10: 0000000000000000 R11: 0000000000000000 R12:
>>> ffff88031e6bec80
>>> [1463111.320342] R13: ffff880035d70700 R14: ffff8801a9a54400 R15:
>>> ffff88031e6bec80
>>> [1463111.320344] FS: 00007f9ae4924840(0000) GS:ffff8801a9a40000(0000)
>>> knlGS:0000000000000000
>>> [1463111.320346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [1463111.320347] CR2: 00007f9ae43bc280 CR3: 00000001a3adc000 CR4:
>>> 00000000000407e0
>>> [1463111.320349] Stack:
>>> [1463111.320350] ffff8801a3e05af8 ffffffff8178f5ae 0000000000000000
>>> 0000000000000000
>>> [1463111.320353] 0000000000000000 0000000000000000 0000000000000000
>>> ffff8801a9a54480
>>> [1463111.320357] 0000000000000002 0000000000000001 ffff8801a3e05b70
>>> 0000000000000004
>>> [1463111.320360] Call Trace:
>>> [1463111.320364] [<ffffffff8178f5ae>] ?
>>> schedule_hrtimeout_range_clock+0xce/0x170
>>> [1463111.320367] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
>>> [1463111.320369] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>>> [1463111.320372] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>>> [1463111.320375] [<ffffffff81095e80>] ?
>>> __migrate_swap_task.part.68+0x80/0x80
>>> [1463111.320378] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
>>> [1463111.320381] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
>>> [1463111.320384] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
>>> [1463111.320386] [<ffffffff8109f8ef>] task_numa_fault+0x26f/0x890
>>> [1463111.320390] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
>>> [1463111.320394] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
>>> [1463111.320396] [<ffffffff8117bc30>] ? change_protection+0x690/0x720
>>> [1463111.320399] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
>>> [1463111.320403] [<ffffffff81191b0b>] ? change_prot_numa+0x1b/0x40
>>> [1463111.320405] [<ffffffff8109c806>] ? task_numa_work+0x266/0x300
>>> [1463111.320408] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
>>> [1463111.320412] [<ffffffff81012e67>] ? do_notify_resume+0x97/0xb0
>>> [1463111.320414] [<ffffffff81794348>] page_fault+0x28/0x30
>>> [1463111.320415] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
>>> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
>>> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
>>> 89 76
>>> [1463111.344253] BUG: soft lockup - CPU#6 stuck for 22s! [Xorg:1842]
>>> [1463111.344254] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
>>> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
>>> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
>>> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
>>> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
>>> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
>>> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
>>> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
>>> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
>>> pps_core
>>> [1463111.344298] CPU: 6 PID: 1842 Comm: Xorg Tainted: G D W
>>> 3.13.0Write-Hole-Monitor #47
>>> [1463111.344299] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
>>> 2.1.3 05/21/2014
>>> [1463111.344302] task: ffff8800367b8000 ti: ffff880035f9e000 task.ti:
>>> ffff880035f9e000
>>> [1463111.344303] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
>>> smp_call_function_single+0xc6/0x190
>>> [1463111.344306] RSP: 0000:ffff880035f9fa60 EFLAGS: 00003202
>>> [1463111.344308] RAX: 0000000000000001 RBX: 0000000000400000 RCX:
>>> 0000000000000000
>>> [1463111.344309] RDX: ffff880035f9fad8 RSI: ffff8801a9a74e80 RDI:
>>> 0000000000000003
>>> [1463111.344311] RBP: ffff880035f9fac8 R08: ffff880035f9fb70 R09:
>>> 0000000000000000
>>> [1463111.344312] R10: 0000000000000000 R11: 0000000000000000 R12:
>>> 00000000003fc02a
>>> [1463111.344313] R13: ffffffff811ca5fc R14: ffff880035f9f9f0 R15:
>>> ffffffff81150fa3
>>> [1463111.344315] FS: 00007f2f338019c0(0000) GS:ffff8801a9a60000(0000)
>>> knlGS:0000000000000000
>>> [1463111.344317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [1463111.344318] CR2: 00007f2f344fdedc CR3: 000000031dd2b000 CR4:
>>> 00000000000407e0
>>> [1463111.344320] Stack:
>>> [1463111.344321] ffff880035f9fdf8 ffff880035f9fdf0 0000000000000000
>>> 0000000000000000
>>> [1463111.344324] 0000000000000000 0000000000000000 0000000000000000
>>> ffff8801a9a74480
>>> [1463111.344327] 0000000000000002 0000000000000003 ffff880035f9fb70
>>> 0000000000000006
>>> [1463111.344330] Call Trace:
>>> [1463111.344334] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
>>> [1463111.344336] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>>> [1463111.344339] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>>> [1463111.344342] [<ffffffff81095e80>] ?
>>> __migrate_swap_task.part.68+0x80/0x80
>>> [1463111.344345] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
>>> [1463111.344347] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
>>> [1463111.344351] [<ffffffff811ca500>] ?
>>> poll_select_copy_remaining+0x130/0x130
>>> [1463111.344354] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
>>> [1463111.344356] [<ffffffff8109fd00>] task_numa_fault+0x680/0x890
>>> [1463111.344360] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
>>> [1463111.344363] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
>>> [1463111.344366] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
>>> [1463111.344368] [<ffffffff811b7579>] ? do_readv_writev+0x169/0x220
>>> [1463111.344371] [<ffffffff813624f4>] ? timerqueue_del+0x24/0x70
>>> [1463111.344374] [<ffffffff8108baa6>] ? __remove_hrtimer+0x46/0xa0
>>> [1463111.344377] [<ffffffff8108bec8>] ? hrtimer_try_to_cancel+0x48/0xe0
>>> [1463111.344380] [<ffffffff81068b53>] ? do_setitimer+0xe3/0x2a0
>>> [1463111.344382] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
>>> [1463111.344385] [<ffffffff81794348>] page_fault+0x28/0x30
>>> [1463111.344386] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
>>> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
>>> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
>>> 89 76
>>> [root] [ /home/arh ]
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* (unknown),
From: DeadManMoving @ 2015-02-18 19:42 UTC (permalink / raw)
To: linux-raid
unsubscribe linux-raid
^ permalink raw reply
* Re: [PATCH 0/1] RFC: Use /dev/md/X as default name
From: Jes Sorensen @ 2015-02-18 18:12 UTC (permalink / raw)
To: neilb; +Cc: linux-raid
In-Reply-To: <1423684389-10977-1-git-send-email-Jes.Sorensen@redhat.com>
Jes.Sorensen@redhat.com writes:
> From: Jes Sorensen <Jes.Sorensen@redhat.com>
>
> I have received some issues for when creating an array using a
> /dev/mdX name, the matching symlink in /dev/md/X isn't
> created. Whereas if you create /dev/md/X, /dev/mdX is created
> automatically.
>
> I was trying to see if there was a better way of dealing with this,
> but I couldn't find one. If you have suggestions for a better solution
> I am all ears.
>
> Thoughts?
Hi Neil,
Any thoughts on this one?
Cheers,
Jes
>
> Jes
>
> Jes Sorensen (1):
> mddev_create(): choose /dev/md/ name over /dev/mdX
>
> mdopen.c | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
^ permalink raw reply
* Re: Potential Bug in "echo 0 > /dev/md0"
From: Alireza Haghdoost @ 2015-02-18 16:25 UTC (permalink / raw)
To: Jes Sorensen; +Cc: Linux RAID, Neil Brown
In-Reply-To: <wrfjtwyjb3b1.fsf@redhat.com>
Here you are:
[root] [ /home/arh ]
# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5]
[raid4] [multipath] [faulty]
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
314374656 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
[root] [ /home/arh ]
# uname -a
Linux mist01-umh 3.13.0 #47 SMP Sun Feb 1 10:27:24 CST 2015 x86_64
x86_64 x86_64 GNU/Linux
[root] [ /home/arh ]
# cat /etc/issue
Ubuntu 14.04.1 LTS \n \l
[root] [ /home/arh ]
# mdadm --version
mdadm - v3.2.5 - 18th May 2012
[root] [ /home/arh ]
# mdadm --detail /dev/md0
<No output, Freezes since kernel is crashed !>
On Wed, Feb 18, 2015 at 10:19 AM, Jes Sorensen <Jes.Sorensen@redhat.com> wrote:
> Alireza Haghdoost <alireza@cs.umn.edu> writes:
>> I understand this is not the right way to talk with md device but my
>> understanding is that if some one run this command by mistake (or
>> vandalism) , it should not results a kernel crash:
>>
>> [root] [ /home/arh ]
>> # echo 0 > /dev/md0
>
> It shouldn't, however before anyone can debug this, you need to provide
> a proper bug report with information about the kernel version you are
> running, the configuration of /dev/md0 etc.
>
> Please include 'cat /proc/mdstat' output with it.
>
> JEs
>
>>
>> [root] [ /home/arh ]
>> # dmesg
>> [1463111.320277] BUG: soft lockup - CPU#4 stuck for 22s! [whoopsie:1829]
>> [1463111.320284] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
>> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
>> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
>> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
>> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
>> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
>> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
>> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
>> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
>> pps_core
>> [1463111.320326] CPU: 4 PID: 1829 Comm: whoopsie Tainted: G D W
>> 3.13.0Write-Hole-Monitor #47
>> [1463111.320328] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
>> 2.1.3 05/21/2014
>> [1463111.320329] task: ffff8801a425c7d0 ti: ffff8801a3e04000 task.ti:
>> ffff8801a3e04000
>> [1463111.320331] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
>> smp_call_function_single+0xc6/0x190
>> [1463111.320335] RSP: 0000:ffff8801a3e05a60 EFLAGS: 00000202
>> [1463111.320336] RAX: 0000000000000001 RBX: ffffffff813624f4 RCX:
>> 0000000000000000
>> [1463111.320338] RDX: ffff8801a3e05ad8 RSI: ffff8801a9a54e80 RDI:
>> 0000000000000001
>> [1463111.320339] RBP: ffff8801a3e05ac8 R08: ffff8801a3e05b70 R09:
>> 0000000000000000
>> [1463111.320341] R10: 0000000000000000 R11: 0000000000000000 R12:
>> ffff88031e6bec80
>> [1463111.320342] R13: ffff880035d70700 R14: ffff8801a9a54400 R15:
>> ffff88031e6bec80
>> [1463111.320344] FS: 00007f9ae4924840(0000) GS:ffff8801a9a40000(0000)
>> knlGS:0000000000000000
>> [1463111.320346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [1463111.320347] CR2: 00007f9ae43bc280 CR3: 00000001a3adc000 CR4:
>> 00000000000407e0
>> [1463111.320349] Stack:
>> [1463111.320350] ffff8801a3e05af8 ffffffff8178f5ae 0000000000000000
>> 0000000000000000
>> [1463111.320353] 0000000000000000 0000000000000000 0000000000000000
>> ffff8801a9a54480
>> [1463111.320357] 0000000000000002 0000000000000001 ffff8801a3e05b70
>> 0000000000000004
>> [1463111.320360] Call Trace:
>> [1463111.320364] [<ffffffff8178f5ae>] ?
>> schedule_hrtimeout_range_clock+0xce/0x170
>> [1463111.320367] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
>> [1463111.320369] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>> [1463111.320372] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>> [1463111.320375] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
>> [1463111.320378] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
>> [1463111.320381] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
>> [1463111.320384] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
>> [1463111.320386] [<ffffffff8109f8ef>] task_numa_fault+0x26f/0x890
>> [1463111.320390] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
>> [1463111.320394] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
>> [1463111.320396] [<ffffffff8117bc30>] ? change_protection+0x690/0x720
>> [1463111.320399] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
>> [1463111.320403] [<ffffffff81191b0b>] ? change_prot_numa+0x1b/0x40
>> [1463111.320405] [<ffffffff8109c806>] ? task_numa_work+0x266/0x300
>> [1463111.320408] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
>> [1463111.320412] [<ffffffff81012e67>] ? do_notify_resume+0x97/0xb0
>> [1463111.320414] [<ffffffff81794348>] page_fault+0x28/0x30
>> [1463111.320415] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
>> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
>> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
>> 89 76
>> [1463111.344253] BUG: soft lockup - CPU#6 stuck for 22s! [Xorg:1842]
>> [1463111.344254] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
>> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
>> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
>> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
>> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
>> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
>> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
>> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
>> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
>> pps_core
>> [1463111.344298] CPU: 6 PID: 1842 Comm: Xorg Tainted: G D W
>> 3.13.0Write-Hole-Monitor #47
>> [1463111.344299] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
>> 2.1.3 05/21/2014
>> [1463111.344302] task: ffff8800367b8000 ti: ffff880035f9e000 task.ti:
>> ffff880035f9e000
>> [1463111.344303] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
>> smp_call_function_single+0xc6/0x190
>> [1463111.344306] RSP: 0000:ffff880035f9fa60 EFLAGS: 00003202
>> [1463111.344308] RAX: 0000000000000001 RBX: 0000000000400000 RCX:
>> 0000000000000000
>> [1463111.344309] RDX: ffff880035f9fad8 RSI: ffff8801a9a74e80 RDI:
>> 0000000000000003
>> [1463111.344311] RBP: ffff880035f9fac8 R08: ffff880035f9fb70 R09:
>> 0000000000000000
>> [1463111.344312] R10: 0000000000000000 R11: 0000000000000000 R12:
>> 00000000003fc02a
>> [1463111.344313] R13: ffffffff811ca5fc R14: ffff880035f9f9f0 R15:
>> ffffffff81150fa3
>> [1463111.344315] FS: 00007f2f338019c0(0000) GS:ffff8801a9a60000(0000)
>> knlGS:0000000000000000
>> [1463111.344317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [1463111.344318] CR2: 00007f2f344fdedc CR3: 000000031dd2b000 CR4:
>> 00000000000407e0
>> [1463111.344320] Stack:
>> [1463111.344321] ffff880035f9fdf8 ffff880035f9fdf0 0000000000000000
>> 0000000000000000
>> [1463111.344324] 0000000000000000 0000000000000000 0000000000000000
>> ffff8801a9a74480
>> [1463111.344327] 0000000000000002 0000000000000003 ffff880035f9fb70
>> 0000000000000006
>> [1463111.344330] Call Trace:
>> [1463111.344334] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
>> [1463111.344336] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>> [1463111.344339] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
>> [1463111.344342] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
>> [1463111.344345] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
>> [1463111.344347] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
>> [1463111.344351] [<ffffffff811ca500>] ? poll_select_copy_remaining+0x130/0x130
>> [1463111.344354] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
>> [1463111.344356] [<ffffffff8109fd00>] task_numa_fault+0x680/0x890
>> [1463111.344360] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
>> [1463111.344363] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
>> [1463111.344366] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
>> [1463111.344368] [<ffffffff811b7579>] ? do_readv_writev+0x169/0x220
>> [1463111.344371] [<ffffffff813624f4>] ? timerqueue_del+0x24/0x70
>> [1463111.344374] [<ffffffff8108baa6>] ? __remove_hrtimer+0x46/0xa0
>> [1463111.344377] [<ffffffff8108bec8>] ? hrtimer_try_to_cancel+0x48/0xe0
>> [1463111.344380] [<ffffffff81068b53>] ? do_setitimer+0xe3/0x2a0
>> [1463111.344382] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
>> [1463111.344385] [<ffffffff81794348>] page_fault+0x28/0x30
>> [1463111.344386] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
>> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
>> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
>> 89 76
>> [root] [ /home/arh ]
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Potential Bug in "echo 0 > /dev/md0"
From: Jes Sorensen @ 2015-02-18 16:19 UTC (permalink / raw)
To: Alireza Haghdoost; +Cc: Linux RAID, Neil Brown
In-Reply-To: <CAB-428mj-P-+M0MnKhmojRe9ZQQzsSTBfCSS91ZK5KRy5p1mSg@mail.gmail.com>
Alireza Haghdoost <alireza@cs.umn.edu> writes:
> I understand this is not the right way to talk with md device but my
> understanding is that if some one run this command by mistake (or
> vandalism) , it should not results a kernel crash:
>
> [root] [ /home/arh ]
> # echo 0 > /dev/md0
It shouldn't, however before anyone can debug this, you need to provide
a proper bug report with information about the kernel version you are
running, the configuration of /dev/md0 etc.
Please include 'cat /proc/mdstat' output with it.
JEs
>
> [root] [ /home/arh ]
> # dmesg
> [1463111.320277] BUG: soft lockup - CPU#4 stuck for 22s! [whoopsie:1829]
> [1463111.320284] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
> pps_core
> [1463111.320326] CPU: 4 PID: 1829 Comm: whoopsie Tainted: G D W
> 3.13.0Write-Hole-Monitor #47
> [1463111.320328] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
> 2.1.3 05/21/2014
> [1463111.320329] task: ffff8801a425c7d0 ti: ffff8801a3e04000 task.ti:
> ffff8801a3e04000
> [1463111.320331] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
> smp_call_function_single+0xc6/0x190
> [1463111.320335] RSP: 0000:ffff8801a3e05a60 EFLAGS: 00000202
> [1463111.320336] RAX: 0000000000000001 RBX: ffffffff813624f4 RCX:
> 0000000000000000
> [1463111.320338] RDX: ffff8801a3e05ad8 RSI: ffff8801a9a54e80 RDI:
> 0000000000000001
> [1463111.320339] RBP: ffff8801a3e05ac8 R08: ffff8801a3e05b70 R09:
> 0000000000000000
> [1463111.320341] R10: 0000000000000000 R11: 0000000000000000 R12:
> ffff88031e6bec80
> [1463111.320342] R13: ffff880035d70700 R14: ffff8801a9a54400 R15:
> ffff88031e6bec80
> [1463111.320344] FS: 00007f9ae4924840(0000) GS:ffff8801a9a40000(0000)
> knlGS:0000000000000000
> [1463111.320346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [1463111.320347] CR2: 00007f9ae43bc280 CR3: 00000001a3adc000 CR4:
> 00000000000407e0
> [1463111.320349] Stack:
> [1463111.320350] ffff8801a3e05af8 ffffffff8178f5ae 0000000000000000
> 0000000000000000
> [1463111.320353] 0000000000000000 0000000000000000 0000000000000000
> ffff8801a9a54480
> [1463111.320357] 0000000000000002 0000000000000001 ffff8801a3e05b70
> 0000000000000004
> [1463111.320360] Call Trace:
> [1463111.320364] [<ffffffff8178f5ae>] ?
> schedule_hrtimeout_range_clock+0xce/0x170
> [1463111.320367] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
> [1463111.320369] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.320372] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.320375] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
> [1463111.320378] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
> [1463111.320381] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
> [1463111.320384] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
> [1463111.320386] [<ffffffff8109f8ef>] task_numa_fault+0x26f/0x890
> [1463111.320390] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
> [1463111.320394] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
> [1463111.320396] [<ffffffff8117bc30>] ? change_protection+0x690/0x720
> [1463111.320399] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
> [1463111.320403] [<ffffffff81191b0b>] ? change_prot_numa+0x1b/0x40
> [1463111.320405] [<ffffffff8109c806>] ? task_numa_work+0x266/0x300
> [1463111.320408] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
> [1463111.320412] [<ffffffff81012e67>] ? do_notify_resume+0x97/0xb0
> [1463111.320414] [<ffffffff81794348>] page_fault+0x28/0x30
> [1463111.320415] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
> 89 76
> [1463111.344253] BUG: soft lockup - CPU#6 stuck for 22s! [Xorg:1842]
> [1463111.344254] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
> bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
> coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
> crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
> acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
> gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
> wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
> lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
> pps_core
> [1463111.344298] CPU: 6 PID: 1842 Comm: Xorg Tainted: G D W
> 3.13.0Write-Hole-Monitor #47
> [1463111.344299] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
> 2.1.3 05/21/2014
> [1463111.344302] task: ffff8800367b8000 ti: ffff880035f9e000 task.ti:
> ffff880035f9e000
> [1463111.344303] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
> smp_call_function_single+0xc6/0x190
> [1463111.344306] RSP: 0000:ffff880035f9fa60 EFLAGS: 00003202
> [1463111.344308] RAX: 0000000000000001 RBX: 0000000000400000 RCX:
> 0000000000000000
> [1463111.344309] RDX: ffff880035f9fad8 RSI: ffff8801a9a74e80 RDI:
> 0000000000000003
> [1463111.344311] RBP: ffff880035f9fac8 R08: ffff880035f9fb70 R09:
> 0000000000000000
> [1463111.344312] R10: 0000000000000000 R11: 0000000000000000 R12:
> 00000000003fc02a
> [1463111.344313] R13: ffffffff811ca5fc R14: ffff880035f9f9f0 R15:
> ffffffff81150fa3
> [1463111.344315] FS: 00007f2f338019c0(0000) GS:ffff8801a9a60000(0000)
> knlGS:0000000000000000
> [1463111.344317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [1463111.344318] CR2: 00007f2f344fdedc CR3: 000000031dd2b000 CR4:
> 00000000000407e0
> [1463111.344320] Stack:
> [1463111.344321] ffff880035f9fdf8 ffff880035f9fdf0 0000000000000000
> 0000000000000000
> [1463111.344324] 0000000000000000 0000000000000000 0000000000000000
> ffff8801a9a74480
> [1463111.344327] 0000000000000002 0000000000000003 ffff880035f9fb70
> 0000000000000006
> [1463111.344330] Call Trace:
> [1463111.344334] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
> [1463111.344336] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.344339] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
> [1463111.344342] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
> [1463111.344345] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
> [1463111.344347] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
> [1463111.344351] [<ffffffff811ca500>] ? poll_select_copy_remaining+0x130/0x130
> [1463111.344354] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
> [1463111.344356] [<ffffffff8109fd00>] task_numa_fault+0x680/0x890
> [1463111.344360] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
> [1463111.344363] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
> [1463111.344366] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
> [1463111.344368] [<ffffffff811b7579>] ? do_readv_writev+0x169/0x220
> [1463111.344371] [<ffffffff813624f4>] ? timerqueue_del+0x24/0x70
> [1463111.344374] [<ffffffff8108baa6>] ? __remove_hrtimer+0x46/0xa0
> [1463111.344377] [<ffffffff8108bec8>] ? hrtimer_try_to_cancel+0x48/0xe0
> [1463111.344380] [<ffffffff81068b53>] ? do_setitimer+0xe3/0x2a0
> [1463111.344382] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
> [1463111.344385] [<ffffffff81794348>] page_fault+0x28/0x30
> [1463111.344386] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
> 4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
> 0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
> 89 76
> [root] [ /home/arh ]
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Potential Bug in "echo 0 > /dev/md0"
From: Alireza Haghdoost @ 2015-02-18 16:11 UTC (permalink / raw)
To: Linux RAID; +Cc: Neil Brown
I understand this is not the right way to talk with md device but my
understanding is that if some one run this command by mistake (or
vandalism) , it should not results a kernel crash:
[root] [ /home/arh ]
# echo 0 > /dev/md0
[root] [ /home/arh ]
# dmesg
[1463111.320277] BUG: soft lockup - CPU#4 stuck for 22s! [whoopsie:1829]
[1463111.320284] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
pps_core
[1463111.320326] CPU: 4 PID: 1829 Comm: whoopsie Tainted: G D W
3.13.0Write-Hole-Monitor #47
[1463111.320328] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
2.1.3 05/21/2014
[1463111.320329] task: ffff8801a425c7d0 ti: ffff8801a3e04000 task.ti:
ffff8801a3e04000
[1463111.320331] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
smp_call_function_single+0xc6/0x190
[1463111.320335] RSP: 0000:ffff8801a3e05a60 EFLAGS: 00000202
[1463111.320336] RAX: 0000000000000001 RBX: ffffffff813624f4 RCX:
0000000000000000
[1463111.320338] RDX: ffff8801a3e05ad8 RSI: ffff8801a9a54e80 RDI:
0000000000000001
[1463111.320339] RBP: ffff8801a3e05ac8 R08: ffff8801a3e05b70 R09:
0000000000000000
[1463111.320341] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff88031e6bec80
[1463111.320342] R13: ffff880035d70700 R14: ffff8801a9a54400 R15:
ffff88031e6bec80
[1463111.320344] FS: 00007f9ae4924840(0000) GS:ffff8801a9a40000(0000)
knlGS:0000000000000000
[1463111.320346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1463111.320347] CR2: 00007f9ae43bc280 CR3: 00000001a3adc000 CR4:
00000000000407e0
[1463111.320349] Stack:
[1463111.320350] ffff8801a3e05af8 ffffffff8178f5ae 0000000000000000
0000000000000000
[1463111.320353] 0000000000000000 0000000000000000 0000000000000000
ffff8801a9a54480
[1463111.320357] 0000000000000002 0000000000000001 ffff8801a3e05b70
0000000000000004
[1463111.320360] Call Trace:
[1463111.320364] [<ffffffff8178f5ae>] ?
schedule_hrtimeout_range_clock+0xce/0x170
[1463111.320367] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
[1463111.320369] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
[1463111.320372] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
[1463111.320375] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
[1463111.320378] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
[1463111.320381] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
[1463111.320384] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
[1463111.320386] [<ffffffff8109f8ef>] task_numa_fault+0x26f/0x890
[1463111.320390] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
[1463111.320394] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
[1463111.320396] [<ffffffff8117bc30>] ? change_protection+0x690/0x720
[1463111.320399] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
[1463111.320403] [<ffffffff81191b0b>] ? change_prot_numa+0x1b/0x40
[1463111.320405] [<ffffffff8109c806>] ? task_numa_work+0x266/0x300
[1463111.320408] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
[1463111.320412] [<ffffffff81012e67>] ? do_notify_resume+0x97/0xb0
[1463111.320414] [<ffffffff81794348>] page_fault+0x28/0x30
[1463111.320415] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
89 76
[1463111.344253] BUG: soft lockup - CPU#6 stuck for 22s! [Xorg:1842]
[1463111.344254] Modules linked in: ib_iser rdma_cm ib_addr iw_cm
ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi bnep rfcomm
bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp nfsd
coretemp kvm_intel kvm joydev auth_rpcgss crct10dif_pclmul
crc32_pclmul gpio_ich mei_me ghash_clmulni_intel nfs_acl
acpi_power_meter aesni_intel aes_x86_64 nfs glue_helper mei lrw
gf128mul lpc_ich ablk_helper cryptd sb_edac dcdbas edac_core shpchp
wmi mac_hid lockd sunrpc parport_pc ppdev ipmi_si ipmi_devintf fscache
lp parport hid_generic ixgbe tg3 usbhid dca ahci mdio hid libahci ptp
pps_core
[1463111.344298] CPU: 6 PID: 1842 Comm: Xorg Tainted: G D W
3.13.0Write-Hole-Monitor #47
[1463111.344299] Hardware name: Dell Inc. PowerEdge R420/0JD6X3, BIOS
2.1.3 05/21/2014
[1463111.344302] task: ffff8800367b8000 ti: ffff880035f9e000 task.ti:
ffff880035f9e000
[1463111.344303] RIP: 0010:[<ffffffff810d8446>] [<ffffffff810d8446>]
smp_call_function_single+0xc6/0x190
[1463111.344306] RSP: 0000:ffff880035f9fa60 EFLAGS: 00003202
[1463111.344308] RAX: 0000000000000001 RBX: 0000000000400000 RCX:
0000000000000000
[1463111.344309] RDX: ffff880035f9fad8 RSI: ffff8801a9a74e80 RDI:
0000000000000003
[1463111.344311] RBP: ffff880035f9fac8 R08: ffff880035f9fb70 R09:
0000000000000000
[1463111.344312] R10: 0000000000000000 R11: 0000000000000000 R12:
00000000003fc02a
[1463111.344313] R13: ffffffff811ca5fc R14: ffff880035f9f9f0 R15:
ffffffff81150fa3
[1463111.344315] FS: 00007f2f338019c0(0000) GS:ffff8801a9a60000(0000)
knlGS:0000000000000000
[1463111.344317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1463111.344318] CR2: 00007f2f344fdedc CR3: 000000031dd2b000 CR4:
00000000000407e0
[1463111.344320] Stack:
[1463111.344321] ffff880035f9fdf8 ffff880035f9fdf0 0000000000000000
0000000000000000
[1463111.344324] 0000000000000000 0000000000000000 0000000000000000
ffff8801a9a74480
[1463111.344327] 0000000000000002 0000000000000003 ffff880035f9fb70
0000000000000006
[1463111.344330] Call Trace:
[1463111.344334] [<ffffffff810f1e8c>] stop_two_cpus+0x14c/0x1a0
[1463111.344336] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
[1463111.344339] [<ffffffff810f18a0>] ? cpu_stop_should_run+0x50/0x50
[1463111.344342] [<ffffffff81095e80>] ? __migrate_swap_task.part.68+0x80/0x80
[1463111.344345] [<ffffffff8109624a>] migrate_swap+0x8a/0xa0
[1463111.344347] [<ffffffff8109e133>] task_numa_migrate+0x1d3/0x480
[1463111.344351] [<ffffffff811ca500>] ? poll_select_copy_remaining+0x130/0x130
[1463111.344354] [<ffffffff8109e433>] numa_migrate_preferred+0x53/0x60
[1463111.344356] [<ffffffff8109fd00>] task_numa_fault+0x680/0x890
[1463111.344360] [<ffffffff8117269e>] do_numa_page+0x13e/0x1a0
[1463111.344363] [<ffffffff81173823>] handle_mm_fault+0x5e3/0xe30
[1463111.344366] [<ffffffff81797ac4>] __do_page_fault+0x154/0x570
[1463111.344368] [<ffffffff811b7579>] ? do_readv_writev+0x169/0x220
[1463111.344371] [<ffffffff813624f4>] ? timerqueue_del+0x24/0x70
[1463111.344374] [<ffffffff8108baa6>] ? __remove_hrtimer+0x46/0xa0
[1463111.344377] [<ffffffff8108bec8>] ? hrtimer_try_to_cancel+0x48/0xe0
[1463111.344380] [<ffffffff81068b53>] ? do_setitimer+0xe3/0x2a0
[1463111.344382] [<ffffffff81797efa>] do_page_fault+0x1a/0x70
[1463111.344385] [<ffffffff81794348>] page_fault+0x28/0x30
[1463111.344386] Code: 00 00 00 85 c9 48 8d 74 24 10 75 1b 48 c7 c6 80
4e 01 00 65 48 03 34 25 c8 dc 00 00 0f b7 46 20 a8 01 74 0b 90 f3 90
0f b7 46 20 <a8> 01 75 f6 83 c8 01 66 89 46 20 0f ae f0 48 89 56 18 4c
89 76
[root] [ /home/arh ]
^ permalink raw reply
* help with the little script (erc timout fix)
From: Chris @ 2015-02-18 15:04 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20150217104906.62d36c62@notabene.brown>
Hello,
by adapting what I could find, I compiled the following short snippet now.
Could list members please look at this novice code and suggest a way to
determine the containing disk device $HDD_DEV from the parition/disk,
before I dare to test this.
In udev-md-raid-assembly.rules, below LABEL="md_inc" (section only handling
all md suppported devices) add:
# fix timouts for redundant raids, if possible
IMPORT{program}="BINDIR/mdadm --examine --export $tempnode"
TEST="/usr/sbin/smartctl", ENV{MD_LEVEL}=="raid[1-9]*",
RUN+="BINDIR/mdadm-erc-timout-fix.sh $tempnode"
And in a new mdadm-erc-timout-fix.sh file implement:
#! /bin/sh
HDD_DEV= $1 somehow stipping off the tailing numbers?
if smartctl -l scterc ${HDD_DEV} | grep -q Disabled ; then
/usr/sbin/smartctl -l scterc,70,70 ${HDD_DEV}
else
if ! smartctl -l scterc ${HDD_DEV} | grep -q seconds ; then
echo 180 >/sys/block/${HDD_DEV}/device/timeout
fi
fi
Correct execution during boot would seem to require that distro
package managers hook smartctl and the script into the initramfs
generation.
Regards,
Chris
^ permalink raw reply
* Re: RAID1 might_sleep() warning on 3.19-rc7
From: Peter Zijlstra @ 2015-02-18 13:47 UTC (permalink / raw)
To: NeilBrown; +Cc: Tony Battersby, linux-raid, lkml, axboe, Linus Torvalds
In-Reply-To: <20150218120936.3c7beed2@notabene.brown>
On Wed, Feb 18, 2015 at 12:09:36PM +1100, NeilBrown wrote:
> On Fri, 13 Feb 2015 15:48:03 +0100 Peter Zijlstra <peterz@infradead.org>
> wrote:
>
> > On Fri, Feb 13, 2015 at 11:27:46AM +0100, Peter Zijlstra wrote:
> >
> > > > I've moved blk_flush_plug to the beginning of the function.
> > >
> > > > I wondered if it really make sense to call blk_flush_plug with nr_iowait
> > > > elevated and delayacct_blkio active. blk_flush_plug() could call schedule()
> > > > for non-"io" reasons and maybe that could upset stuff???
> > >
> > > Yeah, good question that. Lemme ponder that a bit.
> >
> > Yes, I thikn your version makes most sense as, you say, even regular
> > schedule() call nested in my version would go towards blk delayacct --
> > and I doubt that was the intent; even though the current kernel works
> > that way.
> >
> > I'll move the now rudimentary io_schedule() into sched.h as an inline.
>
> Thanks.
>
> Are you OK with this going to -stable for 3.19?
OK, lets do that.
^ permalink raw reply
* mdadm monitoring for RAID-1 - no events after pulling out all disks.
From: Iakiv Shchelkunov @ 2015-02-18 11:42 UTC (permalink / raw)
To: linux-raid
Hello everyone. Please, help me with one mdadm monitoring issue.
I use mdadm 3.2.5 on Debian 7.6. Besides system HDD, I have 2 SSDs
with the same partition table.
I have an array RAID1 over those SSDs with the same partitions in it.
As I need to monitor mdadm errors, I have a tool for monitoring.
So, I do next:
mdadm –assemble /dev/md1 /dev/sda1 /dev/sdb1
//run the md which was created earlier
mdadm --monitor --daemonize --scan --syslog
--program=/usr/local/mdmonitor //monitoring
The md /dev/md1 is running, everything is okey.
When I pull out the first SSD and try to write anything on active
array, mdadm calls my program with parameters “Fail” (or “FailSpare”),
the md device name /dev/md1 and partition name, as it’s supposed to
do.
But when I pull out second SSD and there is no one SSD is present,
then try to write data on the md, mdadm doesn’t run my program to
inform about fail.
Also /proc/mdstat isn’t changed – the array is still ‘active’, even
without any physically present SSD. Of course, I can’t do any write
operations, I’ve got an error trying to do that, but I have no event
from mdadm monitoring.
Has anybody faced with similar troubles? Is there any solution?
Thanks in advance for your time, Iakiv.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Chris @ 2015-02-18 11:04 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAJCQCtTv-HMX77GWqA+_1rfkWodsTETp_Y3w5En-N4nGCymWsA@mail.gmail.com>
>
Hello all,
the discussion about SCTERC boils down to letting the drive attempt ERC a
little more or less. For any given disk experience seems to tell the slight
difference is, that if ERC is allowed longer you may see the first
unrecoverable erros (UREs) just a little (maybe only a month) later.
UREs are inevitable. Thus, if I run a filesystem on just a single drive it
will get corrupted at some point, nothing to do about it.
Wait, except..., use a redundant raid! And here it makes a lot of a
difference that the drive's ERC actually terminates before the controller
timeout, to not loose all your redundacy again and be in hight risk of UREs
showing up during the re-sync.
So for a proper comparison we need to look at the difference it makes in the
usage scenarios (error delay vs. loosing redundant error resilence + URE
triggering), not at the single recoverable/unrecoverable error incidence. It
looks to me, that it makes a lot of a differnce to redundant raids and no
qualitative difference to single disk filesystems.
And we need to keep in mind that single disk filesystems do also depend on
the disk to stop grinding away with ERC attempts before the controller
timout. Otherwise disk reset may make the system clear buffers and loose
open files? Without prolonging the linux default controller timout, SCTERC
can prevent that where supported.
> in any case the proper place to change the default kernel command
> timer value is in the kernel, not with a udev rule.
Right. And as you write increasing the controller timout has clear downsides.
Noteing as well, as long as the proposed script (a temporary safety measure)
maximizes the controller timeout to remedy for disks that don's support
SCTERC, this would even fix the timout mismatch for single disk filesystems.
(Letting the controller wait until the disk finally succeeds or fails its
recovery attempts.)
So the proposed script actually provides a case that brings benefit for
raid0 setups as well (as long as the linux default is not adaptive to the
disk parameters), but increasing the controller timout in all cases would
introduce long and unreported i/o blocking into all redundant setups.
> I don't know if a udev rule can say "If the drive exclusively uses md,
> lvm, btrfs, zfs raid1, 4+ or nested of those, and if the drive does
> not support configurable SCT ERC, then change the kernel command timer
> for those devices to ~120 seconds" then that might be a plausible
> solution to use consumer drives the manufacturer rather explicitly
> proscribes from use in raid...
The script called by the udev rule could do that, but can be kept as simple
as proposed, and can set SCTERC regardles, because setting SCTERC below the
controller timout makes a qualitative difference in running the redundant
arrays and a marginal difference in running non-redundant filesystems. (And
nevertheless, set long controller timout for devices that don's support SCTERC.)
After all, this looks like a quite simple change is appropriate:
In udev-md-raid-assembly.rules, below LABEL="md_inc" (only handling all md
suppported devices) add one rule:
# fix timouts for redundant raids, if possible
TEST="/usr/sbin/smartctl", ENV{MD_LEVEL}=="raid[1-9]*",
RUN+="/usr/bin/mdadm-erc-timout-fix"
And in a new /usr/bin/mdadm-erc-timout-fix file implement:
if smartctl -l scterc ${HDD_DEV} returns "Disabled"
/usr/sbin/smartctl -l scterc,70,70 ${HDD_DEV}
else
if smartctl -l scterc ${HDD_DEV} does not return "seconds"
echo 180 >/sys/block/${HDD_DEV}/device/timeout
Regards,
Chris
^ permalink raw reply
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Jes Sorensen @ 2015-02-18 5:05 UTC (permalink / raw)
To: NeilBrown; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <20150218121656.0584e09d@notabene.brown>
NeilBrown <neilb@suse.de> writes:
> On Tue, 17 Feb 2015 20:07:24 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
> wrote:
>
>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> > NeilBrown <neilb@suse.de> writes:
>> >> On Tue, 17 Feb 2015 19:03:30 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
>> >> wrote:
>> >>
>> >>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> >>> > Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> >>> >> NeilBrown <neilb@suse.de> writes:
>> >>> >>> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P
>> >>> >>> <pmanibalan@amiindia.co.in>
>> >>> >>> wrote:
>> >>> >>>
>> >>> >>>> Dear All,
>> >>> >>>> Any updates on this issue.
>> >>> >>>
>> >>> >>> Probably the same as:
>> >>> >>>
>> >>> >>> http://marc.info/?l=linux-raid&m=142283560704091&w=2
>> >>> >>
>> >>> >> Hi Neil,
>> >>> >>
>> >>> >> I ran some tests on this one against the latest Linus' tree as of today
>> >>> >> (1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
>> >>> >> your pending 3.20 patches.
>> >>> >>
>> >>> >> I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
>> >>> >> run fio on top of a device while it is resyncing and I fail one of the
>> >>> >> devices.
>> >>> >
>> >>> > Since Manibalan mentioned this issue wasn't present in earlier kernels,
>> >>> > I started trying to track down what change caused it.
>> >>> >
>> >>> > So far I have been able to reproduce the hang as far back as 3.10.
>> >>>
>> >>> After a lot of bisecting I finally traced the issue back to this commit:
>> >>>
>> >>> a7854487cd7128a30a7f4f5259de9f67d5efb95f is the first bad commit
>> >>> commit a7854487cd7128a30a7f4f5259de9f67d5efb95f
>> >>> Author: Alexander Lyakas <alex.bolshoy@gmail.com>
>> >>> Date: Thu Oct 11 13:50:12 2012 +1100
>> >>>
>> >>> md: When RAID5 is dirty, force reconstruct-write instead of
>> >>> read-modify-write.
>> >>>
>> >>> Signed-off-by: Alex Lyakas <alex@zadarastorage.com>
>> >>> Suggested-by: Yair Hershko <yair@zadarastorage.com>
>> >>> Signed-off-by: NeilBrown <neilb@suse.de>
>> >>>
>> >>> If I revert that one I cannot reproduce the hang, applying it reproduces
>> >>> the hang consistently.
>> >>
>> >> Thanks for all the research!
>> >>
>> >> That is consistent with what you already reported.
>> >> You noted that it doesn't affect RAID6, and RAID6 doesn't have an RMW cycle.
>> >>
>> >> Also, one of the early emails from Manibalan contained:
>> >>
>> >> handling stripe 273480328, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1
>> >> , check:0, reconstruct:0
>> >> check 5: state 0x10 read (null) write (null) written (null)
>> >> check 4: state 0x11 read (null) write (null) written (null)
>> >> check 3: state 0x0 read (null) write (null) written (null)
>> >> check 2: state 0x11 read (null) write (null) written (null)
>> >> check 1: state 0x11 read (null) write (null) written (null)
>> >> check 0: state 0x18 read (null) write ffff8808029b6b00 written (null)
>> >> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1
>> >> force RCW max_degraded=1, recovery_cp=7036944 sh->sector=273480328
>> >> for sector 273480328, rmw=2 rcw=1
>> >>
>> >> So it is forcing RCW, even though a single block update is usually handled
>> >> with RMW.
>> >>
>> >> In this stripe, the parity disk is '5' and disk 3 has failed.
>> >> That means to perform an RCW, we need to read the parity block in order
>> >> to reconstruct the content of the failed disk. And if we were to do that,
>> >> we may as well just do an RMW.
>> >>
>> >> So I think the correct fix would be to only force RCW when the array
>> >> is not degraded.
>> >>
>> >> So something like this:
>> >>
>> >> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
>> >> index aa76865b804b..fa8f8b94bfa8 100644
>> >> --- a/drivers/md/raid5.c
>> >> +++ b/drivers/md/raid5.c
>> >> @@ -3170,7 +3170,8 @@ static void handle_stripe_dirtying(struct r5conf *conf,
>> >> * generate correct data from the parity.
>> >> */
>> >> if (conf->max_degraded == 2 ||
>> >> - (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
>> >> + (recovery_cp < MaxSector && sh->sector >= recovery_cp &&
>> >> + s->failed == 0)) {
>> >> /* Calculate the real rcw later - for now make it
>> >> * look like rcw is cheaper
>> >> */
>> >>
>> >>
>> >> I think reverting the whole patch is not necessary and discards useful
>> >> functionality while the array is not degraded.
>> >>
>> >> Can you test this patch please?
>> >
>> > Actually I just tried this one - I was on my way home and grabbed food
>> > on the way, and thought there was a better solution than to revert.
>> >
>> > I'll give your solution a spin too.
>>
>> I tried your patch, as expected that also resolves the problem. Not sure
>> which solution is better, so I'll let you pick.
>
> Thanks!
>
>>
>> Note whichever patch you choose it is applicable for stable-3.6+
>
> 3.6??
>
> $ git describe --contains a7854487cd7128a30a7f4f5259
> v3.7-rc1~10^2~7
>
> so I assume 3.7.
> Doesn't apply to 3.6, so I'll assume a typo.
Too many kernels in one day, you win - 3.7 it is :)
Jes
^ permalink raw reply
* Re: [PATCH 0/3] md raid: enhancements to support the device mapper dm-raid target
From: NeilBrown @ 2015-02-18 2:03 UTC (permalink / raw)
To: heinzm; +Cc: linux-raid, dm-devel
In-Reply-To: <1423853282-6218-1-git-send-email-heinzm@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 2701 bytes --]
On Fri, 13 Feb 2015 19:47:59 +0100 heinzm@redhat.com wrote:
> From: Heinz Mauelshagen <heinzm@redhat.com>
>
> I'm enhancing the device mapper raid target (dm-raid) to take
> advantage of so far unused md raid kernel funtionality:
> takeover, reshape, resize, addition and removal of devices to/from raid sets.
>
> This series of patches remove constraints doing so.
>
>
> Patch #1:
> add 2 API functions to allow dm-raid to access the raid takeover
> and resize functionality (namely md_takeover() and md_resize());
> reshape APIs are not needed in lieu of the existing personalilty ones
>
> Patch #2:
> because device mapper core manages a request queue per mapped device
> utilizing the md make_request API to pass on bios via the dm-raid target,
> no md instance underneath it needs to manage a request queue of its own.
> Thus dm-raid can't use the md raid0 personality as is, because the latter
> accesses the request queue unconditionally in 3 places via mddev->queue
> which this patch addresses.
>
> Patch #3:
> when dm-raid processes a down takeover to raid0, it needs to destroy
> any existing bitmap, because raid0 does not require one. The patch
> exports the bitmap_destroy() API to allow dm-raid to remove bitmaps.
>
>
> Heinz Mauelshagen (3):
> md core: add 2 API functions for takeover and resize to support dm-raid
> md raid0: access mddev->queue (request queue member) conditionally
> because it is not set when accessed from dm-raid
> md bitmap: export bitmap_destroy() to support dm-raid down takover to raid0
>
> drivers/md/bitmap.c | 1 +
> drivers/md/md.c | 39 ++++++++++++++++++++++++++++++---------
> drivers/md/md.h | 3 +++
> drivers/md/raid0.c | 48 +++++++++++++++++++++++++++---------------------
> 4 files changed, 61 insertions(+), 30 deletions(-)
>
Hi Heinz,
I don't object to these patches if you will find the exported functionality
useful, but I am a little surprised by them.
I would expect that dm-raid wouldn't ask md to 'takeover' from one level to
another, but instead would
- suspend the dm device
- dismantle the array using the old level
- assemble the array using the new level
- resume the dm device
The reason md needs 'takeover' is because it doesn't have the same
device/target separation that dm does.
I was particularly surprised that you wanted to use md/raid0.c It is no
better than dm/dm-stripe.c and managing two different stripe engines under
LVM doesn't see like a good idea.
Is there some reason that I have missed which makes it easier to use
'takeover' rather than suspend/resume?
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* [GIT PULL REQUEST] some more bugfixes for md.
From: NeilBrown @ 2015-02-18 1:31 UTC (permalink / raw)
To: Linus Torvalds; +Cc: lkml, linux RAID, Nate Dailey, Jes Sorensen, Manibalan P
[-- Attachment #1: Type: text/plain, Size: 1174 bytes --]
hi Linus,
Please pull these three fixes, one tagged for -stable.
Thanks,
NeilBrown
The following changes since commit 53a6ab4d3f6d6dc87ec8f14998b4b5536ee2968c:
md/raid10: fix conversion from RAID0 to RAID10 (2015-02-12 14:09:57 +1100)
are available in the git repository at:
git://neil.brown.name/md/ tags/md/3.20-fixes
for you to fetch changes up to 26ac107378c4742978216be1005b7291b799c7b2:
md/raid5: Fix livelock when array is both resyncing and degraded. (2015-02-18 11:35:14 +1100)
----------------------------------------------------------------
3 bug md fixes for 3.20
yet-another-livelock in raid5, and a problem with write errors
to 4K-block devices.
----------------------------------------------------------------
Nate Dailey (1):
md/raid1: round up to bdev_logical_block_size in narrow_write_error
NeilBrown (2):
md/raid10: round up to bdev_logical_block_size in narrow_write_error.
md/raid5: Fix livelock when array is both resyncing and degraded.
drivers/md/raid1.c | 3 ++-
drivers/md/raid10.c | 3 ++-
drivers/md/raid5.c | 3 ++-
3 files changed, 6 insertions(+), 3 deletions(-)
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: NeilBrown @ 2015-02-18 1:16 UTC (permalink / raw)
To: Jes Sorensen; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <wrfjsie4do37.fsf@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 5172 bytes --]
On Tue, 17 Feb 2015 20:07:24 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
wrote:
> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> > NeilBrown <neilb@suse.de> writes:
> >> On Tue, 17 Feb 2015 19:03:30 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
> >> wrote:
> >>
> >>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> >>> > Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> >>> >> NeilBrown <neilb@suse.de> writes:
> >>> >>> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P <pmanibalan@amiindia.co.in>
> >>> >>> wrote:
> >>> >>>
> >>> >>>> Dear All,
> >>> >>>> Any updates on this issue.
> >>> >>>
> >>> >>> Probably the same as:
> >>> >>>
> >>> >>> http://marc.info/?l=linux-raid&m=142283560704091&w=2
> >>> >>
> >>> >> Hi Neil,
> >>> >>
> >>> >> I ran some tests on this one against the latest Linus' tree as of today
> >>> >> (1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
> >>> >> your pending 3.20 patches.
> >>> >>
> >>> >> I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
> >>> >> run fio on top of a device while it is resyncing and I fail one of the
> >>> >> devices.
> >>> >
> >>> > Since Manibalan mentioned this issue wasn't present in earlier kernels,
> >>> > I started trying to track down what change caused it.
> >>> >
> >>> > So far I have been able to reproduce the hang as far back as 3.10.
> >>>
> >>> After a lot of bisecting I finally traced the issue back to this commit:
> >>>
> >>> a7854487cd7128a30a7f4f5259de9f67d5efb95f is the first bad commit
> >>> commit a7854487cd7128a30a7f4f5259de9f67d5efb95f
> >>> Author: Alexander Lyakas <alex.bolshoy@gmail.com>
> >>> Date: Thu Oct 11 13:50:12 2012 +1100
> >>>
> >>> md: When RAID5 is dirty, force reconstruct-write instead of
> >>> read-modify-write.
> >>>
> >>> Signed-off-by: Alex Lyakas <alex@zadarastorage.com>
> >>> Suggested-by: Yair Hershko <yair@zadarastorage.com>
> >>> Signed-off-by: NeilBrown <neilb@suse.de>
> >>>
> >>> If I revert that one I cannot reproduce the hang, applying it reproduces
> >>> the hang consistently.
> >>
> >> Thanks for all the research!
> >>
> >> That is consistent with what you already reported.
> >> You noted that it doesn't affect RAID6, and RAID6 doesn't have an RMW cycle.
> >>
> >> Also, one of the early emails from Manibalan contained:
> >>
> >> handling stripe 273480328, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1
> >> , check:0, reconstruct:0
> >> check 5: state 0x10 read (null) write (null) written (null)
> >> check 4: state 0x11 read (null) write (null) written (null)
> >> check 3: state 0x0 read (null) write (null) written (null)
> >> check 2: state 0x11 read (null) write (null) written (null)
> >> check 1: state 0x11 read (null) write (null) written (null)
> >> check 0: state 0x18 read (null) write ffff8808029b6b00 written (null)
> >> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1
> >> force RCW max_degraded=1, recovery_cp=7036944 sh->sector=273480328
> >> for sector 273480328, rmw=2 rcw=1
> >>
> >> So it is forcing RCW, even though a single block update is usually handled
> >> with RMW.
> >>
> >> In this stripe, the parity disk is '5' and disk 3 has failed.
> >> That means to perform an RCW, we need to read the parity block in order
> >> to reconstruct the content of the failed disk. And if we were to do that,
> >> we may as well just do an RMW.
> >>
> >> So I think the correct fix would be to only force RCW when the array
> >> is not degraded.
> >>
> >> So something like this:
> >>
> >> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> >> index aa76865b804b..fa8f8b94bfa8 100644
> >> --- a/drivers/md/raid5.c
> >> +++ b/drivers/md/raid5.c
> >> @@ -3170,7 +3170,8 @@ static void handle_stripe_dirtying(struct r5conf *conf,
> >> * generate correct data from the parity.
> >> */
> >> if (conf->max_degraded == 2 ||
> >> - (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
> >> + (recovery_cp < MaxSector && sh->sector >= recovery_cp &&
> >> + s->failed == 0)) {
> >> /* Calculate the real rcw later - for now make it
> >> * look like rcw is cheaper
> >> */
> >>
> >>
> >> I think reverting the whole patch is not necessary and discards useful
> >> functionality while the array is not degraded.
> >>
> >> Can you test this patch please?
> >
> > Actually I just tried this one - I was on my way home and grabbed food
> > on the way, and thought there was a better solution than to revert.
> >
> > I'll give your solution a spin too.
>
> I tried your patch, as expected that also resolves the problem. Not sure
> which solution is better, so I'll let you pick.
Thanks!
>
> Note whichever patch you choose it is applicable for stable-3.6+
3.6??
$ git describe --contains a7854487cd7128a30a7f4f5259
v3.7-rc1~10^2~7
so I assume 3.7.
Doesn't apply to 3.6, so I'll assume a typo.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: RAID1 might_sleep() warning on 3.19-rc7
From: NeilBrown @ 2015-02-18 1:09 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: Tony Battersby, linux-raid, lkml, axboe, Linus Torvalds
In-Reply-To: <20150213144803.GA11574@worktop.ger.corp.intel.com>
[-- Attachment #1: Type: text/plain, Size: 907 bytes --]
On Fri, 13 Feb 2015 15:48:03 +0100 Peter Zijlstra <peterz@infradead.org>
wrote:
> On Fri, Feb 13, 2015 at 11:27:46AM +0100, Peter Zijlstra wrote:
>
> > > I've moved blk_flush_plug to the beginning of the function.
> >
> > > I wondered if it really make sense to call blk_flush_plug with nr_iowait
> > > elevated and delayacct_blkio active. blk_flush_plug() could call schedule()
> > > for non-"io" reasons and maybe that could upset stuff???
> >
> > Yeah, good question that. Lemme ponder that a bit.
>
> Yes, I thikn your version makes most sense as, you say, even regular
> schedule() call nested in my version would go towards blk delayacct --
> and I doubt that was the intent; even though the current kernel works
> that way.
>
> I'll move the now rudimentary io_schedule() into sched.h as an inline.
Thanks.
Are you OK with this going to -stable for 3.19?
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH] Revert "md: When RAID5 is dirty, force reconstruct-write instead of read-modify-write."
From: Jes Sorensen @ 2015-02-18 1:08 UTC (permalink / raw)
To: neilb; +Cc: linux-raid
In-Reply-To: <1424218444-4022-1-git-send-email-Jes.Sorensen@redhat.com>
Jes.Sorensen@redhat.com writes:
> From: Jes Sorensen <Jes.Sorensen@redhat.com>
>
> This reverts commit a7854487cd7128a30a7f4f5259de9f67d5efb95f.
>
> The above patch would cause lockups of RAID4/5 arrays if a drive is
> going faulty during reconstruction.
>
> Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
> ---
> drivers/md/raid5.c | 19 +++----------------
> 1 file changed, 3 insertions(+), 16 deletions(-)
Please ignore this one - per previous email, there is a better way<tm>
Cheers,
Jes
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index aa76865..bc3a085 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -3159,25 +3159,12 @@ static void handle_stripe_dirtying(struct r5conf *conf,
> int disks)
> {
> int rmw = 0, rcw = 0, i;
> - sector_t recovery_cp = conf->mddev->recovery_cp;
> -
> - /* RAID6 requires 'rcw' in current implementation.
> - * Otherwise, check whether resync is now happening or should start.
> - * If yes, then the array is dirty (after unclean shutdown or
> - * initial creation), so parity in some stripes might be inconsistent.
> - * In this case, we need to always do reconstruct-write, to ensure
> - * that in case of drive failure or read-error correction, we
> - * generate correct data from the parity.
> - */
> - if (conf->max_degraded == 2 ||
> - (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
> - /* Calculate the real rcw later - for now make it
> + if (conf->max_degraded == 2) {
> + /* RAID6 requires 'rcw' in current implementation
> + * Calculate the real rcw later - for now fake it
> * look like rcw is cheaper
> */
> rcw = 1; rmw = 2;
> - pr_debug("force RCW max_degraded=%u, recovery_cp=%llu sh->sector=%llu\n",
> - conf->max_degraded, (unsigned long long)recovery_cp,
> - (unsigned long long)sh->sector);
> } else for (i = disks; i--; ) {
> /* would I have to read this buffer for read_modify_write */
> struct r5dev *dev = &sh->dev[i];
^ permalink raw reply
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Jes Sorensen @ 2015-02-18 1:07 UTC (permalink / raw)
To: NeilBrown; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <wrfjh9ukf2y6.fsf@redhat.com>
Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> NeilBrown <neilb@suse.de> writes:
>> On Tue, 17 Feb 2015 19:03:30 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
>> wrote:
>>
>>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>>> > Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>>> >> NeilBrown <neilb@suse.de> writes:
>>> >>> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P <pmanibalan@amiindia.co.in>
>>> >>> wrote:
>>> >>>
>>> >>>> Dear All,
>>> >>>> Any updates on this issue.
>>> >>>
>>> >>> Probably the same as:
>>> >>>
>>> >>> http://marc.info/?l=linux-raid&m=142283560704091&w=2
>>> >>
>>> >> Hi Neil,
>>> >>
>>> >> I ran some tests on this one against the latest Linus' tree as of today
>>> >> (1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
>>> >> your pending 3.20 patches.
>>> >>
>>> >> I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
>>> >> run fio on top of a device while it is resyncing and I fail one of the
>>> >> devices.
>>> >
>>> > Since Manibalan mentioned this issue wasn't present in earlier kernels,
>>> > I started trying to track down what change caused it.
>>> >
>>> > So far I have been able to reproduce the hang as far back as 3.10.
>>>
>>> After a lot of bisecting I finally traced the issue back to this commit:
>>>
>>> a7854487cd7128a30a7f4f5259de9f67d5efb95f is the first bad commit
>>> commit a7854487cd7128a30a7f4f5259de9f67d5efb95f
>>> Author: Alexander Lyakas <alex.bolshoy@gmail.com>
>>> Date: Thu Oct 11 13:50:12 2012 +1100
>>>
>>> md: When RAID5 is dirty, force reconstruct-write instead of
>>> read-modify-write.
>>>
>>> Signed-off-by: Alex Lyakas <alex@zadarastorage.com>
>>> Suggested-by: Yair Hershko <yair@zadarastorage.com>
>>> Signed-off-by: NeilBrown <neilb@suse.de>
>>>
>>> If I revert that one I cannot reproduce the hang, applying it reproduces
>>> the hang consistently.
>>
>> Thanks for all the research!
>>
>> That is consistent with what you already reported.
>> You noted that it doesn't affect RAID6, and RAID6 doesn't have an RMW cycle.
>>
>> Also, one of the early emails from Manibalan contained:
>>
>> handling stripe 273480328, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1
>> , check:0, reconstruct:0
>> check 5: state 0x10 read (null) write (null) written (null)
>> check 4: state 0x11 read (null) write (null) written (null)
>> check 3: state 0x0 read (null) write (null) written (null)
>> check 2: state 0x11 read (null) write (null) written (null)
>> check 1: state 0x11 read (null) write (null) written (null)
>> check 0: state 0x18 read (null) write ffff8808029b6b00 written (null)
>> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1
>> force RCW max_degraded=1, recovery_cp=7036944 sh->sector=273480328
>> for sector 273480328, rmw=2 rcw=1
>>
>> So it is forcing RCW, even though a single block update is usually handled
>> with RMW.
>>
>> In this stripe, the parity disk is '5' and disk 3 has failed.
>> That means to perform an RCW, we need to read the parity block in order
>> to reconstruct the content of the failed disk. And if we were to do that,
>> we may as well just do an RMW.
>>
>> So I think the correct fix would be to only force RCW when the array
>> is not degraded.
>>
>> So something like this:
>>
>> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
>> index aa76865b804b..fa8f8b94bfa8 100644
>> --- a/drivers/md/raid5.c
>> +++ b/drivers/md/raid5.c
>> @@ -3170,7 +3170,8 @@ static void handle_stripe_dirtying(struct r5conf *conf,
>> * generate correct data from the parity.
>> */
>> if (conf->max_degraded == 2 ||
>> - (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
>> + (recovery_cp < MaxSector && sh->sector >= recovery_cp &&
>> + s->failed == 0)) {
>> /* Calculate the real rcw later - for now make it
>> * look like rcw is cheaper
>> */
>>
>>
>> I think reverting the whole patch is not necessary and discards useful
>> functionality while the array is not degraded.
>>
>> Can you test this patch please?
>
> Actually I just tried this one - I was on my way home and grabbed food
> on the way, and thought there was a better solution than to revert.
>
> I'll give your solution a spin too.
I tried your patch, as expected that also resolves the problem. Not sure
which solution is better, so I'll let you pick.
Note whichever patch you choose it is applicable for stable-3.6+
Cheers,
Jes
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Chris Murphy @ 2015-02-18 1:02 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <54E3C51C.2080106@websitemanagers.com.au>
On Tue, Feb 17, 2015 at 3:47 PM, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
> If we enable SCT ERC on every drive that supports it, and we are using the
> drive (only) in a RAID0/linear array then what is the downside?
Unnecessary data loss.
> As I
> understand it, the drive will no longer try for > 120sec to recover the data
> stored in the "bad" sector, and instead return an unreadable error message
> in a short amount of time (well below 30 seconds) which means the driver
> will be able to return a read error to the application (or FS or MD) and the
> system as a whole will carry on.
Not necessarily, it depends what's in that sector. If it's user data,
this means a sector (or possibly more) of data loss. If it's file
system metadata it means progressive file system corruption.
Configuring the drive to give up too soon is completely inappropriate
for single, raid0 or linear configurations.
Arguably the drive should have already recovered this data. If a
longer recovery can recover, then why isn't the drive writing the data
back to that sector so that next time it isn't so ambiguous that it
requires long recovery? I can't answer that question. In some case
that appears to happen in other cases it's not. But the followup is
that there really ought to be some way for user space to get access to
these kinds of errors rather than them accumulating until disaster
strikes.
The contra argument to that is, it's still cheaper to buy the proper
use case specified drive.
>If we didn't enable SCT ERC, then the
> entire drive would vanish, (because the timeout wasn't changed for the
> driver) and the current read and every future read/write will all fail, and
> the system will probably crash (well, depending on the application, FS
> layout, etc).
Umm no. If SCT ERC remains a high value or disable, while also
increasing the kernel command timer, the drive has a longer chance to
recover. That's the appropriate configuration for single, linear, and
raid0.
>
> So, IMHO, it seems that by default, every SCT ERC capable drive should have
> this enabled by default. As a part of error recovery (ie, crap that really
> important data stored on those few unreadable sectors) the user could
> manually disable SCT ERC and re-attempt to request the data from the drive
> (eg, during dd_rescue or similar).
If you do this for single, linear, or raid0 it will increase the
incident of data loss that would otherwise not occur if deep/long
recovery times were available.
Before changing these settings, there should be some better
understanding of what the manufacturer defined recovery times in the
real world actually are, and whether or not these long recoveries are
helpful. Presumably they'd say they are helpful, but I think we need
facts to contradict their position before second guessing the default
settings. And we have such facts to do exactly that when it comes to
raid1, 5, 6 with such drives which is why the recommendation is to
change SCT ERC if supported.
> Secondly, changing the timeout for those drives that don't support SCT ERC,
> again, it is fairly similar to above, we get the error from the drive before
> the timeout, except we will avoid the only possible downside above (failing
> to read a very unlikely but possible to read sector). Again, we will avoid
> dropping the entire drive, even if all operations on this drive will stop
> for a longer period of time, it is probably better than stopping
> permanently.
Not by default. You can't assume any drive hang is due to bad sectors
that merely need a longer recovery time. It could be some other error
condition, in which case doing a 120 or 180 second *by default* delay
means no error messages at all for upwards of 3 minutes.
And in any case the proper place to change the default kernel command
timer value is in the kernel, not with a udev rule.
I don't know if a udev rule can say "If the drive exclusively uses md,
lvm, btrfs, zfs raid1, 4+ or nested of those, and if the drive does
not support configurable SCT ERC, then change the kernel command timer
for those devices to ~120 seconds" then that might be a plausible
solution to use consumer drives the manufacturer rather explicitly
proscribes from use in raid...
But the contra argument to that is, why should anyone do this work for
(sorry) basically cheap users who don't want to buy the proper drive
for the specific use case? There are limited resources for this work.
And in fact the problem has a work around, if not a solution.
What we still don't have is something that reports any such problems
to user space.
--
Chris Murphy
^ permalink raw reply
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Jes Sorensen @ 2015-02-18 1:01 UTC (permalink / raw)
To: NeilBrown; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <20150218112741.08495514@notabene.brown>
NeilBrown <neilb@suse.de> writes:
> On Tue, 17 Feb 2015 19:03:30 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
> wrote:
>
>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> > Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> >> NeilBrown <neilb@suse.de> writes:
>> >>> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P <pmanibalan@amiindia.co.in>
>> >>> wrote:
>> >>>
>> >>>> Dear All,
>> >>>> Any updates on this issue.
>> >>>
>> >>> Probably the same as:
>> >>>
>> >>> http://marc.info/?l=linux-raid&m=142283560704091&w=2
>> >>
>> >> Hi Neil,
>> >>
>> >> I ran some tests on this one against the latest Linus' tree as of today
>> >> (1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
>> >> your pending 3.20 patches.
>> >>
>> >> I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
>> >> run fio on top of a device while it is resyncing and I fail one of the
>> >> devices.
>> >
>> > Since Manibalan mentioned this issue wasn't present in earlier kernels,
>> > I started trying to track down what change caused it.
>> >
>> > So far I have been able to reproduce the hang as far back as 3.10.
>>
>> After a lot of bisecting I finally traced the issue back to this commit:
>>
>> a7854487cd7128a30a7f4f5259de9f67d5efb95f is the first bad commit
>> commit a7854487cd7128a30a7f4f5259de9f67d5efb95f
>> Author: Alexander Lyakas <alex.bolshoy@gmail.com>
>> Date: Thu Oct 11 13:50:12 2012 +1100
>>
>> md: When RAID5 is dirty, force reconstruct-write instead of
>> read-modify-write.
>>
>> Signed-off-by: Alex Lyakas <alex@zadarastorage.com>
>> Suggested-by: Yair Hershko <yair@zadarastorage.com>
>> Signed-off-by: NeilBrown <neilb@suse.de>
>>
>> If I revert that one I cannot reproduce the hang, applying it reproduces
>> the hang consistently.
>
> Thanks for all the research!
>
> That is consistent with what you already reported.
> You noted that it doesn't affect RAID6, and RAID6 doesn't have an RMW cycle.
>
> Also, one of the early emails from Manibalan contained:
>
> handling stripe 273480328, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1
> , check:0, reconstruct:0
> check 5: state 0x10 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x11 read (null) write (null) written (null)
> check 1: state 0x11 read (null) write (null) written (null)
> check 0: state 0x18 read (null) write ffff8808029b6b00 written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1
> force RCW max_degraded=1, recovery_cp=7036944 sh->sector=273480328
> for sector 273480328, rmw=2 rcw=1
>
> So it is forcing RCW, even though a single block update is usually handled
> with RMW.
>
> In this stripe, the parity disk is '5' and disk 3 has failed.
> That means to perform an RCW, we need to read the parity block in order
> to reconstruct the content of the failed disk. And if we were to do that,
> we may as well just do an RMW.
>
> So I think the correct fix would be to only force RCW when the array
> is not degraded.
>
> So something like this:
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index aa76865b804b..fa8f8b94bfa8 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -3170,7 +3170,8 @@ static void handle_stripe_dirtying(struct r5conf *conf,
> * generate correct data from the parity.
> */
> if (conf->max_degraded == 2 ||
> - (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
> + (recovery_cp < MaxSector && sh->sector >= recovery_cp &&
> + s->failed == 0)) {
> /* Calculate the real rcw later - for now make it
> * look like rcw is cheaper
> */
>
>
> I think reverting the whole patch is not necessary and discards useful
> functionality while the array is not degraded.
>
> Can you test this patch please?
Actually I just tried this one - I was on my way home and grabbed food
on the way, and thought there was a better solution than to revert.
I'll give your solution a spin too.
Jes
From 63e7c81c955d99e1eb7ab70956689a12e02eb856 Mon Sep 17 00:00:00 2001
From: Jes Sorensen <Jes.Sorensen@redhat.com>
Date: Tue, 17 Feb 2015 19:48:49 -0500
Subject: [PATCH] [md] raid5.c: Do not force reconstruct writes on a degraded
array
If an array has no writeable spares, do not try to force
reconstruct-write to it.
Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
drivers/md/raid5.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index aa76865..c0036c4 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -3170,7 +3170,8 @@ static void handle_stripe_dirtying(struct r5conf *conf,
* generate correct data from the parity.
*/
if (conf->max_degraded == 2 ||
- (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
+ (conf->mddev->degraded < (conf->max_degraded - 1) &&
+ recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
/* Calculate the real rcw later - for now make it
* look like rcw is cheaper
*/
--
2.1.0
^ permalink raw reply related
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: NeilBrown @ 2015-02-18 0:27 UTC (permalink / raw)
To: Jes Sorensen; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <wrfj61b0gk6l.fsf@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 3998 bytes --]
On Tue, 17 Feb 2015 19:03:30 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
wrote:
> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> > Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> >> NeilBrown <neilb@suse.de> writes:
> >>> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P <pmanibalan@amiindia.co.in>
> >>> wrote:
> >>>
> >>>> Dear All,
> >>>> Any updates on this issue.
> >>>
> >>> Probably the same as:
> >>>
> >>> http://marc.info/?l=linux-raid&m=142283560704091&w=2
> >>
> >> Hi Neil,
> >>
> >> I ran some tests on this one against the latest Linus' tree as of today
> >> (1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
> >> your pending 3.20 patches.
> >>
> >> I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
> >> run fio on top of a device while it is resyncing and I fail one of the
> >> devices.
> >
> > Since Manibalan mentioned this issue wasn't present in earlier kernels,
> > I started trying to track down what change caused it.
> >
> > So far I have been able to reproduce the hang as far back as 3.10.
>
> After a lot of bisecting I finally traced the issue back to this commit:
>
> a7854487cd7128a30a7f4f5259de9f67d5efb95f is the first bad commit
> commit a7854487cd7128a30a7f4f5259de9f67d5efb95f
> Author: Alexander Lyakas <alex.bolshoy@gmail.com>
> Date: Thu Oct 11 13:50:12 2012 +1100
>
> md: When RAID5 is dirty, force reconstruct-write instead of read-modify-write.
>
> Signed-off-by: Alex Lyakas <alex@zadarastorage.com>
> Suggested-by: Yair Hershko <yair@zadarastorage.com>
> Signed-off-by: NeilBrown <neilb@suse.de>
>
> If I revert that one I cannot reproduce the hang, applying it reproduces
> the hang consistently.
Thanks for all the research!
That is consistent with what you already reported.
You noted that it doesn't affect RAID6, and RAID6 doesn't have an RMW cycle.
Also, one of the early emails from Manibalan contained:
handling stripe 273480328, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1
, check:0, reconstruct:0
check 5: state 0x10 read (null) write (null) written (null)
check 4: state 0x11 read (null) write (null) written (null)
check 3: state 0x0 read (null) write (null) written (null)
check 2: state 0x11 read (null) write (null) written (null)
check 1: state 0x11 read (null) write (null) written (null)
check 0: state 0x18 read (null) write ffff8808029b6b00 written (null)
locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1
force RCW max_degraded=1, recovery_cp=7036944 sh->sector=273480328
for sector 273480328, rmw=2 rcw=1
So it is forcing RCW, even though a single block update is usually handled
with RMW.
In this stripe, the parity disk is '5' and disk 3 has failed.
That means to perform an RCW, we need to read the parity block in order
to reconstruct the content of the failed disk. And if we were to do that,
we may as well just do an RMW.
So I think the correct fix would be to only force RCW when the array
is not degraded.
So something like this:
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index aa76865b804b..fa8f8b94bfa8 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -3170,7 +3170,8 @@ static void handle_stripe_dirtying(struct r5conf *conf,
* generate correct data from the parity.
*/
if (conf->max_degraded == 2 ||
- (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
+ (recovery_cp < MaxSector && sh->sector >= recovery_cp &&
+ s->failed == 0)) {
/* Calculate the real rcw later - for now make it
* look like rcw is cheaper
*/
I think reverting the whole patch is not necessary and discards useful
functionality while the array is not degraded.
Can you test this patch please?
Thanks!
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* [PATCH] Revert "md: When RAID5 is dirty, force reconstruct-write instead of read-modify-write."
From: Jes.Sorensen @ 2015-02-18 0:14 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, Jes Sorensen
From: Jes Sorensen <Jes.Sorensen@redhat.com>
This reverts commit a7854487cd7128a30a7f4f5259de9f67d5efb95f.
The above patch would cause lockups of RAID4/5 arrays if a drive is
going faulty during reconstruction.
Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
drivers/md/raid5.c | 19 +++----------------
1 file changed, 3 insertions(+), 16 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index aa76865..bc3a085 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -3159,25 +3159,12 @@ static void handle_stripe_dirtying(struct r5conf *conf,
int disks)
{
int rmw = 0, rcw = 0, i;
- sector_t recovery_cp = conf->mddev->recovery_cp;
-
- /* RAID6 requires 'rcw' in current implementation.
- * Otherwise, check whether resync is now happening or should start.
- * If yes, then the array is dirty (after unclean shutdown or
- * initial creation), so parity in some stripes might be inconsistent.
- * In this case, we need to always do reconstruct-write, to ensure
- * that in case of drive failure or read-error correction, we
- * generate correct data from the parity.
- */
- if (conf->max_degraded == 2 ||
- (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
- /* Calculate the real rcw later - for now make it
+ if (conf->max_degraded == 2) {
+ /* RAID6 requires 'rcw' in current implementation
+ * Calculate the real rcw later - for now fake it
* look like rcw is cheaper
*/
rcw = 1; rmw = 2;
- pr_debug("force RCW max_degraded=%u, recovery_cp=%llu sh->sector=%llu\n",
- conf->max_degraded, (unsigned long long)recovery_cp,
- (unsigned long long)sh->sector);
} else for (i = disks; i--; ) {
/* would I have to read this buffer for read_modify_write */
struct r5dev *dev = &sh->dev[i];
--
1.8.3.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox