From: NeilBrown <neilb@suse.de>
To: "Kwolek, Adam" <adam.kwolek@intel.com>
Cc: "linux-raid@vger.kernel.org" <linux-raid@vger.kernel.org>,
"Ciechanowski, Ed" <ed.ciechanowski@intel.com>,
"Labun, Marcin" <Marcin.Labun@intel.com>,
"Williams, Dan J" <dan.j.williams@intel.com>
Subject: Re: [PATCH] md: Add ability for disable bad block management
Date: Fri, 9 Dec 2011 14:53:15 +1100 [thread overview]
Message-ID: <20111209145315.55a0704d@notabene.brown> (raw)
In-Reply-To: <79556383A0E1384DB3A3903742AAC04A056305@IRSMSX101.ger.corp.intel.com>
[-- Attachment #1: Type: text/plain, Size: 11600 bytes --]
On Thu, 8 Dec 2011 15:36:43 +0000 "Kwolek, Adam" <adam.kwolek@intel.com>
wrote:
>
>
> > -----Original Message-----
> > From: NeilBrown [mailto:neilb@suse.de]
> > Sent: Thursday, December 08, 2011 5:02 AM
> > To: Kwolek, Adam
> > Cc: linux-raid@vger.kernel.org; Ciechanowski, Ed; Labun, Marcin; Williams,
> > Dan J
> > Subject: Re: [PATCH] md: Add ability for disable bad block management
> >
> > On Wed, 7 Dec 2011 11:10:06 +0000 "Kwolek, Adam"
> > <adam.kwolek@intel.com>
> > wrote:
> >
> > >
> > >
> > > > -----Original Message-----
> > > > From: NeilBrown [mailto:neilb@suse.de]
> >
> > > > I cannot reproduce this.
> > > > I didn't physically remove devices, but I used
> > > > echo 1 > /sys/block/sdc/device/delete which should be nearly
> > > > identical from the perspective of md and mdadm.
> > >
> > > I've checked that when I'm deleting device using sysfs everything works
> > perfect.
> > > When when device is pulled out, reshape stops in md/mdstat.
> > >
> > > > If you could give me the exact set of steps that you follow to
> > > > produce the problem that would help - maybe a script? Just a description
> > is OK.
> > >
> > >
> > > #used disks sdb, sdc, sdd, sde
> > > export IMSM_NO_PLATFORM=1
> > > #create container
> > > mdadm -C /dev/md/imsm0 -amd -e imsm -n 3 /dev/sdb /dev/sdc /dev/sde
> > -R
> > > #create vol mdadm -C /dev/md/raid5vol_0 -amd -l 5 --chunk 32 --size
> > > 104850 -n 3 /dev/sdb /dev/sdc /dev/sde -R #add spare mdadm --add
> > > /dev/md/imsm0 /dev/sdd #run OLCE mdadm --grow /dev/md/imsm0
> > > --raid-devices 4 #when reshape starts, I'm (physically) pulling device
> > > out
> > >
> > > > Also you say it is blocking in md_do_sync. Is that at the
> > > >
> > > > wait_event(mddev->recovery_wait, !atomic_read(&mddev-
> > > > >recovery_active));
> > > >
> > > > call just after the "out:" label?
> > >
> > > None of those 2 places.
> > > It enters sync_request() function. Md_error() is called.
> > > More is visible on thread stack information below
> > (md_wait_for_blocked_rdev()).
> > >
> > >
> > > >
> > > > What is the raid thread doing at this point?
> > > > cat /proc/PID/stack
> > > > might help.
> > >
> > > [md126_raid5]
> > > [<ffffffff8121d843>] md_wait_for_blocked_rdev+0xbc/0x10f
> > > [<ffffffffa01d87ce>] handle_stripe+0x1c5c/0x2c99 [raid456]
> > > [<ffffffffa01d9d0d>] raid5d+0x502/0x564 [raid456] [<ffffffff8121eca5>]
> > > md_thread+0x101/0x11f [<ffffffff81049e0e>] kthread+0x81/0x89
> > > [<ffffffff812cc4f4>] kernel_thread_helper+0x4/0x10
> > > [<ffffffffffffffff>] 0xffffffffffffffff
> > >
> > > [md126_reshape]
> > > [<ffffffffa02455a2>] sync_request+0x90a/0xbfb [raid456]
> > > [<ffffffff8121e151>] md_do_sync+0x7aa/0xc40 [<ffffffff8121ecb3>]
> > > md_thread+0x101/0x11f [<ffffffff81049e0e>] kthread+0x81/0x89
> > > [<ffffffff812cc4f4>] kernel_thread_helper+0x4/0x10
> > > [<ffffffffffffffff>] 0xffffffffffffffff
> > >
> > > >
> > > > What are the contents of all the sysfs files?
> > > > grep . /sys/block/mdXXX/md/*
> > > array_state ->active
> > > degraded ->1
> > > max_read_errors ->20
> > > reshape_position ->12288
> > > resync_start ->none
> > > sync_completed ->4096 / 209664
> > >
> > >
> > > > grep . /sys/block/mdXXX/md/dev-*/*
> > >
> > > When removed is sdd /sys/block/mdXXX/md/dev-sdd/*
> > > bad_blocks ->4096 512
> > > ->4608 128
> > > ->4736 384
> > > block ->MISSING link is not valid
> > > errors ->0
> > > offset ->0
> > > recovery_start ->4096
> > > size ->104832
> > > slot ->3
> > > state ->faulty,write_error
> > > unacknowledged_bad_blocks ->4096 512
> > > ->4608 128
> > > ->4736 384
> > >
> > > I hope this helps.
> >
> > Yes it does, thanks.
> >
> > Can you try with this patch as well please.
> >
> > Thanks,
> > NeilBrown
> >
> >
> > diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index ea6dce9..6cf0f6a
> > 100644
> > --- a/drivers/md/raid5.c
> > +++ b/drivers/md/raid5.c
> > @@ -3175,6 +3175,8 @@ static void analyse_stripe(struct stripe_head *sh,
> > struct stripe_head_state *s)
> > rdev = rcu_dereference(conf->disks[i].rdev);
> > clear_bit(R5_ReadRepl, &dev->flags);
> > }
> > + if (rdev && test_bit(Faulty, &rdev->flags))
> > + rdev = NULL;
> > if (rdev) {
> > is_bad = is_badblock(rdev, sh->sector,
> > STRIPE_SECTORS,
> > &first_bad, &bad_sectors);
>
> I've didn't succeed with this patch only, but when I've switch to newest md from today's neil_for-linus branch things went better.
> During migration it seems that it is OK.
>
> Problems are when during rebuild/resync additional disk is failed (physical pull). Metadata react correctly (mdadm/mdmon) but md stops again. This time:
>
> [md126_resync]
> [<ffffffffa027037d>] get_active_stripe+0x295/0x598 [raid456]
> [<ffffffffa02757da>] sync_request+0xb1c/0xba7 [raid456]
> [<ffffffff8121e656>] md_do_sync+0x772/0xbc4
> [<ffffffff8121f174>] md_thread+0x101/0x11f
> [<ffffffff81049ebe>] kthread+0x81/0x89
> [<ffffffff812cc934>] kernel_thread_helper+0x4/0x10
> [<ffffffffffffffff>] 0xffffffffffffffff
>
> Thread [md126_raid5] is missing, but in mdstat raid5 resync/rebuild is visible
> During initialization one time it was executed correctly, second time it stops exactly as rebuild in get_active_stripe() and [md126_raid5] thread was missing also.
> Any 'mdadm -Ss' causes system hung (not very surprising without raid5 thread)
>
> In /var/log/messages we have:
> Dec 8 12:39:49 gklab-128-013 kernel: Modules linked in: raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx ext2 nvidia(P) snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device ipv6 af_packet cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod snd_hda_codec_hdmi snd_hda_codec_idt snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd intel_agp iTCO_wdt tpm_tis tpm soundcore e100 pcspkr mii tpm_bios snd_page_alloc sr_mod cdrom serio_raw i2c_i801 i2c_core iTCO_vendor_support sg intel_gtt button agpgart usbhid hid uhci_hcd sd_mod crc_t10dif ehci_hcd usbcore usb_common edd ext3 mbcache jbd fan processor ide_pci_generic ide_core ata_generic ahci libahci pata_marvell libata scsi_mod thermal thermal_sys hwmon
> Dec 8 12:39:49 gklab-128-013 kernel:
> Dec 8 12:39:49 gklab-128-013 kernel: Pid: 4584, comm: md126_raid5 Tainted: P 3.2.0-rc1-SLE11_BRANCH_ADK #10 /DP35DP
> Dec 8 12:39:49 gklab-128-013 kernel: RIP: 0010:[<ffffffffa0280e67>] [<ffffffffa0280e67>] handle_stripe+0x2f5/0x2cbf [raid456]
> Dec 8 12:39:49 gklab-128-013 kernel: RSP: 0018:ffff8800d61cdb80 EFLAGS: 00010002
> Dec 8 12:39:49 gklab-128-013 kernel: RAX: 0000000000008001 RBX: 0000000000000000 RCX: 0000000000000002
> Dec 8 12:39:49 gklab-128-013 kernel: RDX: 0000000000000000 RSI: ffff880114462800 RDI: ffff8801144629a8
> Dec 8 12:39:49 gklab-128-013 kernel: RBP: ffff8800d61cdd40 R08: ffff8800379256c0 R09: 0000000300000000
> Dec 8 12:39:49 gklab-128-013 kernel: R10: ffff88010e5bfa00 R11: 0000000100000001 R12: ffff8800372602c8
> Dec 8 12:39:49 gklab-128-013 kernel: R13: ffff880037260048 R14: ffff8800372602d0 R15: ffff8801144638b0
> Dec 8 12:39:49 gklab-128-013 kernel: FS: 0000000000000000(0000) GS:ffff88011bc00000(0000) knlGS:0000000000000000
> Dec 8 12:39:49 gklab-128-013 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> Dec 8 12:39:49 gklab-128-013 kernel: CR2: 00000000000000b0 CR3: 00000000379b3000 CR4: 00000000000006f0
> Dec 8 12:39:49 gklab-128-013 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Dec 8 12:39:49 gklab-128-013 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Dec 8 12:39:49 gklab-128-013 kernel: Process md126_raid5 (pid: 4584, threadinfo ffff8800d61cc000, task ffff88003715a7c0)
> Dec 8 12:39:49 gklab-128-013 kernel: Stack:
> Dec 8 12:39:49 gklab-128-013 kernel: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> Dec 8 12:39:49 gklab-128-013 kernel: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> Dec 8 12:39:49 gklab-128-013 kernel: 0000000000000400 0000000000000400 0000000300000000 ffff88010e749280
> Dec 8 12:39:49 gklab-128-013 kernel: Call Trace:
> Dec 8 12:39:49 gklab-128-013 kernel: [<ffffffff81221fd4>] ? md_check_recovery+0x60d/0x630
> Dec 8 12:39:49 gklab-128-013 kernel: [<ffffffffa027ef28>] ? __release_stripe+0x174/0x18f [raid456]
> Dec 8 12:39:49 gklab-128-013 kernel: [<ffffffffa0283d33>] raid5d+0x502/0x564 [raid456]
> Dec 8 12:39:49 gklab-128-013 kernel: [<ffffffff812c3e6c>] ? schedule_timeout+0x35/0x1e8
> Dec 8 12:39:49 gklab-128-013 kernel: [<ffffffff8121f174>] md_thread+0x101/0x11f
> Dec 8 12:39:49 gklab-128-013 kernel: [<ffffffff8104a2ad>] ? wake_up_bit+0x23/0x23
> Dec 8 12:39:49 gklab-128-013 kernel: [<ffffffff8121f073>] ? md_register_thread+0xd6/0xd6
> Dec 8 12:39:50 gklab-128-013 kernel: [<ffffffff81049ebe>] kthread+0x81/0x89
> Dec 8 12:39:50 gklab-128-013 kernel: [<ffffffff812cc934>] kernel_thread_helper+0x4/0x10
> Dec 8 12:39:50 gklab-128-013 kernel: [<ffffffff81049e3d>] ? kthread_worker_fn+0x145/0x145
> Dec 8 12:39:50 gklab-128-013 kernel: [<ffffffff812cc930>] ? gs_change+0xb/0xb
> Dec 8 12:39:50 gklab-128-013 kernel: Code: 75 11 49 8b 45 30 48 83 c0 08 48 3b 83 e0 00 00 00 77 07 f0 41 80 4c 24 08 08 49 8b 44 24 08 66 85 c0 79 2c f0 41 80 64 24 08 f7
> Dec 8 12:39:50 gklab-128-013 kernel: <48> 8b 83 b0 00 00 00 a8 02 75 10 c7 45 80 01 00 00 00 f0 ff 83
> Dec 8 12:39:50 gklab-128-013 kernel: RIP [<ffffffffa0280e67>] handle_stripe+0x2f5/0x2cbf [raid456]
> Dec 8 12:39:50 gklab-128-013 kernel: RSP <ffff8800d61cdb80>
> Dec 8 12:39:50 gklab-128-013 kernel: CR2: 00000000000000b0
>
>
> The problem is caused by access to just cleaned rdev a few lines below in raid5.c.
> The following patch corrects it.
>
> >From fbaa3fdff634721e5c2c09e07b8429385494ee02 Mon Sep 17 00:00:00 2001
> From: Adam Kwolek <adam.kwolek@intel.com>
> Date: Thu, 8 Dec 2011 15:34:09 +0100
> Subject: [PATCH] md: raid5 crash during degradation
>
> NULL pointer access causes crash in raid5 module.
>
> Signed-off-by: Adam Kwolek <adam.kwolek@intel.com>
Ahh, thanks. Obvious once you see it :-)
Thanks,
I've sent this and the other fixes off to Linus.
NeilBrown
> ---
> drivers/md/raid5.c | 4 ++--
> 1 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index b0dec01..da4997c 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -3070,7 +3070,7 @@ static void analyse_stripe(struct stripe_head *sh, struct stripe_head_state *s)
> if (sh->sector + STRIPE_SECTORS <= rdev->recovery_offset)
> set_bit(R5_Insync, &dev->flags);
> }
> - if (test_bit(R5_WriteError, &dev->flags)) {
> + if (test_bit(R5_WriteError, &dev->flags) && rdev) {
> clear_bit(R5_Insync, &dev->flags);
> if (!test_bit(Faulty, &rdev->flags)) {
> s->handle_bad_blocks = 1;
> @@ -3078,7 +3078,7 @@ static void analyse_stripe(struct stripe_head *sh, struct stripe_head_state *s)
> } else
> clear_bit(R5_WriteError, &dev->flags);
> }
> - if (test_bit(R5_MadeGood, &dev->flags)) {
> + if (test_bit(R5_MadeGood, &dev->flags) && rdev) {
> if (!test_bit(Faulty, &rdev->flags)) {
> s->handle_bad_blocks = 1;
> atomic_inc(&rdev->nr_pending);
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
prev parent reply other threads:[~2011-12-09 3:53 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-11-24 12:19 [PATCH] md: Add ability for disable bad block management Adam Kwolek
2011-11-24 12:23 ` Paul Menzel
2011-11-24 12:28 ` Kwolek, Adam
2011-11-24 12:48 ` Paul Menzel
2011-11-30 0:14 ` NeilBrown
2011-11-30 8:17 ` Kwolek, Adam
2011-12-06 6:05 ` NeilBrown
2011-12-06 13:02 ` Kwolek, Adam
2011-12-07 1:52 ` NeilBrown
2011-12-07 11:10 ` Kwolek, Adam
2011-12-08 4:02 ` NeilBrown
2011-12-08 15:36 ` Kwolek, Adam
2011-12-09 3:53 ` NeilBrown [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20111209145315.55a0704d@notabene.brown \
--to=neilb@suse.de \
--cc=Marcin.Labun@intel.com \
--cc=adam.kwolek@intel.com \
--cc=dan.j.williams@intel.com \
--cc=ed.ciechanowski@intel.com \
--cc=linux-raid@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).