* failed reshape! @ 2011-12-09 13:53 Gavin Peters (蓋文彼德斯) 2011-12-12 3:39 ` NeilBrown 0 siblings, 1 reply; 3+ messages in thread From: Gavin Peters (蓋文彼德斯) @ 2011-12-09 13:53 UTC (permalink / raw) To: linux-raid I tried to reshape today, a raid6 array from seven devices up to eight. I ran mdadm 3.2.2, something like # mdadm /dev/md2 --grow -n 8 --layout=preserve and then, blammo! Dec 8 22:30:10 avclub kernel: [ 527.094708] RAID5 conf printout: Dec 8 22:30:10 avclub kernel: [ 527.094712] --- rd:8 wd:8 Dec 8 22:30:10 avclub kernel: [ 527.094714] disk 0, o:1, dev:sdc6 Dec 8 22:30:10 avclub kernel: [ 527.094715] disk 1, o:1, dev:sdf6 Dec 8 22:30:10 avclub kernel: [ 527.094717] disk 2, o:1, dev:sda6 Dec 8 22:30:10 avclub kernel: [ 527.094718] disk 3, o:1, dev:sdd6 Dec 8 22:30:10 avclub kernel: [ 527.094719] disk 4, o:1, dev:sdb6 Dec 8 22:30:10 avclub kernel: [ 527.094720] disk 5, o:1, dev:sde6 Dec 8 22:30:10 avclub kernel: [ 527.094721] disk 6, o:1, dev:sdg6 Dec 8 22:30:10 avclub kernel: [ 527.094722] disk 7, o:1, dev:sdh6 Dec 8 22:30:10 avclub kernel: [ 527.094876] md: reshape of RAID array md2 Dec 8 22:30:10 avclub kernel: [ 527.094886] md: minimum _guaranteed_ speed: 40000 KB/sec/disk. Dec 8 22:30:10 avclub kernel: [ 527.094892] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape. Dec 8 22:30:10 avclub kernel: [ 527.094912] md: using 128k window, over a total of 1371476928 blocks. Dec 8 22:30:11 avclub mdadm[2959]: RebuildStarted event detected on md device /dev/md2 Dec 8 22:30:11 avclub kernel: [ 527.515359] general protection fault: 0000 [#1] SMP Dec 8 22:30:11 avclub kernel: [ 527.515370] last sysfs file: /sys/devices/virtual/block/md2/md/sync_speed Dec 8 22:30:11 avclub kernel: [ 527.515376] CPU 5 Dec 8 22:30:11 avclub kernel: [ 527.515381] Modules linked in: binfmt_misc nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc snd_usb_audio snd_usb_lib snd_hda_codec_atihdmi fbcon tileblit font bitblit softcursor vga16fb vgastate snd_hda_codec_via snd_hda_intel snd_pcm_oss snd_hda_codec snd_mixer_ oss snd_hwdep snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device radeon ttm asus_atk0110 drm_kms_helper ppdev snd drm i2c_algo_bit parport_pc edac_core edac_mce_amd gspca_zc3xx gspca_main videodev v4l1_compat v4l2_compat_ioctl32 soundcore snd_page_alloc i2c_piix4 shpchp lp parport tcp_vegas raid10 raid456 async_pq async_xor xor async_memcpy usbhid async_raid6_recov hid raid6_pq async_tx raid1 raid0 pata_atiixp r8169 mii multipath ahci linear [last unloaded: kvm] Dec 8 22:30:11 avclub kernel: [ 527.515500] Pid: 528, comm: md2_raid6 Not tainted 2.6.32-32-generic #62-Ubuntu System Product Name Dec 8 22:30:11 avclub kernel: [ 527.515507] RIP: 0010:[<ffffffff812be15b>] [<ffffffff812be15b>] memcpy_c+0xb/0x20 Dec 8 22:30:11 avclub kernel: [ 527.515526] RSP: 0018:ffff880408985c18 EFLAGS: 00010246 Dec 8 22:30:11 avclub kernel: [ 527.515531] RAX: db73880000000000 RBX: ffff880408984000 RCX: 0000000000000200 Dec 8 22:30:11 avclub kernel: [ 527.515537] RDX: 0000000000000000 RSI: ffff880369717000 RDI: db73880000000000 Dec 8 22:30:11 avclub kernel: [ 527.515543] RBP: ffff880408985c80 R08: 0000000000001000 R09: ffff880408985ca0 Dec 8 22:30:11 avclub kernel: [ 527.515548] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880408985ca0 Dec 8 22:30:11 avclub kernel: [ 527.515553] R13: ffff880369741290 R14: 0000000000000000 R15: 0000000000000000 Dec 8 22:30:11 avclub kernel: [ 527.515560] FS: 00007f465923d7a0(0000) GS:ffff880028340000(0000) knlGS:00000000f6990760 Dec 8 22:30:11 avclub kernel: [ 527.515566] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Dec 8 22:30:11 avclub kernel: [ 527.515571] CR2: 00007fe6aaf92000 CR3: 00000003c3a5e000 CR4: 00000000000006e0 Dec 8 22:30:11 avclub kernel: [ 527.515576] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Dec 8 22:30:11 avclub kernel: [ 527.515582] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Dec 8 22:30:11 avclub kernel: [ 527.515589] Process md2_raid6 (pid: 528, threadinfo ffff880408984000, task ffff88040b210000) Dec 8 22:30:11 avclub kernel: [ 527.515593] Stack: Dec 8 22:30:11 avclub kernel: [ 527.515596] ffffffffa004a0e7 ffff880408985c50 0000000000000000 0000000000000000 Dec 8 22:30:11 avclub kernel: [ 527.515604] <0> ffffea000bf10d08 0000000000000000 0000000000001000 ffff880408985c80 Dec 8 22:30:11 avclub kernel: [ 527.515614] <0> 0000000000000000 ffff8803696a6930 ffff880369741290 ffff880408985d70 Dec 8 22:30:11 avclub kernel: [ 527.515624] Call Trace: Dec 8 22:30:11 avclub kernel: [ 527.515639] [<ffffffffa004a0e7>] ? async_memcpy+0xe7/0x25c [async_memcpy] Dec 8 22:30:11 avclub kernel: [ 527.515654] [<ffffffffa00aaabb>] handle_stripe_expansion+0x14b/0x1e0 [raid456] Dec 8 22:30:11 avclub kernel: [ 527.515668] [<ffffffffa00ab113>] handle_stripe6+0x5c3/0xb40 [raid456] Dec 8 22:30:11 avclub kernel: [ 527.515680] [<ffffffffa00a794c>] ? __release_stripe+0xcc/0x1c0 [raid456] Dec 8 22:30:11 avclub kernel: [ 527.515692] [<ffffffffa00ac055>] handle_stripe+0x25/0x30 [raid456] Dec 8 22:30:11 avclub kernel: [ 527.515703] [<ffffffffa00ac452>] raid5d+0x202/0x320 [raid456] Dec 8 22:30:11 avclub kernel: [ 527.515716] [<ffffffff815416b9>] ? _spin_unlock_irqrestore+0x19/0x30 Dec 8 22:30:11 avclub kernel: [ 527.515725] [<ffffffff8141704c>] md_thread+0x5c/0x130 Dec 8 22:30:11 avclub kernel: [ 527.515735] [<ffffffff81084cb0>] ? autoremove_wake_function+0x0/0x40 Dec 8 22:30:11 avclub kernel: [ 527.515743] [<ffffffff81416ff0>] ? md_thread+0x0/0x130 Dec 8 22:30:11 avclub kernel: [ 527.515750] [<ffffffff81084936>] kthread+0x96/0xa0 Dec 8 22:30:11 avclub kernel: [ 527.515758] [<ffffffff810131ea>] child_rip+0xa/0x20 Dec 8 22:30:11 avclub kernel: [ 527.515766] [<ffffffff810848a0>] ? kthread+0x0/0xa0 Dec 8 22:30:11 avclub kernel: [ 527.515772] [<ffffffff810131e0>] ? child_rip+0x0/0x20 Dec 8 22:30:11 avclub kernel: [ 527.515776] Code: 81 ea d8 1f 00 00 48 3b 42 20 73 07 48 8b 50 f9 31 c0 c3 31 d2 48 c7 c0 f2 ff ff ff c3 90 90 90 48 89 f8 89 d1 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 Dec 8 22:30:11 avclub kernel: [ 527.515842] RIP [<ffffffff812be15b>] memcpy_c+0xb/0x20 Dec 8 22:30:11 avclub kernel: [ 527.515850] RSP <ffff880408985c18> Dec 8 22:30:11 avclub kernel: [ 527.515857] ---[ end trace 5146b1cc8ebe8dc1 ]--- Dec 8 22:30:11 avclub kernel: [ 527.515865] note: md2_raid6[528] exited with preempt_count 2 Dec 8 22:32:52 avclub kernel: Kernel logging (proc) stopped. I believe that last line shows me giving up. I am sad. Thankfully, after rebooting into single user mode, I was able to mdadm --assemble the array, and it appears to be working. Boy that was a rush! $ uname -aLinux avclub 2.6.32-32-generic #62-Ubuntu SMP Wed Apr 20 21:52:38 UTC 2011 x86_64 GNU/Linux Let me know if I can provide any other information. - Gavin -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: failed reshape! 2011-12-09 13:53 failed reshape! Gavin Peters (蓋文彼德斯) @ 2011-12-12 3:39 ` NeilBrown 2011-12-12 12:12 ` Gavin Peters (蓋文彼德斯) 0 siblings, 1 reply; 3+ messages in thread From: NeilBrown @ 2011-12-12 3:39 UTC (permalink / raw) To: Gavin Peters (蓋文彼德斯); +Cc: linux-raid [-- Attachment #1: Type: text/plain, Size: 8092 bytes --] On Fri, 9 Dec 2011 08:53:42 -0500 Gavin Peters (蓋文彼德斯) <gavin@ytz.ca> wrote: > I tried to reshape today, a raid6 array from seven devices up to > eight. I ran mdadm 3.2.2, something like > > # mdadm /dev/md2 --grow -n 8 --layout=preserve > > and then, blammo! > > Dec 8 22:30:10 avclub kernel: [ 527.094708] RAID5 conf printout: > Dec 8 22:30:10 avclub kernel: [ 527.094712] --- rd:8 wd:8 > Dec 8 22:30:10 avclub kernel: [ 527.094714] disk 0, o:1, dev:sdc6 > Dec 8 22:30:10 avclub kernel: [ 527.094715] disk 1, o:1, dev:sdf6 > Dec 8 22:30:10 avclub kernel: [ 527.094717] disk 2, o:1, dev:sda6 > Dec 8 22:30:10 avclub kernel: [ 527.094718] disk 3, o:1, dev:sdd6 > Dec 8 22:30:10 avclub kernel: [ 527.094719] disk 4, o:1, dev:sdb6 > Dec 8 22:30:10 avclub kernel: [ 527.094720] disk 5, o:1, dev:sde6 > Dec 8 22:30:10 avclub kernel: [ 527.094721] disk 6, o:1, dev:sdg6 > Dec 8 22:30:10 avclub kernel: [ 527.094722] disk 7, o:1, dev:sdh6 > Dec 8 22:30:10 avclub kernel: [ 527.094876] md: reshape of RAID array md2 > Dec 8 22:30:10 avclub kernel: [ 527.094886] md: minimum _guaranteed_ > speed: 40000 KB/sec/disk. > Dec 8 22:30:10 avclub kernel: [ 527.094892] md: using maximum > available idle IO bandwidth (but not more than 200000 KB/sec) for > reshape. > Dec 8 22:30:10 avclub kernel: [ 527.094912] md: using 128k window, > over a total of 1371476928 blocks. > Dec 8 22:30:11 avclub mdadm[2959]: RebuildStarted event detected on > md device /dev/md2 > Dec 8 22:30:11 avclub kernel: [ 527.515359] general protection > fault: 0000 [#1] SMP > Dec 8 22:30:11 avclub kernel: [ 527.515370] last sysfs file: > /sys/devices/virtual/block/md2/md/sync_speed > Dec 8 22:30:11 avclub kernel: [ 527.515376] CPU 5 > Dec 8 22:30:11 avclub kernel: [ 527.515381] Modules linked in: > binfmt_misc nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc > snd_usb_audio snd_usb_lib snd_hda_codec_atihdmi fbcon tileblit font > bitblit softcursor > vga16fb vgastate snd_hda_codec_via snd_hda_intel snd_pcm_oss > snd_hda_codec snd_mixer_ > oss snd_hwdep snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi > snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device radeon > ttm asus_atk0110 drm_kms_helper ppdev snd drm i2c_algo_bit parport_pc > edac_core edac_mce_amd gspca_zc3xx gspca_main videodev v4l1_compat > v4l2_compat_ioctl32 soundcore snd_page_alloc i2c_piix4 shpchp lp > parport tcp_vegas raid10 raid456 async_pq async_xor xor async_memcpy > usbhid async_raid6_recov hid raid6_pq async_tx raid1 raid0 pata_atiixp > r8169 mii multipath ahci linear [last unloaded: kvm] > Dec 8 22:30:11 avclub kernel: [ 527.515500] Pid: 528, comm: > md2_raid6 Not tainted 2.6.32-32-generic #62-Ubuntu System Product Name > Dec 8 22:30:11 avclub kernel: [ 527.515507] RIP: > 0010:[<ffffffff812be15b>] [<ffffffff812be15b>] memcpy_c+0xb/0x20 > Dec 8 22:30:11 avclub kernel: [ 527.515526] RSP: > 0018:ffff880408985c18 EFLAGS: 00010246 > Dec 8 22:30:11 avclub kernel: [ 527.515531] RAX: db73880000000000 > RBX: ffff880408984000 RCX: 0000000000000200 > Dec 8 22:30:11 avclub kernel: [ 527.515537] RDX: 0000000000000000 > RSI: ffff880369717000 RDI: db73880000000000 > Dec 8 22:30:11 avclub kernel: [ 527.515543] RBP: ffff880408985c80 > R08: 0000000000001000 R09: ffff880408985ca0 > Dec 8 22:30:11 avclub kernel: [ 527.515548] R10: 0000000000000000 > R11: 0000000000000000 R12: ffff880408985ca0 > Dec 8 22:30:11 avclub kernel: [ 527.515553] R13: ffff880369741290 > R14: 0000000000000000 R15: 0000000000000000 > Dec 8 22:30:11 avclub kernel: [ 527.515560] FS: > 00007f465923d7a0(0000) GS:ffff880028340000(0000) > knlGS:00000000f6990760 > Dec 8 22:30:11 avclub kernel: [ 527.515566] CS: 0010 DS: 0018 ES: > 0018 CR0: 000000008005003b > Dec 8 22:30:11 avclub kernel: [ 527.515571] CR2: 00007fe6aaf92000 > CR3: 00000003c3a5e000 CR4: 00000000000006e0 > Dec 8 22:30:11 avclub kernel: [ 527.515576] DR0: 0000000000000000 > DR1: 0000000000000000 DR2: 0000000000000000 > Dec 8 22:30:11 avclub kernel: [ 527.515582] DR3: 0000000000000000 > DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Dec 8 22:30:11 avclub kernel: [ 527.515589] Process md2_raid6 (pid: > 528, threadinfo ffff880408984000, task ffff88040b210000) > Dec 8 22:30:11 avclub kernel: [ 527.515593] Stack: > Dec 8 22:30:11 avclub kernel: [ 527.515596] ffffffffa004a0e7 > ffff880408985c50 0000000000000000 0000000000000000 > Dec 8 22:30:11 avclub kernel: [ 527.515604] <0> ffffea000bf10d08 > 0000000000000000 0000000000001000 ffff880408985c80 > Dec 8 22:30:11 avclub kernel: [ 527.515614] <0> 0000000000000000 > ffff8803696a6930 ffff880369741290 ffff880408985d70 > Dec 8 22:30:11 avclub kernel: [ 527.515624] Call Trace: > Dec 8 22:30:11 avclub kernel: [ 527.515639] [<ffffffffa004a0e7>] ? > async_memcpy+0xe7/0x25c [async_memcpy] > Dec 8 22:30:11 avclub kernel: [ 527.515654] [<ffffffffa00aaabb>] > handle_stripe_expansion+0x14b/0x1e0 [raid456] > Dec 8 22:30:11 avclub kernel: [ 527.515668] [<ffffffffa00ab113>] > handle_stripe6+0x5c3/0xb40 [raid456] > Dec 8 22:30:11 avclub kernel: [ 527.515680] [<ffffffffa00a794c>] ? > __release_stripe+0xcc/0x1c0 [raid456] > Dec 8 22:30:11 avclub kernel: [ 527.515692] [<ffffffffa00ac055>] > handle_stripe+0x25/0x30 [raid456] > Dec 8 22:30:11 avclub kernel: [ 527.515703] [<ffffffffa00ac452>] > raid5d+0x202/0x320 [raid456] > Dec 8 22:30:11 avclub kernel: [ 527.515716] [<ffffffff815416b9>] ? > _spin_unlock_irqrestore+0x19/0x30 > Dec 8 22:30:11 avclub kernel: [ 527.515725] [<ffffffff8141704c>] > md_thread+0x5c/0x130 > Dec 8 22:30:11 avclub kernel: [ 527.515735] [<ffffffff81084cb0>] ? > autoremove_wake_function+0x0/0x40 > Dec 8 22:30:11 avclub kernel: [ 527.515743] [<ffffffff81416ff0>] ? > md_thread+0x0/0x130 > Dec 8 22:30:11 avclub kernel: [ 527.515750] [<ffffffff81084936>] > kthread+0x96/0xa0 > Dec 8 22:30:11 avclub kernel: [ 527.515758] [<ffffffff810131ea>] > child_rip+0xa/0x20 > Dec 8 22:30:11 avclub kernel: [ 527.515766] [<ffffffff810848a0>] ? > kthread+0x0/0xa0 > Dec 8 22:30:11 avclub kernel: [ 527.515772] [<ffffffff810131e0>] ? > child_rip+0x0/0x20 > Dec 8 22:30:11 avclub kernel: [ 527.515776] Code: 81 ea d8 1f 00 00 > 48 3b 42 20 73 07 48 8b 50 f9 31 c0 c3 31 d2 48 c7 c0 f2 ff ff ff c3 > 90 90 90 48 89 f8 89 d1 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 66 > 66 66 66 2e 0f 1f 84 00 00 00 00 00 > Dec 8 22:30:11 avclub kernel: [ 527.515842] RIP > [<ffffffff812be15b>] memcpy_c+0xb/0x20 > Dec 8 22:30:11 avclub kernel: [ 527.515850] RSP <ffff880408985c18> > Dec 8 22:30:11 avclub kernel: [ 527.515857] ---[ end trace > 5146b1cc8ebe8dc1 ]--- > Dec 8 22:30:11 avclub kernel: [ 527.515865] note: md2_raid6[528] > exited with preempt_count 2 > Dec 8 22:32:52 avclub kernel: Kernel logging (proc) stopped. > > I believe that last line shows me giving up. I am sad. > Thankfully, after rebooting into single user mode, I was able to mdadm > --assemble the array, and it appears to be working. Boy that was a > rush! > $ uname -aLinux avclub 2.6.32-32-generic #62-Ubuntu SMP Wed Apr 20 > 21:52:38 UTC 2011 x86_64 GNU/Linux > Let me know if I can provide any other information. > Thanks for the report. It seems that as part of the reshape, md is trying to copy to an invalid memory address. It copies from 0xffff880369717000 (RSI) to 0xdb73880000000000 (rdi). The latter is clearly invalid. I have no idea how this might be happening. My best guess is that 'ddidx' in handle_stripe_expansion is getting a bad value but I cannot see how that would happen. If you have reasonable backups you could try again and see if it still fails. Maybe it was a one-off. Not sure what else to suggest. It might be fixed in a newer kernel, or it might not... NeilBrown [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 828 bytes --] ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: failed reshape! 2011-12-12 3:39 ` NeilBrown @ 2011-12-12 12:12 ` Gavin Peters (蓋文彼德斯) 0 siblings, 0 replies; 3+ messages in thread From: Gavin Peters (蓋文彼德斯) @ 2011-12-12 12:12 UTC (permalink / raw) To: NeilBrown; +Cc: linux-raid 2011/12/11 NeilBrown <neilb@suse.de>: > On Fri, 9 Dec 2011 08:53:42 -0500 Gavin Peters (蓋文彼德斯) <gavin@ytz.ca> > wrote: > >> I tried to reshape today, a raid6 array from seven devices up to >> eight. I ran mdadm 3.2.2, something like >> >> # mdadm /dev/md2 --grow -n 8 --layout=preserve >> >> and then, blammo! >> >> Dec 8 22:30:10 avclub kernel: [ 527.094708] RAID5 conf printout: >> Dec 8 22:30:10 avclub kernel: [ 527.094712] --- rd:8 wd:8 >> Dec 8 22:30:10 avclub kernel: [ 527.094714] disk 0, o:1, dev:sdc6 >> Dec 8 22:30:10 avclub kernel: [ 527.094715] disk 1, o:1, dev:sdf6 >> Dec 8 22:30:10 avclub kernel: [ 527.094717] disk 2, o:1, dev:sda6 >> Dec 8 22:30:10 avclub kernel: [ 527.094718] disk 3, o:1, dev:sdd6 >> Dec 8 22:30:10 avclub kernel: [ 527.094719] disk 4, o:1, dev:sdb6 >> Dec 8 22:30:10 avclub kernel: [ 527.094720] disk 5, o:1, dev:sde6 >> Dec 8 22:30:10 avclub kernel: [ 527.094721] disk 6, o:1, dev:sdg6 >> Dec 8 22:30:10 avclub kernel: [ 527.094722] disk 7, o:1, dev:sdh6 >> Dec 8 22:30:10 avclub kernel: [ 527.094876] md: reshape of RAID array md2 >> Dec 8 22:30:10 avclub kernel: [ 527.094886] md: minimum _guaranteed_ >> speed: 40000 KB/sec/disk. >> Dec 8 22:30:10 avclub kernel: [ 527.094892] md: using maximum >> available idle IO bandwidth (but not more than 200000 KB/sec) for >> reshape. >> Dec 8 22:30:10 avclub kernel: [ 527.094912] md: using 128k window, >> over a total of 1371476928 blocks. >> Dec 8 22:30:11 avclub mdadm[2959]: RebuildStarted event detected on >> md device /dev/md2 >> Dec 8 22:30:11 avclub kernel: [ 527.515359] general protection >> fault: 0000 [#1] SMP >> Dec 8 22:30:11 avclub kernel: [ 527.515370] last sysfs file: >> /sys/devices/virtual/block/md2/md/sync_speed >> Dec 8 22:30:11 avclub kernel: [ 527.515376] CPU 5 >> Dec 8 22:30:11 avclub kernel: [ 527.515381] Modules linked in: >> binfmt_misc nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc >> snd_usb_audio snd_usb_lib snd_hda_codec_atihdmi fbcon tileblit font >> bitblit softcursor >> vga16fb vgastate snd_hda_codec_via snd_hda_intel snd_pcm_oss >> snd_hda_codec snd_mixer_ >> oss snd_hwdep snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi >> snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device radeon >> ttm asus_atk0110 drm_kms_helper ppdev snd drm i2c_algo_bit parport_pc >> edac_core edac_mce_amd gspca_zc3xx gspca_main videodev v4l1_compat >> v4l2_compat_ioctl32 soundcore snd_page_alloc i2c_piix4 shpchp lp >> parport tcp_vegas raid10 raid456 async_pq async_xor xor async_memcpy >> usbhid async_raid6_recov hid raid6_pq async_tx raid1 raid0 pata_atiixp >> r8169 mii multipath ahci linear [last unloaded: kvm] >> Dec 8 22:30:11 avclub kernel: [ 527.515500] Pid: 528, comm: >> md2_raid6 Not tainted 2.6.32-32-generic #62-Ubuntu System Product Name >> Dec 8 22:30:11 avclub kernel: [ 527.515507] RIP: >> 0010:[<ffffffff812be15b>] [<ffffffff812be15b>] memcpy_c+0xb/0x20 >> Dec 8 22:30:11 avclub kernel: [ 527.515526] RSP: >> 0018:ffff880408985c18 EFLAGS: 00010246 >> Dec 8 22:30:11 avclub kernel: [ 527.515531] RAX: db73880000000000 >> RBX: ffff880408984000 RCX: 0000000000000200 >> Dec 8 22:30:11 avclub kernel: [ 527.515537] RDX: 0000000000000000 >> RSI: ffff880369717000 RDI: db73880000000000 >> Dec 8 22:30:11 avclub kernel: [ 527.515543] RBP: ffff880408985c80 >> R08: 0000000000001000 R09: ffff880408985ca0 >> Dec 8 22:30:11 avclub kernel: [ 527.515548] R10: 0000000000000000 >> R11: 0000000000000000 R12: ffff880408985ca0 >> Dec 8 22:30:11 avclub kernel: [ 527.515553] R13: ffff880369741290 >> R14: 0000000000000000 R15: 0000000000000000 >> Dec 8 22:30:11 avclub kernel: [ 527.515560] FS: >> 00007f465923d7a0(0000) GS:ffff880028340000(0000) >> knlGS:00000000f6990760 >> Dec 8 22:30:11 avclub kernel: [ 527.515566] CS: 0010 DS: 0018 ES: >> 0018 CR0: 000000008005003b >> Dec 8 22:30:11 avclub kernel: [ 527.515571] CR2: 00007fe6aaf92000 >> CR3: 00000003c3a5e000 CR4: 00000000000006e0 >> Dec 8 22:30:11 avclub kernel: [ 527.515576] DR0: 0000000000000000 >> DR1: 0000000000000000 DR2: 0000000000000000 >> Dec 8 22:30:11 avclub kernel: [ 527.515582] DR3: 0000000000000000 >> DR6: 00000000ffff0ff0 DR7: 0000000000000400 >> Dec 8 22:30:11 avclub kernel: [ 527.515589] Process md2_raid6 (pid: >> 528, threadinfo ffff880408984000, task ffff88040b210000) >> Dec 8 22:30:11 avclub kernel: [ 527.515593] Stack: >> Dec 8 22:30:11 avclub kernel: [ 527.515596] ffffffffa004a0e7 >> ffff880408985c50 0000000000000000 0000000000000000 >> Dec 8 22:30:11 avclub kernel: [ 527.515604] <0> ffffea000bf10d08 >> 0000000000000000 0000000000001000 ffff880408985c80 >> Dec 8 22:30:11 avclub kernel: [ 527.515614] <0> 0000000000000000 >> ffff8803696a6930 ffff880369741290 ffff880408985d70 >> Dec 8 22:30:11 avclub kernel: [ 527.515624] Call Trace: >> Dec 8 22:30:11 avclub kernel: [ 527.515639] [<ffffffffa004a0e7>] ? >> async_memcpy+0xe7/0x25c [async_memcpy] >> Dec 8 22:30:11 avclub kernel: [ 527.515654] [<ffffffffa00aaabb>] >> handle_stripe_expansion+0x14b/0x1e0 [raid456] >> Dec 8 22:30:11 avclub kernel: [ 527.515668] [<ffffffffa00ab113>] >> handle_stripe6+0x5c3/0xb40 [raid456] >> Dec 8 22:30:11 avclub kernel: [ 527.515680] [<ffffffffa00a794c>] ? >> __release_stripe+0xcc/0x1c0 [raid456] >> Dec 8 22:30:11 avclub kernel: [ 527.515692] [<ffffffffa00ac055>] >> handle_stripe+0x25/0x30 [raid456] >> Dec 8 22:30:11 avclub kernel: [ 527.515703] [<ffffffffa00ac452>] >> raid5d+0x202/0x320 [raid456] >> Dec 8 22:30:11 avclub kernel: [ 527.515716] [<ffffffff815416b9>] ? >> _spin_unlock_irqrestore+0x19/0x30 >> Dec 8 22:30:11 avclub kernel: [ 527.515725] [<ffffffff8141704c>] >> md_thread+0x5c/0x130 >> Dec 8 22:30:11 avclub kernel: [ 527.515735] [<ffffffff81084cb0>] ? >> autoremove_wake_function+0x0/0x40 >> Dec 8 22:30:11 avclub kernel: [ 527.515743] [<ffffffff81416ff0>] ? >> md_thread+0x0/0x130 >> Dec 8 22:30:11 avclub kernel: [ 527.515750] [<ffffffff81084936>] >> kthread+0x96/0xa0 >> Dec 8 22:30:11 avclub kernel: [ 527.515758] [<ffffffff810131ea>] >> child_rip+0xa/0x20 >> Dec 8 22:30:11 avclub kernel: [ 527.515766] [<ffffffff810848a0>] ? >> kthread+0x0/0xa0 >> Dec 8 22:30:11 avclub kernel: [ 527.515772] [<ffffffff810131e0>] ? >> child_rip+0x0/0x20 >> Dec 8 22:30:11 avclub kernel: [ 527.515776] Code: 81 ea d8 1f 00 00 >> 48 3b 42 20 73 07 48 8b 50 f9 31 c0 c3 31 d2 48 c7 c0 f2 ff ff ff c3 >> 90 90 90 48 89 f8 89 d1 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 66 >> 66 66 66 2e 0f 1f 84 00 00 00 00 00 >> Dec 8 22:30:11 avclub kernel: [ 527.515842] RIP >> [<ffffffff812be15b>] memcpy_c+0xb/0x20 >> Dec 8 22:30:11 avclub kernel: [ 527.515850] RSP <ffff880408985c18> >> Dec 8 22:30:11 avclub kernel: [ 527.515857] ---[ end trace >> 5146b1cc8ebe8dc1 ]--- >> Dec 8 22:30:11 avclub kernel: [ 527.515865] note: md2_raid6[528] >> exited with preempt_count 2 >> Dec 8 22:32:52 avclub kernel: Kernel logging (proc) stopped. >> >> I believe that last line shows me giving up. I am sad. >> Thankfully, after rebooting into single user mode, I was able to mdadm >> --assemble the array, and it appears to be working. Boy that was a >> rush! >> $ uname -aLinux avclub 2.6.32-32-generic #62-Ubuntu SMP Wed Apr 20 >> 21:52:38 UTC 2011 x86_64 GNU/Linux >> Let me know if I can provide any other information. >> > > Thanks for the report. > > It seems that as part of the reshape, md is trying to copy to an invalid > memory address. > It copies from 0xffff880369717000 (RSI) to 0xdb73880000000000 (rdi). > The latter is clearly invalid. > > I have no idea how this might be happening. My best guess is that 'ddidx' in > handle_stripe_expansion is getting a bad value but I cannot see how that > would happen. > > If you have reasonable backups you could try again and see if it still fails. > Maybe it was a one-off. That appears to be the case. After I rebooted, I was able to assemble the raid, and it continued the sync with no apparent loss of data. Naturally, during that resync, I lost a drive, which I'm now doing a series of resyncs to recover from (the drive reappeared on reboot, so I can't quite figure out what's up with it).... Hopefully, sometime in the next day or two I'll have a computer that works, hasn't lost data, and is not running a RAID resync! If I can be of any help debugging the problem I had, let me know. Otherwise, I hope you don't hear from me again. :-D - Gavin > > Not sure what else to suggest. It might be fixed in a newer kernel, or it > might not... > > NeilBrown > > -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2011-12-12 12:12 UTC | newest] Thread overview: 3+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-12-09 13:53 failed reshape! Gavin Peters (蓋文彼德斯) 2011-12-12 3:39 ` NeilBrown 2011-12-12 12:12 ` Gavin Peters (蓋文彼德斯)
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox