* crash/dmesg - RAID locking up? @ 2010-03-28 22:31 Mark Knecht [not found] ` <4BB0267E.8080605@stud.tu-ilmenau.de> 0 siblings, 1 reply; 2+ messages in thread From: Mark Knecht @ 2010-03-28 22:31 UTC (permalink / raw) To: Linux-RAID So I'm giving up on mdadm/RAID1/WD10EARS Green Drive experiment for the next day or two. I've tried 4 times in the last two days to see if I can get this machine to boot using RAID. I keep running into crashes that look like the one below. I will try a non-RAID install and see if the hardware is stable and then come back to this later if appropriate. I was running a March, 2010 Gentoo install CD when this happened. One terminal doing bash completion of all things was hung with 1 CPU stuck in a 100% wait condition. New terminals continued to work if I didn't try to access the hard drive. I can run things from the CD. I have never seen a Gentoo install use nearly 2GB so something is really going wrong here. Cheers, Mark livecd ~ # cat /proc/mdstat Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md3 : active raid1 sdc3[2] sdb3[1] sda3[0] 31463232 blocks [3/3] [UUU] unused devices: <none> livecd ~ # livecd ~ # top top - 06:24:39 up 1:28, 3 users, load average: 1.95, 2.52, 2.36 Tasks: 182 total, 1 running, 181 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu5 : 0.0%us, 0.0%sy, 0.0%ni, 0.0%id,100.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 6105068k total, 1822336k used, 4282732k free, 172016k buffers Swap: 12602964k total, 0k used, 12602964k free, 1326360k cached livecd ~ # uname -a Linux livecd 2.6.31-gentoo-r6 #1 SMP Thu Mar 4 13:09:55 UTC 2010 x86_64 Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz GenuineIntel GNU/Linux livecd ~ # Adding 4200988k swap on /dev/sda2. Priority:1 extents:1 across:4200988k Adding 4200988k swap on /dev/sdb2. Priority:1 extents:1 across:4200988k Adding 4200988k swap on /dev/sdc2. Priority:1 extents:1 across:4200988k kjournald starting. Commit interval 5 seconds EXT3 FS on md3, internal journal EXT3-fs: mounted filesystem with writeback data mode. INFO: task bash:17348 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. bash D ffff8800280d3e00 0 17348 17345 0x00000000 ffff8801afb1a670 0000000000000082 0000000000000000 00000000000000ff ffff8801a788f038 0000000000004000 000000000000fe00 000000000000c878 ffff8801af1e39b8 ffff8801af1e39d0 ffff8801af91c730 ffff8801afb1a8e8 Call Trace: [<ffffffff81209a92>] ? generic_make_request+0x220/0x26a [<ffffffff8109c248>] ? sync_buffer+0x0/0x40 [<ffffffff8137a4fc>] ? io_schedule+0x2d/0x3a [<ffffffff8109c283>] ? sync_buffer+0x3b/0x40 [<ffffffff8137a879>] ? __wait_on_bit+0x41/0x70 [<ffffffff8109c248>] ? sync_buffer+0x0/0x40 [<ffffffff8137a913>] ? out_of_line_wait_on_bit+0x6b/0x77 [<ffffffff810438b2>] ? wake_bit_function+0x0/0x23 [<ffffffff81132dff>] ? log_do_checkpoint+0x366/0x429 [<ffffffff81132f92>] ? __log_wait_for_space+0xd0/0x176 [<ffffffff81130e8e>] ? start_this_handle+0x2b2/0x2fb [<ffffffff8108a698>] ? filldir+0x0/0xb7 [<ffffffff8108a698>] ? filldir+0x0/0xb7 [<ffffffff8113102d>] ? journal_start+0x9a/0xce [<ffffffff810f3ae2>] ? ext3_dirty_inode+0x28/0x7b [<ffffffff8109745d>] ? __mark_inode_dirty+0x26/0xf5 [<ffffffff8108ebca>] ? touch_atime+0x10a/0x113 [<ffffffff8108a85c>] ? vfs_readdir+0x7a/0x97 [<ffffffff8108a9b1>] ? sys_getdents+0x7a/0xc1 [<ffffffff8100adab>] ? system_call_fastpath+0x16/0x1b INFO: task kjournald:17466 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kjournald D ffff8800280bbe00 0 17466 2 0x00000000 ffff8801adf9d890 0000000000000046 0000000000000000 0000000000000000 ffff8801adcbde44 0000000000004000 000000000000fe00 000000000000c878 0000000800000050 ffff88017a99aa40 ffff8801af90a150 ffff8801adf9db08 Call Trace: [<ffffffff812dd063>] ? md_make_request+0xb6/0xf1 [<ffffffff8109c248>] ? sync_buffer+0x0/0x40 [<ffffffff8137a4fc>] ? io_schedule+0x2d/0x3a [<ffffffff8109c283>] ? sync_buffer+0x3b/0x40 [<ffffffff8137a879>] ? __wait_on_bit+0x41/0x70 [<ffffffff8109c248>] ? sync_buffer+0x0/0x40 [<ffffffff8137a913>] ? out_of_line_wait_on_bit+0x6b/0x77 [<ffffffff810438b2>] ? wake_bit_function+0x0/0x23 [<ffffffff8109c637>] ? sync_dirty_buffer+0x72/0xaa [<ffffffff81131b8e>] ? journal_commit_transaction+0xa74/0xde2 [<ffffffff8103abcc>] ? lock_timer_base+0x26/0x4b [<ffffffff81043884>] ? autoremove_wake_function+0x0/0x2e [<ffffffff81134804>] ? kjournald+0xe3/0x206 [<ffffffff81043884>] ? autoremove_wake_function+0x0/0x2e [<ffffffff81134721>] ? kjournald+0x0/0x206 [<ffffffff81043591>] ? kthread+0x8b/0x93 [<ffffffff8100bd3a>] ? child_rip+0xa/0x20 [<ffffffff81043506>] ? kthread+0x0/0x93 [<ffffffff8100bd30>] ? child_rip+0x0/0x20 livecd ~ # ^ permalink raw reply [flat|nested] 2+ messages in thread
[parent not found: <4BB0267E.8080605@stud.tu-ilmenau.de>]
* Re: crash/dmesg - RAID locking up? [not found] ` <4BB0267E.8080605@stud.tu-ilmenau.de> @ 2010-03-29 12:44 ` Mark Knecht 0 siblings, 0 replies; 2+ messages in thread From: Mark Knecht @ 2010-03-29 12:44 UTC (permalink / raw) To: st0ff; +Cc: Linux-RAID On Sun, Mar 28, 2010 at 9:03 PM, Stefan /*St0fF*/ Hübner <stefan.huebner@stud.tu-ilmenau.de> wrote: > Am 29.03.2010 00:31, schrieb Mark Knecht: >> So I'm giving up on mdadm/RAID1/WD10EARS Green Drive experiment for >> the next day or two. I've tried 4 times in the last two days to see if >> I can get this machine to boot using RAID. I keep running into crashes >> that look like the one below. I will try a non-RAID install and see if >> the hardware is stable and then come back to this later if >> appropriate. >> >> I was running a March, 2010 Gentoo install CD when this happened. One >> terminal doing bash completion of all things was hung with 1 CPU stuck >> in a 100% wait condition. New terminals continued to work if I didn't >> try to access the hard drive. I can run things from the CD. > > Hi Mark, that's been the desktop-nature of the disks. Please run > smartctl -A /dev/sdX on those disks and see if any have _raw_ values of > "pending_sector_count" larger than zero. > If so, let the disks get exchanged by your vendor, stating: "data loss > already occurred" or "computer freezes with this disk connected". > They'll (at least WD) exchange the disk without questions. > > To clarify: it seems to me that at least one disk has bad magnetization > on at least one platter, causing it to enter error correction code all > the time and by that not responding... > <SNIP> None of the items that look like errors to me have counts greater than 0 so far. I do not see "pending_sector_Count" reported by these drives. - Mark keeper ~ # smartctl -A /dev/sda smartctl 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 136 134 021 Pre-fail Always - 6175 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 13 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 21 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 11 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 10 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 345 194 Temperature_Celsius 0x0022 125 102 000 Old_age Always - 22 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 keeper ~ # smartctl -A /dev/sdb smartctl 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 134 130 021 Pre-fail Always - 6258 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 14 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 21 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 12 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 11 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 203 194 Temperature_Celsius 0x0022 125 098 000 Old_age Always - 22 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 keeper ~ # smartctl -A /dev/sdc smartctl 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 131 128 021 Pre-fail Always - 6416 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 14 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 21 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 12 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 11 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 199 194 Temperature_Celsius 0x0022 125 099 000 Old_age Always - 22 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 keeper ~ # I ran the short selftest in the devices and there were no errors reported. I've started the long selftest but that takes 4 hours to complete so I'll have to report on that later. keeper ~ # smartctl -l error /dev/sda smartctl 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART Error Log Version: 1 No Errors Logged keeper ~ # smartctl -l error /dev/sdb smartctl 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART Error Log Version: 1 No Errors Logged keeper ~ # smartctl -l error /dev/sdc smartctl 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART Error Log Version: 1 No Errors Logged keeper ~ # Thanks, Mark -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2010-03-29 12:44 UTC | newest] Thread overview: 2+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-03-28 22:31 crash/dmesg - RAID locking up? Mark Knecht [not found] ` <4BB0267E.8080605@stud.tu-ilmenau.de> 2010-03-29 12:44 ` Mark Knecht
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).