* XFS shutting down due to IO timeout on SATA disk (pata_via for CX700)
@ 2008-09-11 17:35 Bruno Prémont
2008-09-15 8:55 ` Tejun Heo
0 siblings, 1 reply; 11+ messages in thread
From: Bruno Prémont @ 2008-09-11 17:35 UTC (permalink / raw)
To: Linux Kernel, linux-ide, Jeff Garzik
Since some time one of my systems "freezes" after limited uptime (a few hours), usually
during package compilation process.
This seems to happen only with recent kernel versions (2.6.27-rc*), don't remember if
it also happened with 2.6.26 (though I'm pretty sure it did not happen with early
2.6.2x series)
Unfortunately this always shutdowns the root filesystem rendering system unusable.
The kernel output below was generated by 2.6.27-rc5-git9, same symptoms happened with
other -rc releases of 2.6.27 though I couldn't look at dmesg because it happens to /
and I only enabled networked syslog pretty recently on that box in order to find out
what happens.
Unfortunately either the chipset or the BIOS do not support AHCI, for the SATA
controller as the only choice for SATA offered by BIOS is: IDE.
Is this a known issue? At least there seem to be similar ATA exceptions happening
lately according to search results returned by google when looking for the error
messages (exception and originating command).
-- improvement suggestion --
To keep the system running it would be nice if the failing command could be re-issued
after resetting the link and rediscovering the drive, that is, pushing the error to
upper layers only after new failure when retrying the operation following the reset.
-- end of suggestion --
If kernel config or complete output of dmesg is of some help, please let me know.
In case there are some tuning options to try in order to pinpoint the cause I can try
them out, that system is not in production use. (according to some of the messages I
found it could be related to drive cache flushing)
Bruno
lspci output:
00:00.0 Host bridge [0600]: VIA Technologies, Inc. CX700 Host Bridge [1106:0324] (rev 03)
00:00.1 Host bridge [0600]: VIA Technologies, Inc. CX700 Host Bridge [1106:1324]
00:00.2 Host bridge [0600]: VIA Technologies, Inc. CX700 Host Bridge [1106:2324]
00:00.3 Host bridge [0600]: VIA Technologies, Inc. CX700 Host Bridge [1106:3324]
00:00.4 Host bridge [0600]: VIA Technologies, Inc. CX700 Host Bridge [1106:4324]
00:00.7 Host bridge [0600]: VIA Technologies, Inc. CX700 Host Bridge [1106:7324]
00:01.0 PCI bridge [0604]: VIA Technologies, Inc. VT8237 PCI Bridge [1106:b198]
00:0f.0 IDE interface [0101]: VIA Technologies, Inc. Device [1106:0581]
00:10.0 USB Controller [0c03]: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller [1106:3038] (rev 90)
00:10.1 USB Controller [0c03]: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller [1106:3038] (rev 90)
00:10.2 USB Controller [0c03]: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller [1106:3038] (rev 90)
00:10.4 USB Controller [0c03]: VIA Technologies, Inc. USB 2.0 [1106:3104] (rev 90)
00:11.0 ISA bridge [0601]: VIA Technologies, Inc. CX700 PCI to ISA Bridge [1106:8324]
00:11.7 Host bridge [0600]: VIA Technologies, Inc. CX700 Internal Module Bus [1106:324e]
00:13.0 Host bridge [0600]: VIA Technologies, Inc. CX700 Host Bridge [1106:324b]
00:13.1 PCI bridge [0604]: VIA Technologies, Inc. CX700 PCI to PCI Bridge [1106:324a]
01:00.0 VGA compatible controller [0300]: VIA Technologies, Inc. CX700M2 UniChrome PRO II Graphics [1106:3157] (rev 03)
02:08.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet [10ec:8169] (rev 10)
80:01.0 Audio device [0403]: VIA Technologies, Inc. VIA High Definition Audio Controller [1106:3288] (rev 10)
Hard-drive details as reported by hdparm -I:
/dev/sda:
ATA device, with non-removable media
Model Number: FUJITSU MHY2250BH
Serial Number: K407T7A25THF
Firmware Revision: 0000000B
Transport: Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5; Revision: ATA8-AST T13 Project D1697
Revision 0b
Standards:
Used: ATA-8-ACS revision 3f
Supported: 8 7 6 5
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 488397168
device size with M = 1024*1024: 238475 MBytes
device size with M = 1000*1000: 250059 MBytes (250 GB)
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec'd by Standard, no device specific minimum
R/W multiple sector transfer: Max = 16 Current = 16
Advanced power management level: 128
Recommended acoustic management value: 254, current value: 254
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* SMART feature set
Security Mode feature set
* Power Management feature set
* Write cache
* Look-ahead
Host Protected Area feature set
* WRITE_BUFFER command
* READ_BUFFER command
* DOWNLOAD_MICROCODE
* Advanced Power Management feature set
SET_MAX security extension
* Automatic Acoustic Management feature set
* 48-bit Address feature set
* Device Configuration Overlay feature set
* Mandatory FLUSH_CACHE
* FLUSH_CACHE_EXT
* SMART error logging
* SMART self-test
* General Purpose Logging feature set
* WRITE_{DMA|MULTIPLE}_FUA_EXT
* 64-bit World wide name
* IDLE_IMMEDIATE with UNLOAD
Disable Data Transfer After Error Detection
* WRITE_UNCORRECTABLE_EXT command
* {READ,WRITE}_DMA_EXT_GPL commands
* Segmented DOWNLOAD_MICROCODE
* SATA-I signaling speed (1.5Gb/s)
* Native Command Queueing (NCQ)
* Host-initiated interface power management
* Phy event counters
DMA Setup Auto-Activate optimization
Device-initiated interface power management
* Software settings preservation
* SMART Command Transport (SCT) feature set
* SCT LBA Segment Access (AC2)
* SCT Error Recovery Control (AC3)
* SCT Features Control (AC4)
* SCT Data Tables (AC5)
Security:
Master password revision code = 65534
supported
not enabled
not locked
not frozen
not expired: security count
not supported: enhanced erase
250min for SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 5000e040f1a7bd
NAA : 5
IEEE OUI : e
Unique ID : 040f1a7bd
Checksum: correct
Kernel messages related to driver initialization:
[ 2.568109] pata_via 0000:00:0f.0: version 0.3.3
[ 2.568313] scsi0 : pata_via
[ 2.568748] scsi1 : pata_via
[ 2.573314] ata1: PATA max UDMA/133 cmd 0x1f0 ctl 0x3f6 bmdma 0xff00 irq 14
[ 2.573418] ata2: PATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0xff08 irq 15
[ 2.760280] ata1.00: ATA-8: FUJITSU MHY2250BH, 0000000B, max UDMA/100
[ 2.760422] ata1.00: 488397168 sectors, multi 16: LBA48 NCQ (depth 0/32)
[ 2.800304] ata1.00: configured for UDMA/100
[ 2.971844] scsi 0:0:0:0: Direct-Access ATA FUJITSU MHY2250B 0000 PQ: 0 ANSI: 5
[ 2.972976] sd 0:0:0:0: [sda] 488397168 512-byte hardware sectors (250059 MB)
[ 2.973192] sd 0:0:0:0: [sda] Write Protect is off
[ 2.973321] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 2.973453] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 2.973938] sd 0:0:0:0: [sda] 488397168 512-byte hardware sectors (250059 MB)
[ 2.974142] sd 0:0:0:0: [sda] Write Protect is off
[ 2.974270] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 2.974399] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 2.974588] sda: sda1 sda2 sda3 sda4 sda5 sda6
[ 3.201488] sd 0:0:0:0: [sda] Attached SCSI disk
Kernel error output related to XFS shutdown:
[ 9352.420180] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[ 9352.420247] ata1.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
[ 9352.420261] res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 9352.420289] ata1.00: status: { DRDY }
[ 9352.420353] ata1: soft resetting link
[ 9352.650317] ata1.00: configured for UDMA/100
[ 9352.650374] end_request: I/O error, dev sda, sector 6410215
[ 9352.650432] ata1: EH complete
[ 9352.650654] I/O error in filesystem ("sda3") meta-data dev sda3 block 0x203fa6 ("xlog_iodone") error 5 buf count 32768
[ 9352.650824] xfs_force_shutdown(sda3,0x2) called from line 1027 of file /usr/src/linux-2.6.27-rc5-git9/fs/xfs/xfs_log.c. Return address = 0xc020ccba
[ 9352.651304] Filesystem "sda3": Log I/O Error Detected. Shutting down filesystem: sda3
[ 9352.651332] Please umount the filesystem, and rectify the problem(s)
[ 9352.651395] xfs_force_shutdown(sda3,0x2) called from line 790 of file /usr/src/linux-2.6.27-rc5-git9/fs/xfs/xfs_log.c. Return address = 0xc020dfce
[ 9352.654454] sd 0:0:0:0: [sda] 488397168 512-byte hardware sectors (250059 MB)
[ 9352.659345] xfs_force_shutdown(sda3,0x2) called from line 790 of file /usr/src/linux-2.6.27-rc5-git9/fs/xfs/xfs_log.c. Return address = 0xc020dfce
[ 9352.988239] sd 0:0:0:0: [sda] Write Protect is off
[ 9352.988277] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 9353.026123] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 9383.090107] Filesystem "sda3": xfs_log_force: error 5 returned.
[ 9413.090091] Filesystem "sda3": xfs_log_force: error 5 returned.
[ 9443.090112] Filesystem "sda3": xfs_log_force: error 5 returned.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS shutting down due to IO timeout on SATA disk (pata_via for CX700)
2008-09-11 17:35 XFS shutting down due to IO timeout on SATA disk (pata_via for CX700) Bruno Prémont
@ 2008-09-15 8:55 ` Tejun Heo
2008-09-15 17:02 ` Bruno Prémont
2008-09-15 20:30 ` Mark Lord
0 siblings, 2 replies; 11+ messages in thread
From: Tejun Heo @ 2008-09-15 8:55 UTC (permalink / raw)
To: Bruno Prémont; +Cc: Linux Kernel, linux-ide, Jeff Garzik
(please try to wrap paragraphs for 80 column)
Hello,
Bruno Prémont wrote:
> Since some time one of my systems "freezes" after limited uptime (a
> few hours), usually during package compilation process. This seems
> to happen only with recent kernel versions (2.6.27-rc*), don't
> remember if it also happened with 2.6.26 (though I'm pretty sure it
> did not happen with early 2.6.2x series) Unfortunately this always
> shutdowns the root filesystem rendering system unusable.
>
> The kernel output below was generated by 2.6.27-rc5-git9, same
> symptoms happened with other -rc releases of 2.6.27 though I
> couldn't look at dmesg because it happens to / and I only enabled
> networked syslog pretty recently on that box in order to find out
> what happens.
>
> Unfortunately either the chipset or the BIOS do not support AHCI,
> for the SATA controller as the only choice for SATA offered by BIOS
> is: IDE.
>
> Is this a known issue? At least there seem to be similar ATA
> exceptions happening lately according to search results returned by
> google when looking for the error messages (exception and
> originating command).
> Kernel error output related to XFS shutdown:
> [ 9352.420180] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
> [ 9352.420247] ata1.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
> [ 9352.420261] res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Timeout on FLUSH_EXT. That's a bad sign. Patch to retry FLUSH is
pending but at any rate FLUSH failure is often accompanied by loss of
data and XFS is doing the right thing of giving up on it.
Can you please post the result of "smartctl -a /dev/sda"?
--
tejun
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS shutting down due to IO timeout on SATA disk (pata_via for CX700)
2008-09-15 8:55 ` Tejun Heo
@ 2008-09-15 17:02 ` Bruno Prémont
2008-09-15 20:43 ` Tejun Heo
2008-09-15 20:30 ` Mark Lord
1 sibling, 1 reply; 11+ messages in thread
From: Bruno Prémont @ 2008-09-15 17:02 UTC (permalink / raw)
To: Tejun Heo; +Cc: Linux Kernel, linux-ide, Jeff Garzik
Hello,
On Mon, 15 September 2008 Tejun Heo <tj@kernel.org> wrote:
> (please try to wrap paragraphs for 80 column)
I try not to break lines from dmesg, lspci and and other commands'
(formatted) output as those tend to get pretty hard to read when
line-wrapped.
Sorry if I wrapped my text after 80 columns.
> > Kernel error output related to XFS shutdown:
> > [ 9352.420180] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0
> > action 0x6 frozen [ 9352.420247] ata1.00: cmd
> > ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0 [ 9352.420261]
> > res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
>
> Timeout on FLUSH_EXT. That's a bad sign. Patch to retry FLUSH is
> pending but at any rate FLUSH failure is often accompanied by loss of
> data and XFS is doing the right thing of giving up on it.
>
> Can you please post the result of "smartctl -a /dev/sda"?
I checked it though there were no errors logged nor any other information
that would catch attention. The disk/machine is pretty unused (a year old
but low uptime, a few hours those days with uptime)
Anyhow smaprtctl's output is blow.
Bruno
smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
=== START OF INFORMATION SECTION ===
Device Model: FUJITSU MHY2250BH
Serial Number: K407T7A25THF
Firmware Version: 0000000B
User Capacity: 250,059,350,016 bytes
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 3f
Local Time is: Mon Sep 15 18:52:53 2008 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (1009) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 143) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 100 046 Pre-fail Always - 189618
2 Throughput_Performance 0x0005 100 100 030 Pre-fail Offline - 53870592
3 Spin_Up_Time 0x0003 100 100 025 Pre-fail Always - 1
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 168
5 Reallocated_Sector_Ct 0x0033 100 100 024 Pre-fail Always - 8589934592000
7 Seek_Error_Rate 0x000f 100 100 047 Pre-fail Always - 679
8 Seek_Time_Performance 0x0005 100 100 019 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 230
10 Spin_Retry_Count 0x0013 100 100 020 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 149
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 18
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 6674
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 27 (Lifetime Min/Max 22/36)
195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 606
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 441778176
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 253 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x000f 100 100 060 Pre-fail Always - 10312
203 Run_Out_Cancel 0x0002 100 100 000 Old_age Always - 3728055731610
240 Head_Flying_Hours 0x003e 200 200 000 Old_age Always - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS shutting down due to IO timeout on SATA disk (pata_via for CX700)
2008-09-15 8:55 ` Tejun Heo
2008-09-15 17:02 ` Bruno Prémont
@ 2008-09-15 20:30 ` Mark Lord
2008-09-15 20:37 ` Tejun Heo
1 sibling, 1 reply; 11+ messages in thread
From: Mark Lord @ 2008-09-15 20:30 UTC (permalink / raw)
To: Tejun Heo; +Cc: Bruno Prémont, Linux Kernel, linux-ide, Jeff Garzik
Tejun Heo wrote:
> Bruno Prémont wrote:
>> Since some time one of my systems "freezes" after limited uptime (a
>> few hours), usually during package compilation process. This seems
>> to happen only with recent kernel versions (2.6.27-rc*), don't
>> remember if it also happened with 2.6.26 (though I'm pretty sure it
>> did not happen with early 2.6.2x series) Unfortunately this always
>> shutdowns the root filesystem rendering system unusable.
>>
>> The kernel output below was generated by 2.6.27-rc5-git9, same
>> symptoms happened with other -rc releases of 2.6.27 though I
>> couldn't look at dmesg because it happens to / and I only enabled
>> networked syslog pretty recently on that box in order to find out
>> what happens.
...
>> Kernel error output related to XFS shutdown:
>> [ 9352.420180] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
>> [ 9352.420247] ata1.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
>> [ 9352.420261] res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
..
Bruno, please also post the output from these commands:
hdparm --Istdout /dev/sda
smartctl -data -a /dev/sda
Thanks.
> Timeout on FLUSH_EXT. That's a bad sign. Patch to retry FLUSH is
> pending but at any rate FLUSH failure is often accompanied by loss of
> data and XFS is doing the right thing of giving up on it.
..
Tejun, are we *sure* that's really a timeout?
The status shows 0x40 "drive ready" there, aka. "command complete".
I have a client who is also seeing this exact scenario on 750GB drives,
using a patched SLES10 kernel (2.6.16 + libata from 2.6.18 or so).
Smartctl output is clean (no logged errors), and the drives themselves
are fine after a reboot -- necessary since libata/scsi kicked the drive out
of the RAID array.
Something strange is going on here.
????
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS shutting down due to IO timeout on SATA disk (pata_via for CX700)
2008-09-15 20:30 ` Mark Lord
@ 2008-09-15 20:37 ` Tejun Heo
2008-09-16 3:49 ` Mark Lord
0 siblings, 1 reply; 11+ messages in thread
From: Tejun Heo @ 2008-09-15 20:37 UTC (permalink / raw)
To: Mark Lord; +Cc: Bruno Prémont, Linux Kernel, linux-ide, Jeff Garzik
Mark Lord wrote:
>> Timeout on FLUSH_EXT. That's a bad sign. Patch to retry FLUSH is
>> pending but at any rate FLUSH failure is often accompanied by loss of
>> data and XFS is doing the right thing of giving up on it.
> ..
>
> Tejun, are we *sure* that's really a timeout?
> The status shows 0x40 "drive ready" there, aka. "command complete".
Heh... on timeout, libata EH doesn't touch status register as some
controllers lock the whole machine up on that, so the 0x40 is just the
fill value libata used during qc initialization. It definitely
requires clarification.
> I have a client who is also seeing this exact scenario on 750GB drives,
> using a patched SLES10 kernel (2.6.16 + libata from 2.6.18 or so).
Hmm.. most of FLUSH timeouts I've seen are either a dying drive or bad
PSU. There just isn't much which can go wrong from the driver side.
IIRC, there was a problem when the unused part of TF is not cleared
but that was the only one.
> Smartctl output is clean (no logged errors), and the drives themselves
> are fine after a reboot -- necessary since libata/scsi kicked the drive out
> of the RAID array.
>
> Something strange is going on here.
Any chance you can trick the client to hook up the drive to a separate
PSU?
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS shutting down due to IO timeout on SATA disk (pata_via for CX700)
2008-09-15 17:02 ` Bruno Prémont
@ 2008-09-15 20:43 ` Tejun Heo
2008-09-16 8:21 ` Bruno Prémont
2008-09-16 15:07 ` Grant Grundler
0 siblings, 2 replies; 11+ messages in thread
From: Tejun Heo @ 2008-09-15 20:43 UTC (permalink / raw)
To: Bruno Prémont; +Cc: Linux Kernel, linux-ide, Jeff Garzik
Hello,
Bruno Prémont wrote:
> On Mon, 15 September 2008 Tejun Heo <tj@kernel.org> wrote:
>> (please try to wrap paragraphs for 80 column)
> I try not to break lines from dmesg, lspci and and other commands'
> (formatted) output as those tend to get pretty hard to read when
> line-wrapped. Sorry if I wrapped my text after 80 columns.
Yeap, I was talking only about the text. Not wrapping outputs and
code snippets is definitely better.
>> Timeout on FLUSH_EXT. That's a bad sign. Patch to retry FLUSH is
>> pending but at any rate FLUSH failure is often accompanied by loss of
>> data and XFS is doing the right thing of giving up on it.
>>
>> Can you please post the result of "smartctl -a /dev/sda"?
> I checked it though there were no errors logged nor any other information
> that would catch attention. The disk/machine is pretty unused (a year old
> but low uptime, a few hours those days with uptime)
>
> Anyhow smaprtctl's output is blow.
>
> 5 Reallocated_Sector_Ct 0x0033 100 100 024 Pre-fail Always - 8589934592000
Whee... That's unusally high realloc count but I'm not sure whether it
indicates actual problem or it's just the drive's way of saying I'm
okay. But this does look quite suspicious.
> 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 441778176
Hmmm.. Do you happen to have drives of the same model? If so, can you
please check what other drives are reporting?
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS shutting down due to IO timeout on SATA disk (pata_via for CX700)
2008-09-15 20:37 ` Tejun Heo
@ 2008-09-16 3:49 ` Mark Lord
0 siblings, 0 replies; 11+ messages in thread
From: Mark Lord @ 2008-09-16 3:49 UTC (permalink / raw)
To: Tejun Heo; +Cc: Bruno Prémont, Linux Kernel, linux-ide, Jeff Garzik
Tejun Heo wrote:
>..
> Hmm.. most of FLUSH timeouts I've seen are either a dying drive or bad
> PSU. There just isn't much which can go wrong from the driver side.
> IIRC, there was a problem when the unused part of TF is not cleared
> but that was the only one.
>
>> Smartctl output is clean (no logged errors), and the drives themselves
>> are fine after a reboot -- necessary since libata/scsi kicked the drive out
>> of the RAID array.
>>
>> Something strange is going on here.
>
> Any chance you can trick the client to hook up the drive to a separate PSU?
..
No, the failures happen randomly at customer sites, and only since they
"upgraded" to SLES10 with libata. I think the PSUs are probably just fine.
Time to hack the drivers to give proper status on the timeouts, too;
otherwise we won't ever have any clue as to what is really happening.
Cheers
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS shutting down due to IO timeout on SATA disk (pata_via for CX700)
2008-09-15 20:43 ` Tejun Heo
@ 2008-09-16 8:21 ` Bruno Prémont
2008-09-16 17:35 ` Tejun Heo
2008-09-16 15:07 ` Grant Grundler
1 sibling, 1 reply; 11+ messages in thread
From: Bruno Prémont @ 2008-09-16 8:21 UTC (permalink / raw)
To: Tejun Heo; +Cc: Linux Kernel, linux-ide, Jeff Garzik, Mark Lord
Hello,
On Mon, 15 Sep 2008 Tejun Heo <tj@kernel.org> wrote:
> >> Timeout on FLUSH_EXT. That's a bad sign. Patch to retry FLUSH is
> >> pending but at any rate FLUSH failure is often accompanied by loss
> >> of data and XFS is doing the right thing of giving up on it.
> >>
> >> Can you please post the result of "smartctl -a /dev/sda"?
> > I checked it though there were no errors logged nor any other
> > information that would catch attention. The disk/machine is pretty
> > unused (a year old but low uptime, a few hours those days with
> > uptime)
> >
> > Anyhow smaprtctl's output is blow.
> >
>
> > 5 Reallocated_Sector_Ct 0x0033 100 100 024 Pre-fail
> > Always - 8589934592000
>
> Whee... That's unusally high realloc count but I'm not sure whether it
> indicates actual problem or it's just the drive's way of saying I'm
> okay. But this does look quite suspicious.
>
> > 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age
> > Always - 441778176
>
> Hmmm.. Do you happen to have drives of the same model? If so, can you
> please check what other drives are reporting?
>
I don't have other drives of the same model, but 3 ones slightly smaller
in size and previous generation, those are just a few months older (but
also have way more uptime), see below for the smartctl output. I've never
seen ATA errors on them.
These other drives are running on AMD based platform, SB600 SATA with AHCI.
Just for the record, on the VIA there were no errors yet with
2.6.27-rc6-git3 though I ran some compiling on it. As such it *could* be
that one of the patches pushed by Jeff helped:
[libata] LBA28/LBA48 off-by-one bug in ata.h
sata_inic162x: enable LED blinking
ata: duplicate variable sparse warning
Bruno
=============== first drive ====================
smartctl version 5.38 [x86_64-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
=== START OF INFORMATION SECTION ===
Model Family: Fujitsu MHW2 BH
Device Model: FUJITSU MHW2160BH
Serial Number: K102T7727YER
Firmware Version: 0000001C
User Capacity: 160,041,885,696 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 3b
Local Time is: Tue Sep 16 09:50:34 2008 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 649) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 92) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 100 046 Pre-fail Always - 117083
2 Throughput_Performance 0x0005 100 100 030 Pre-fail Offline - 38797312
3 Spin_Up_Time 0x0003 100 100 025 Pre-fail Always - 1
4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 452
5 Reallocated_Sector_Ct 0x0033 100 100 024 Pre-fail Always - 8589934592000
7 Seek_Error_Rate 0x000f 100 100 047 Pre-fail Always - 1461
8 Seek_Time_Performance 0x0005 100 100 019 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 092 092 000 Old_age Always - 4125
10 Spin_Retry_Count 0x0013 100 100 020 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 451
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 13
193 Load_Cycle_Count 0x0032 098 098 000 Old_age Always - 42607
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 30 (Lifetime Min/Max 20/39)
195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 31
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 457375744
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x000f 100 100 060 Pre-fail Always - 1978
203 Run_Out_Cancel 0x0002 100 100 000 Old_age Always - 2628559175426
240 Head_Flying_Hours 0x003e 200 200 000 Old_age Always - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
=================== second drive ====================
smartctl version 5.38 [x86_64-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
=== START OF INFORMATION SECTION ===
Model Family: Fujitsu MHW2 BH
Device Model: FUJITSU MHW2160BH
Serial Number: K102T7727YH5
Firmware Version: 0000001C
User Capacity: 160,041,885,696 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 3b
Local Time is: Tue Sep 16 10:16:17 2008 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 649) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 92) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 100 046 Pre-fail Always - 107032
2 Throughput_Performance 0x0005 100 100 030 Pre-fail Offline - 39190528
3 Spin_Up_Time 0x0003 100 100 025 Pre-fail Always - 1
4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 454
5 Reallocated_Sector_Ct 0x0033 100 100 024 Pre-fail Always - 8589934592000
7 Seek_Error_Rate 0x000f 100 100 047 Pre-fail Always - 841
8 Seek_Time_Performance 0x0005 100 100 019 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 092 092 000 Old_age Always - 4158
10 Spin_Retry_Count 0x0013 100 100 020 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 453
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 10
193 Load_Cycle_Count 0x0032 098 098 000 Old_age Always - 56896
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 29 (Lifetime Min/Max 19/36)
195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 412
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 412483584
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x000f 100 100 060 Pre-fail Always - 8190
203 Run_Out_Cancel 0x0002 100 100 000 Old_age Always - 3732267335218
240 Head_Flying_Hours 0x003e 200 200 000 Old_age Always - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
=========== third drive ========================
smartctl version 5.38 [x86_64-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
=== START OF INFORMATION SECTION ===
Model Family: Fujitsu MHW2 BH
Device Model: FUJITSU MHW2160BH
Serial Number: K102T7727YFK
Firmware Version: 0000001C
User Capacity: 160,041,885,696 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 3b
Local Time is: Tue Sep 16 10:17:49 2008 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 649) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 92) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 100 046 Pre-fail Always - 183032
2 Throughput_Performance 0x0005 100 100 030 Pre-fail Offline - 38862848
3 Spin_Up_Time 0x0003 100 100 025 Pre-fail Always - 1
4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 452
5 Reallocated_Sector_Ct 0x0033 100 100 024 Pre-fail Always - 8589934592000
7 Seek_Error_Rate 0x000f 100 100 047 Pre-fail Always - 997
8 Seek_Time_Performance 0x0005 100 100 019 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 092 092 000 Old_age Always - 4114
10 Spin_Retry_Count 0x0013 100 100 020 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 451
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 15
193 Load_Cycle_Count 0x0032 098 098 000 Old_age Always - 46692
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 30 (Lifetime Min/Max 20/39)
195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 492
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 422903808
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x000f 100 100 060 Pre-fail Always - 5669
203 Run_Out_Cancel 0x0002 100 100 000 Old_age Always - 3732273955621
240 Head_Flying_Hours 0x003e 200 200 000 Old_age Always - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS shutting down due to IO timeout on SATA disk (pata_via for CX700)
2008-09-15 20:43 ` Tejun Heo
2008-09-16 8:21 ` Bruno Prémont
@ 2008-09-16 15:07 ` Grant Grundler
1 sibling, 0 replies; 11+ messages in thread
From: Grant Grundler @ 2008-09-16 15:07 UTC (permalink / raw)
To: Tejun Heo; +Cc: Bruno Prémont, Linux Kernel, linux-ide, Jeff Garzik
On Mon, Sep 15, 2008 at 1:43 PM, Tejun Heo <tj@kernel.org> wrote:
...
>> 5 Reallocated_Sector_Ct 0x0033 100 100 024 Pre-fail Always - 8589934592000
>
> Whee... That's unusally high realloc count but I'm not sure whether it
> indicates actual problem or it's just the drive's way of saying I'm
> okay. But this does look quite suspicious.
The "raw" values are "vendor defined".
Often the 6 bytes of raw data are split up into 1 or 2 byte wide fields.
Any volunteers to submit a patch to smartmontools to dump
"raw values" as hexidecimal by default?
(8589934592000 == 0x7D000000000)
>> 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 441778176
>
> Hmmm.. Do you happen to have drives of the same model? If so, can you
> please check what other drives are reporting?
441778176 == 0x1a550000
Need to see if the vendor will provide documentation to decode these values.
AFAICT, it's not safe to directly compare raw values between vendors since
the raw values don't mean exactly the same thing (e.g. more aggressive retries
might result in few reallocations.)
hth,
grant
>
> Thanks.
>
> --
> tejun
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ide" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS shutting down due to IO timeout on SATA disk (pata_via for CX700)
2008-09-16 8:21 ` Bruno Prémont
@ 2008-09-16 17:35 ` Tejun Heo
2008-09-21 19:51 ` Bruno Prémont
0 siblings, 1 reply; 11+ messages in thread
From: Tejun Heo @ 2008-09-16 17:35 UTC (permalink / raw)
To: Bruno Prémont; +Cc: Linux Kernel, linux-ide, Jeff Garzik, Mark Lord
Hello,
Bruno Prémont wrote:
> I don't have other drives of the same model, but 3 ones slightly smaller
> in size and previous generation, those are just a few months older (but
> also have way more uptime), see below for the smartctl output. I've never
> seen ATA errors on them.
> These other drives are running on AMD based platform, SB600 SATA with AHCI.
So, smartctl output seems fine.
> Just for the record, on the VIA there were no errors yet with
> 2.6.27-rc6-git3 though I ran some compiling on it. As such it *could* be
> that one of the patches pushed by Jeff helped:
> [libata] LBA28/LBA48 off-by-one bug in ata.h
> sata_inic162x: enable LED blinking
> ata: duplicate variable sparse warning
The only patch which could have affected the result is the off-by-one
one; however, FLUSH timeout is a bit unexpected failure mode for the
problem. Can you please keep the system running for a whlie and see
whether the problem occurs?
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: XFS shutting down due to IO timeout on SATA disk (pata_via for CX700)
2008-09-16 17:35 ` Tejun Heo
@ 2008-09-21 19:51 ` Bruno Prémont
0 siblings, 0 replies; 11+ messages in thread
From: Bruno Prémont @ 2008-09-21 19:51 UTC (permalink / raw)
To: Tejun Heo; +Cc: Linux Kernel, linux-ide, Jeff Garzik, Mark Lord
Hello,
On Tue, 16 September 2008 Tejun Heo <tj@kernel.org> wrote:
> > Just for the record, on the VIA there were no errors yet with
> > 2.6.27-rc6-git3 though I ran some compiling on it. As such it
> > *could* be that one of the patches pushed by Jeff helped:
> > [libata] LBA28/LBA48 off-by-one bug in ata.h
> > sata_inic162x: enable LED blinking
> > ata: duplicate variable sparse warning
>
> The only patch which could have affected the result is the off-by-one
> one; however, FLUSH timeout is a bit unexpected failure mode for the
> problem. Can you please keep the system running for a whlie and see
> whether the problem occurs?
I haven't seen the error happen yet with 2.6.27-rc6-git3, so that off
by one bug might have had an incidence.
No warranty on that as I did also slightly change power-saving settings
for the disk around the time I updated kernel (-B254, -S241 parameters
to hdparm) and have not seen the error yet on 2.6.27-rc5-git9 since
then.
Will do more testing as time permits over the next days and report if I
find anything that looks useful (e.g. better way than time to trigger
the error or some of the hdparm options influencing the probability)
In case there is a patch that would eventually be able to get more info
when the error occurs I would be happy to apply it.
Bruno
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2008-09-21 20:02 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-09-11 17:35 XFS shutting down due to IO timeout on SATA disk (pata_via for CX700) Bruno Prémont
2008-09-15 8:55 ` Tejun Heo
2008-09-15 17:02 ` Bruno Prémont
2008-09-15 20:43 ` Tejun Heo
2008-09-16 8:21 ` Bruno Prémont
2008-09-16 17:35 ` Tejun Heo
2008-09-21 19:51 ` Bruno Prémont
2008-09-16 15:07 ` Grant Grundler
2008-09-15 20:30 ` Mark Lord
2008-09-15 20:37 ` Tejun Heo
2008-09-16 3:49 ` Mark Lord
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).