[eSATA] 2.6.28.7: IO errors

[eSATA] 2.6.28.7: IO errors

[Guennadi Liakhovetski]

Hi,

I'm using an external eSATA / USB / Firewire 2x1T RAID from WD in RAID1 
configuration for backups with ext4 under 2.6.28.7. I've got a couple of 
problems with it before, and today a backup aborted with error messages 
below. Actually, those errors come from two backup attempts: the first one 
over NFS, and the second one locally. The system is one year old with an 
Intel chipset:

00:00.0 Host bridge: Intel Corporation Unknown device 29b0 (rev 02)

with a Core-2D 2.66GHz, so, it does have 3Gbps SATA, and below the driver 
drops down to 1.5Gbps.

The problems I had before with the disk were, that it has a badly built 
eSATA connector, the connector doesn't sit firmly enough in it. So, the 
main question, I guess - cann errors below be caused by that? I think, 
I'll have to contact WD for an update, or use Firewire for now...

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer


ata4.00: exception Emask 0x40 SAct 0x0 SErr 0x80800 action 0x6 frozen
ata4: SError: { HostInt 10B8B }
ata4.00: cmd 35/00:f8:27:87:4a/00:03:09:00:00/e0 tag 0 dma 520192 out
         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x44 (timeout)
ata4.00: status: { DRDY }
ata4: hard resetting link
ata4: SATA link down (SStatus 0 SControl 300)
ata4: hard resetting link
ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata4.00: configured for UDMA/133
ata4: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
ata4: irq_stat 0x00000040, connection status changed
ata4.00: configured for UDMA/133
ata4: EH complete
sd 3:0:0:0: [sdb] 1953508784 512-byte hardware sectors: (1.00 TB/931 GiB)
sd 3:0:0:0: [sdb] Write Protect is off
sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 3:0:0:0: [sdb] 1953508784 512-byte hardware sectors: (1.00 TB/931 GiB)
sd 3:0:0:0: [sdb] Write Protect is off
sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
nfsd: last server has exited, flushing export cache
EXT4-fs: mballoc: 517405 blocks 709 reqs (308 success)
EXT4-fs: mballoc: 656 extents scanned, 582 goal hits, 77 2^N hits, 0 breaks, 0 lost
EXT4-fs: mballoc: 99 generated and it took 850048
EXT4-fs: mballoc: 443177 preallocated, 441245 discarded
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period

[here starts the local backup almost an hour later]

EXT4-fs: barriers enabled
kjournald2 starting.  Commit interval 5 seconds
EXT4 FS on sdb1, internal journal on sdb1:8
EXT4-fs: delayed allocation enabled
EXT4-fs: file extents enabled
EXT4-fs: mballoc enabled
EXT4-fs: mounted filesystem with ordered data mode.
ata4.00: exception Emask 0x40 SAct 0x0 SErr 0x80800 action 0x6 frozen
ata4: SError: { HostInt 10B8B }
ata4.00: cmd 35/00:00:4f:98:44/00:04:31:00:00/e0 tag 0 dma 524288 out
         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x44 (timeout)
ata4.00: status: { DRDY }
ata4: hard resetting link
ata4: SATA link down (SStatus 0 SControl 300)
ata4: hard resetting link
ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata4.00: configured for UDMA/133
ata4: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
ata4: irq_stat 0x00000040, connection status changed
ata4.00: configured for UDMA/133
ata4: EH complete
sd 3:0:0:0: [sdb] 1953508784 512-byte hardware sectors: (1.00 TB/931 GiB)
sd 3:0:0:0: [sdb] Write Protect is off
sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 3:0:0:0: [sdb] 1953508784 512-byte hardware sectors: (1.00 TB/931 GiB)
sd 3:0:0:0: [sdb] Write Protect is off
sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
ata4.00: exception Emask 0x40 SAct 0x0 SErr 0x80800 action 0x6 frozen
ata4: SError: { HostInt 10B8B }
ata4.00: cmd 35/00:00:87:51:47/00:04:31:00:00/e0 tag 0 dma 524288 out
         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x44 (timeout)
ata4.00: status: { DRDY }
ata4: hard resetting link
ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata4.00: configured for UDMA/133
ata4: EH complete
sd 3:0:0:0: [sdb] 1953508784 512-byte hardware sectors: (1.00 TB/931 GiB)
sd 3:0:0:0: [sdb] Write Protect is off
sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
ata4: limiting SATA link speed to 1.5 Gbps
ata4.00: exception Emask 0x40 SAct 0x0 SErr 0x80800 action 0x6 frozen
ata4: SError: { HostInt 10B8B }
ata4.00: cmd 35/00:00:af:ad:05/00:04:09:00:00/e0 tag 0 dma 524288 out
         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x44 (timeout)
ata4.00: status: { DRDY }
ata4: hard resetting link
ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata4.00: configured for UDMA/133
ata4: EH complete
sd 3:0:0:0: [sdb] 1953508784 512-byte hardware sectors: (1.00 TB/931 GiB)
sd 3:0:0:0: [sdb] Write Protect is off
sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
ata4.00: exception Emask 0x50 SAct 0x0 SErr 0x280900 action 0x6 frozen
ata4.00: irq_stat 0x08000000, interface fatal error
ata4: SError: { UnrecovData HostInt 10B8B BadCRC }
ata4.00: cmd c8/00:08:57:97:08/00:00:00:00:00/e9 tag 0 dma 4096 in
         res 50/00:00:36:06:34/00:00:01:00:00/41 Emask 0x50 (ATA bus error)
ata4.00: status: { DRDY }
ata4: hard resetting link
ata4: SATA link down (SStatus 0 SControl 310)
ata4: hard resetting link
ata4: SATA link down (SStatus 0 SControl 310)
ata4: hard resetting link
ata4: SATA link down (SStatus 0 SControl 310)
ata4.00: disabled
ata4: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
ata4: irq_stat 0x00400040, connection status changed
ata4: hard resetting link
ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata4.00: ATA-6: WD      My Book         1105, 1010, max UDMA/133
ata4.00: 1953508784 sectors, multi 0: LBA48
ata4.00: configured for UDMA/133
sd 3:0:0:0: [sdb] Result: hostbyte=DID_OK 
driverbyte=DRIVER_SENSE,SUGGEST_OK
sd 3:0:0:0: [sdb] Sense Key : Aborted Command [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
        72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
        01 34 06 36
sd 3:0:0:0: [sdb] Add. Sense: No additional sense information
end_request: I/O error, dev sdb, sector 151557975
sd 3:0:0:0: rejecting I/O to offline device
Buffer I/O error on device sdb1, logical block 2099128
lost page write due to I/O error on sdb1
Buffer I/O error on device sdb1, logical block 2099129
lost page write due to I/O error on sdb1
Buffer I/O error on device sdb1, logical block 2099130
lost page write due to I/O error on sdb1
Buffer I/O error on device sdb1, logical block 2099131
lost page write due to I/O error on sdb1
Buffer I/O error on device sdb1, logical block 2099132
lost page write due to I/O error on sdb1
sd 3:0:0:0: rejecting I/O to offline device
Buffer I/O error on device sdb1, logical block 2101059
lost page write due to I/O error on sdb1
sd 3:0:0:0: rejecting I/O to offline device
Buffer I/O error on device sdb1, logical block 2101205
lost page write due to I/O error on sdb1
sd 3:0:0:0: rejecting I/O to offline device
Buffer I/O error on device sdb1, logical block 2104385
lost page write due to I/O error on sdb1
Buffer I/O error on device sdb1, logical block 2104386
lost page write due to I/O error on sdb1
sd 3:0:0:0: rejecting I/O to offline device
Buffer I/O error on device sdb1, logical block 2104398
lost page write due to I/O error on sdb1
sd 3:0:0:0: rejecting I/O to offline device

[last message repeated multiple times]

sd 3:0:0:0: rejecting I/O to offline device
ata4: EH complete
sd 3:0:0:0: rejecting I/O to offline device
sd 3:0:0:0: rejecting I/O to offline device
EXT4-fs error (device sdb1): ext4_find_entry: reading directory #4741319 offset 0
sd 3:0:0:0: rejecting I/O to offline device
ext4: I/O error while writing superblock for sdb1.
sd 3:0:0:0: rejecting I/O to offline device

[repeated again]

sd 3:0:0:0: rejecting I/O to offline device
ata4.00: detaching (SCSI 3:0:0:0)
Aborting journal on device sdb1:8.
JBD2: I/O error detected when updating journal superblock for sdb1:8.
journal commit I/O error
EXT4-fs error (device sdb1): ext4_find_entry: reading directory #2 offset 0
ext4: I/O error while writing superblock for sdb1.
sd 3:0:0:0: [sdb] Synchronizing SCSI cache
sd 3:0:0:0: [sdb] Stopping disk
scsi 3:0:0:0: Direct-Access     ATA      WD      My Book  1010 PQ: 0 ANSI: 5
sd 3:0:0:0: [sdc] 1953508784 512-byte hardware sectors: (1.00 TB/931 GiB)
sd 3:0:0:0: [sdc] Write Protect is off
sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 3:0:0:0: [sdc] 1953508784 512-byte hardware sectors: (1.00 TB/931 GiB)
sd 3:0:0:0: [sdc] Write Protect is off
sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdc: sdc1 sdc2
sd 3:0:0:0: [sdc] Attached SCSI disk
sd 3:0:0:0: Attached scsi generic sg2 type 0

Re: [eSATA] 2.6.28.7: IO errors

[Matthew Wilcox]

On Sat, Mar 28, 2009 at 12:18:38AM +0100, Guennadi Liakhovetski wrote:
> I'm using an external eSATA / USB / Firewire 2x1T RAID from WD in RAID1 

Looks like your problems are solved by
9062712fa9ed13b531dfc2228086650b8bd6a255

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: [eSATA] 2.6.28.7: IO errors

[Guennadi Liakhovetski]

Hi Matthew,

On Fri, 27 Mar 2009, Matthew Wilcox wrote:

> On Sat, Mar 28, 2009 at 12:18:38AM +0100, Guennadi Liakhovetski wrote:
> > I'm using an external eSATA / USB / Firewire 2x1T RAID from WD in RAID1 
> 
> Looks like your problems are solved by
> 9062712fa9ed13b531dfc2228086650b8bd6a255

(added Tejun as the patch author and owner of similar hardware)

AFAIU, that patch improves or fixes error handling on those drives, and 
the symptom, that Tejun was observing, was that the drive was hanging and 
required a power cycle.

In my case the drive didn't need a power cycle, and error handling did 
work, AFAICS, but even just having that error seems not healthy for the 
data, which is confirmed by ext4 errors. Am I right? (BTW, I was 
impressed, that after replugging the drive to firewire and running fsck on 
a 800G big ext4 partition, it only needed around a second or two to replay 
the journal and then it was ok! I hope the fileystem also has indeed been 
repaired...)

Error handling is great, but I think what I want is to avoid such errors 
altogether. So, my question was - can those errors be indeed caused by the 
unstable eSATA connector in the drive? Tejun, you have similar drives, do 
yours also have such a problem? How are you solving it? Is the only way to 
improve the connector stability by milling the plastic casing to actually 
be able to insert the connector properly or by contacting the retailer for 
a replacement?... Sorry, these are not directly Linux driver questions, 
but the questions are really what those error messages tell me about the 
possible reasons for the problem:-)

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer

Re: [eSATA] 2.6.28.7: IO errors

[Tejun Heo]

Hello,

Guennadi Liakhovetski wrote:
> Hi Matthew,
> 
> On Fri, 27 Mar 2009, Matthew Wilcox wrote:
> 
>> On Sat, Mar 28, 2009 at 12:18:38AM +0100, Guennadi Liakhovetski wrote:
>>> I'm using an external eSATA / USB / Firewire 2x1T RAID from WD in RAID1 
>> Looks like your problems are solved by
>> 9062712fa9ed13b531dfc2228086650b8bd6a255
> 
> (added Tejun as the patch author and owner of similar hardware)

Thanks for forwarding.

> AFAIU, that patch improves or fixes error handling on those drives, and 
> the symptom, that Tejun was observing, was that the drive was hanging and 
> required a power cycle.
> 
> In my case the drive didn't need a power cycle, and error handling did 
> work, AFAICS, but even just having that error seems not healthy for the 
> data, which is confirmed by ext4 errors. Am I right?

No, your drive checked out after the last error even after the link
speed had been lowered to 1.5Gbps, so the EH did everything it could
but no candy.  The drive still checked out.

> Error handling is great, but I think what I want is to avoid such errors 
> altogether. So, my question was - can those errors be indeed caused by the 
> unstable eSATA connector in the drive? Tejun, you have similar drives, do 
> yours also have such a problem? How are you solving it?

I don't use the drives extensively but shorter cables seem to solve a
lot of the issues during testing.  Or rather, longer cables triggered
a LOT of problems.

> Is the only way to improve the connector stability by milling the
> plastic casing to actually be able to insert the connector properly
> or by contacting the retailer for a replacement?... Sorry, these are
> not directly Linux driver questions, but the questions are really
> what those error messages tell me about the possible reasons for the
> problem:-)

Give a shot at shorter cable.

Good luck!

-- 
tejun

Re: [eSATA] 2.6.28.7: IO errors

[Guennadi Liakhovetski]

On Mon, 30 Mar 2009, Tejun Heo wrote:

> Hello,
> 
> Guennadi Liakhovetski wrote:
> > Hi Matthew,
> > 
> > On Fri, 27 Mar 2009, Matthew Wilcox wrote:
> > 
> >> On Sat, Mar 28, 2009 at 12:18:38AM +0100, Guennadi Liakhovetski wrote:
> >>> I'm using an external eSATA / USB / Firewire 2x1T RAID from WD in RAID1 
> >> Looks like your problems are solved by
> >> 9062712fa9ed13b531dfc2228086650b8bd6a255
> > 
> > (added Tejun as the patch author and owner of similar hardware)
> 
> Thanks for forwarding.

Thanks for replying:-)

> > AFAIU, that patch improves or fixes error handling on those drives, and 
> > the symptom, that Tejun was observing, was that the drive was hanging and 
> > required a power cycle.
> > 
> > In my case the drive didn't need a power cycle, and error handling did 
> > work, AFAICS, but even just having that error seems not healthy for the 
> > data, which is confirmed by ext4 errors. Am I right?
> 
> No, your drive checked out after the last error even after the link
> speed had been lowered to 1.5Gbps, so the EH did everything it could
> but no candy.  The drive still checked out.
> 
> > Error handling is great, but I think what I want is to avoid such errors 
> > altogether. So, my question was - can those errors be indeed caused by the 
> > unstable eSATA connector in the drive? Tejun, you have similar drives, do 
> > yours also have such a problem? How are you solving it?
> 
> I don't use the drives extensively but shorter cables seem to solve a
> lot of the issues during testing.  Or rather, longer cables triggered
> a LOT of problems.
> 
> > Is the only way to improve the connector stability by milling the
> > plastic casing to actually be able to insert the connector properly
> > or by contacting the retailer for a replacement?... Sorry, these are
> > not directly Linux driver questions, but the questions are really
> > what those error messages tell me about the possible reasons for the
> > problem:-)
> 
> Give a shot at shorter cable.

Hm, using a 62cm long cable ATM, should it be even shorter? It's going to 
be difficult physically then:-( But what about your drives - dies the 
connector fit well in the slot and sits firmaly there? With my disk looks 
likt the connector cannot be inserted completely because of too small a 
hole in plastic:-(

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer

Re: [eSATA] 2.6.28.7: IO errors

[Tejun Heo]

Hello,

Guennadi Liakhovetski wrote:
> Hm, using a 62cm long cable ATM, should it be even shorter? It's going to 
> be difficult physically then:-(

62cm looks short enough.  My long cable is much longer like 2m

> But what about your drives - dies the 
> connector fit well in the slot and sits firmaly there? With my disk looks 
> likt the connector cannot be inserted completely because of too small a 
> hole in plastic:-(

But mine fits well and firmly, so maybe you need to take out the knife
after all.

-- 
tejun

Re: [eSATA] 2.6.28.7: IO errors

[Guennadi Liakhovetski]

On Mon, 30 Mar 2009, Tejun Heo wrote:

> Hello,
> 
> Guennadi Liakhovetski wrote:
> > Hm, using a 62cm long cable ATM, should it be even shorter? It's going to 
> > be difficult physically then:-(
> 
> 62cm looks short enough.  My long cable is much longer like 2m
> 
> > But what about your drives - dies the 
> > connector fit well in the slot and sits firmaly there? With my disk looks 
> > likt the connector cannot be inserted completely because of too small a 
> > hole in plastic:-(
> 
> But mine fits well and firmly, so maybe you need to take out the knife
> after all.

yeah, warranty, you know:-( I contacted the retailer, will see. but it's 
strange - I bought mine in December 08, so, it's new enough, so, why?... 
The opening in the case is ~4.8mm, the plastic part of the connector on 
the eSATA cable is ~6.8mm... Ok, that's getting way OT, sorry.

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer

Re: [eSATA] 2.6.28.7: IO errors

[Mark Lord]

Guennadi Liakhovetski wrote:
> On Mon, 30 Mar 2009, Tejun Heo wrote:
> 
>> Hello,
>>
>> Guennadi Liakhovetski wrote:
>>> Hm, using a 62cm long cable ATM, should it be even shorter? It's going to 
>>> be difficult physically then:-(
>> 62cm looks short enough.  My long cable is much longer like 2m
>>
>>> But what about your drives - dies the 
>>> connector fit well in the slot and sits firmaly there? With my disk looks 
>>> likt the connector cannot be inserted completely because of too small a 
>>> hole in plastic:-(
>> But mine fits well and firmly, so maybe you need to take out the knife
>> after all.
> 
> yeah, warranty, you know:-( I contacted the retailer, will see. but it's 
> strange - I bought mine in December 08, so, it's new enough, so, why?... 
> The opening in the case is ~4.8mm, the plastic part of the connector on 
> the eSATA cable is ~6.8mm... Ok, that's getting way OT, sorry.
..

Speaking of which.. dealextreme.com sells surprisingly excellent eSATA
cables with really sturdy connectors.  Very cheap pricing, too,
with free shipping worldwide.  An easy way to eliminate cable fuss.

Cheers

Re: [eSATA] 2.6.28.7: IO errors

[Guennadi Liakhovetski]

On Mon, 30 Mar 2009, Mark Lord wrote:

> Guennadi Liakhovetski wrote:
> > On Mon, 30 Mar 2009, Tejun Heo wrote:
> > 
> > > Hello,
> > > 
> > > Guennadi Liakhovetski wrote:
> > > > Hm, using a 62cm long cable ATM, should it be even shorter? It's going
> > > > to be difficult physically then:-(
> > > 62cm looks short enough.  My long cable is much longer like 2m
> > > 
> > > > But what about your drives - dies the connector fit well in the slot and
> > > > sits firmaly there? With my disk looks likt the connector cannot be
> > > > inserted completely because of too small a hole in plastic:-(
> > > But mine fits well and firmly, so maybe you need to take out the knife
> > > after all.
> > 
> > yeah, warranty, you know:-( I contacted the retailer, will see. but it's
> > strange - I bought mine in December 08, so, it's new enough, so, why?... The
> > opening in the case is ~4.8mm, the plastic part of the connector on the
> > eSATA cable is ~6.8mm... Ok, that's getting way OT, sorry.
> ..
> 
> Speaking of which.. dealextreme.com sells surprisingly excellent eSATA
> cables with really sturdy connectors.  Very cheap pricing, too,
> with free shipping worldwide.  An easy way to eliminate cable fuss.

Haven't tried that cable, but here's a confirmation of my observation:

http://forums.techguy.org/hardware/580140-esata-drive-not-recognized.html

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer