SATA driver issues?

SATA driver issues?

[Danny Sung]

Hi guys, I'm just writing to report some strange problems I'm seeing on
an IBM system with SATA drives.  I'm following the comments in the
sata_svw.c file, so hopefully I'm writing the correct people.

First my configuration:

 - IBM 7969AC1 - xSeries 326m  (2x dual core AMD Opteron 285, 2GB RAM)
 - Fedora Core 5, kernel 2.6.18  (package 2.6.18-1.220.fc5smp)
   I specifically chose to run the 32-bit version of Fedora as well.
 - IBM BIOS has the latest firmware from IBM (which did resolve the
issue of the system crashing).
 - Hard drives are 2x WD2500JS's
 - This particular server is using the BCM5785 (aka HT-1000) broadcom
chipset
 - If it matters I am running the drives under linux software RAID
mirroring, with a mixture of ext3 and xfs filesystems.

I've run short and long SMART tests on both drives, so I know it's not a
drive problem.  And this is a completely new server, just got it a
couple weeks ago.  The MFG date is 09/23/2006.


Prior to the firmware update, I got errors like these, along with
periodic system hangs:
Oct 15 03:50:40 aeolus kernel: ata2: command 0xea timeout, stat 0x50
host_stat 0x0
Oct 15 03:50:40 aeolus kernel: ata2: status=0x50 { DriveReady
SeekComplete }
Oct 15 03:50:40 aeolus kernel: ata2: error=0x01 { AddrMarkNotFound }

The firmware release notes did mention specifically having timing
problems that would cause Suse to hang.  So that's not entirely
unexpected.

But now I'm getting errors like these:
Oct 23 12:33:00 aeolus kernel: ATA: abnormal status 0xD0 on port
0xF881C11C
Oct 23 12:33:00 aeolus kernel: ATA: abnormal status 0xD0 on port
0xF881C11C
Oct 23 12:33:00 aeolus kernel: ATA: abnormal status 0xD0 on port
0xF881C11C
Oct 23 12:33:00 aeolus kernel: ATA: abnormal status 0xD0 on port
0xF881C01C
Oct 23 12:33:00 aeolus last message repeated 2 times
Oct 23 12:33:30 aeolus kernel: ata2.00: exception Emask 0x10 SAct 0x0
SErr 0x44050000 action 0x2 frozen
Oct 23 12:33:30 aeolus kernel: ata2.00: tag 0 cmd 0xea Emask 0x14 stat
0x40 err 0x0 (ATA bus error)
Oct 23 12:33:30 aeolus kernel: ata1.00: exception Emask 0x10 SAct 0x0
SErr 0x44050000 action 0x2 frozen
Oct 23 12:33:30 aeolus kernel: ata1.00: tag 0 cmd 0xea Emask 0x14 stat
0x40 err 0x0 (ATA bus error)
Oct 23 12:33:30 aeolus kernel: ata2: soft resetting port
Oct 23 12:33:30 aeolus kernel: ata1: soft resetting port
Oct 23 12:33:31 aeolus kernel: ata2: SATA link up 1.5 Gbps (SStatus 113
SControl 300)
Oct 23 12:33:31 aeolus kernel: ata2.00: configured for UDMA/133
Oct 23 12:33:31 aeolus kernel: ata2: EH complete
Oct 23 12:33:31 aeolus kernel: SCSI device sdb: 488397168 512-byte hdwr
sectors (250059 MB)
Oct 23 12:33:31 aeolus kernel: sdb: Write Protect is off
Oct 23 12:33:31 aeolus kernel: SCSI device sdb: drive cache: write back
Oct 23 12:33:31 aeolus kernel: ata1: SATA link up 1.5 Gbps (SStatus 113
SControl 300)

The system no longer hangs completely.  There is a good pause during the
soft reset (probably about 5 to 10 seconds), but then things seem to
return to normal.  Some of my filesystems are a little corrupted (may
have been due to the firmware bug, though), but I can more or less use
the system.  Obviously I'd like to fix the problem and
reformat/reinstall everything.  But I'm just trying to narrow down the
problem.

My latest test was to boot up with the following kernel parameters:
"pci=noacpi idebus=66".  I'm not quite sure if I should be using 66 or
133.  But I figured it was worth a shot.  The "noacpi" flag was given to
me by other kernel messages on bootup.

It's only been a couple hours now without errors, so it's hard to say if
that's completely fixed my problem.  But I thought I'd report it in case
anyone else has seen the problem or has any suggestions on how I can fix
this.

Even going the RHEL route seems problematic as IBM only lists RHEL3
drivers on their website.  In addition, that division of Broadcom seems
to have been sold to another company and they're in transition in
offering driver support.  *sigh* to closed drivers...

Thanks in advance for any advice/suggestions,
Danny Sung
**********************************************************************
This e-mail is the property of Lantronix. It is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential, or otherwise protected from disclosure. Distribution or copying of this e-mail, or the information contained herein, to anyone other than the intended recipient is prohibited.

Re: SATA driver issues?

[Benjamin Herrenschmidt]

On Mon, 2006-10-23 at 18:27 -0700, Danny Sung wrote:
> Hi guys, I'm just writing to report some strange problems I'm seeing on
> an IBM system with SATA drives.  I'm following the comments in the
> sata_svw.c file, so hopefully I'm writing the correct people.
> 
> First my configuration:
> 
>  - IBM 7969AC1 - xSeries 326m  (2x dual core AMD Opteron 285, 2GB RAM)
>  - Fedora Core 5, kernel 2.6.18  (package 2.6.18-1.220.fc5smp)
>    I specifically chose to run the 32-bit version of Fedora as well.
>  - IBM BIOS has the latest firmware from IBM (which did resolve the
> issue of the system crashing).
>  - Hard drives are 2x WD2500JS's
>  - This particular server is using the BCM5785 (aka HT-1000) broadcom
> chipset

The HT1000 is using a frodo SATA cell ? (me doesn't know for sure :)

What is the lspci output ?

Ben.

Re: SATA driver issues?

[Benjamin Herrenschmidt]

> Thanks in advance for any advice/suggestions,
> Danny Sung
> **********************************************************************
> This e-mail is the property of Lantronix. It is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential, or otherwise protected from disclosure. Distribution or copying of this e-mail, or the information contained herein, to anyone other than the intended recipient is prohibited.

BTW. Please remove that completely bogus disclaimer at the end of your
emails, especially when posting to a public list. It's a total nonsense
and may cause quite a few people to decide to ignore your message
completely.

Ben.

Re: SATA driver issues?

[Alan Cox]

Ar Llu, 2006-10-23 am 18:27 -0700, ysgrifennodd Danny Sung:
> Prior to the firmware update, I got errors like these, along with
> periodic system hangs:
> Oct 15 03:50:40 aeolus kernel: ata2: command 0xea timeout, stat 0x50
> host_stat 0x0
> Oct 15 03:50:40 aeolus kernel: ata2: status=0x50 { DriveReady
> SeekComplete }
> Oct 15 03:50:40 aeolus kernel: ata2: error=0x01 { AddrMarkNotFound }

Your drive is reporting an error that it cannot find a disk sector.

> My latest test was to boot up with the following kernel parameters:
> "pci=noacpi idebus=66".  I'm not quite sure if I should be using 66 or

Well unless your system is very strange your idebus is 33Mhz, you only
need idebus= on old 486 systems with vesa local bus.

RE: SATA driver issues?

[Danny Sung]

 

-----Original Message-----
From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org] 
Sent: Monday, October 23, 2006 6:56 PM
To: Danny Sung
Cc: Jeff Garzik; linux-ide@vger.kernel.org
Subject: Re: SATA driver issues?


Interestingly, I didn't get any drive resets last night.  Maybe one of
those flags (pci=noacpi idebus=66) fixed the problem.  I'll try removing
the idebus line since Alan Cox said I shouldn't need it.

> The HT1000 is using a frodo SATA cell ? (me doesn't know for sure :)
> What is the lspci output ?

00:01.0 PCI bridge: Broadcom HT1000 PCI/PCI-X bridge
00:02.0 Host bridge: Broadcom HT1000 Legacy South Bridge
00:02.1 IDE interface: Broadcom HT1000 Legacy IDE controller
00:02.2 ISA bridge: Broadcom HT1000 LPC Bridge
00:03.0 USB Controller: Broadcom HT1000 USB Controller (rev 01)
00:03.1 USB Controller: Broadcom HT1000 USB Controller (rev 01)
00:03.2 USB Controller: Broadcom HT1000 USB Controller (rev 01)
00:04.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)
00:06.0 PCI bridge: Broadcom HT1000 PCI-X bridge (rev a3)
00:07.0 PCI bridge: Broadcom HT1000 PCI-X bridge (rev a3)
00:08.0 PCI bridge: Broadcom HT1000 PCI-Express bridge (rev a3)
00:09.0 PCI bridge: Broadcom HT1000 PCI-Express bridge (rev a3)
00:0a.0 PCI bridge: Broadcom HT1000 PCI-Express bridge (rev a3)
00:0b.0 PCI bridge: Broadcom HT1000 PCI-Express bridge (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
01:0d.0 PCI bridge: Broadcom HT1000 PCI/PCI-X bridge (rev b2)
01:0e.0 IDE interface: Broadcom BCM5785 (HT1000) SATA Native SATA Mode
01:0e.1 IDE interface: Broadcom BCM5785 (HT1000) SATA Native SATA Mode
04:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5780
Gigabit Ethernet (rev 03)
04:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5780
Gigabit Ethernet (rev 03)

(verbose at bottom)

>BTW. Please remove that completely bogus disclaimer at the end of your
emails,
>especially when posting to a public list. It's a total nonsense and may
cause
>quite a few people to decide to ignore your message completely.

I know, it's horribly embarrassing.  I wish I could do something about
it, but it's from my company's outgoing mail server.  I'll see what I
can get out of my IT people, but I doubt there's much I can do.  Many
sincere appologies.  Can I legally put a disclaimer that nullifies the
disclaimer? ;-)



Here's the lspci -v output:

00:01.0 PCI bridge: Broadcom HT1000 PCI/PCI-X bridge (prog-if 00 [Normal
decode])
        Flags: bus master, fast devsel, latency 64
        Bus: primary=00, secondary=01, subordinate=02, sec-latency=64
        I/O behind bridge: 00003000-00003fff
        Memory behind bridge: e8100000-e81fffff
        Prefetchable memory behind bridge:
0000000088000000-0000000088000000
        Capabilities: [90] PCI-X bridge device
        Capabilities: [a0] HyperTransport: MSI Mapping
        Capabilities: [b0] HyperTransport: Slave or Primary Interface

00:02.0 Host bridge: Broadcom HT1000 Legacy South Bridge
        Subsystem: Broadcom Unknown device 0201
        Flags: bus master, medium devsel, latency 64

00:02.1 IDE interface: Broadcom HT1000 Legacy IDE controller (prog-if 8a
[Master SecP PriP])
        Subsystem: Broadcom HT1000 Legacy IDE controller
        Flags: bus master, medium devsel, latency 64
        I/O ports at <ignored>
        I/O ports at <ignored>
        I/O ports at <ignored>
        I/O ports at <ignored>
        I/O ports at 2000 [size=16]
        Capabilities: [b0] Power Management version 2

00:02.2 ISA bridge: Broadcom HT1000 LPC Bridge
        Subsystem: Broadcom Unknown device 0230
        Flags: bus master, medium devsel, latency 0

00:03.0 USB Controller: Broadcom HT1000 USB Controller (rev 01) (prog-if
10 [OHCI])
        Subsystem: Broadcom HT1000 USB Controller
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 10
        Memory at e8011000 (32-bit, non-prefetchable) [size=4K]
        I/O ports at 1000 [size=256]
        Capabilities: [dc] Power Management version 2

00:03.1 USB Controller: Broadcom HT1000 USB Controller (rev 01) (prog-if
10 [OHCI])
        Subsystem: Broadcom HT1000 USB Controller
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 10
        Memory at e8012000 (32-bit, non-prefetchable) [size=4K]
        I/O ports at 1400 [size=256]
        Capabilities: [dc] Power Management version 2

00:03.2 USB Controller: Broadcom HT1000 USB Controller (rev 01) (prog-if
20 [EHCI])
        Subsystem: Broadcom HT1000 USB Controller
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 10
        Memory at e8013000 (32-bit, non-prefetchable) [size=4K]
        I/O ports at 1800 [size=256]
        Capabilities: [dc] Power Management version 2

00:04.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)
(prog-if 00 [VGA])
        Subsystem: IBM Unknown device 0325
        Flags: stepping, medium devsel, IRQ 169
        Memory at f0000000 (32-bit, prefetchable) [size=128M]
        I/O ports at 1c00 [size=256]
        Memory at e8000000 (32-bit, non-prefetchable) [size=64K]
        [virtual] Expansion ROM at 88100000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 2

00:06.0 PCI bridge: Broadcom HT1000 PCI-X bridge (rev a3) (prog-if 00
[Normal decode])
        Flags: bus master, fast devsel, latency 64
        Bus: primary=00, secondary=03, subordinate=03, sec-latency=64
        Capabilities: [90] PCI-X bridge device
        Capabilities: [a0] HyperTransport: MSI Mapping
        Capabilities: [b0] HyperTransport: Slave or Primary Interface

00:07.0 PCI bridge: Broadcom HT1000 PCI-X bridge (rev a3) (prog-if 00
[Normal decode])
        Flags: bus master, fast devsel, latency 64
        Bus: primary=00, secondary=04, subordinate=04, sec-latency=64
        Memory behind bridge: e8200000-e82fffff
        Capabilities: [90] PCI-X bridge device
        Capabilities: [a0] HyperTransport: MSI Mapping

00:08.0 PCI bridge: Broadcom HT1000 PCI-Express bridge (rev a3) (prog-if
00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=05, subordinate=05, sec-latency=64
        Capabilities: [a0] HyperTransport: MSI Mapping
        Capabilities: [b0] Express Root Port (Slot-) IRQ 0

00:09.0 PCI bridge: Broadcom HT1000 PCI-Express bridge (rev a3) (prog-if
00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=06, subordinate=06, sec-latency=64
        Capabilities: [a0] HyperTransport: MSI Mapping
        Capabilities: [b0] Express Root Port (Slot-) IRQ 0

00:0a.0 PCI bridge: Broadcom HT1000 PCI-Express bridge (rev a3) (prog-if
00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=07, subordinate=07, sec-latency=64
        Capabilities: [a0] HyperTransport: MSI Mapping
        Capabilities: [b0] Express Root Port (Slot-) IRQ 0

00:0b.0 PCI bridge: Broadcom HT1000 PCI-Express bridge (rev a3) (prog-if
00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=08, subordinate=08, sec-latency=64
        Capabilities: [a0] HyperTransport: MSI Mapping
        Capabilities: [b0] Express Root Port (Slot-) IRQ 0

00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuratio
n
        Flags: fast devsel
        Capabilities: [80] HyperTransport: Host or Secondary Interface
        Capabilities: [a0] HyperTransport: Host or Secondary Interface
        Capabilities: [c0] HyperTransport: Host or Secondary Interface

00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Address Map
        Flags: fast devsel

00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
DRAM Controller
        Flags: fast devsel

00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
        Flags: fast devsel

00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuratio
n
        Flags: fast devsel
        Capabilities: [80] HyperTransport: Host or Secondary Interface
        Capabilities: [a0] HyperTransport: Host or Secondary Interface
        Capabilities: [c0] HyperTransport: Host or Secondary Interface

00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Address Map
        Flags: fast devsel

00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
DRAM Controller
        Flags: fast devsel

00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
        Flags: fast devsel

01:0d.0 PCI bridge: Broadcom HT1000 PCI/PCI-X bridge (rev b2) (prog-if
00 [Normal decode])
        Flags: bus master, 66MHz, medium devsel, latency 64
        Bus: primary=01, secondary=02, subordinate=02, sec-latency=64
        Capabilities: [90] PCI-X bridge device

01:0e.0 IDE interface: Broadcom BCM5785 (HT1000) SATA Native SATA Mode
(prog-if 8f [Master SecP SecO PriP PriO
])
        Subsystem: Broadcom BCM5785 (HT1000) SATA Native SATA Mode
        Flags: bus master, medium devsel, latency 64, IRQ 11
        I/O ports at 3040 [size=8]
        I/O ports at 3034 [size=4]
        I/O ports at 3038 [size=8]
        I/O ports at 3030 [size=4]
        I/O ports at 3000 [size=16]
        Memory at e8100000 (32-bit, non-prefetchable) [size=8K]
        [virtual] Expansion ROM at 88000000 [disabled] [size=128K]
        Capabilities: [60] PCI-X non-bridge device
        Capabilities: [90] Power Management version 2
        Capabilities: [a0] Message Signalled Interrupts: 64bit-
Queue=0/0 Enable-

01:0e.1 IDE interface: Broadcom BCM5785 (HT1000) SATA Native SATA Mode
(prog-if 8f [Master SecP SecO PriP PriO
])
        Subsystem: Broadcom BCM5785 (HT1000) SATA Native SATA Mode
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 11
        I/O ports at 3058 [size=8]
        I/O ports at 304c [size=4]
        I/O ports at 3050 [size=8]
        I/O ports at 3048 [size=4]
        I/O ports at 3020 [size=16]
        Capabilities: [60] PCI-X non-bridge device
        Capabilities: [90] Power Management version 2
        Capabilities: [a0] Message Signalled Interrupts: 64bit-
Queue=0/0 Enable-

04:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5780
Gigabit Ethernet (rev 03)
        Subsystem: IBM Unknown device 0329
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 201
        Memory at e8210000 (64-bit, non-prefetchable) [size=64K]
        Memory at e8200000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: [40] PCI-X non-bridge device
        Capabilities: [48] Power Management version 2
        Capabilities: [50] Vital Product Data
        Capabilities: [58] Message Signalled Interrupts: 64bit+
Queue=0/3 Enable+

04:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5780
Gigabit Ethernet (rev 03)
        Subsystem: IBM Unknown device 0329
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 185
        Memory at e8230000 (64-bit, non-prefetchable) [size=64K]
        Memory at e8220000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: [40] PCI-X non-bridge device
        Capabilities: [48] Power Management version 2
        Capabilities: [50] Vital Product Data
        Capabilities: [58] Message Signalled Interrupts: 64bit+
Queue=0/3 Enable-
**********************************************************************
This e-mail is the property of Lantronix. It is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential, or otherwise protected from disclosure. Distribution or copying of this e-mail, or the information contained herein, to anyone other than the intended recipient is prohibited.

RE: SATA driver issues?

[Danny Sung]

An update to some of the SATA issues I've had with the IBM xSeries 326m.
I decided to try out RedHat Enterprise Linux 4 (Update 4), since that's
theoretically something that IBM supports and has tested with.  In all
cases I've been using i386 (Fedora and Redhat) even though this is a
64-bit Opteron.

Turns out the first motherboard actually did have some sort of bizarre
motherboard/controller problems.  IBM's hardware support service is
always first class, and they got that switched out for me that day.  The
new hardware passed the diagnostics that came with the computer,
definitely a good sign.  However, both Fedora and RHEL still saw drive
problems.  At this point, I'm not sure there's anything I can do but
return the hardware.

But I'm just curious if anyone's used this machine successfully with
Linux and SATA drives?  Searching around the net, I did find numerous
reports of people with "abnormal status 0xd0" problems with nary a
resolution.

On a similar topic does anyone have suggestions for good quality stable
machines (either through IBM or otherwise) they can recommend?  I know
it's not completely on topic for this list, but where do you go when
looking for server-type machines with good support that are Linux
friendly?

Thanks,
Danny

(Kindly ignore the disclaimer below)
**********************************************************************
This e-mail is the property of Lantronix. It is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential, or otherwise protected from disclosure. Distribution or copying of this e-mail, or the information contained herein, to anyone other than the intended recipient is prohibited.