Re: sata_sil data corruption, possible workarounds

[Robert Hancock <hancockrwd@gmail.com>]

On 12/15/2012 02:02 AM, bl0 wrote:
> I have a PCI card based on Silicon Image 3114 SATA controller. Like many
> people in the past I have experienced silent data corruption.
> I am lucky to have a hardware configuration where it is easy to reproduce
> this behavior with 100% rate by copying a file from a USB stick plugged
> into another PCI card. My motherboard has nvidia chipset.
>
> Going through messages and bug reports about this problem, someone mentioned
> that PCI cache line size may be relevant. I did some testing with different
> CLS values and found that the problem of data corruption is solved if
> either
> A). CLS is set to 0, before or after sata_sil kernel driver is loaded
>    # setpci -d 1095:3114 CACHE_LINE_SIZE=0
> where 1095:3114 is the device id as shown by 'lspci -nn'. The same command
> can also be used in grub2 (recent versions) shell or configuration file
> before booting linux.
> or
> B). CLS is set to a sufficiently large value, only after sata_sil is loaded.
>    # setpci -d 1095:3114 CACHE_LINE_SIZE=28
> (value is hexadecimal, in 4-byte units, here it's 160 bytes)
> What is a sufficiently large value depends on the value that is set before
> the driver is loaded. If the value before the driver is loaded is 32 or 64
> bytes, I have to increase it (after the driver is loaded) to 128 or 160
> bytes, respectively.
>
> In sata_sil.c source in sil_init_controller it writes some hardware-specific
> value depending on PCI cache line size. By lowering this value I can get it
> to work with lower CLS. The lowest value 0 works with CLS 64 bytes. If the
> CLS is 32 bytes, I have to increase the CLS.

The meaning of that value from the datasheet is: "This bit field is used 
to specify the system cacheline size in terms of 32-bit words. The upper 
2 bits are not used, resulting a maximum size of 64 32-bit words. With 
the SiI3114 as a master, initiating a read transaction, it issues PCI 
command Read Multiple in place, when empty space in its FIFO is larger 
than the value programmed in this register."

I think this value is likely the key. The cache line size itself 
shouldn't make any difference with this controller as it only really 
affects Memory Write & Invalidate (MWI) and the driver doesn't try to 
enable that for this device. But it's being used to derive the value 
written into this register.

Can you add in some output to figure out what values are being written 
to this register and see which values are working or not working?

>
> Data corruption is the biggest problem for me and these workarounds help but
> another problem remains, sometimes when accessing multiple PCI devices at
> the same time sata becomes inaccessible and times out with log messages
> similar to:
> [  411.351805] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6
> frozen
> [  411.351824] ata3.00: cmd c8/00:00:00:af:00/00:00:00:00:00/e0 tag 0 dma
> 131072 in
> [  411.351826]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4
> (timeout)
> [  411.351830] ata3.00: status: { DRDY }
> [  411.351843] ata3: hard resetting link
> [  411.671775] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
> [  411.697059] ata3.00: configured for UDMA/100
> [  411.697080] ata3: EH complete
>
> Reboot is needed to access sata drives again. If I had the root filesystem
> on a sata drive it would probably crash the system.
>
> Another thing that may be related. Comparing lspci output reveals that when
> accessing multiple PCI devices at the same time, the flag DiscTmrStat
> (Discard Timer Status) gets toggled on for device "00:08.0 PCI bridge:
> nVidia Corporation nForce2 External PCI Bridge". I don't know if it's
> normal or not.

I'm not an expert on the whole PCI bridge/delayed completion stuff but 
it appears that this means that a device (either the host bridge/CPU or 
a device behind that bridge) initiated a delayed transaction for a read, 
but then didn't retry the request to pick up the read data later. From 
what I can tell this seems abnormal, at least in most cases.

Can you post the full lspci -vv output? Do the problems only occur if 
there are multiple devices plugged in behind that bridge?

>
> Finally, the same simple test that I use on Linux does not produce data
> corruption on FreeBSD. Either this problem doesn't occur there or it's not
> trivial to reproduce.
>
> This bug has been around for so long. I hope someone will find this
> information useful.
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ide" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>