* Au1500 hardware cache coherency @ 2003-03-31 12:08 Hartvig Ekner 2003-03-31 14:41 ` Eric DeVolder 2003-03-31 18:35 ` Pete Popov 0 siblings, 2 replies; 12+ messages in thread From: Hartvig Ekner @ 2003-03-31 12:08 UTC (permalink / raw) To: Pete Popov, Linux MIPS mailing list Hi Pete, I am attempting to use the HW coherency feature of the AU1500 to avoid SW flushes and increase the performance. In the config-shared.in file, I can see that the CONFIG_NONCOHERENT_IO define is always set for the AMD eval boards, which results in SW cache flushes when dma_cache_xxx functions are called. If HW coherency is working, this define should not be set. However, in your drivers, you only call the dma_cache functions from au1000/common/usbdev.c, but not from e.g. the ethernet driver or the audio driver. Is this intentional? It seems that the ethernet & audio driver already relies on HW coherency to function correctly (and it also sets the MAC enable bits accordingly, to force all ETH DMA accesses to be coherent), so why not USB also? When turning off NONCOHERENT_IO, there are some bugs (not in AU1000 code) which prevents the code from compiling, but I have fixed these. And the kernel boots, but during some large disk-copy tests, I get occasional data errors which I'm looking in to. So before spending more time on debugging this, and creating patches for using HW coherency, I wanted to hear your input - maybe there are known problems in the Au1500 coherency implementation? /Hartvig ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Au1500 hardware cache coherency 2003-03-31 12:08 Au1500 hardware cache coherency Hartvig Ekner @ 2003-03-31 14:41 ` Eric DeVolder 2003-03-31 15:14 ` Hartvig Ekner 2003-03-31 18:35 ` Pete Popov 1 sibling, 1 reply; 12+ messages in thread From: Eric DeVolder @ 2003-03-31 14:41 UTC (permalink / raw) To: Hartvig Ekner; +Cc: Pete Popov, Linux MIPS mailing list There are data cache snoop bugs with respect to PCI on the Au1500. For the Au1500s soldered on DbAu1500 boards to-date, PCI can not use coherent transactions. Details in the Specification Update for the Au1500 available from: www.amd.com/pcs -> Technical Resources -> AMD Alchemy Solutions Development Board Support Login with your board and you'll be presented with various docs, including the spec update. Regards, Eric Hartvig Ekner wrote: >Hi Pete, > >I am attempting to use the HW coherency feature of the AU1500 to avoid SW flushes and increase the performance. >In the config-shared.in file, I can see that the CONFIG_NONCOHERENT_IO define is always set for the AMD >eval boards, which results in SW cache flushes when dma_cache_xxx functions are called. If HW coherency is >working, this define should not be set. > >However, in your drivers, you only call the dma_cache functions from au1000/common/usbdev.c, but not from e.g. the ethernet >driver or the audio driver. Is this intentional? It seems that the ethernet & audio driver already relies on HW >coherency to function correctly (and it also sets the MAC enable bits accordingly, to force all ETH DMA >accesses to be coherent), so why not USB also? > >When turning off NONCOHERENT_IO, there are some bugs (not in AU1000 code) which prevents the code from >compiling, but I have fixed these. And the kernel boots, but during some large disk-copy tests, I get occasional >data errors which I'm looking in to. > >So before spending more time on debugging this, and creating patches for using HW coherency, I wanted to hear >your input - maybe there are known problems in the Au1500 coherency implementation? > >/Hartvig > > > > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Au1500 hardware cache coherency 2003-03-31 14:41 ` Eric DeVolder @ 2003-03-31 15:14 ` Hartvig Ekner 2003-03-31 18:06 ` Eric DeVolder 0 siblings, 1 reply; 12+ messages in thread From: Hartvig Ekner @ 2003-03-31 15:14 UTC (permalink / raw) To: Eric DeVolder; +Cc: Pete Popov, Linux MIPS mailing list Hi Eric, is that errata #12 you are referring to? (only present on silicon stepping AB?) Or are there others also? While using HW coherency for copying large files on disk, I do get occasional data errors which always look like this: 00061000 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 33 32 0a Linie: 24832. 00061010 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 33 33 0a Linie: 24833. 00061020 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 33 34 0a Linie: 24834. 00061030 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 33 35 0a Linie: 24835. 00061040 38 33 35 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 835.Linie: 24 00061050 38 33 36 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 836.Linie: 24 00061060 38 33 37 0a 38 33 37 0a 4c 69 6e 69 65 3a 20 20 837.837.Linie: 00061070 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 33 39 0a Linie: 24839. 00061080 38 33 39 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 839.Linie: 24 00061090 38 34 30 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 840.Linie: 24 000610a0 38 34 31 0a 38 34 31 0a 4c 69 6e 69 65 3a 20 20 841.841.Linie: 000610b0 20 20 32 34 65 3a 20 20 20 20 32 34 38 34 33 0a 24e: 24843. 000610c0 38 34 33 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 843.Linie: 24 000610d0 38 34 34 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 844.Linie: 24 000610e0 38 34 35 0a 38 34 35 0a 4c 69 6e 69 65 3a 20 20 845.845.Linie: 000610f0 20 20 32 34 65 3a 20 20 20 20 32 34 38 34 37 0a 24e: 24847. 00061100 38 34 37 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 847.Linie: 24 00061110 38 34 38 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 848.Linie: 24 00061120 38 34 39 0a 38 34 39 0a 4c 69 6e 69 65 3a 20 20 849.849.Linie: 00061130 20 20 32 34 65 3a 20 20 20 20 32 34 38 35 31 0a 24e: 24851. 00061140 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 32 0a Linie: 24852. 00061150 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 33 0a Linie: 24853. 00061160 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 34 0a Linie: 24854. 00061170 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 35 0a Linie: 24855. 00061180 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 36 0a Linie: 24856. 00061190 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 37 0a Linie: 24857. 000611a0 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 38 0a Linie: 24858. 000611b0 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 39 0a Linie: 24859. 000611c0 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 36 30 0a Linie: 24860. The file consists of 16-byte lines with increasing line numbers. Note that most of the errors are cacheline oriented, but there are also "skews" of 4 bytes going into the next cacheline. However, interestingly enough, switching HW coherency off entirely (in PCI_CFG) and using uncached PCI buffers, there are still failures: 001ec060 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 35 38 0a Linie: 125958. 001ec070 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 35 39 0a Linie: 125959. 001ec080 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 30 0a Linie: 125960. 001ec090 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 31 0a Linie: 125961. 001ec0a0 00 00 00 00 00 00 00 00 03 00 09 00 00 00 00 00 ................ 001ec0b0 00 00 00 00 00 00 00 00 03 00 0a 00 00 00 00 00 ................ 001ec0c0 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 34 0a Linie: 125964. 001ec0d0 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 35 0a Linie: 125965. 001ec0e0 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 36 0a Linie: 125966. 001ec0f0 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 37 0a Linie: 125967. 001ec100 00 00 00 00 00 00 00 00 10 00 00 00 c7 00 00 00 ................ 001ec110 00 00 00 00 00 00 00 00 10 00 00 00 dd 00 00 00 ................ 001ec120 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 37 30 0a Linie: 125970. 001ec130 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 37 31 0a Linie: 125971. 001ec140 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 37 32 0a Linie: 125972. 001ec150 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 37 33 0a Linie: 125973. 001ec160 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 37 34 0a Linie: 125974. Is there anything else than bit 16 in the PCI_CFG register which needs to be set to force non-coherent PCI accesses and avoid the PCI errata? /Hartvig Eric DeVolder wrote: > There are data cache snoop bugs with respect to PCI on the Au1500. For the > Au1500s soldered on DbAu1500 boards to-date, PCI can not use coherent > transactions. Details in the Specification Update for the Au1500 > available from: > > www.amd.com/pcs > -> Technical Resources > -> AMD Alchemy Solutions Development Board Support > > Login with your board and you'll be presented with various docs, > including the > spec update. > > Regards, > Eric > > Hartvig Ekner wrote: > > >Hi Pete, > > > >I am attempting to use the HW coherency feature of the AU1500 to avoid SW flushes and increase the performance. > >In the config-shared.in file, I can see that the CONFIG_NONCOHERENT_IO define is always set for the AMD > >eval boards, which results in SW cache flushes when dma_cache_xxx functions are called. If HW coherency is > >working, this define should not be set. > > > >However, in your drivers, you only call the dma_cache functions from au1000/common/usbdev.c, but not from e.g. the ethernet > >driver or the audio driver. Is this intentional? It seems that the ethernet & audio driver already relies on HW > >coherency to function correctly (and it also sets the MAC enable bits accordingly, to force all ETH DMA > >accesses to be coherent), so why not USB also? > > > >When turning off NONCOHERENT_IO, there are some bugs (not in AU1000 code) which prevents the code from > >compiling, but I have fixed these. And the kernel boots, but during some large disk-copy tests, I get occasional > >data errors which I'm looking in to. > > > >So before spending more time on debugging this, and creating patches for using HW coherency, I wanted to hear > >your input - maybe there are known problems in the Au1500 coherency implementation? > > > >/Hartvig > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Au1500 hardware cache coherency 2003-03-31 15:14 ` Hartvig Ekner @ 2003-03-31 18:06 ` Eric DeVolder 2003-03-31 19:24 ` Hartvig Ekner 0 siblings, 1 reply; 12+ messages in thread From: Eric DeVolder @ 2003-03-31 18:06 UTC (permalink / raw) To: Hartvig Ekner; +Cc: Pete Popov, Linux MIPS mailing list [-- Attachment #1: Type: text/plain, Size: 7089 bytes --] Yes, this is precisely the errata; my apologies for failing to list it explicitly earlier. The only other one to be aware of is that Config[OD] must be set, and the YAMON booter does this, but if you ever do a R-M-W of Config, then Config[OD] is reset to zero (this is errata #4). Also, keep in mind that you need CONFIG_NONCOHERENT_IO set so that the dma cache routines do a flush (and invalidate) to memory, and that these routines need to be called at the appropriate places in the driver. In your data dump where there are lines "missing", it sort of suggests to me that perhaps these memory addresses were already in the cache, and were written to memory, but because coherent transactions were disabled, the cache never saw them. So, in summary, for AB stepping, you need pci_config[NC]=1, Config[OD]=1, and CONFIG_NONCOHERENT_IO=y. Hartvig Ekner wrote: >Hi Eric, > >is that errata #12 you are referring to? (only present on silicon stepping AB?) Or are there >others also? > >While using HW coherency for copying large files on disk, I do get occasional data errors >which always look like this: > > 00061000 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 33 32 0a Linie: 24832. > 00061010 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 33 33 0a Linie: 24833. > 00061020 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 33 34 0a Linie: 24834. > 00061030 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 33 35 0a Linie: 24835. > 00061040 38 33 35 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 835.Linie: 24 > 00061050 38 33 36 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 836.Linie: 24 > 00061060 38 33 37 0a 38 33 37 0a 4c 69 6e 69 65 3a 20 20 837.837.Linie: > 00061070 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 33 39 0a Linie: 24839. > 00061080 38 33 39 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 839.Linie: 24 > 00061090 38 34 30 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 840.Linie: 24 > 000610a0 38 34 31 0a 38 34 31 0a 4c 69 6e 69 65 3a 20 20 841.841.Linie: > 000610b0 20 20 32 34 65 3a 20 20 20 20 32 34 38 34 33 0a 24e: 24843. > 000610c0 38 34 33 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 843.Linie: 24 > 000610d0 38 34 34 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 844.Linie: 24 > 000610e0 38 34 35 0a 38 34 35 0a 4c 69 6e 69 65 3a 20 20 845.845.Linie: > 000610f0 20 20 32 34 65 3a 20 20 20 20 32 34 38 34 37 0a 24e: 24847. > 00061100 38 34 37 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 847.Linie: 24 > 00061110 38 34 38 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 848.Linie: 24 > 00061120 38 34 39 0a 38 34 39 0a 4c 69 6e 69 65 3a 20 20 849.849.Linie: > 00061130 20 20 32 34 65 3a 20 20 20 20 32 34 38 35 31 0a 24e: 24851. > 00061140 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 32 0a Linie: 24852. > 00061150 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 33 0a Linie: 24853. > 00061160 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 34 0a Linie: 24854. > 00061170 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 35 0a Linie: 24855. > 00061180 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 36 0a Linie: 24856. > 00061190 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 37 0a Linie: 24857. > 000611a0 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 38 0a Linie: 24858. > 000611b0 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 39 0a Linie: 24859. > 000611c0 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 36 30 0a Linie: 24860. > >The file consists of 16-byte lines with increasing line numbers. Note that most of >the errors are cacheline oriented, but there are also "skews" of 4 bytes going >into the next cacheline. > >However, interestingly enough, switching HW coherency off entirely >(in PCI_CFG) and using uncached PCI buffers, there are still failures: > > 001ec060 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 35 38 0a Linie: 125958. > 001ec070 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 35 39 0a Linie: 125959. > 001ec080 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 30 0a Linie: 125960. > 001ec090 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 31 0a Linie: 125961. > 001ec0a0 00 00 00 00 00 00 00 00 03 00 09 00 00 00 00 00 ................ > 001ec0b0 00 00 00 00 00 00 00 00 03 00 0a 00 00 00 00 00 ................ > 001ec0c0 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 34 0a Linie: 125964. > 001ec0d0 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 35 0a Linie: 125965. > 001ec0e0 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 36 0a Linie: 125966. > 001ec0f0 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 37 0a Linie: 125967. > 001ec100 00 00 00 00 00 00 00 00 10 00 00 00 c7 00 00 00 ................ > 001ec110 00 00 00 00 00 00 00 00 10 00 00 00 dd 00 00 00 ................ > 001ec120 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 37 30 0a Linie: 125970. > 001ec130 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 37 31 0a Linie: 125971. > 001ec140 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 37 32 0a Linie: 125972. > 001ec150 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 37 33 0a Linie: 125973. > 001ec160 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 37 34 0a Linie: 125974. > >Is there anything else than bit 16 in the PCI_CFG register which needs to be set >to force non-coherent PCI accesses and avoid the PCI errata? > >/Hartvig > > > >Eric DeVolder wrote: > > > >>There are data cache snoop bugs with respect to PCI on the Au1500. For the >>Au1500s soldered on DbAu1500 boards to-date, PCI can not use coherent >>transactions. Details in the Specification Update for the Au1500 >>available from: >> >>www.amd.com/pcs >> -> Technical Resources >> -> AMD Alchemy Solutions Development Board Support >> >>Login with your board and you'll be presented with various docs, >>including the >>spec update. >> >>Regards, >>Eric >> >>Hartvig Ekner wrote: >> >> >> >>>Hi Pete, >>> >>>I am attempting to use the HW coherency feature of the AU1500 to avoid SW flushes and increase the performance. >>>In the config-shared.in file, I can see that the CONFIG_NONCOHERENT_IO define is always set for the AMD >>>eval boards, which results in SW cache flushes when dma_cache_xxx functions are called. If HW coherency is >>>working, this define should not be set. >>> >>>However, in your drivers, you only call the dma_cache functions from au1000/common/usbdev.c, but not from e.g. the ethernet >>>driver or the audio driver. Is this intentional? It seems that the ethernet & audio driver already relies on HW >>>coherency to function correctly (and it also sets the MAC enable bits accordingly, to force all ETH DMA >>>accesses to be coherent), so why not USB also? >>> >>>When turning off NONCOHERENT_IO, there are some bugs (not in AU1000 code) which prevents the code from >>>compiling, but I have fixed these. And the kernel boots, but during some large disk-copy tests, I get occasional >>>data errors which I'm looking in to. >>> >>>So before spending more time on debugging this, and creating patches for using HW coherency, I wanted to hear >>>your input - maybe there are known problems in the Au1500 coherency implementation? >>> >>>/Hartvig >>> >>> >>> > > > > > [-- Attachment #2: Type: text/html, Size: 7507 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Au1500 hardware cache coherency 2003-03-31 18:06 ` Eric DeVolder @ 2003-03-31 19:24 ` Hartvig Ekner 2003-03-31 20:33 ` Pete Popov 0 siblings, 1 reply; 12+ messages in thread From: Hartvig Ekner @ 2003-03-31 19:24 UTC (permalink / raw) To: Eric DeVolder; +Cc: Linux MIPS mailing list [-- Attachment #1: Type: text/plain, Size: 6681 bytes --] Hi Eric, I did a quick check of a complete kernel disassembly, and there are tons of direct or indirect RMW's to config, which do not explicitly insure that Config[0D] is set. Pete - are you aware of this? Thus, there seems to be a potential problem lurking here for anybody who is using USB. However, I am not using USB at all, and it is configured out of the kernel. So I assume this is not errata #3 we're seeing here? So, to summarize: The first set of problems in my email below seem to be fully explained by errata #14. Note that any kernel compiled from the current CVS exhibits this problem: Because although NONCOHEHENT_IO is set, the NC bit in PCI_CFG is not set. I have verified that the problem occurs when NC is cleared, regardless of the .config option. So some code needs to be changed in au1000/xxx/setup.c... (set NC if NONCOHERENT_IO is enabled). But - much wore worrisome: I did this modification, and with the NC bit set, and NONCOHERENT_IO set, I get the second set of errors, although it takes much longer time. The wback_inv calls are made through the generic code in the subroutine pci_alloc_consistent() (in arch/mips/kernel/pci-dma.c). So something is wrong.... Anybody at AMD who would care to continue the debug? /Hartvig Eric DeVolder wrote: > Yes, this is precisely the errata; my apologies for failing to list it explicitly earlier. > The only other one to be aware of is that Config[OD] must be set, and the YAMON > booter does this, but if you ever do a R-M-W of Config, then Config[OD] is reset to > zero (this is errata #4). > > Also, keep in mind that you need CONFIG_NONCOHERENT_IO set so that > the dma cache routines do a flush (and invalidate) to memory, and that these routines > need to be called at the appropriate places in the driver. > > In your data dump where there are lines "missing", it sort of suggests to me that perhaps > these memory addresses were already in the cache, and were written to memory, but > because coherent transactions were disabled, the cache never saw them. > > So, in summary, for AB stepping, you need pci_config[NC]=1, Config[OD]=1, and > CONFIG_NONCOHERENT_IO=y. > > Hartvig Ekner wrote: > >> Hi Eric, >> >> is that errata #12 you are referring to? (only present on silicon stepping AB?) Or are there >> others also? >> >> While using HW coherency for copying large files on disk, I do get occasional data errors >> which always look like this: >> >> 00061000 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 33 32 0a Linie: 24832. >> 00061010 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 33 33 0a Linie: 24833. >> 00061020 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 33 34 0a Linie: 24834. >> 00061030 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 33 35 0a Linie: 24835. >> 00061040 38 33 35 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 835.Linie: 24 >> 00061050 38 33 36 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 836.Linie: 24 >> 00061060 38 33 37 0a 38 33 37 0a 4c 69 6e 69 65 3a 20 20 837.837.Linie: >> 00061070 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 33 39 0a Linie: 24839. >> 00061080 38 33 39 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 839.Linie: 24 >> 00061090 38 34 30 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 840.Linie: 24 >> 000610a0 38 34 31 0a 38 34 31 0a 4c 69 6e 69 65 3a 20 20 841.841.Linie: >> 000610b0 20 20 32 34 65 3a 20 20 20 20 32 34 38 34 33 0a 24e: 24843. >> 000610c0 38 34 33 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 843.Linie: 24 >> 000610d0 38 34 34 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 844.Linie: 24 >> 000610e0 38 34 35 0a 38 34 35 0a 4c 69 6e 69 65 3a 20 20 845.845.Linie: >> 000610f0 20 20 32 34 65 3a 20 20 20 20 32 34 38 34 37 0a 24e: 24847. >> 00061100 38 34 37 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 847.Linie: 24 >> 00061110 38 34 38 0a 4c 69 6e 69 65 3a 20 20 20 20 32 34 848.Linie: 24 >> 00061120 38 34 39 0a 38 34 39 0a 4c 69 6e 69 65 3a 20 20 849.849.Linie: >> 00061130 20 20 32 34 65 3a 20 20 20 20 32 34 38 35 31 0a 24e: 24851. >> 00061140 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 32 0a Linie: 24852. >> 00061150 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 33 0a Linie: 24853. >> 00061160 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 34 0a Linie: 24854. >> 00061170 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 35 0a Linie: 24855. >> 00061180 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 36 0a Linie: 24856. >> 00061190 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 37 0a Linie: 24857. >> 000611a0 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 38 0a Linie: 24858. >> 000611b0 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 35 39 0a Linie: 24859. >> 000611c0 4c 69 6e 69 65 3a 20 20 20 20 32 34 38 36 30 0a Linie: 24860. >> >> The file consists of 16-byte lines with increasing line numbers. Note that most of >> the errors are cacheline oriented, but there are also "skews" of 4 bytes going >> into the next cacheline. >> >> However, interestingly enough, switching HW coherency off entirely >> (in PCI_CFG) and using uncached PCI buffers, there are still failures: >> >> 001ec060 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 35 38 0a Linie: 125958. >> 001ec070 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 35 39 0a Linie: 125959. >> 001ec080 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 30 0a Linie: 125960. >> 001ec090 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 31 0a Linie: 125961. >> 001ec0a0 00 00 00 00 00 00 00 00 03 00 09 00 00 00 00 00 ................ >> 001ec0b0 00 00 00 00 00 00 00 00 03 00 0a 00 00 00 00 00 ................ >> 001ec0c0 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 34 0a Linie: 125964. >> 001ec0d0 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 35 0a Linie: 125965. >> 001ec0e0 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 36 0a Linie: 125966. >> 001ec0f0 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 36 37 0a Linie: 125967. >> 001ec100 00 00 00 00 00 00 00 00 10 00 00 00 c7 00 00 00 ................ >> 001ec110 00 00 00 00 00 00 00 00 10 00 00 00 dd 00 00 00 ................ >> 001ec120 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 37 30 0a Linie: 125970. >> 001ec130 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 37 31 0a Linie: 125971. >> 001ec140 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 37 32 0a Linie: 125972. >> 001ec150 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 37 33 0a Linie: 125973. >> 001ec160 4c 69 6e 69 65 3a 20 20 20 31 32 35 39 37 34 0a Linie: 125974. >> >> Is there anything else than bit 16 in the PCI_CFG register which needs to be set >> to force non-coherent PCI accesses and avoid the PCI errata? >> >> /Hartvig >> >> >> >> [-- Attachment #2: Type: text/html, Size: 8369 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Au1500 hardware cache coherency 2003-03-31 19:24 ` Hartvig Ekner @ 2003-03-31 20:33 ` Pete Popov 2003-04-01 7:51 ` Hartvig Ekner 0 siblings, 1 reply; 12+ messages in thread From: Pete Popov @ 2003-03-31 20:33 UTC (permalink / raw) To: Hartvig Ekner; +Cc: Eric DeVolder, Linux MIPS mailing list On Mon, 2003-03-31 at 11:24, Hartvig Ekner wrote: > Hi Eric, > > I did a quick check of a complete kernel disassembly, and there are > tons of direct or indirect RMW's to config, which do not explicitly > insure that Config[0D] is set. > Pete - are you aware of this? Config[OD] is set in setup.c and should not be cleared afterward. > Thus, there seems to be a potential problem lurking here for anybody > who is using USB. > > However, I am not using USB at all, and it is configured out of the > kernel. So I assume this is not errata #3 we're seeing here? > > So, to summarize: The first set of problems in my email below seem to > be fully explained by errata #14. Note that any kernel compiled from > the current CVS exhibits this problem: > Because although NONCOHEHENT_IO is set, the NC bit in PCI_CFG is not > set. Hmm, ok, I'll check that out. > I have verified that the problem occurs when NC is cleared, regardless > of the .config option. So some code needs to be changed in > au1000/xxx/setup.c... (set NC if NONCOHERENT_IO > is enabled). > But - much wore worrisome: I did this modification, and with the NC > bit set, and NONCOHERENT_IO set, I get the second set of errors, > although it takes much longer time. The wback_inv calls are made > through the generic code in the subroutine > pci_alloc_consistent() (in arch/mips/kernel/pci-dma.c). > So something is wrong.... Anybody at AMD who would care to continue > the debug? Can you send me your test and exact instructions on how you're duplicating the error? I won't have time to look at it until after 4/20 though. Pete ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Au1500 hardware cache coherency 2003-03-31 20:33 ` Pete Popov @ 2003-04-01 7:51 ` Hartvig Ekner 2003-04-01 18:22 ` Pete Popov 0 siblings, 1 reply; 12+ messages in thread From: Hartvig Ekner @ 2003-04-01 7:51 UTC (permalink / raw) To: Pete, Popov; +Cc: Linux MIPS mailing list Hi Pete, Pete Popov wrote: > On Mon, 2003-03-31 at 11:24, Hartvig Ekner wrote: > > Hi Eric, > > > > I did a quick check of a complete kernel disassembly, and there are > > tons of direct or indirect RMW's to config, which do not explicitly > > insure that Config[0D] is set. > > Pete - are you aware of this? > > Config[OD] is set in setup.c and should not be cleared afterward. > Due to errata #4, it is cleared whenever macroes like set_c0_config or change_c0_config is called. This happens in several places: au1000_restart (probably doesn't matter?) cache parity error exception (doesn't matter, we're probably dying anyway) ld_mmu_mips32 (in c-mips32.c) I'm not quite sure whether ld_mmu_mips32 is called after au1x00 setup, but if it is, the bit is cleared, never to be set again. Maybe the c0_config macroes should be changed due to errata #4? > > So, to summarize: The first set of problems in my email below seem to > > be fully explained by errata #14. Note that any kernel compiled from > > the current CVS exhibits this problem: > > Because although NONCOHEHENT_IO is set, the NC bit in PCI_CFG is not > > set. > > Hmm, ok, I'll check that out. > > > > I have verified that the problem occurs when NC is cleared, regardless > > of the .config option. So some code needs to be changed in > > au1000/xxx/setup.c... (set NC if NONCOHERENT_IO > > is enabled). > > > But - much wore worrisome: I did this modification, and with the NC > > bit set, and NONCOHERENT_IO set, I get the second set of errors, > > although it takes much longer time. The wback_inv calls are made > > through the generic code in the subroutine > > pci_alloc_consistent() (in arch/mips/kernel/pci-dma.c). > > > So something is wrong.... Anybody at AMD who would care to continue > > the debug? > > Can you send me your test and exact instructions on how you're > duplicating the error? I won't have time to look at it until after 4/20 > though. > Sure. However, I will first try to make sure that the kernel does not have the same problem on another non AU1500 platform. BTW, are you using the HPT onboard IDE controller? Last time I tried, it wasn't functional, the kernel crashed during boot when kudzu was doing some HW probing on the IDE stuff. I'm using a plug-in promise card (20268 based). /Hartvig ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Au1500 hardware cache coherency 2003-04-01 7:51 ` Hartvig Ekner @ 2003-04-01 18:22 ` Pete Popov 2003-04-01 18:23 ` Pete Popov 2003-04-02 12:17 ` Maciej W. Rozycki 0 siblings, 2 replies; 12+ messages in thread From: Pete Popov @ 2003-04-01 18:22 UTC (permalink / raw) To: Hartvig Ekner; +Cc: Pete, Popov, Linux MIPS mailing list > > Config[OD] is set in setup.c and should not be cleared afterward. > Due to errata #4, it is cleared whenever macroes like set_c0_config or change_c0_config is > called. This happens in several places: > au1000_restart (probably doesn't matter?) > cache parity error exception (doesn't matter, we're probably dying anyway) > ld_mmu_mips32 (in c-mips32.c) Thanks for bringing this to my attention. I'll take a look at it, but I'm leaving this Thursday for two weeks and won't be able to get to it until after 4/21. > I'm not quite sure whether ld_mmu_mips32 is called after au1x00 setup, but if it is, > the bit is cleared, never to be set again. Maybe the c0_config macroes should be changed > due to errata #4? I doubt Ralf is going to change common macros to fix a specific bug. > > Can you send me your test and exact instructions on how you're > > duplicating the error? I won't have time to look at it until after 4/20 > > though. > > > > Sure. However, I will first try to make sure that the kernel does not have the same problem on another > non AU1500 platform. > > BTW, are you using the HPT onboard IDE controller? Last time I tried, it wasn't functional, the kernel crashed > during boot when kudzu was doing some HW probing on the IDE stuff. I'm using a plug-in promise card > (20268 based). The Pb1500 HPT370 driver works fine for me, and now the Db1500 HPT371 seems to work as well. However, with the 2.4.20-pre4 kernel, both boards crash with a kernel panic when doing a very large 'cp -a'. I don't know at this point what the problem is. The MV 2.4.18 based kernel passes the same stress tests repeatedly on the Pb1500. So either something got broken between 2.4.18 and 2.4.20-pre4, or the 2.4.18 kernel is getting lucky. The boards are still 'usable' with a 2.4.20-pre4 and a hard disk cause I can boot with a disk based root fs and run lmbench of the disk. But it's quite possible that the problem you observed is caused by the same bug I'm encountering. I think I have a Promise IDE card at home and I'll run a test with it when I get back. It would be interesting to see if that driver causes the same problems. Pete ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Au1500 hardware cache coherency 2003-04-01 18:22 ` Pete Popov @ 2003-04-01 18:23 ` Pete Popov 2003-04-02 12:17 ` Maciej W. Rozycki 1 sibling, 0 replies; 12+ messages in thread From: Pete Popov @ 2003-04-01 18:23 UTC (permalink / raw) To: Hartvig Ekner; +Cc: Pete, Popov, Linux MIPS mailing list As Ralf pointed out, s/2.4.20-pre4/2.4.21-pre4 :) Pete On Tue, 2003-04-01 at 10:22, Pete Popov wrote: > > > Config[OD] is set in setup.c and should not be cleared afterward. > > > Due to errata #4, it is cleared whenever macroes like set_c0_config or change_c0_config is > > called. This happens in several places: > > > au1000_restart (probably doesn't matter?) > > cache parity error exception (doesn't matter, we're probably dying anyway) > > ld_mmu_mips32 (in c-mips32.c) > > Thanks for bringing this to my attention. I'll take a look at it, but > I'm leaving this Thursday for two weeks and won't be able to get to it > until after 4/21. > > > I'm not quite sure whether ld_mmu_mips32 is called after au1x00 setup, but if it is, > > the bit is cleared, never to be set again. Maybe the c0_config macroes should be changed > > due to errata #4? > > I doubt Ralf is going to change common macros to fix a specific bug. > > > > Can you send me your test and exact instructions on how you're > > > duplicating the error? I won't have time to look at it until after 4/20 > > > though. > > > > > > > Sure. However, I will first try to make sure that the kernel does not have the same problem on another > > non AU1500 platform. > > > > BTW, are you using the HPT onboard IDE controller? Last time I tried, it wasn't functional, the kernel crashed > > during boot when kudzu was doing some HW probing on the IDE stuff. I'm using a plug-in promise card > > (20268 based). > > The Pb1500 HPT370 driver works fine for me, and now the Db1500 HPT371 > seems to work as well. However, with the 2.4.20-pre4 kernel, both boards > crash with a kernel panic when doing a very large 'cp -a'. I don't know > at this point what the problem is. The MV 2.4.18 based kernel passes the > same stress tests repeatedly on the Pb1500. So either something got > broken between 2.4.18 and 2.4.20-pre4, or the 2.4.18 kernel is getting > lucky. The boards are still 'usable' with a 2.4.20-pre4 and a hard disk > cause I can boot with a disk based root fs and run lmbench of the disk. > But it's quite possible that the problem you observed is caused by the > same bug I'm encountering. > > I think I have a Promise IDE card at home and I'll run a test with it > when I get back. It would be interesting to see if that driver causes > the same problems. > > Pete ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Au1500 hardware cache coherency 2003-04-01 18:22 ` Pete Popov 2003-04-01 18:23 ` Pete Popov @ 2003-04-02 12:17 ` Maciej W. Rozycki 1 sibling, 0 replies; 12+ messages in thread From: Maciej W. Rozycki @ 2003-04-02 12:17 UTC (permalink / raw) To: Pete Popov; +Cc: Hartvig Ekner, Pete, Popov, Linux MIPS mailing list On 1 Apr 2003, Pete Popov wrote: > > I'm not quite sure whether ld_mmu_mips32 is called after au1x00 setup, but if it is, > > the bit is cleared, never to be set again. Maybe the c0_config macroes should be changed > > due to errata #4? > > I doubt Ralf is going to change common macros to fix a specific bug. You'd better ask instead of guessing. I would see no problem with such a workaround IF done cleanly. -- + Maciej W. Rozycki, Technical University of Gdansk, Poland + +--------------------------------------------------------------+ + e-mail: macro@ds2.pg.gda.pl, PGP key available + ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Au1500 hardware cache coherency 2003-03-31 12:08 Au1500 hardware cache coherency Hartvig Ekner 2003-03-31 14:41 ` Eric DeVolder @ 2003-03-31 18:35 ` Pete Popov 2003-04-01 11:47 ` Ralf Baechle 1 sibling, 1 reply; 12+ messages in thread From: Pete Popov @ 2003-03-31 18:35 UTC (permalink / raw) To: Hartvig Ekner; +Cc: Linux MIPS mailing list > So before spending more time on debugging this, and creating patches > for using HW coherency, I wanted to hear your input - maybe there are > known problems in the Au1500 coherency implementation? Looks like Eric already replied to your questions. Patches to get the code to compile with NONCOHERENT_IO turned off would be good, even if you don't want to do that yet. FYI, in the latest kernel I'm getting a panic when doing a very large 'cp -a' from nfs to ata100 disk (don't know if the disk itself matters) (Pb1500 and Db1500). The same test passes with a 2.4.18 kernel, so it seems like it's not a hardware issue, unless the 2.4.18 kernel is just getting lucky. I'll be gone till 4/20 so I won't have time to debug it until after I get back. Pete ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Au1500 hardware cache coherency 2003-03-31 18:35 ` Pete Popov @ 2003-04-01 11:47 ` Ralf Baechle 0 siblings, 0 replies; 12+ messages in thread From: Ralf Baechle @ 2003-04-01 11:47 UTC (permalink / raw) To: Pete Popov; +Cc: Hartvig Ekner, Linux MIPS mailing list On Mon, Mar 31, 2003 at 10:35:14AM -0800, Pete Popov wrote: > > So before spending more time on debugging this, and creating patches > > for using HW coherency, I wanted to hear your input - maybe there are > > known problems in the Au1500 coherency implementation? > > Looks like Eric already replied to your questions. > > Patches to get the code to compile with NONCOHERENT_IO turned off would > be good, even if you don't want to do that yet. > > FYI, in the latest kernel I'm getting a panic when doing a very large > 'cp -a' from nfs to ata100 disk (don't know if the disk itself matters) > (Pb1500 and Db1500). The same test passes with a 2.4.18 kernel, so it > seems like it's not a hardware issue, unless the 2.4.18 kernel is just > getting lucky. I'll be gone till 4/20 so I won't have time to debug it > until after I get back. Not that this is Linux 2.4.21-pre4 which did replace essentially the entire IDE code due to major problems with the old IDE core. The problems were huge - large enough to warrant the rewrite of such a large subsystem in a stable series of kernels and certainly the dust hasn't fully settled yet. Ralf ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2003-04-02 12:17 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2003-03-31 12:08 Au1500 hardware cache coherency Hartvig Ekner 2003-03-31 14:41 ` Eric DeVolder 2003-03-31 15:14 ` Hartvig Ekner 2003-03-31 18:06 ` Eric DeVolder 2003-03-31 19:24 ` Hartvig Ekner 2003-03-31 20:33 ` Pete Popov 2003-04-01 7:51 ` Hartvig Ekner 2003-04-01 18:22 ` Pete Popov 2003-04-01 18:23 ` Pete Popov 2003-04-02 12:17 ` Maciej W. Rozycki 2003-03-31 18:35 ` Pete Popov 2003-04-01 11:47 ` Ralf Baechle
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox