Re: HPPA problems with the PCI

Re: HPPA problems with the PCI

[Grant Grundler]

+linux-parisc mailing list (public list since link at bottom is public too)

On Wed, May 22, 2019 at 9:16 AM Carlo Pisani <carlojpisani@gmail.com> wrote:
> hi
> I have been testing a few PCI-X SATA cards on C3600 and they do not
> work properly for weird not yet clear reason.
>
> Here (1) you can see what I tested, how, etc.

Hi Carlo,
It's awesome that you are testing SATA cards and running linux on a C3600.

 "not work properly" isn't a lot of information to go by.  You've been
keeping good notes at (1) and I encourage others on the parisc mailing
list to take a look.

> Cards have been recently tested on a PowerMacG4 and they do work
> perfectly. Note the PowerMacG4 is PCI64, not PCI-X, so it's the same
> as the C3600.

While PCI bus type is the same, most problems with device drivers are
at DMA Coherency/Memory ordering level.  C3600 has PA8600 processor
and you might learn more about PA-8600 processor, Astro (IOMMU), and
Elroy (PCI host controller) from
https://www.openpa.net/systems/hp-visualize_b1000_c3000_c3600.html

If you can try some experiments, start adding mb() calls after the
driver adds or removes an IO request from any list or queue.

> My feelings about a PCI-X card in the PCI64 of the C3600 is that there
> is something wrong, probably the PCIXCAP pin is missing on the BUS?

I have no idea. It's been > 10 years since I've had access to HP HW
documents or used a PCI or PCI-X bus analyzer.

I'm skeptical on your hypothesis though - HP was very thorough in
complying with PCI spec - assuming PCIXCAP pin was defined by PCI
spec.

You'd have to get a PCI protocol analyzer to collect signaling
information to see where the card fails to initialize or operate
correctly.

> What do you think?
> Carlo
>
> (1) http://www.downthebunker.com/reloaded/space/viewtopic.php?f=50&t=337

cheers,
grant

Re: HPPA problems with the PCI

[John David Anglin]

On 2019-05-22 8:38 p.m., Grant Grundler wrote:
> While PCI bus type is the same, most problems with device drivers are
> at DMA Coherency/Memory ordering level.  C3600 has PA8600 processor
> and you might learn more about PA-8600 processor, Astro (IOMMU), and
> Elroy (PCI host controller) from
> https://www.openpa.net/systems/hp-visualize_b1000_c3000_c3600.html
>
> If you can try some experiments, start adding mb() calls after the
> driver adds or removes an IO request from any list or queue.
That's an interesting comment.  On a UP kernel, mb() is currently just a compiler
memory barrier.  On a SMP kernel, mb() generates a "sync" instruction.  We also
use "ldcw" as a barrier in spinlocks.

I'm thinking dma_rmb() and dma_wmb() may need to be stronger.  Is "sync" or "syncdma"
a better choice for these defines?

Cheers,
Dave

-- 
John David Anglin  dave.anglin@bell.net

Re: HPPA problems with the PCI

[Grant Grundler]

[resending as plain-text. :(  gmail is such a PITA - it should just
know by now that anything going to vger needs to be plain text. :/ ]

On Wed, May 22, 2019 at 6:09 PM John David Anglin <dave.anglin@bell.net> wrote:
>
> On 2019-05-22 8:38 p.m., Grant Grundler wrote:
> > While PCI bus type is the same, most problems with device drivers are
> > at DMA Coherency/Memory ordering level.  C3600 has PA8600 processor
> > and you might learn more about PA-8600 processor, Astro (IOMMU), and
> > Elroy (PCI host controller) from
> > https://www.openpa.net/systems/hp-visualize_b1000_c3000_c3600.html
> >
> > If you can try some experiments, start adding mb() calls after the
> > driver adds or removes an IO request from any list or queue.
> That's an interesting comment.

I was also being very lazy and imprecise. :)

More specifically, I was thinking the mb() should be placed AFTER
adding any IOs to data structure in memory the device will read but
BEFORE the driver tells the device more IO requests.   I didn't look
if such sequences even exist in the drivers mentioned. If the devices
use "mail boxes", completely different issues around ordering can come
up.

> On a UP kernel, mb() is currently just a compiler
> memory barrier.  On a SMP kernel, mb() generates a "sync" instruction.  We also
> use "ldcw" as a barrier in spinlocks.

Yeah, I'm not sure how strong the mb() needs to be and maybe I'm
giving the wrong advice: use dma_wmb() for the case I've described
above.  Then use dma_rmb() before reading data structures updated by
the device. See examples in the existing code:
   https://elixir.bootlin.com/linux/v4.20/ident/dma_wmb

[It's interesting that mostly networking drivers are using these
memory barriers and very few block devices do.]

OTOH, limiting the compiler is often sufficient to get "expected
behavior" since the caches and memory controllers usually aren't that
busy/backlogged that memory accesses can get very much out of order -
especially memory reads.  Especially for drivers only tested on x86
where everything is pretty strongly ordered and compiler doesn't have
many registers to work with. These older drivers are more likely to
have issues with RISC compiler ordering accesses into sequences that
are "unexpected" (even if perfectly correct from compilers PoV).

> I'm thinking dma_rmb() and dma_wmb() may need to be stronger.  Is "sync" or "syncdma"
> a better choice for these defines?

Hrm, maybe but seems like we should be using dma_*mb() calls since
they are available.

cheers,
grant

>
> Cheers,
> Dave
>
> --
> John David Anglin  dave.anglin@bell.net
>

Re: HPPA problems with the PCI

[John David Anglin]

On 2019-05-23 12:43 a.m., Grant Grundler wrote:
>> On a UP kernel, mb() is currently just a compiler
>> memory barrier.  On a SMP kernel, mb() generates a "sync" instruction.  We also
>> use "ldcw" as a barrier in spinlocks.
> Yeah, I'm not sure how strong the mb() needs to be and maybe I'm
> giving the wrong advice: use dma_wmb() for the case I've described
> above.  Then use dma_rmb() before reading data structures updated by
> the device. See examples in the existing code:
>    https://elixir.bootlin.com/linux/v4.20/ident/dma_wmb
> 
Looking at arm and arm64, I think sync should be used for mb(), rmb() and wmb().  Possibly,
ldcw can be used for dma_rmb() and dma_wmb() although sync should be okay.  Sync is heavier
than ldcw.  The __smp barriers could use ldcw.

Arm64 doesn't distinguish between UP and SMP.  32-bit arm has this config option,
CONFIG_ARM_DMA_MEM_BUFFERABLE, that enables stronger barriers when defined.  I think we
should use the same barriers on UP and SMP on parisc to ensure we properly synchronize I/O
operations.

Dave
-- 
John David Anglin  dave.anglin@bell.net

Re: HPPA problems with the PCI

[Carlo Pisani]

I have just recompiled the kernel with the SMP support and tested the
sil24 driver.
No dice, it has failed again.

I have a PCI64 SCSI card and a SCSI U160 harddrive, so I am going to
test it with the same workbench that I am using for testing SATA.

Re: HPPA problems with the PCI

[Grant Grundler]

On Thu, May 23, 2019 at 9:10 AM John David Anglin <dave.anglin@bell.net> wrote:
>
> On 2019-05-23 12:43 a.m., Grant Grundler wrote:
> >> On a UP kernel, mb() is currently just a compiler
> >> memory barrier.  On a SMP kernel, mb() generates a "sync" instruction.  We also
> >> use "ldcw" as a barrier in spinlocks.
> > Yeah, I'm not sure how strong the mb() needs to be and maybe I'm
> > giving the wrong advice: use dma_wmb() for the case I've described
> > above.  Then use dma_rmb() before reading data structures updated by
> > the device. See examples in the existing code:
> >    https://elixir.bootlin.com/linux/v4.20/ident/dma_wmb
> >
> Looking at arm and arm64, I think sync should be used for mb(), rmb() and wmb().  Possibly,
> ldcw can be used for dma_rmb() and dma_wmb() although sync should be okay.  Sync is heavier
> than ldcw.  The __smp barriers could use ldcw.
>
> Arm64 doesn't distinguish between UP and SMP.  32-bit arm has this config option,
> CONFIG_ARM_DMA_MEM_BUFFERABLE, that enables stronger barriers when defined.  I think we
> should use the same barriers on UP and SMP on parisc to ensure we properly synchronize I/O
> operations.

dma_wmb/rmb were added to linux kernel in 2014 - well after I stopped
working on this. So I never had to think about this while details of
parisc internals were a bit fresher in my brain. Helge might know
enough to determine this.

I suspect ARMs approach is correct: CPU memory model doesn't change
depending on whether kernel is compiled for UP or SMP. Using same type
of barriers for DMA makes sense.  Just becareful that the
implementation for something called "sync" could be completely
different on ARM64 - even on different ARM64 implementations.

cheers,
grant

>
> Dave
> --
> John David Anglin  dave.anglin@bell.net

Re: HPPA problems with the PCI

[John David Anglin]

On 2019-05-23 12:37 a.m., Grant Grundler wrote:
>
>     If you can try some experiments, start adding mb() calls after the
>     > driver adds or removes an IO request from any list or queue.
>     That's an interesting comment.
>
>
> I was also being very lazy and imprecise. :)
>
> More specifically, I was thinking the mb() should be placed AFTER adding any IOs to data structure in memory the device will read but BEFORE
> the driver tells the device more IO requests.   I didn't look if such sequences even exist in the drivers mentioned. If the devices use "mail
> boxes", completely different issues around ordering can come up.
>
>       On a UP kernel, mb() is currently just a compiler
>     memory barrier.  On a SMP kernel, mb() generates a "sync" instruction.  We also
>     use "ldcw" as a barrier in spinlocks.
>
>
> Yeah, I'm not sure how strong the mb() needs to be and maybe I'm giving the wrong advice: use dma_wmb() for the case I've described above. 
> Then use dma_rmb() before reading data structures updated by the device. See examples in the existing code:
>    https://elixir.bootlin.com/linux/v4.20/ident/dma_wmb
>
I brought up the issue because PA 2.0 processors do out-of-order execution of loads and stores.  This
will be visible to I/O processors even on a UP machine.

Carlo, are you testing with UP or SMP kernel?

Dave

-- 
John David Anglin  dave.anglin@bell.net

Re: HPPA problems with the PCI

[Carlo Pisani]

> Carlo, are you testing with UP or SMP kernel?

Kernel without SMP.
I am recompiling with SMP just right now

Re: HPPA problems with the PCI

[Carlo Pisani]

maybe irrelevant, but I have just noticed this line in the dmesg
(kernel 4.16, running on C3600)

"runtime IRQ mapping not provided by arch"

drivers/pci/setup-irq.c:                pci_dbg(dev, "runtime IRQ
mapping not provided by arch\n");

It's not exactly clear to me, googling I have found this
https://patchwork.kernel.org/patch/9775659

what do you think?

Re: HPPA problems with the PCI

[Grant Grundler]

On Thu, May 23, 2019 at 6:44 AM Carlo Pisani <carlojpisani@gmail.com> wrote:
>
> maybe irrelevant, but I have just noticed this line in the dmesg
> (kernel 4.16, running on C3600)
>
> "runtime IRQ mapping not provided by arch"
>
> drivers/pci/setup-irq.c:                pci_dbg(dev, "runtime IRQ
> mapping not provided by arch\n");
>
> It's not exactly clear to me, googling I have found this
> https://patchwork.kernel.org/patch/9775659
>
> what do you think?

Red Herring (ie not a problem).

AFAIK, parisc never used firmware to route IRQ lines. For starters,
the CPUs never had an IRQ input and always used messages (equivalent
to MMIO writes) directed at a CPUs EIEM register.

IIRC, the PCI Host bus controller to route IRQ lines to slots in a
specific pattern which is hard coded in the PCI host controller
driver.  See dino (?) and elroy PCI host controller drivers.

cheers,
grant

Re: HPPA problems with the PCI

[Sven Schnelle]

Hi Grant,

On Thu, May 23, 2019 at 04:09:34PM -0700, Grant Grundler wrote:
> 
> AFAIK, parisc never used firmware to route IRQ lines. For starters,
> the CPUs never had an IRQ input and always used messages (equivalent
> to MMIO writes) directed at a CPUs EIEM register.

I think you mean EIRR here?

Regards
Sven

Re: HPPA problems with the PCI

[Grant Grundler]

On Thu, May 23, 2019 at 11:39 PM Sven Schnelle <svens@stackframe.org> wrote:
>
> Hi Grant,
>
> On Thu, May 23, 2019 at 04:09:34PM -0700, Grant Grundler wrote:
> >
> > AFAIK, parisc never used firmware to route IRQ lines. For starters,
> > the CPUs never had an IRQ input and always used messages (equivalent
> > to MMIO writes) directed at a CPUs EIEM register.
>
> I think you mean EIRR here?

Yes - most likely. :D

"EIEM" just slipped out more naturally. :D

cheers,
grant

>
> Regards
> Sven

Re: HPPA problems with the PCI

[Carlo Pisani]

on  kernel 5.1, from the git repository, compiled with SMP

dmesg | grep altern
[    2.551002] alternatives: applied 156 out of 175 patches

this patch has made my day!
It seems the solution for the problem!

the card is still working after 10 hours of burn-in test!

--- a/arch/parisc/include/asm/cache.h
+++ b/arch/parisc/include/asm/cache.h
@@ -52,7 +52,6 @@ void parisc_setup_cache_timing(void);

 #define asm_io_fdc(addr) asm volatile("fdc %%r0(%0)" \
                        ALTERNATIVE(ALT_COND_NO_DCACHE, INSN_NOP) \
-                       ALTERNATIVE(ALT_COND_NO_IOC_FDC, INSN_NOP) \
                        : : "r" (addr) : "memory")
 #define asm_io_sync()  asm volatile("sync" \
                        ALTERNATIVE(ALT_COND_NO_DCACHE, INSN_NOP) \

Re: HPPA problems with the PCI

[John David Anglin]

On 2019-05-25 5:01 a.m., Carlo Pisani wrote:
> on  kernel 5.1, from the git repository, compiled with SMP
>
> dmesg | grep altern
> [    2.551002] alternatives: applied 156 out of 175 patches
>
> this patch has made my day!
> It seems the solution for the problem!
>
> the card is still working after 10 hours of burn-in test!
>
> --- a/arch/parisc/include/asm/cache.h
> +++ b/arch/parisc/include/asm/cache.h
> @@ -52,7 +52,6 @@ void parisc_setup_cache_timing(void);
>
>  #define asm_io_fdc(addr) asm volatile("fdc %%r0(%0)" \
>                         ALTERNATIVE(ALT_COND_NO_DCACHE, INSN_NOP) \
> -                       ALTERNATIVE(ALT_COND_NO_IOC_FDC, INSN_NOP) \
>                         : : "r" (addr) : "memory")
>  #define asm_io_sync()  asm volatile("sync" \
>                         ALTERNATIVE(ALT_COND_NO_DCACHE, INSN_NOP) \
The change isn't fully correct as the alternative should also have been removed from
the asm_io_sync() define.

If the change actually fixes controller, then there must be a problem with the test in
arch/parisc/kernel/alternative.c that determines whether or not to patch out fdc instruction
on c3600.  See Helge's patch to fix alternative coding for PCX-U CPUs.  It would help to
print boot_cpu_data.pdc.capabilities and boot_cpu_data.cpu_type on your machine.

Dave

-- 
John David Anglin  dave.anglin@bell.net

Re: HPPA problems with the PCI

[Helge Deller]

On 25.05.19 16:13, John David Anglin wrote:
> On 2019-05-25 5:01 a.m., Carlo Pisani wrote:
>> on  kernel 5.1, from the git repository, compiled with SMP
>>
>> dmesg | grep altern
>> [    2.551002] alternatives: applied 156 out of 175 patches
>>
>> this patch has made my day!
>> It seems the solution for the problem!
>>
>> the card is still working after 10 hours of burn-in test!
>
>> --- a/arch/parisc/include/asm/cache.h
>> +++ b/arch/parisc/include/asm/cache.h
>> @@ -52,7 +52,6 @@ void parisc_setup_cache_timing(void);
>>
>>  #define asm_io_fdc(addr) asm volatile("fdc %%r0(%0)" \
>>                         ALTERNATIVE(ALT_COND_NO_DCACHE, INSN_NOP) \
>> -                       ALTERNATIVE(ALT_COND_NO_IOC_FDC, INSN_NOP) \
>>                         : : "r" (addr) : "memory")
>>  #define asm_io_sync()  asm volatile("sync" \
>>                         ALTERNATIVE(ALT_COND_NO_DCACHE, INSN_NOP) \
> The change isn't fully correct as the alternative should also have been removed from
> the asm_io_sync() define.
>
> If the change actually fixes controller, then there must be a problem with the test in
> arch/parisc/kernel/alternative.c that determines whether or not to patch out fdc instruction
> on c3600.  See Helge's patch to fix alternative coding for PCX-U CPUs.  It would help to
> print boot_cpu_data.pdc.capabilities and boot_cpu_data.cpu_type on your machine.

I wonder as well. c3600 seems to have a PA-8600 CPUs (according to openpa.net) which is pcxw_
(according to  arch/parisc/kernel/hardware.c) and it should be fine to replace the fdc by nop
if the PDC_MODEL_IOPDIR_FDC capabilities bit is not set.
Carlo, can you run "cat /proc/cpuinfo" and paste the output here?

Helge

Re: HPPA problems with the PCI

[Carlo Pisani]

> Carlo, can you run "cat /proc/cpuinfo" and paste the output here?

# cat /proc/cpuinfo
processor       : 0
cpu family      : PA-RISC 2.0
cpu             : PA8600 (PCX-W+)
cpu MHz         : 552.000000
physical id     : 0
siblings        : 1
core id         : 0
capabilities    : os32 os64 nva_supported (0x03)
model           : 9000/785/C3600
model name      : Allegro W+
hversion        : 0x00005cf0
sversion        : 0x00000481
I-cache         : 512 KB
D-cache         : 1024 KB (WB, direct mapped)
ITLB entries    : 160
DTLB entries    : 160 - shared with ITLB
bogomips        : 1093.63
software id     : 2007455457
PCI failmode    : soft

Re: HPPA problems with the PCI

[John David Anglin]

On 2019-05-25 1:19 p.m., Carlo Pisani wrote:
>> Carlo, can you run "cat /proc/cpuinfo" and paste the output here?
> # cat /proc/cpuinfo
> processor       : 0
> cpu family      : PA-RISC 2.0
> cpu             : PA8600 (PCX-W+)
> cpu MHz         : 552.000000
> physical id     : 0
> siblings        : 1
> core id         : 0
> capabilities    : os32 os64 nva_supported (0x03)
> model           : 9000/785/C3600
> model name      : Allegro W+
> hversion        : 0x00005cf0
> sversion        : 0x00000481
> I-cache         : 512 KB
> D-cache         : 1024 KB (WB, direct mapped)
> ITLB entries    : 160
> DTLB entries    : 160 - shared with ITLB
> bogomips        : 1093.63
> software id     : 2007455457
> PCI failmode    : soft
I think the PDC is buggy on c3600.  The NP (iopdir_fdc) bit is set on my c3750:

dave@hiauly6:~/linux/D$ cat /proc/cpuinfo
processor       : 0
cpu family      : PA-RISC 2.0
cpu             : PA8700 (PCX-W2)
cpu MHz         : 875.000000
capabilities    : os32 os64 iopdir_fdc nva_supported (0x07)
model           : 9000/785/C3750
model name      : Allegro W2
hversion        : 0x00005dc0
sversion        : 0x00000481
I-cache         : 768 KB
D-cache         : 1536 KB (WB, direct mapped)
ITLB entries    : 240
DTLB entries    : 240 - shared with ITLB
bogomips        : 1737.72
software id     : 2011455108

Dave

-- 
John David Anglin  dave.anglin@bell.net

Re: HPPA problems with the PCI

[Sven Schnelle]

On Sat, May 25, 2019 at 01:50:01PM -0400, John David Anglin wrote:
> On 2019-05-25 1:19 p.m., Carlo Pisani wrote:
> >> Carlo, can you run "cat /proc/cpuinfo" and paste the output here?
> > # cat /proc/cpuinfo
> > processor       : 0
> > cpu family      : PA-RISC 2.0
> > cpu             : PA8600 (PCX-W+)
> > cpu MHz         : 552.000000
> > physical id     : 0
> > siblings        : 1
> > core id         : 0
> > capabilities    : os32 os64 nva_supported (0x03)
> > model           : 9000/785/C3600
> > model name      : Allegro W+
> > hversion        : 0x00005cf0
> > sversion        : 0x00000481
> > I-cache         : 512 KB
> > D-cache         : 1024 KB (WB, direct mapped)
> > ITLB entries    : 160
> > DTLB entries    : 160 - shared with ITLB
> > bogomips        : 1093.63
> > software id     : 2007455457
> > PCI failmode    : soft
> I think the PDC is buggy on c3600.  The NP (iopdir_fdc) bit is set on my c3750:
> [..] 

It's also set on my C3750, but not on my J5000 (PDC FW 5.0):

root@j5000:~# cat /proc/cpuinfo 
processor	: 0
cpu family	: PA-RISC 2.0
cpu		: PA8500 (PCX-W)
cpu MHz		: 440.000000
physical id	: 0
siblings	: 1
core id		: 0
capabilities	: os32 os64 nva_supported (0x03)
model		: 9000/785/J5000
model name	: Forte W 2-way
hversion	: 0x00005bd0
sversion	: 0x00000491
I-cache		: 512 KB
D-cache		: 1024 KB (WB, direct mapped)
ITLB entries	: 160
DTLB entries	: 160 - shared with ITLB
bogomips	: 875.72
software id	: 2006803539

processor	: 1
cpu family	: PA-RISC 2.0
cpu		: PA8500 (PCX-W)
cpu MHz		: 440.000000
physical id	: 1
siblings	: 1
core id		: 0
capabilities	: os32 os64 nva_supported (0x03)
model		: 9000/785/J5000
model name	: Forte W 2-way
hversion	: 0x00005bd0
sversion	: 0x00000491
I-cache		: 512 KB
D-cache		: 1024 KB (WB, direct mapped)
ITLB entries	: 160
DTLB entries	: 160 - shared with ITLB
bogomips	: 875.72
software id	: 2006803539

Sven

Re: HPPA problems with the PCI

[Carlo Pisani]

> It's also set on my C3750, but not on my J5000 (PDC FW 5.0):

maybe it's related to the firmware version?

Re: HPPA problems with the PCI

[Helge Deller]

On 25.05.19 20:13, Sven Schnelle wrote:
> On Sat, May 25, 2019 at 01:50:01PM -0400, John David Anglin wrote:
>> On 2019-05-25 1:19 p.m., Carlo Pisani wrote:
>>>> Carlo, can you run "cat /proc/cpuinfo" and paste the output here?
>>> # cat /proc/cpuinfo
>>> cpu             : PA8600 (PCX-W+)
>>> capabilities    : os32 os64 nva_supported (0x03)
>> I think the PDC is buggy on c3600.  The NP (iopdir_fdc) bit is set on my c3750:
>> cpu             : PA8700 (PCX-W2)
>> cpu MHz         : 875.000000
>> capabilities    : os32 os64 iopdir_fdc nva_supported (0x07)
>> model           : 9000/785/C3750
>
> It's also set on my C3750, but not on my J5000 (PDC FW 5.0):
>
> root@j5000:~# cat /proc/cpuinfo
> cpu		: PA8500 (PCX-W)
> capabilities	: os32 os64 nva_supported (0x03)
> model		: 9000/785/J5000
> model name	: Forte W 2-way

Carlo & Sven, do you run 32- or 64-bit kernel?
Maybe the 32-bit PDC does not return the NP (iopdir_fdc) while the 64-bit PDC does?
Otherwise we maybe should only trust this bit, if it's a "PA8700 (PCX-W2)" or higher CPU ?

Helge

Re: HPPA problems with the PCI

[Carlo Pisani]

> Carlo & Sven, do you run 32- or 64-bit kernel?

8Gbyte of ram, I am on 64-bit kernel.

> Maybe the 32-bit PDC does not return the NP (iopdir_fdc) while the 64-bit PDC does?
> Otherwise we maybe should only trust this bit, if it's a "PA8700 (PCX-W2)" or higher CPU ?

Good question. I do not know. But, maybe it's related to the firmware?
C3750 seems to have an updated one.

Re: HPPA problems with the PCI

[Helge Deller]

On 25.05.19 21:20, Carlo Pisani wrote:
>> Carlo & Sven, do you run 32- or 64-bit kernel?
>
> 8Gbyte of ram, I am on 64-bit kernel.

Run "uname -m" or "uname -a" and paste output here.
64bit kernel reports "parisc64", 32bit kernel says "parisc".

Same output via "lscpu", or "dmesg | grep 'has started'"
[    0.000000] The 64-bit Kernel has started...

>> Maybe the 32-bit PDC does not return the NP (iopdir_fdc) while the 64-bit PDC does?
>> Otherwise we maybe should only trust this bit, if it's a "PA8700 (PCX-W2)" or higher CPU ?
>
> Good question. I do not know. But, maybe it's related to the firmware?
> C3750 seems to have an updated one.

Firmware versions most likely differ in version between models...

Helge

Re: HPPA problems with the PCI

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 677 bytes --]

Carlo Pisani wrote:
> > Carlo & Sven, do you run 32- or 64-bit kernel?
> 
> 8Gbyte of ram, I am on 64-bit kernel.

What have you done with the 2GB that were in the machine before? And what 
should have happen to them?

> > Maybe the 32-bit PDC does not return the NP (iopdir_fdc) while the 64-bit
> > PDC does? Otherwise we maybe should only trust this bit, if it's a
> > "PA8700 (PCX-W2)" or higher CPU ?
> Good question. I do not know. But, maybe it's related to the firmware?
> C3750 seems to have an updated one.

I have a C3600 with only 1GB of RAM, and my /proc/cpuinfo is exactly the same. 
So at least what is in that output does not depend on the kernel bitsize.

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: HPPA problems with the PCI

[Carlo Pisani]

> do you run 32- or 64-bit kernel?

# uname -r
5.1.4-c3600-64bit-SMP-experimental-r049

# uname -m
parisc64

Re: HPPA problems with the PCI

[Sven Schnelle]

Hi Helge,

On Sat, May 25, 2019 at 09:09:04PM +0200, Helge Deller wrote:

> > It's also set on my C3750, but not on my J5000 (PDC FW 5.0):
> >
> > root@j5000:~# cat /proc/cpuinfo
> > cpu		: PA8500 (PCX-W)
> > capabilities	: os32 os64 nva_supported (0x03)
> > model		: 9000/785/J5000
> > model name	: Forte W 2-way
> 
> Carlo & Sven, do you run 32- or 64-bit kernel?
> Maybe the 32-bit PDC does not return the NP (iopdir_fdc) while the 64-bit PDC does?
> Otherwise we maybe should only trust this bit, if it's a "PA8700 (PCX-W2)" or higher CPU ?

This was on 32 bit, but i tested it on 64 bit now and the iopdir_fdc flag is
also not present. Can't say whether your >PA-8700 idea is good or bad, don't
know enough about PA-RISC CPUs.

Sven

Re: HPPA problems with the PCI

[John David Anglin]

On 2019-05-25 1:15 p.m., Helge Deller wrote:
> On 25.05.19 16:13, John David Anglin wrote:
>> On 2019-05-25 5:01 a.m., Carlo Pisani wrote:
>>> on  kernel 5.1, from the git repository, compiled with SMP
>>>
>>> dmesg | grep altern
>>> [    2.551002] alternatives: applied 156 out of 175 patches
>>>
>>> this patch has made my day!
>>> It seems the solution for the problem!
>>>
>>> the card is still working after 10 hours of burn-in test!
>>> --- a/arch/parisc/include/asm/cache.h
>>> +++ b/arch/parisc/include/asm/cache.h
>>> @@ -52,7 +52,6 @@ void parisc_setup_cache_timing(void);
>>>
>>>  #define asm_io_fdc(addr) asm volatile("fdc %%r0(%0)" \
>>>                         ALTERNATIVE(ALT_COND_NO_DCACHE, INSN_NOP) \
>>> -                       ALTERNATIVE(ALT_COND_NO_IOC_FDC, INSN_NOP) \
>>>                         : : "r" (addr) : "memory")
>>>  #define asm_io_sync()  asm volatile("sync" \
>>>                         ALTERNATIVE(ALT_COND_NO_DCACHE, INSN_NOP) \
>> The change isn't fully correct as the alternative should also have been removed from
>> the asm_io_sync() define.
>>
>> If the change actually fixes controller, then there must be a problem with the test in
>> arch/parisc/kernel/alternative.c that determines whether or not to patch out fdc instruction
>> on c3600.  See Helge's patch to fix alternative coding for PCX-U CPUs.  It would help to
>> print boot_cpu_data.pdc.capabilities and boot_cpu_data.cpu_type on your machine.
> I wonder as well. c3600 seems to have a PA-8600 CPUs (according to openpa.net) which is pcxw_
> (according to  arch/parisc/kernel/hardware.c) and it should be fine to replace the fdc by nop
> if the PDC_MODEL_IOPDIR_FDC capabilities bit is not set.
> Carlo, can you run "cat /proc/cpuinfo" and paste the output here?
I'm thinking that machines with Elroy PCI bridges may need flush but I don't know why.

Dave

-- 
John David Anglin  dave.anglin@bell.net

Re: HPPA problems with the PCI

[John David Anglin]

On 2019-05-25 5:01 a.m., Carlo Pisani wrote:
> on  kernel 5.1, from the git repository, compiled with SMP
>
> dmesg | grep altern
> [    2.551002] alternatives: applied 156 out of 175 patches
>
Prior to the patch which added ALT_COND_NO_IOC_FDC support, sba_iommu.c checked
the capability bit directly and set the variable ioc_needs_fdc.  This explains why earlier kernels
didn't work for Carlo (ioc_needs_fdc would need to be 1 to get flushes).

Carlo, would you confirm that Helge's "fix alternative coding for PCX-U CPUs" patch with "pcxw"
changed to "pcxw2" fixes the Syba SATA card?

Dave

-- 
John David Anglin  dave.anglin@bell.net

Re: HPPA problems with the PCI

[Sven Schnelle]

On Sat, May 25, 2019 at 11:01:27AM +0200, Carlo Pisani wrote:
> on  kernel 5.1, from the git repository, compiled with SMP
> 
> dmesg | grep altern
> [    2.551002] alternatives: applied 156 out of 175 patches
> 
> this patch has made my day!
> It seems the solution for the problem!
> 
> the card is still working after 10 hours of burn-in test!
> 
> --- a/arch/parisc/include/asm/cache.h
> +++ b/arch/parisc/include/asm/cache.h
> @@ -52,7 +52,6 @@ void parisc_setup_cache_timing(void);
> 
>  #define asm_io_fdc(addr) asm volatile("fdc %%r0(%0)" \
>                         ALTERNATIVE(ALT_COND_NO_DCACHE, INSN_NOP) \
> -                       ALTERNATIVE(ALT_COND_NO_IOC_FDC, INSN_NOP) \
>                         : : "r" (addr) : "memory")
>  #define asm_io_sync()  asm volatile("sync" \
>                         ALTERNATIVE(ALT_COND_NO_DCACHE, INSN_NOP) \

Just for clarification: This change was never meant to be a real fix - it
was just a change i made to figure out which alternative patch breaks things
on my C240.

Sven

Re: HPPA problems with the PCI

[John David Anglin]

Here is another question.  I see this comment in lba-pci.c:

/* FIXME: B2K/C3600 workaround is always use old method... */
        /* if (!LBA_SKIP_PROBE(d)) */ {
                /* original - Generate config cycle on broken elroy
                  with risk we will miss PCI bus errors. */
                *data = lba_rd_cfg(d, tok, pos, size);
                DBG_CFG("%s(%x+%2x) -> 0x%x (a)\n", __func__, tok, pos, *data);
                return 0;
        }

Which machines are affected by this problem (i.e., are more recent elroy and mercury machines
affected by this bug?

Dave
-- 

John David Anglin  dave.anglin@bell.net

Re: HPPA problems with the PCI

[Grant Grundler]

On Mon, May 27, 2019 at 10:43 AM John David Anglin <dave.anglin@bell.net> wrote:
>
> Here is another question.  I see this comment in lba-pci.c:
>
> /* FIXME: B2K/C3600 workaround is always use old method... */
>         /* if (!LBA_SKIP_PROBE(d)) */ {
>                 /* original - Generate config cycle on broken elroy
>                   with risk we will miss PCI bus errors. */
>                 *data = lba_rd_cfg(d, tok, pos, size);
>                 DBG_CFG("%s(%x+%2x) -> 0x%x (a)\n", __func__, tok, pos, *data);
>                 return 0;
>         }
>
> Which machines are affected by this problem (i.e., are more recent elroy and mercury machines
> affected by this bug?

I've been hoping someone else would answer since I don't know. Sorry. :(

cheers,
grant

Re: HPPA problems with the PCI

[Sven Schnelle]

Hi Carlo,

On Thu, May 23, 2019 at 02:16:39PM +0200, Carlo Pisani wrote:
> > Carlo, are you testing with UP or SMP kernel?
> 
> Kernel without SMP.
> I am recompiling with SMP just right now

I think you've seen that problem also with older Kernel versions, but i have
some trouble with my C240 (see my other post), so it might be worth a try to
take the latest kernel and disabling alternatives patching (add no-alternatives
to the kernel command line)

To check that alternative patching is disabled, check that there's no message like
this in your dmesg output:

[   33.419615] alternatives: applied 158 out of 183 patches

Thanks for testing Linux on PA-RISC :-)

Regards
Sven