qla1280 driver for qlogic-1040 on alpha

qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

Hi,


I've been running linux on alpha (alphaserver es40) for a while, using
a qlogic-1040 scsi controller. A few weeks ago I added more RAM to the
es40, but as soon as I got above 2GB RAM I started seeing file system
corruptions on the drive attached to the qlogic controller. I
re-compiled the driver to force DMA_BIT_MASK of 32 bits and everything
was fine again. I believe that on alpha the
CONFIG_ARCH_DMA_ADDR_T_64BIT flag gets set in the kernel config, which
will enable 64-bit support in the qla1280 driver. This works as long
as there is less than 2GB RAM in the system (which is the case for my
other alpha hardware). The nvram flag "enable_64bit_addressing" on the
qlogic board is not checked nor set by the driver. What is the best
way of using the qla1280 driver? I might want
CONFIG_ARCH_DMA_ADDR_T_64BIT still enabled for other hardware on my
es40.
I've not yet tested the card on non-alpha hardware so I don't know for
sure if this is platform specific for qlogic on alpha.

Regards

Magnus Lindholm

Re: qla1280 driver for qlogic-1040 on alpha

[Martin K. Petersen]

Hi Magnus!

> I've been running linux on alpha (alphaserver es40) for a while, using
> a qlogic-1040 scsi controller. A few weeks ago I added more RAM to the
> es40, but as soon as I got above 2GB RAM I started seeing file system
> corruptions on the drive attached to the qlogic controller.

The qla1280 driver has been used extensively on 64-bit platforms.

Is your isp1040 original to the ES40? My Alphas used 53c8xx series
controllers if I remember correctly. And with the ES40 being fairly
recent (21264), I would have thought it would have used a slightly more
modern controller than a 1040.

> The nvram flag "enable_64bit_addressing" on the qlogic board is not
> checked nor set by the driver.

That would be a good place to start. Maybe if you could dump the NVRAM
contents and validate if that is set by the 1040 firmware? I'm afraid I
don't have a databook. But the 1040 was current right around the time
the industry transitioned from 32 to 64-bit so it could very well be
broken.

If you can establish whether that flag is unset on your controller,
we could use that as a heuristic for configuring DMA.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

Hi,


As you can see below, the driver  says "enable 64bit addressing=0"
when dumping nvram settings. The qlogic-1040 card is only a 32-bit
card plugged into a 64-bit slot. I guess the card is not original to
the ES40, I have other cards that I can use, i.e a  53c8xx is also
present in the system. Would it make sense to have the driver honor
the nvram flag for 64-bit addressing, at least for the 1040 cards? I'm
thinking I should give it a go... btw. the card runs fine even when
the driver is compiled in 64bit mode, just as long as I stay below 2GB
ram or when explicitly setting the DMA_BIT_MASK when building the
driver. (I guess most systems around when the qla1040 was hot stuff
had less than 2GB ram).

This is the output I get from running the driver with debug enabled:

[   19.109365] qla1280: QLA1040 found on PCI bus 2, dev 4
[   19.110341] scsi(0): 64 Bit PCI Addressing Enabled
[   19.110341] Configure PCI space for adapter...
[   19.111318] scsi(2): Verifying chip
[   19.111318] qla1280_chip_diag: Checking mailboxes of chip
[   19.111318] Loading firmware: qlogic/1040.bin
[   19.749013] qla1280_start_firmware: Verifying checksum of loaded RISC code.
[   19.757802] qla1280_start_firmware: start firmware running.
[   19.760732] scsi(2): Configure NVRAM parameters
[   19.760732] Using defaults for NVRAM:
[   19.760732] qla1280 : initiator scsi id bus[0]=7
[   19.760732] qla1280 : initiator scsi id bus[1]=7
[   19.760732] qla1280 : bus reset delay[0]=3
[   19.760732] qla1280 : bus reset delay[1]=3
[   19.760732] qla1280 : retry count[0]=0
[   19.760732] qla1280 : retry delay[0]=1
[   19.760732] qla1280 : retry count[1]=0
[   19.760732] qla1280 : retry delay[1]=1
[   19.760732] qla1280 : async data setup time[0]=6
[   19.760732] qla1280 : async data setup time[1]=6
[   19.760732] qla1280 : req/ack active negation[0]=1
[   19.760732] qla1280 : req/ack active negation[1]=1
[   19.760732] qla1280 : data line active negation[0]=1
[   19.760732] qla1280 : data line active negation[1]=1
[   19.760732] qla1280 : disable loading risc code=0
[   19.760732] qla1280 : enable 64bit addressing=0
[   19.760732] qla1280 : selection timeout limit[0]=250
[   19.760732] qla1280 : selection timeout limit[1]=250
[   19.760732] qla1280 : max queue depth[0]=32
[   19.760732] qla1280 : max queue depth[1]=32
[   19.772450] scsi(2:0): Resetting SCSI BUS
[   22.812488] scsi host2: QLogic QLA1040 PCI to SCSI Host Adapter
                      Firmware version:  7.65.06, Driver version 3.27.2


On Fri, Oct 25, 2024 at 10:00 PM Martin K. Petersen
<martin.petersen@oracle.com> wrote:
>
>
> Hi Magnus!
>
> > I've been running linux on alpha (alphaserver es40) for a while, using
> > a qlogic-1040 scsi controller. A few weeks ago I added more RAM to the
> > es40, but as soon as I got above 2GB RAM I started seeing file system
> > corruptions on the drive attached to the qlogic controller.
>
> The qla1280 driver has been used extensively on 64-bit platforms.
>
> Is your isp1040 original to the ES40? My Alphas used 53c8xx series
> controllers if I remember correctly. And with the ES40 being fairly
> recent (21264), I would have thought it would have used a slightly more
> modern controller than a 1040.
>
> > The nvram flag "enable_64bit_addressing" on the qlogic board is not
> > checked nor set by the driver.
>
> That would be a good place to start. Maybe if you could dump the NVRAM
> contents and validate if that is set by the 1040 firmware? I'm afraid I
> don't have a databook. But the 1040 was current right around the time
> the industry transitioned from 32 to 64-bit so it could very well be
> broken.
>
> If you can establish whether that flag is unset on your controller,
> we could use that as a heuristic for configuring DMA.
>
> --
> Martin K. Petersen      Oracle Linux Engineering

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

Hi,

I've made some changes to the qla1280 driver, the changes include
things like checking if the card is in a 64-bit slot and setting
DMA_BIT_MASK and enable_64bit_addressing accordingly. Also in the
driver information string, it now shows hardware revision on 1040
chips as well as printing info on its PCI slot (32 or 64 bit). I've
tested it with a ISP1040B card and a ISP1080 and it seems to work
fine. This may be of interest to others still running legacy qlogic
SCSI-controllers?

On Fri, Oct 25, 2024 at 10:48 PM Magnus Lindholm <linmag7@gmail.com> wrote:
>
> Hi,
>
>
> As you can see below, the driver  says "enable 64bit addressing=0"
> when dumping nvram settings. The qlogic-1040 card is only a 32-bit
> card plugged into a 64-bit slot. I guess the card is not original to
> the ES40, I have other cards that I can use, i.e a  53c8xx is also
> present in the system. Would it make sense to have the driver honor
> the nvram flag for 64-bit addressing, at least for the 1040 cards? I'm
> thinking I should give it a go... btw. the card runs fine even when
> the driver is compiled in 64bit mode, just as long as I stay below 2GB
> ram or when explicitly setting the DMA_BIT_MASK when building the
> driver. (I guess most systems around when the qla1040 was hot stuff
> had less than 2GB ram).
>
> This is the output I get from running the driver with debug enabled:
>
> [   19.109365] qla1280: QLA1040 found on PCI bus 2, dev 4
> [   19.110341] scsi(0): 64 Bit PCI Addressing Enabled
> [   19.110341] Configure PCI space for adapter...
> [   19.111318] scsi(2): Verifying chip
> [   19.111318] qla1280_chip_diag: Checking mailboxes of chip
> [   19.111318] Loading firmware: qlogic/1040.bin
> [   19.749013] qla1280_start_firmware: Verifying checksum of loaded RISC code.
> [   19.757802] qla1280_start_firmware: start firmware running.
> [   19.760732] scsi(2): Configure NVRAM parameters
> [   19.760732] Using defaults for NVRAM:
> [   19.760732] qla1280 : initiator scsi id bus[0]=7
> [   19.760732] qla1280 : initiator scsi id bus[1]=7
> [   19.760732] qla1280 : bus reset delay[0]=3
> [   19.760732] qla1280 : bus reset delay[1]=3
> [   19.760732] qla1280 : retry count[0]=0
> [   19.760732] qla1280 : retry delay[0]=1
> [   19.760732] qla1280 : retry count[1]=0
> [   19.760732] qla1280 : retry delay[1]=1
> [   19.760732] qla1280 : async data setup time[0]=6
> [   19.760732] qla1280 : async data setup time[1]=6
> [   19.760732] qla1280 : req/ack active negation[0]=1
> [   19.760732] qla1280 : req/ack active negation[1]=1
> [   19.760732] qla1280 : data line active negation[0]=1
> [   19.760732] qla1280 : data line active negation[1]=1
> [   19.760732] qla1280 : disable loading risc code=0
> [   19.760732] qla1280 : enable 64bit addressing=0
> [   19.760732] qla1280 : selection timeout limit[0]=250
> [   19.760732] qla1280 : selection timeout limit[1]=250
> [   19.760732] qla1280 : max queue depth[0]=32
> [   19.760732] qla1280 : max queue depth[1]=32
> [   19.772450] scsi(2:0): Resetting SCSI BUS
> [   22.812488] scsi host2: QLogic QLA1040 PCI to SCSI Host Adapter
>                       Firmware version:  7.65.06, Driver version 3.27.2
>
>
> On Fri, Oct 25, 2024 at 10:00 PM Martin K. Petersen
> <martin.petersen@oracle.com> wrote:
> >
> >
> > Hi Magnus!
> >
> > > I've been running linux on alpha (alphaserver es40) for a while, using
> > > a qlogic-1040 scsi controller. A few weeks ago I added more RAM to the
> > > es40, but as soon as I got above 2GB RAM I started seeing file system
> > > corruptions on the drive attached to the qlogic controller.
> >
> > The qla1280 driver has been used extensively on 64-bit platforms.
> >
> > Is your isp1040 original to the ES40? My Alphas used 53c8xx series
> > controllers if I remember correctly. And with the ES40 being fairly
> > recent (21264), I would have thought it would have used a slightly more
> > modern controller than a 1040.
> >
> > > The nvram flag "enable_64bit_addressing" on the qlogic board is not
> > > checked nor set by the driver.
> >
> > That would be a good place to start. Maybe if you could dump the NVRAM
> > contents and validate if that is set by the 1040 firmware? I'm afraid I
> > don't have a databook. But the 1040 was current right around the time
> > the industry transitioned from 32 to 64-bit so it could very well be
> > broken.
> >
> > If you can establish whether that flag is unset on your controller,
> > we could use that as a heuristic for configuring DMA.
> >
> > --
> > Martin K. Petersen      Oracle Linux Engineering

Re: qla1280 driver for qlogic-1040 on alpha

[Martin K. Petersen]

Magnus,

> I've made some changes to the qla1280 driver, the changes include
> things like checking if the card is in a 64-bit slot and setting
> DMA_BIT_MASK and enable_64bit_addressing accordingly. Also in the
> driver information string, it now shows hardware revision on 1040
> chips as well as printing info on its PCI slot (32 or 64 bit). I've
> tested it with a ISP1040B card and a ISP1080 and it seems to work
> fine. This may be of interest to others still running legacy qlogic
> SCSI-controllers?

It would be great for the driver to have a solid heuristic for running
the older ISP cards in 32-bit mode.

You don't happen to have a qla1280, do you? I'm afraid I don't have one
anymore and it's the model that occasionally pops up in bug reports.
Would be nice to validate your changes against that ASIC.

In the meantime I'll see if I can locate the qla12160 I believe I still
have.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

Hi,

I have  qlogic 1020,1040 and 1080 cards and actually a 12160 on the
way from ebay. Those are the cards I'll be able to test for now. I'll
put together a patch with the changes I've made to qla1280.c and post
it to the list. My ES40 will not run the qla1040 without getting
filesystem corruption, when I put more than 2GB ram in it. I guess
qla1040 and large memory 64-bit systems were never a frequent
combination?

On Tue, Oct 29, 2024 at 3:19 AM Martin K. Petersen
<martin.petersen@oracle.com> wrote:
>
>
> Magnus,
>
> > I've made some changes to the qla1280 driver, the changes include
> > things like checking if the card is in a 64-bit slot and setting
> > DMA_BIT_MASK and enable_64bit_addressing accordingly. Also in the
> > driver information string, it now shows hardware revision on 1040
> > chips as well as printing info on its PCI slot (32 or 64 bit). I've
> > tested it with a ISP1040B card and a ISP1080 and it seems to work
> > fine. This may be of interest to others still running legacy qlogic
> > SCSI-controllers?
>
> It would be great for the driver to have a solid heuristic for running
> the older ISP cards in 32-bit mode.
>
> You don't happen to have a qla1280, do you? I'm afraid I don't have one
> anymore and it's the model that occasionally pops up in bug reports.
> Would be nice to validate your changes against that ASIC.
>
> In the meantime I'll see if I can locate the qla12160 I believe I still
> have.
>
> --
> Martin K. Petersen      Oracle Linux Engineering

Re: qla1280 driver for qlogic-1040 on alpha

[Maciej W. Rozycki]

On Mon, 28 Oct 2024, Martin K. Petersen wrote:

> > I've made some changes to the qla1280 driver, the changes include
> > things like checking if the card is in a 64-bit slot and setting
> > DMA_BIT_MASK and enable_64bit_addressing accordingly. Also in the
> > driver information string, it now shows hardware revision on 1040
> > chips as well as printing info on its PCI slot (32 or 64 bit). I've
> > tested it with a ISP1040B card and a ISP1080 and it seems to work
> > fine. This may be of interest to others still running legacy qlogic
> > SCSI-controllers?
> 
> It would be great for the driver to have a solid heuristic for running
> the older ISP cards in 32-bit mode.

 The use of the 32-bit form factor does not preclude 64-bit addressing as 
long as the PCI bus logic supports the DAC command, both in the system's 
relevant host bridge and the device concerned.  Is there documentation 
available for the ISP devices that would provide such information or is 
that all we have is trial and error?

  Maciej

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

Hi,

I've found a few datasheets from qlogic, on ISP1040x, ISP1080 and
ISP10160. The ISP1040 doesn't mention any 64-bit capabilities in the
chip.
This first appears on the ISP1080 and higher. This information is
consistent with what I find in trial and error tests on my ES40.
Once bus addresses go beyond 32-bits in allocating space for
S/G-buffers, I start seeing file system corruption with the
ISP1040-cards.
Thats said, according to the datasheet for the ISP1040 the chip
conforms to the PCI 2.1 specifications, which has "64-Bit Bus
Extension Pins" as optional
for DAC commands and REQ64# and ACK64#. This capability is never
mentioned in the datasheet and there are no pins for REQ64 or ACK64 on
the chip,
they appear in later ships (1080 and onwards). In the qla1280.c driver
there is an enable_64_bit_addressing in NVRAM for 1040 cards, but when
I look at
the NetBSD driver for the same card, this feature is only available in
ISP1080 cards and higher. So there seems to be some inconsistency
here.

Magnus

Re: qla1280 driver for qlogic-1040 on alpha

[Thomas Bogendoerfer]

On Wed, 30 Oct 2024 08:52:44 +0100
Magnus Lindholm <linmag7@gmail.com> wrote:

> I've found a few datasheets from qlogic, on ISP1040x, ISP1080 and
> ISP10160. The ISP1040 doesn't mention any 64-bit capabilities in the
> chip.

the datasheet is wrong. QL1040 supports 64bit addresses via DAC,
otherwise it wouldn't work on SGI Origin and SGI Octane systems.
Your patch breaks them (verified on a Octane).

So your problem might not be in the scsi driver, but a PCI setup
problem for the bus of the 32bit PCI slot in your system. If this
bus doesn't support DAC it IMHO shoulnd't allow 64bit PCI addresses.

Thomas.

-- 
SUSE Software Solutions Germany GmbH
HRB 36809 (AG Nürnberg)
Geschäftsführer: Ivo Totev, Andrew McDonald, Werner Knoblich

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

Hi,

Thanks for your reply, I guess this suggests that this is an
alpha-platform specific problem then? I can run the QL1040 card fine
with the original qla1280.c driver ( in a 64-bit slot on the ES40 )
just as long as I don't put more than 2GB ram in the system, this is
when I start seeing the issues with file system corruption. As soon as
S/G buffers are allocated on addresses above 32-bit, the QL1040 cannot
deal with this, unless enforcing a DMA_BIT_MASK(32). The same card
runs fine under Tru64 UNIX on the same system, however, I don't know
how the Tru64 driver sets the DMA mask, if it enables full 64-bit
addressing or not.

Magnus

> the datasheet is wrong. QL1040 supports 64bit addresses via DAC,
> otherwise it wouldn't work on SGI Origin and SGI Octane systems.
> Your patch breaks them (verified on a Octane).
>
> So your problem might not be in the scsi driver, but a PCI setup
> problem for the bus of the 32bit PCI slot in your system. If this
> bus doesn't support DAC it IMHO shoulnd't allow 64bit PCI addresses.
>
> Thomas.
>
> --
> SUSE Software Solutions Germany GmbH
> HRB 36809 (AG Nürnberg)
> Geschäftsführer: Ivo Totev, Andrew McDonald, Werner Knoblich

Re: qla1280 driver for qlogic-1040 on alpha

[Maciej W. Rozycki]

On Wed, 30 Oct 2024, Magnus Lindholm wrote:

> Thanks for your reply, I guess this suggests that this is an
> alpha-platform specific problem then? I can run the QL1040 card fine
> with the original qla1280.c driver ( in a 64-bit slot on the ES40 )
> just as long as I don't put more than 2GB ram in the system, this is
> when I start seeing the issues with file system corruption. As soon as
> S/G buffers are allocated on addresses above 32-bit, the QL1040 cannot
> deal with this, unless enforcing a DMA_BIT_MASK(32). The same card
> runs fine under Tru64 UNIX on the same system, however, I don't know
> how the Tru64 driver sets the DMA mask, if it enables full 64-bit
> addressing or not.

 I find this interesting.  Documentation for the Tsunami chipset indicates 
DAC support[1]:

"In case of a PCI dual-address cycle command, the high-order PCI address 
bits <63:40> are compared to the constant value 0x0000_01 (that is, bit 
<40> = 1; all other bits = 0).  If these bits match, a monster window hit 
has occurred and the low-order PCI address bits <34:0> are used unchanged 
as the system address bits <34:0>.  PCI address bits <39:35> are ignored. 
The high-order 32 PCI address bits are available on b_ad<31:0> in the 
second cycle of a DAC, and also on b_ad<63:32> in the first cycle of a DAC 
if b_req64_l is asserted."

and I can see it handled in arch/alpha/kernel/pci_iommu.c; allocations can 
be handed out with the TSUNAMI_DAC_OFFSET bit set.  You might be able to 
see if it triggers by defining DEBUG_ALLOC to 2 and checking messages from 
DMA mappings in the kernel log.

 I did a little research however and discovered that the DAC capability is 
documented in the ISP1040C datasheet, but not in the ISP1040B one.  So it 
seems to me like it's the matter of the chip revision.  I've skimmed over 
the driver and as far as I can tell there's everything supplied there to 
get this sorted now.

References:

[1] "Tsunami/Typhoon 21272 Chipset Hardware Reference Manual", Revision 
    4.0, Compaq Computer Corporation, Order Number: DS-0025A-TE, 21 
    October 1999, Section 10.1.4.4 "Monster Window DMA Address 
    Translation", p. 10-13

  Maciej

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

HI,

I don't have a QLA1040 revC to test with, but I've put together a much
smaller patch which just limits the DMA_BIT_MASK to 32-bit on
1020/1040 cards this should at least not break anything while fixing
the most urgent problems with file system corruption on some
combination of hardware/memory. Thanks for the pointers to tsunami
chipsets though, I'll try to enable some debugging to see what's going
on when I hit this problem with the qlogic card.
I'll clean up the chip revision reporting code to see if I can improve
things there, this will be as a separate patch then.

I have one question regarding the hardware revision on 1040 chips,
according to qla1280.h we have this:

#define ISP_CFG0_HWMSK   0x000f /* Hardware revision mask */
#define ISP_CFG0_1020    BIT_0  /* ISP1020 */
#define ISP_CFG0_1020A   BIT_1  /* ISP1020A */
#define ISP_CFG0_1040    BIT_2  /* ISP1040 */
#define ISP_CFG0_1040A   BIT_3  /* ISP1040A */
#define ISP_CFG0_1040B   BIT_4  /* ISP1040B */
#define ISP_CFG0_1040C   BIT_5  /* ISP1040C */


But when I examine the register it looks more like:

#define ISP_CFG0_HWMSK   0x000f /* Hardware revision mask */
#define ISP_CFG0_1020     1  /* ISP1020 *
#define ISP_CFG0_1020A   2  /* ISP1020A */
#define ISP_CFG0_1040     3  /* ISP1040 */
#define ISP_CFG0_1040A   4  /* ISP1040A */
#define ISP_CFG0_1040B   5  /* ISP1040B */
#define ISP_CFG0_1040C   6  /* ISP1040C */

Which is consistent with how NetBSD does things in their ISP driver.
Also, in this case the HWMSK actually works for filtering out the
hardware revision part of the CFG0 register, since it would actually
require a 6-bit mask to match the current definitions in qla1280.c,
right?


Magnus

On Thu, Oct 31, 2024 at 8:37 AM Maciej W. Rozycki <macro@orcam.me.uk> wrote:

>  I find this interesting.  Documentation for the Tsunami chipset indicates
> DAC support[1]:
>
> "In case of a PCI dual-address cycle command, the high-order PCI address
> bits <63:40> are compared to the constant value 0x0000_01 (that is, bit
> <40> = 1; all other bits = 0).  If these bits match, a monster window hit
> has occurred and the low-order PCI address bits <34:0> are used unchanged
> as the system address bits <34:0>.  PCI address bits <39:35> are ignored.
> The high-order 32 PCI address bits are available on b_ad<31:0> in the
> second cycle of a DAC, and also on b_ad<63:32> in the first cycle of a DAC
> if b_req64_l is asserted."
>
> and I can see it handled in arch/alpha/kernel/pci_iommu.c; allocations can
> be handed out with the TSUNAMI_DAC_OFFSET bit set.  You might be able to
> see if it triggers by defining DEBUG_ALLOC to 2 and checking messages from
> DMA mappings in the kernel log.
>
>  I did a little research however and discovered that the DAC capability is
> documented in the ISP1040C datasheet, but not in the ISP1040B one.  So it
> seems to me like it's the matter of the chip revision.  I've skimmed over
> the driver and as far as I can tell there's everything supplied there to
> get this sorted now.
>
> References:
>
> [1] "Tsunami/Typhoon 21272 Chipset Hardware Reference Manual", Revision
>     4.0, Compaq Computer Corporation, Order Number: DS-0025A-TE, 21
>     October 1999, Section 10.1.4.4 "Monster Window DMA Address
>     Translation", p. 10-13
>
>   Maciej
>

Re: qla1280 driver for qlogic-1040 on alpha

[Maciej W. Rozycki]

On Thu, 31 Oct 2024, Magnus Lindholm wrote:

> I don't have a QLA1040 revC to test with, but I've put together a much
> smaller patch which just limits the DMA_BIT_MASK to 32-bit on
> 1020/1040 cards this should at least not break anything while fixing
> the most urgent problems with file system corruption on some
> combination of hardware/memory. Thanks for the pointers to tsunami
> chipsets though, I'll try to enable some debugging to see what's going
> on when I hit this problem with the qlogic card.

 I think Thomas will be unhappy about disabling DAC completely for the 
1040: as I infer from his response it is indeed required for his Octane to 
operate correctly, and which also presumably implies he does have revision 
C of the chip to verify your fix with.  Thomas?

> I'll clean up the chip revision reporting code to see if I can improve
> things there, this will be as a separate patch then.

 Great!

> I have one question regarding the hardware revision on 1040 chips,
> according to qla1280.h we have this:
> 
> #define ISP_CFG0_HWMSK   0x000f /* Hardware revision mask */
> #define ISP_CFG0_1020    BIT_0  /* ISP1020 */
> #define ISP_CFG0_1020A   BIT_1  /* ISP1020A */
> #define ISP_CFG0_1040    BIT_2  /* ISP1040 */
> #define ISP_CFG0_1040A   BIT_3  /* ISP1040A */
> #define ISP_CFG0_1040B   BIT_4  /* ISP1040B */
> #define ISP_CFG0_1040C   BIT_5  /* ISP1040C */
> 
> 
> But when I examine the register it looks more like:
> 
> #define ISP_CFG0_HWMSK   0x000f /* Hardware revision mask */
> #define ISP_CFG0_1020     1  /* ISP1020 *
> #define ISP_CFG0_1020A   2  /* ISP1020A */
> #define ISP_CFG0_1040     3  /* ISP1040 */
> #define ISP_CFG0_1040A   4  /* ISP1040A */
> #define ISP_CFG0_1040B   5  /* ISP1040B */
> #define ISP_CFG0_1040C   6  /* ISP1040C */
> 
> Which is consistent with how NetBSD does things in their ISP driver.
> Also, in this case the HWMSK actually works for filtering out the
> hardware revision part of the CFG0 register, since it would actually
> require a 6-bit mask to match the current definitions in qla1280.c,
> right?

 Well spotted!  This predates kernel.org git history, but I was able to 
track the origin down with a copy of the LMO git tree, and it has always 
been like this since the introduction of these macros in 2.6.9 with 
<https://lore.kernel.org/r/20040606125728.GB31063@lst.de/>.

 This also means that the ISP_CFG0_1040A check also added in 2.6.9 with 
<https://lore.kernel.org/r/20040606125825.GE31063@lst.de/> will never 
match, possibly meaning that this code wasn't actually ever verified with 
affected 1040A hardware.  This might also explain why a later change made 
with commit 0888f4c33128 ("[SCSI] qla1280: don't use bitfields for 
hardware access in isp_config") went unnoticed that changed the semantics 
of the workaround from keeping bursts unconditionally disabled with the 
1040A to making them enabled in the absence of NVRAM.

 NB comments for the FIFO threshold surely are suspicious too.

 Christoph can you please have a look into it?  It seems like something 
you ought to be quite familiar with if not for the passage of time.

  Maciej

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

On Thu, Oct 31, 2024 at 6:31 PM Maciej W. Rozycki <macro@orcam.me.uk> wrote:
>
> On Thu, 31 Oct 2024, Magnus Lindholm wrote:
>
> > I don't have a QLA1040 revC to test with, but I've put together a much
> > smaller patch which just limits the DMA_BIT_MASK to 32-bit on
> > 1020/1040 cards this should at least not break anything while fixing
> > the most urgent problems with file system corruption on some
> > combination of hardware/memory. Thanks for the pointers to tsunami
> > chipsets though, I'll try to enable some debugging to see what's going
> > on when I hit this problem with the qlogic card.
>
>  I think Thomas will be unhappy about disabling DAC completely for the
> 1040: as I infer from his response it is indeed required for his Octane to
> operate correctly, and which also presumably implies he does have revision
> C of the chip to verify your fix with.  Thomas?


From the Tsunami paper:

"The DMA monster window is enabled by PCTL<MWIN>. If enabled, this
window lies from 100.0000.0000 to
100.FFFF.FFFF, which requires a dual-address cycle (DAC) access from
the PCI bus. This window maps to system memory as defined in Section
10.1.4"

When I run the qla1280.c driver with some debugging enabled I see this
output after a while:

es40 kernel: S/G Segment Cont. phys_addr=100 7fff8000, len=0x10000
es40 kernel: S/G Segment Cont. phys_addr=100 80008000, len=0x10000
es40 kernel: S/G Segment Cont. phys_addr=100 8011e000, len=0x10000

Right after the above messages in log files I see failed read/writes
on the drive attached to the qla1040B. So as soon as I hit the
"monster window" in DMA-addressing my files get messed up.  I guess
this never happens if I don't have enough memory in the system, or if
I set the DMA_BIT_MASK. I'll try to enable some more debug logs from
DMA and IOMMU stuff.



> > I'll clean up the chip revision reporting code to see if I can improve
> > things there, this will be as a separate patch then.
>
>  Great!
>
> > I have one question regarding the hardware revision on 1040 chips,
> > according to qla1280.h we have this:
> >
> > #define ISP_CFG0_HWMSK   0x000f /* Hardware revision mask */
> > #define ISP_CFG0_1020    BIT_0  /* ISP1020 */
> > #define ISP_CFG0_1020A   BIT_1  /* ISP1020A */
> > #define ISP_CFG0_1040    BIT_2  /* ISP1040 */
> > #define ISP_CFG0_1040A   BIT_3  /* ISP1040A */
> > #define ISP_CFG0_1040B   BIT_4  /* ISP1040B */
> > #define ISP_CFG0_1040C   BIT_5  /* ISP1040C */
> >
> >
> > But when I examine the register it looks more like:
> >
> > #define ISP_CFG0_HWMSK   0x000f /* Hardware revision mask */
> > #define ISP_CFG0_1020     1  /* ISP1020 *
> > #define ISP_CFG0_1020A   2  /* ISP1020A */
> > #define ISP_CFG0_1040     3  /* ISP1040 */
> > #define ISP_CFG0_1040A   4  /* ISP1040A */
> > #define ISP_CFG0_1040B   5  /* ISP1040B */
> > #define ISP_CFG0_1040C   6  /* ISP1040C */
> >
> > Which is consistent with how NetBSD does things in their ISP driver.
> > Also, in this case the HWMSK actually works for filtering out the
> > hardware revision part of the CFG0 register, since it would actually
> > require a 6-bit mask to match the current definitions in qla1280.c,
> > right?
>

Thanks! I guess changing these definitions in the header file should
be safe, I don't think they are really used anywhere in the code as of
now. Would make a patch for reporting on the hardware revisions of
1040/1020 more consistent with the headers.


>  Well spotted!  This predates kernel.org git history, but I was able to
> track the origin down with a copy of the LMO git tree, and it has always
> been like this since the introduction of these macros in 2.6.9 with
> <https://lore.kernel.org/r/20040606125728.GB31063@lst.de/>.
>
>  This also means that the ISP_CFG0_1040A check also added in 2.6.9 with
> <https://lore.kernel.org/r/20040606125825.GE31063@lst.de/> will never
> match, possibly meaning that this code wasn't actually ever verified with
> affected 1040A hardware.  This might also explain why a later change made
> with commit 0888f4c33128 ("[SCSI] qla1280: don't use bitfields for
> hardware access in isp_config") went unnoticed that changed the semantics
> of the workaround from keeping bursts unconditionally disabled with the
> 1040A to making them enabled in the absence of NVRAM.
>
>  NB comments for the FIFO threshold surely are suspicious too.
>
>  Christoph can you please have a look into it?  It seems like something
> you ought to be quite familiar with if not for the passage of time.
>
>   Maciej
>

Re: qla1280 driver for qlogic-1040 on alpha

[Maciej W. Rozycki]

On Thu, 31 Oct 2024, Magnus Lindholm wrote:

> >  I think Thomas will be unhappy about disabling DAC completely for the
> > 1040: as I infer from his response it is indeed required for his Octane to
> > operate correctly, and which also presumably implies he does have revision
> > C of the chip to verify your fix with.  Thomas?
> 
> 
> >From the Tsunami paper:
> 
> "The DMA monster window is enabled by PCTL<MWIN>. If enabled, this
> window lies from 100.0000.0000 to
> 100.FFFF.FFFF, which requires a dual-address cycle (DAC) access from
> the PCI bus. This window maps to system memory as defined in Section
> 10.1.4"
> 
> When I run the qla1280.c driver with some debugging enabled I see this
> output after a while:
> 
> es40 kernel: S/G Segment Cont. phys_addr=100 7fff8000, len=0x10000
> es40 kernel: S/G Segment Cont. phys_addr=100 80008000, len=0x10000
> es40 kernel: S/G Segment Cont. phys_addr=100 8011e000, len=0x10000
> 
> Right after the above messages in log files I see failed read/writes
> on the drive attached to the qla1040B. So as soon as I hit the
> "monster window" in DMA-addressing my files get messed up.  I guess
> this never happens if I don't have enough memory in the system, or if
> I set the DMA_BIT_MASK. I'll try to enable some more debug logs from
> DMA and IOMMU stuff.

 So it is as I expected.  Please do coordinate with Thomas and implement 
the DMA mask on a per-revision basis as I suggested.

> > > Which is consistent with how NetBSD does things in their ISP driver.
> > > Also, in this case the HWMSK actually works for filtering out the
> > > hardware revision part of the CFG0 register, since it would actually
> > > require a 6-bit mask to match the current definitions in qla1280.c,
> > > right?
> >
> 
> Thanks! I guess changing these definitions in the header file should
> be safe, I don't think they are really used anywhere in the code as of
> now. Would make a patch for reporting on the hardware revisions of
> 1040/1020 more consistent with the headers.

 Well ISP_CFG0_1040A does get used as I also noted.  I do hope Christoph 
chimes in on the erratum concerned.

  Maciej

Re: qla1280 driver for qlogic-1040 on alpha

[Christoph Hellwig]

On Thu, Oct 31, 2024 at 05:30:31PM +0000, Maciej W. Rozycki wrote:
>  This also means that the ISP_CFG0_1040A check also added in 2.6.9 with 
> <https://lore.kernel.org/r/20040606125825.GE31063@lst.de/> will never 
> match, possibly meaning that this code wasn't actually ever verified with 
> affected 1040A hardware.  This might also explain why a later change made 
> with commit 0888f4c33128 ("[SCSI] qla1280: don't use bitfields for 
> hardware access in isp_config") went unnoticed that changed the semantics 
> of the workaround from keeping bursts unconditionally disabled with the 
> 1040A to making them enabled in the absence of NVRAM.
> 
>  NB comments for the FIFO threshold surely are suspicious too.
> 
>  Christoph can you please have a look into it?  It seems like something 
> you ought to be quite familiar with if not for the passage of time.

Somewhat surprisingly I don't remember that details of a drive by
cleanup 20 years ago :)

So whatever fixes you have based on other implementations are probably
correct.

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

Hi

I'm making another attempt at fixing the qla1280.c driver for ISP1040x
on Alpha, while trying not to break anything on other platforms, like
IP-30/MIPS. This time I'm using dma_get_required_mask(). Is my
understanding that this function should provide the minimum required
mask for the platform, assuming this works it should return something
greater than 32-bits on IP-30/MIPS. From what I can tell by looking at
the kernel source it should return something like a 64-bit MASK for
the sgi/octane but I'm not in the possession of such a system so I'm
unable to verify this.
Maybe Thomas can test the new patch? When I examine other scsi drivers
it seems like most of them select bitmasks that are 32-bit on alpha
systems. Any bitmask below 40-bits will avoid hitting the "monster
window" which is when I see trouble on alpha. Still a much larger mask
is required on SGI/Octane, my hopes are that dma_get_required_mask()
can assist in sorting this out?


Magnus

On Mon, Nov 4, 2024 at 8:41 AM Christoph Hellwig <hch@infradead.org> wrote:
>
> On Thu, Oct 31, 2024 at 05:30:31PM +0000, Maciej W. Rozycki wrote:
> >  This also means that the ISP_CFG0_1040A check also added in 2.6.9 with
> > <https://lore.kernel.org/r/20040606125825.GE31063@lst.de/> will never
> > match, possibly meaning that this code wasn't actually ever verified with
> > affected 1040A hardware.  This might also explain why a later change made
> > with commit 0888f4c33128 ("[SCSI] qla1280: don't use bitfields for
> > hardware access in isp_config") went unnoticed that changed the semantics
> > of the workaround from keeping bursts unconditionally disabled with the
> > 1040A to making them enabled in the absence of NVRAM.
> >
> >  NB comments for the FIFO threshold surely are suspicious too.
> >
> >  Christoph can you please have a look into it?  It seems like something
> > you ought to be quite familiar with if not for the passage of time.
>
> Somewhat surprisingly I don't remember that details of a drive by
> cleanup 20 years ago :)
>
> So whatever fixes you have based on other implementations are probably
> correct.
>

Re: qla1280 driver for qlogic-1040 on alpha

[Maciej W. Rozycki]

On Mon, 4 Nov 2024, Magnus Lindholm wrote:

> I'm making another attempt at fixing the qla1280.c driver for ISP1040x
> on Alpha, while trying not to break anything on other platforms, like
> IP-30/MIPS. This time I'm using dma_get_required_mask(). Is my
> understanding that this function should provide the minimum required
> mask for the platform, assuming this works it should return something
> greater than 32-bits on IP-30/MIPS. From what I can tell by looking at
> the kernel source it should return something like a 64-bit MASK for
> the sgi/octane but I'm not in the possession of such a system so I'm
> unable to verify this.

 This is missing the point, the problem is your card and not the system.  
The chipset in the Alpha can do 64-bit DMA addressing just fine or we 
would have hit it long ago with something else.  Those systems have been 
around for 20+ years.

 Can you please implement what I told you before, that is force 32-bit DMA 
addressing for pre-ISP1040C hardware and let the system determine whether 
64-bit DMA addressing is available for ISP1040C and later devices?  This 
should cover your card, because it's an ISP1040B as you've told us.

 Does your ISP1080 cause trouble with 64-bit DMA as well?

  Maciej

Re: qla1280 driver for qlogic-1040 on alpha

[Martin K. Petersen]

Maciej,

> force 32-bit DMA addressing for pre-ISP1040C hardware and let the
> system determine whether 64-bit DMA addressing is available for
> ISP1040C and later devices?

Yep, that is the correct approach.

Thomas: Can you confirm that your SGI hardware has a C rev 1040 ISP?

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: qla1280 driver for qlogic-1040 on alpha

[Thomas Bogendoerfer]

On Mon, 04 Nov 2024 20:40:40 -0500
"Martin K. Petersen" <martin.petersen@oracle.com> wrote:

> Maciej,
> 
> > force 32-bit DMA addressing for pre-ISP1040C hardware and let the
> > system determine whether 64-bit DMA addressing is available for
> > ISP1040C and later devices?  
> 
> Yep, that is the correct approach.
> 
> Thomas: Can you confirm that your SGI hardware has a C rev 1040 ISP?

they are ISP1040B, so IMHO the revision is not the point.

Thomas.

-- 
SUSE Software Solutions Germany GmbH
HRB 36809 (AG Nürnberg)
Geschäftsführer: Ivo Totev, Andrew McDonald, Werner Knoblich

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

Hi,

If I look at how some of my other SCSI cards are handling this on
Alpha I see that both sym875 and aic7xxx selects 32-bit masks. aic7xxx
uses dma_get_required_mask() and sym has it configurable in kernel
config but seems to default to 32-bit. I guess it's possible to use
dma_get_required_mask() for chip revisions < ISP1040C . Seeing how
other scsi drivers have similar solutions? I know, again this is
missing the point. The ISP1080 does work with a full 64-bit mask.

From the Tsunami manual we have:

"Table 10–5 PCI DMA Address to System Address Via Direct Mapping

Window Size          WSMn<31:20>             Translated Address <34:2>
....                            ....                                 .....
2GB                        111.1111.1111TBA         <34:31>:ad<30:2>
4GB                        N/A
000:ad<34:2> (monster window only)"

"The DMA monster window is enabled by PCTL<MWIN>. If enabled, this
window lies from 100.0000.0000 to 100.FFFF.FFFF, which requires a
dual-address cycle (DAC) access from the PCI bus. This window maps to
system memory as defined in Section 10.1.4. Because the Cchip’s
interface to the CPU and the CAPbus protocol between the Pchip and
Cchip only support 35 bits of addressing..."

From this it seems like any access to the "monster windows" needs to
use DAC, even if the card has full 64-bit addressing capability and is
plugged in 64-bit slot. On Tsunami based alphas, this seems to happen
when you go above 2GB memory according to table 10-5 (or 10-6 for
S/G).

Magnus

On Tue, Nov 5, 2024 at 9:34 AM Thomas Bogendoerfer
<tbogendoerfer@suse.de> wrote:
>
> On Mon, 04 Nov 2024 20:40:40 -0500
> "Martin K. Petersen" <martin.petersen@oracle.com> wrote:
>
> > Maciej,
> >
> > > force 32-bit DMA addressing for pre-ISP1040C hardware and let the
> > > system determine whether 64-bit DMA addressing is available for
> > > ISP1040C and later devices?
> >
> > Yep, that is the correct approach.
> >
> > Thomas: Can you confirm that your SGI hardware has a C rev 1040 ISP?
>
> they are ISP1040B, so IMHO the revision is not the point.
>
> Thomas.
>
> --
> SUSE Software Solutions Germany GmbH
> HRB 36809 (AG Nürnberg)
> Geschäftsführer: Ivo Totev, Andrew McDonald, Werner Knoblich

Re: qla1280 driver for qlogic-1040 on alpha

[Maciej W. Rozycki]

On Tue, 5 Nov 2024, Thomas Bogendoerfer wrote:

> > > force 32-bit DMA addressing for pre-ISP1040C hardware and let the
> > > system determine whether 64-bit DMA addressing is available for
> > > ISP1040C and later devices?  
> > 
> > Yep, that is the correct approach.
> > 
> > Thomas: Can you confirm that your SGI hardware has a C rev 1040 ISP?
> 
> they are ISP1040B, so IMHO the revision is not the point.

 Yet there appears to be a difference somewhere between the two supposedly 
identical chips that makes one work with 64-bit addressing while the other 
does not.  Is it possible that the ISP1040B used in SGI systems has been 
modified on request and not relabeled before QLogic chose to put it on the 
general market?  I can see the spread is very narrow between the dates of 
the ISP1040B and ISP1040C datasheets available: Feb 5th vs Jul 29th, 1997.

 Thomas, Magnus, can you please check what hardware revision is actually 
reported by your devices?  Also a dump of the PCI configuration space 
would be very useful, or at the very least the value of the PCI Revision 
ID register, which is independent from the hardware revision reported via 
the device I/O registers.

 The firmware is loaded by us, so I presume it's the same version on both 
systems (I can see 7.65.06 in Debian, the same as in the report upthread), 
and that only leaves any variable to the PCI bus logic onchip.

 If all else fails, I would still recommend to disable by default 32-bit 
DMA addressing for pre-ISP1040C hardware, as that matches documentation, 
and then have a platform quirk to enable it specifically for ISP1040B for 
the relevant SGI configurations.  But I do hope there is a better approach 
possible.

  Maciej

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

>  Thomas, Magnus, can you please check what hardware revision is actually
> reported by your devices?  Also a dump of the PCI configuration space
> would be very useful, or at the very least the value of the PCI Revision
> ID register, which is independent from the hardware revision reported via
> the device I/O registers.

Below is some info from my Alpha ES40 with an ISP1040. I've added a
printout of hardware revision number to the driver, as previously
pointed out the revision numbers in qla1280.h is wrong so I've used
info from NetBSD
rev5 is a "B" which matches what is actually printed on the chip as
well. This seems to be consistent with PCI Revision ID register.

output from driver:

qla1280: QLA1040 found on PCI bus 2, dev 4
revision=5 <-- printout of cfg0 register 5 = rev B
QLogic QLA1040 PCI to SCSI Host Adapter
Firmware version:  7.65.06, Driver version 3.27.1

#lspci -s 0001:02:04.0 -vvv
0001:02:04.0 SCSI storage controller: QLogic Corp. ISP1020/1040
Fast-wide SCSI (rev 05)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 248, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 52
        Region 0: I/O ports at 200008000 [size=256]
        Region 1: Memory at 209050000 (32-bit, non-prefetchable) [size=4K]
        Expansion ROM at 209040000 [disabled] [size=64K]
        Kernel driver in use: qla1280
        Kernel modules: qla1280


>If all else fails, I would still recommend to disable by default 32-bit
>DMA addressing for pre-ISP1040C hardware, as that matches documentation,
>and then have a platform quirk to enable it specifically for ISP1040B for
>the relevant SGI configurations.  But I do hope there is a better approach
>possible.

I have put together a patch which if we have an ISP1040 chip with chip
revision less than "C" it uses dma_get_required_mask() to se
DMA_BIT_MASK otherwise it will set the full 64-bit mask. If
dma_get_required_mask() works properly on IP30/MIPS it should return a
64bit-something mask, right? Maybe this can be the quirk that makes
this work on both Alpha and SGI/Octane? (Assuming we don't find a
better approach)

Magnus

Re: qla1280 driver for qlogic-1040 on alpha

[Thomas Bogendoerfer]

On Tue, 5 Nov 2024 20:24:58 +0100
Magnus Lindholm <linmag7@gmail.com> wrote:

> >  Thomas, Magnus, can you please check what hardware revision is actually
> > reported by your devices?  Also a dump of the PCI configuration space
> > would be very useful, or at the very least the value of the PCI Revision
> > ID register, which is independent from the hardware revision reported via
> > the device I/O registers.  
> 
> Below is some info from my Alpha ES40 with an ISP1040. I've added a
> printout of hardware revision number to the driver, as previously
> pointed out the revision numbers in qla1280.h is wrong so I've used
> info from NetBSD
> rev5 is a "B" which matches what is actually printed on the chip as
> well. This seems to be consistent with PCI Revision ID register.
> 
> output from driver:
> 
> qla1280: QLA1040 found on PCI bus 2, dev 4
> revision=5 <-- printout of cfg0 register 5 = rev B
> QLogic QLA1040 PCI to SCSI Host Adapter
> Firmware version:  7.65.06, Driver version 3.27.1
> 
> #lspci -s 0001:02:04.0 -vvv
> 0001:02:04.0 SCSI storage controller: QLogic Corp. ISP1020/1040
> Fast-wide SCSI (rev 05)
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium
> >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-  
>         Latency: 248, Cache Line Size: 64 bytes
>         Interrupt: pin A routed to IRQ 52
>         Region 0: I/O ports at 200008000 [size=256]
>         Region 1: Memory at 209050000 (32-bit, non-prefetchable) [size=4K]
>         Expansion ROM at 209040000 [disabled] [size=64K]
>         Kernel driver in use: qla1280
>         Kernel modules: qla1280

0000:00:00.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05)
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 8
	Region 0: I/O ports at 1f200000 [size=256]
	Region 1: Memory at 1f200000 (32-bit, non-prefetchable) [size=4K]
	Expansion ROM at 1f210000 [disabled] [size=64K]
	Kernel driver in use: qla1280

0000:00:01.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05)
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 9
	Region 0: I/O ports at 1f400000 [size=256]
	Region 1: Memory at 1f400000 (32-bit, non-prefetchable) [size=4K]
	Expansion ROM at 1f410000 [disabled] [size=64K]
	Kernel driver in use: qla1280

qla1280: QLA1040 found on PCI bus 0, dev 0
qla1280 0000:00:00.0: enabling device (0006 -> 0007)
random: crng init done
cfg_0 5
scsi(0:0): Resetting SCSI BUS
scsi host0: QLogic QLA1040 PCI to SCSI Host Adapter
       Firmware version:  7.65.06, Driver version 3.27.1
qla1280: QLA1040 found on PCI bus 0, dev 1
qla1280 0000:00:01.0: enabling device (0006 -> 0007)
scsi 0:0:1:0: Direct-Access     FUJITSU  MAW3073NC        0104 PQ: 0 ANSI: 3
scsi(0:0:1:0): Sync: period 10, offset 12, Wide, Tagged queuing: depth 31
scsi 0:0:2:0: Direct-Access     SEAGATE  SX118202LS       B808 PQ: 0 ANSI: 2
scsi(0:0:2:0): Sync: period 10, offset 12, Wide, Tagged queuing: depth 31
cfg_0 5
scsi(1:0): Resetting SCSI BUS

So the Octane 1040 chips are the same revision and working with 64bit addressing.

Thomas.

-- 
SUSE Software Solutions Germany GmbH
HRB 36809 (AG Nürnberg)
Geschäftsführer: Ivo Totev, Andrew McDonald, Werner Knoblich

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

Hi,

Thanks for taking the time to test this on sgi/octane. I guess the
results of your test means that only relying on the chip revision is
not going to do it.

I've put my ISP1040 rev B into a HPZ440 (x86_64) with 128GB RAM. When
booting the system I get
 "PCI Configuration error" when BIOS configures the card. I also see
this in the kernel message log:

"DMAR: [DMA Read NO_PASID] Request device [09:00.0] fault addr
0xfebba000 [fault reason 0x06] PTE Read access is not set"

So the HP workstation and the ISP1040 card do not fully agree.

I've used the standard qla1280 driver and hence enabled full 64-bit
DMA_MASK. Even if I got some generic errors when booting,
the card works and I can mount a drive, format a partition and
copy/paste files without any filesystem corruption.
However, when I enable the debug output I notice that the driver never
uses bus-addresses for DMA transfers that go any
higher than 32-bit. So using DMA_BIT_MASK of 64 or 32 bits does not
have any effect on the actual addresses generated.
The address always stays within a 32-bit mask in both cases.
dma_get_required_mask() returns a 32-bit mask for the
ISP1040 even when the system has 128GB RAM. What does
dma_get_required_mask() return on the sgi/octane?

Magnus

On Tue, Nov 12, 2024 at 2:53 PM Thomas Bogendoerfer
<tbogendoerfer@suse.de> wrote:
>
> So the Octane 1040 chips are the same revision and working with 64bit addressing.
>
> Thomas.
>
> --
> SUSE Software Solutions Germany GmbH
> HRB 36809 (AG Nürnberg)
> Geschäftsführer: Ivo Totev, Andrew McDonald, Werner Knoblich

Re: qla1280 driver for qlogic-1040 on alpha

[Maciej W. Rozycki]

On Sat, 16 Nov 2024, Magnus Lindholm wrote:

> Thanks for taking the time to test this on sgi/octane. I guess the
> results of your test means that only relying on the chip revision is
> not going to do it.

 I had hoped for full `lspci -xxx' dumps actually, which could reveal 
differences perhaps in the device-specific range of config registers.

> I've put my ISP1040 rev B into a HPZ440 (x86_64) with 128GB RAM. When
> booting the system I get
>  "PCI Configuration error" when BIOS configures the card. I also see
> this in the kernel message log:
> 
> "DMAR: [DMA Read NO_PASID] Request device [09:00.0] fault addr
> 0xfebba000 [fault reason 0x06] PTE Read access is not set"

 It looks like an IOMMU fault to me and might mean that the device has 
requested access to a memory location it may not have permission for.  It 
could or could not be a result of address truncation to 32 bits.

> I've used the standard qla1280 driver and hence enabled full 64-bit
> DMA_MASK. Even if I got some generic errors when booting,
> the card works and I can mount a drive, format a partition and
> copy/paste files without any filesystem corruption.
> However, when I enable the debug output I notice that the driver never
> uses bus-addresses for DMA transfers that go any
> higher than 32-bit. So using DMA_BIT_MASK of 64 or 32 bits does not
> have any effect on the actual addresses generated.
> The address always stays within a 32-bit mask in both cases.

 Given that your HPZ440 system appears to have an IOMMU chances are it's
used such as to squeeze all PCI-side DMA mappings into the low 32-bit 
range for the very purpose of avoiding issues with odd devices.

  Maciej

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

On Mon, Nov 25, 2024 at 8:56 PM Maciej W. Rozycki <macro@orcam.me.uk> wrote:

>  I had hoped for full `lspci -xxx' dumps actually, which could reveal
> differences perhaps in the device-specific range of config registers.

Here we go:
SCSI storage controller: QLogic Corp. ISP1020/1040 Fast-wide SCSI (rev 05)
00: 77 10 20 10 07 00 00 02 05 00 00 01 10 40 00 00
10: 01 a0 00 00 00 00 e1 fa 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 e0 fa 00 00 00 00 00 00 00 00 0b 01 00 00
40: 34 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 77 10 20 10 07 00 00 02 05 00 00 01 10 40 00 00
90: 01 a0 00 00 00 00 e1 fa 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 e0 fa 00 00 00 00 00 00 00 00 0b 01 00 00
c0: 34 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00



>  Given that your HPZ440 system appears to have an IOMMU chances are it's
> used such as to squeeze all PCI-side DMA mappings into the low 32-bit
> range for the very purpose of avoiding issues with odd devices.
>

I've gotten my hands on a Supermicro X9SRA board with 128GB of memory.
BIOS is in legacy mode with all the 64-bit PCI stuff enabled.

The ISP1040 card works fine, no errors reported at boot  or in the message file
dma_get_required_mask() returns a 32-bit mask but the card works in wíth
both 32 and 64 bits masks. The DMA_BIT_MASK seems to have no affect
on how the addresses are generated, they always stay below the full
32-bit range in either case. It seems like the x86_64 will not trigger
the issue with 64-bit addresses on the ISP1040 card.

Maybe I can test this on sparc as well, I'll see what I can put together.


/Magnus

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

Hi again,

I'm returning to my qla1280 driver issues on Alpha. To sum things up:
SGI/mips need full 64-bit DMA addressing while anything above 32-bit
DMA addressing on Alpha will result in file-system corruption. The
hypothesis so far has been that some ISP1040 chip revisions don't
support DAC. This feature only appears in data sheets for chip rev C
so it's reasonable to assume that rev B and less do not support it.
The problem arises on Alpha systems with more than 2GB RAM installed.
Up until this point the only Alphas available to me, which has more
than 2GB ram, has been Tsunami based Alphas (21264 UP2000+ and
Alphaserver ES40). I recently got my hands on an Alphaserver-4100
(Rawhide Family) with 4GB ram. On this system I see no file-system
corruption with the ISP1040 rev B card, even though it does seem to
emit bus addresses which are greater than 32-bits, i.e very similar
addresses as on the Tsunami based machines, the so called "monster
window" (where address-bit 40 is set). This gets me thinking that
maybe this really is an issue with how DMA/DAC is handled on the
Tsunami boards, rather than a problem with the qla1280 driver and DAC
support in various chip revisions?

Any thoughts on this?


Btw, does anyone have access to "rawhide system programmer's manual"
referred to in the kernel source?

/Magnus

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

Hi again,

Here are some updates on my continued experiments with the qla1280 driver
This is what I've concluded so far:

*The ISP1040B 32-bitcard works with 64-bit DMA_MASK on a 21164 Rawhide
machine, hence the card supports DAC, even though the papers on the chip
don't officially claim support until rev C (just as Thomas Bogendoerfer
pointed out earlier).

*The ISP1080 (64-bit pci slot/card) works with 64-bit DMA_MASK  on a
21224 Tsunami machine, hence the monster window works fine on Tsunami.

I've made some tests my ES40 (Tsunami) by first filling the drive attached
to the IPS1040 with letter 'A' by just dd:ing a file containing about 40mb
data, that is "dd if=A.txt of=/dev/sdc bs=1M". Then rebooting and loading
the qla1280 driver with 64-bit DMA_MASK and reading back from 20mb from
/dev/sdc using "dd oflag=direct if=/dev/sdc of=dump.img bs=1M count=20"
I expect to get only the letter A in dump.img.

To put some load on the system I have two processes running in background,
unpacking some tar files. These files are unpacked on another drive
attached to a different controller.

When first running this test I just got some random data in one or more
corrupted blocks but after repeating this a few times I started to
recognize the data in the corrupt blocks. It is as if data from the disk
attached to the ISP1040 controller is interleaved with data coming from
another drive attached to another controller. In this case the text is
from stuff belonging to gcc (which was in the tar file being unpacked on
the other drive/controller at the same time).

Data from one test run:

A total of 57 hits on the monster window, 64kb each, during the test a total
of 20mb was transferred from the disk and a total of about 3600kb was
transferred using DAC/DMA mappings to the monster window. Only 64 bytes were
corrupted. This suggests that DAC works on tsunami with the ISP1040 card
but something else is going on that corrupts data along the way?

The text is part of gcc-stuff that was being written to another disk
on another controller simultaneously (that is, a tar.xz file was expanded)

diff between expected output (AAAA) and actual output:

< 012b4c80: 6325 3e00 253c 616e 6425 3e20 6f66 206d  c%>.%<and%> of m
< 012b4c90: 7574 7561 6c6c 7920 6578 636c 7573 6976  utually exclusiv
< 012b4ca0: 6520 6571 7561 6c2d 7465 7374 7320 6973  e equal-tests is
< 012b4cb0: 2061 6c77 6179 7320 3000 253c 6173 6d25   always 0.%<asm%
---
> 012b4c80: 4141 4141 4141 4141 4141 4141 4141 4141  AAAAAAAAAAAAAAAA
> 012b4c90: 4141 4141 4141 4141 4141 4141 4141 4141  AAAAAAAAAAAAAAAA
> 012b4ca0: 4141 4141 4141 4141 4141 4141 4141 4141  AAAAAAAAAAAAAAAA
> 012b4cb0: 4141 4141 4141 4141 4141 4141 4141 4141  AAAAAAAAAAAAAAAA

The amount of data being corrupted each run varies a lot, from no data
corruption at all to several kilobytes. When data is corrupted it is
always in chunks of 64-bytes, which coincides with the cache block size
on the 21264 processor. It's as if every now and then, there is a cache
block that holds stale data and the data written to memory by the dma
controller is not seen by the cpu? this only happens when DAC DMA is
done to 32-bit cards in a 64-bit slot and DMA_MASK is 64 bit.

I've noticed that most of the DAC DMA requests hitting the "monster window"
are allocated as entries in scatter/gather lists. I'm not sure if I
understand this correctly but according to the "Tsunami/Typhoon 21272
Chipset Hardware Reference Manual" the monster window only supports direct
mapping, and if DAC is to be used in S/G mappings, DMA window-3 should be
used instead. This is not the case in how this is implemented in linux for
the tsunami platform. (see section 10.1.2.1,  10.1.4 and table 10-6).


/Magnus

Re: qla1280 driver for qlogic-1040 on alpha

[Martin K. Petersen]

Maciej,

>  Thomas, Magnus, can you please check what hardware revision is
> actually reported by your devices? Also a dump of the PCI
> configuration space would be very useful, or at the very least the
> value of the PCI Revision ID register, which is independent from the
> hardware revision reported via the device I/O registers.

It would also be interesting to know what the 'enable 64-bit addressing'
NVRAM flag is set to on Thomas' system.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

On Tue, Nov 5, 2024 at 8:56 PM Martin K. Petersen
<martin.petersen@oracle.com> wrote:

> It would also be interesting to know what the 'enable 64-bit addressing'
> NVRAM flag is set to on Thomas' system.

This is an interesting note in the revision history of the driver:

    Rev. 3.14 Beta  August 16, 2000   BN  Qlogic
        - Added setting of dma_mask to full 64 bit
          if flags.enable_64bit_addressing is set in NVRAM

This is no longer present in the code, must have been removed along
the way somehow. On my (working full 64bit mask) ISP1080 this flag is
not set (according to the driver). I've tried having it both set/unset
with the ISP1040 but it seems to make no difference. If setting it
through the driver even works?

Magnus

Re: qla1280 driver for qlogic-1040 on alpha

[Thomas Bogendoerfer]

On Tue, 05 Nov 2024 14:56:15 -0500
"Martin K. Petersen" <martin.petersen@oracle.com> wrote:

> Maciej,
> 
> >  Thomas, Magnus, can you please check what hardware revision is
> > actually reported by your devices? Also a dump of the PCI
> > configuration space would be very useful, or at the very least the
> > value of the PCI Revision ID register, which is independent from the
> > hardware revision reported via the device I/O registers.  
> 
> It would also be interesting to know what the 'enable 64-bit addressing'
> NVRAM flag is set to on Thomas' system.

there is no NVRAM on the Octane

Thomas.

-- 
SUSE Software Solutions Germany GmbH
HRB 36809 (AG Nürnberg)
Geschäftsführer: Ivo Totev, Andrew McDonald, Werner Knoblich

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

Hi,

What's your thoughts on his approach:

if    (card is ISP1040 and revision less than C) then use
dma_get_required_mask()
else
     do as before

Assuming dma_get_required_mask() works it should return a
64-something-bit mask on IP30/MIPS but only 32-bits on Alpha.

Magnus

On Tue, Nov 5, 2024 at 10:33 PM Thomas Bogendoerfer
<tbogendoerfer@suse.de> wrote:
>
> On Tue, 05 Nov 2024 14:56:15 -0500
> "Martin K. Petersen" <martin.petersen@oracle.com> wrote:
>
> > Maciej,
> >
> > >  Thomas, Magnus, can you please check what hardware revision is
> > > actually reported by your devices? Also a dump of the PCI
> > > configuration space would be very useful, or at the very least the
> > > value of the PCI Revision ID register, which is independent from the
> > > hardware revision reported via the device I/O registers.
> >
> > It would also be interesting to know what the 'enable 64-bit addressing'
> > NVRAM flag is set to on Thomas' system.
>
> there is no NVRAM on the Octane
>
> Thomas.
>
> --
> SUSE Software Solutions Germany GmbH
> HRB 36809 (AG Nürnberg)
> Geschäftsführer: Ivo Totev, Andrew McDonald, Werner Knoblich

Re: qla1280 driver for qlogic-1040 on alpha

[Maciej W. Rozycki]

On Sat, 9 Nov 2024, Magnus Lindholm wrote:

> What's your thoughts on his approach:
> 
> if    (card is ISP1040 and revision less than C) then use
> dma_get_required_mask()
> else
>      do as before
> 
> Assuming dma_get_required_mask() works it should return a
> 64-something-bit mask on IP30/MIPS but only 32-bits on Alpha.

 It's not required for the mask returned by dma_get_required_mask() to be 
32-bit, so unless your card does support DAC (and everything so far shows 
it does not), it will still fail on another platform where the function 
does return a mask beyond 32 bits.

 Are you able to verify your card with a non-Alpha system that has memory
beyond the low 32-bit DMA space?  I guess not, or you'd have done that 
already, wouldn't you?

 If you really wanted to double-check your Alpha correctly supports DAC, 
you could try wiring your ISP1080 device temporarily via a 32-bit PCI 
riser adapter (or an external 32-bit PCI enclosure) so as to force 64-bit 
addressing via AD[31:0] lines only (assuming that ISP1080 got DAC support 
right though).

  Maciej

Re: qla1280 driver for qlogic-1040 on alpha

[Magnus Lindholm]

On Sun, Nov 10, 2024 at 4:59 PM Maciej W. Rozycki <macro@orcam.me.uk> wrote:

>  It's not required for the mask returned by dma_get_required_mask() to be
> 32-bit, so unless your card does support DAC (and everything so far shows
> it does not), it will still fail on another platform where the function
> does return a mask beyond 32 bits.
>

I guess that's right, It might still break then if DMA_BIT_MASK is set
> 32bits on other platforms

>  Are you able to verify your card with a non-Alpha system that has memory
> beyond the low 32-bit DMA space?  I guess not, or you'd have done that
> already, wouldn't you?
>

No, I don't, my other systems are either PCIe or have less than 4GB RAM.
I might be able to get my hands on a HP Z440 which I've been told has
one 32-bit PCI slot.

>  If you really wanted to double-check your Alpha correctly supports DAC,
> you could try wiring your ISP1080 device temporarily via a 32-bit PCI
> riser adapter (or an external 32-bit PCI enclosure) so as to force 64-bit
> addressing via AD[31:0] lines only (assuming that ISP1080 got DAC support
> right though).


From the Tsunami paper:

"The DMA monster window is enabled by PCTL<MWIN>. If enabled, this window lies
from 100.0000.0000 to 100.FFFF.FFFF, which requires a dual-address cycle (DAC)
access from the PCI bus. This window maps to system memory as defined
in Section 10.1.4"

I interpret this as in order to make the 1080 work DAC is actually
needed/used even when in a 64-bit slot?
Since we're using the "Monster window". From this I'm thinking DAC
works on my ES40 Alpha.

Magnus

Re: qla1280 driver for qlogic-1040 on alpha

[Maciej W. Rozycki]

On Sun, 3 Nov 2024, Christoph Hellwig wrote:

> >  Christoph can you please have a look into it?  It seems like something 
> > you ought to be quite familiar with if not for the passage of time.
> 
> Somewhat surprisingly I don't remember that details of a drive by
> cleanup 20 years ago :)

 Oh dear!  It was worth asking anyway.  Thank you!

  Maciej