Questions about IOMMU & PCIe switch

Questions about IOMMU & PCIe switch

[Raimonds Cicans]

Hello.

We have two kinds of PCIe cards:
A1 - based on chip B
A2 - based on same chip B but behind PCIe switch

Card A1 work flawlessly in any configuration,
but card A2 work flawlessly only if system
lack IOMMU or have disabled IOMMU

Questions:
Do we have to treat card with PCIe switch in special way?
How?
Where (PCI bridge driver? card's driver?)?
Any examples in kernel?

Thank you.

Raimonds Cicans

Re: Questions about IOMMU & PCIe switch

[Clemens Ladisch]

Raimonds Cicans wrote:
> We have two kinds of PCIe cards:
> A1 - based on chip B
> A2 - based on same chip B but behind PCIe switch
>
> Card A1 work flawlessly in any configuration,
> but card A2 work flawlessly only if system
> lack IOMMU or have disabled IOMMU

In theory, such a switch should be transparent.

> Do we have to treat card with PCIe switch in special way?

No.  But the switch has to work correctly, and driver must be
written correctly.

Which switch?  Which driver?


Regards,
Clemens

Re: Questions about IOMMU & PCIe switch

[Raimonds Cicans]

On 07.01.2015 14:09, Clemens Ladisch wrote:
> Raimonds Cicans wrote:
>> We have two kinds of PCIe cards:
>> A1 - based on chip B
>> A2 - based on same chip B but behind PCIe switch
>>
>> Card A1 work flawlessly in any configuration,
>> but card A2 work flawlessly only if system
>> lack IOMMU or have disabled IOMMU
> In theory, such a switch should be transparent.
>
>> Do we have to treat card with PCIe switch in special way?
> No.  But the switch has to work correctly, and driver must be
> written correctly.
>
> Which switch?  Which driver?

We talk about TBS 6285 Quad DVB-T PCIe receiver card.
Starting from version V20 manufacturer added Pericom
PCIe switch PI7C9X2G304SL.

Card have 2 out of kernel drivers:
1) Semi open from TBS:
http://www.tbsdtv.com/download/document/common/tbs-linux-drivers_v141225.zip

2) Open from: https://github.com/ljalves/linux_media

If you want more information you can look at open ticket:
https://github.com/ljalves/linux_media/issues/66


Raimonds Cicans

Re: Questions about IOMMU & PCIe switch

[Clemens Ladisch]

Raimonds Cicans wrote:
> https://github.com/ljalves/linux_media/issues/66

If the TBS driver works, why don't you use it?

The WARNING from the kernel log indicates a hardware bug in the PCIe
bridge.  Do you have the same card, and do you also get this warning
with kernel 3.16?

I/O virtualization requires handling packets from different devices
differently.  Problems with DMAR might indicate that some PCIe device
does not correctly identify itself as the source of PCIe packets.


Regards,
Clemens

Re: Questions about IOMMU & PCIe switch

[Raimonds Cicans]

On 08.01.2015 10:34, Clemens Ladisch wrote:
> Raimonds Cicans wrote:
>> https://github.com/ljalves/linux_media/issues/66
> If the TBS driver works, why don't you use it?
1) driver is not stable in 24x7 setups

2) driver use old DVBAPI. This cause problems with some
      user space programs.

3) TBS recommends to use card in MSI interrupt mode
      but this mode on IOMMU systems do not work:
      card is able to find transponders and tune to it
      but can not receive any data

4) I use GRSecurity patched kernels but external media
     drivers are incompatible with such kernels.
     I am forced to use driver source code injection in kernel
     from project https://github.com/bas-t/saa716x-intree
     which injects few drivers from https://github.com/ljalves/linux_media
     in kernel source tree.

5) I prefer open source drivers, but TBS drivers are half way open -
     open back-end drivers and closed front-end drivers.

>
> The WARNING from the kernel log indicates a hardware bug in the PCIe
> bridge.  Do you have the same card, and do you also get this warning
> with kernel 3.16?
1) I have same card with same revision number

2) affected computer is Intel based but mine is AMD based

3) I can not test kernel 3.16 because I had IOMMU related regression
     in kernels 3.14-3.17 (AMD-Vi: Completion-Wait loop timed out)
     On other kernels I do not have such warning, but starting from
     kernel 3.17.7 I have IOMMU related messages with my other card
    (TBS 6981):

AMD-Vi: Event logged [IO_PAGE_FAULT device=08:00.0 domain=0x001c 
address=0x0000000001355000 flags=0x0000]

and

WARNING: CPU: 0 PID: 13204 at drivers/iommu/amd_iommu.c:2625 
dma_ops_domain_unmap.part.9+0x4d/0x56()

Because TBS 6981 driver is included in kernel I contacted yesterday 
linux-media mailing list, but not received answer yet.



Raimonds Cicans

Re: Questions about IOMMU & PCIe switch

[Manu Abraham]

On Thu, Jan 8, 2015 at 8:35 PM, Raimonds Cicans <ray@apollo.lv> wrote:
> On 08.01.2015 10:34, Clemens Ladisch wrote:
>>
>> Raimonds Cicans wrote:
>>>
>>> https://github.com/ljalves/linux_media/issues/66
>>
>> If the TBS driver works, why don't you use it?
>
> 1) driver is not stable in 24x7 setups
>
> 2) driver use old DVBAPI. This cause problems with some
>      user space programs.

It shouldn't. It is backward compatible.

>
> 3) TBS recommends to use card in MSI interrupt mode
>      but this mode on IOMMU systems do not work:
>      card is able to find transponders and tune to it
>      but can not receive any data

TBS didn't even have the courtesy to say thanks, but still.
They don't have any clue on the driver.

You shouldn't use MSI mode on most systems unless you
are sure that your machine works perfectly well with MSI
mode.

You should simply use standard interrupt mode, if you are
unsure.

In any sense, that card looks a bit odd:

1. It doesn't need a PCIe switch, nor that CPLD in there for
the fact that it is using a SAA7160, rather than an E version,
which thereby support 4 FGPI inputs (4 simultaneous TS's)
(Redundant components)

2. But in any sense, the PCIe switch is transparent, It doesn't
need any programming. All what you can do with the PCIe
switch is to reprogram the lines to it, nothing more. In any
case, if your card is detected, there's no point in playing with
the PCIe switch

3. The TS failure in a while can most likely be attributed to
bad CPLD programming. On the cards that I have worked
with, they've worked pretty well.


Regards,

Manu

Re: Questions about IOMMU & PCIe switch

[Luis Alves]

Hi Manu,

Do you know if your saa716x driver might have any issue in the dma code?
The code doesn't seem to be using chained scatterlists so it should be
ok with the "old" method of just increment the pointer instead of
using sg_next.
(It's still a little blurry to my this IOMMU+dma+scatterlist stuff)

Anyway, I've converted it to use "sg_next" as asked Raimonds to test
(but I think it shouldn't make any difference):
https://github.com/ljalves/linux_media/commit/54362743588bc8271e7988dd2823f818c00bf43d

I also know don't fully understand the whole driver but I say thank
you for it! :)
By the way, when can we get it upstream? If you need help with
anything just mail me. We are waiting for that driver for so long!


Regards,
Luis


On Thu, Jan 8, 2015 at 8:35 PM, Raimonds Cicans <ray@apollo.lv> wrote:
> On 08.01.2015 10:34, Clemens Ladisch wrote:
>>
>> Raimonds Cicans wrote:
>>>
>>> https://github.com/ljalves/linux_media/issues/66
>>
>> If the TBS driver works, why don't you use it?
>
> 1) driver is not stable in 24x7 setups
>
> 2) driver use old DVBAPI. This cause problems with some
>      user space programs.

It shouldn't. It is backward compatible.

>
> 3) TBS recommends to use card in MSI interrupt mode
>      but this mode on IOMMU systems do not work:
>      card is able to find transponders and tune to it
>      but can not receive any data

TBS didn't even have the courtesy to say thanks, but still.
They don't have any clue on the driver.

You shouldn't use MSI mode on most systems unless you
are sure that your machine works perfectly well with MSI
mode.

You should simply use standard interrupt mode, if you are
unsure.

In any sense, that card looks a bit odd:

1. It doesn't need a PCIe switch, nor that CPLD in there for
the fact that it is using a SAA7160, rather than an E version,
which thereby support 4 FGPI inputs (4 simultaneous TS's)
(Redundant components)

2. But in any sense, the PCIe switch is transparent, It doesn't
need any programming. All what you can do with the PCIe
switch is to reprogram the lines to it, nothing more. In any
case, if your card is detected, there's no point in playing with
the PCIe switch

3. The TS failure in a while can most likely be attributed to
bad CPLD programming. On the cards that I have worked
with, they've worked pretty well.


Regards,

Manu
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/