MMIO ioremap() error with PCI passthrough

MMIO ioremap() error with PCI passthrough

[Andy Burns]

I asked this question on the users list a few days ago, but got no 
response, perhaps it's a bit too involved for that list, I hope it's not 
*too* unwelcome here ...

I'm trying to use PCI passthrough of a DVB-T tuner from a fedora8 dom0, 
to a centOS5.2 domU

On the dom0 I have the following kernel/xen installed

kernel-xen.x86_64           2.6.21.7-3.fc8
xen.x86_64                  3.1.2-2.fc8
xen-libs.x86_64             3.1.2-2.fc8

On the domU I have the following kernel installed

kernel-xen.x86_64           2.6.18.92.1.6.el5.centos.plus

Additionally I had to manually compile the saa7134_dvb.ko module for the 
digital section of the card as the centosplus kernel only had the 
saa7134.ko module for the analogue section of the card.

I tried adding a pciback.hide entry to grub.conf for the dom0, but this 
wasn't recognised, presumably because pciback is compiled as a module on 
fedora8 dom0 kernel?

I realise I can add entries to modprobe.conf to ensure no dom0 drivers 
bind to my device, but for now I'm using the following script to get 
dom0 to relinquish the card and make it available for the domU.

rmmod saa7134
modprobe pciback
SLOT=0000:08:01.0
echo -n $SLOT > /sys/bus/pci/drivers/pciback/new_slot
echo -n $SLOT > /sys/bus/pci/drivers/pciback/bind

I then create my domU with this config file

memory = 1024
name = "mythbe"
vif = [ 'mac=00:16:3E:76:E8:92, bridge=eth0' ]
disk = [ 'phy:/dev/vgr1/lvmythroot,xvda,w',
          'phy:/dev/vgr5/lvmythdata,xvdb,w' ]
pci = ['08:01.0']
bootloader = "/usr/bin/pygrub"
vcpus = 1
on_reboot = 'restart'
on_crash = 'restart'

I see a few PCI related messages in the domU dmesg

PCI: Fatal: No PCI config space access function found
PCI: setting up Xen PCI frontend stub
PCI: System does not support PCI
PCI: System does not support PCI
pcifront pci-0: Installing PCI frontend
pcifront pci-0: Creating PCI Frontend Bus 0000:00
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
PCI: Enabling device 0000:00:00.0 (0000 -> 0002)

The DVB-T card is visble in the domU

# lspci -v

00:00.0 Multimedia controller: Philips Semiconductors SAA7130 Video 
Broadcast Decoder (rev 01)
         Subsystem: Compro Technology, Inc. Videomate DVB-T200
         Flags: bus master, medium devsel, latency 64, IRQ 17
         Memory at febffc00 (32-bit, non-prefetchable) [size=1K]
         Capabilities: [40] Power Management version 1

I assume it's normal that the PCI bus/device/function numbers have been 
changed between dom0 and domU?

The driver modules are being loaded by the domU

# lsmod | grep saa

saa7134_dvb            48004  0
dvb_pll                48965  1 saa7134_dvb
mt352                  40133  1 saa7134_dvb
video_buf_dvb          40133  1 saa7134_dvb
nxt200x                46661  1 saa7134_dvb
tda1004x               48581  1 saa7134_dvb
saa7134               159017  1 saa7134_dvb
video_buf              59717  3 saa7134_dvb,video_buf_dvb,saa7134
compat_ioctl32         41793  1 saa7134
ir_kbd_i2c             42961  1 saa7134
i2c_core               56129  7 
saa7134_dvb,dvb_pll,mt352,nxt200x,tda1004x,saa7134,ir_kbd_i2c
ir_common              63173  2 saa7134,ir_kbd_i2c
videodev               58049  1 saa7134
v4l1_compat            44613  2 saa7134,videodev
v4l2_common            57153  3 saa7134,compat_ioctl32,videodev

I see an error in dmesg on the domU

# dmesg | grep -i saa

saa7130/34: v4l2 driver version 0.2.14 loaded
saa7130[0]: found at 0000:00:00.0, rev: 1, irq: 17, latency: 64, mmio: 
0xfebffc00
saa7130[0]: subsystem: 185b:c901, board: Compro Videomate DVB-T200 
[card=71,autodetected]
saa7130[0]: can't ioremap() MMIO memory
saa7134: probe of 0000:00:00.0 failed with error -5

The calling code in domU is here

http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.18.y.git;a=blob;f=drivers/media/video/saa7134/saa7134-core.c;h=be3a81fc90a2498769cb95ca2b4f306c4d71cfc8;hb=HEAD#l911

I also receive a corresponding message in dom0

# xm dmesg

(XEN) mm.c:625:d6 Non-privileged (6) attempt to map I/O space 000fec00

The called code in dom0 is here

http://lxr.xensource.com/lxr/source/xen/arch/x86/mm.c#L674

Am I right in thinking the pciback.permissive parameter is now removed? 
Certainly modinfo is unaware of it.

Any suggestions for how to stop the ioremap() call from failing and thus 
allow the driver to loading properly?

Re: MMIO ioremap() error with PCI passthrough

[Keir Fraser]

On 1/7/08 09:17, "Andy Burns" <lists.xensource.com@adslpipe.co.uk> wrote:

> # xm dmesg
> 
> (XEN) mm.c:625:d6 Non-privileged (6) attempt to map I/O space 000fec00

What does lspci say the resource areas are for this device? It looks like
fec00000 is past the end of the device's mmio region.

> The called code in dom0 is here
> 
> http://lxr.xensource.com/lxr/source/xen/arch/x86/mm.c#L674
> 
> Am I right in thinking the pciback.permissive parameter is now removed?
> Certainly modinfo is unaware of it.

It got turned into a per-device attribute. I think you echo the PCI slot
value to /sys/bus/pci/drivers/pciback/permissive

 -- Keir

> Any suggestions for how to stop the ioremap() call from failing and thus
> allow the driver to loading properly?

Re: MMIO ioremap() error with PCI passthrough

[Andy Burns]

On 01/07/2008 09:30, Keir Fraser wrote:

> What does lspci say the resource areas are for this device? 

Are you asking for anything more than "lspci -vvv" output?

When the card is loaded in dom0 (before I unload the drivers and assign 
the slot to pciback) I see this


08:01.0 Multimedia controller: Philips Semiconductors SAA7130 Video 
Broadcast Decoder (rev 01)
         Subsystem: Compro Technology, Inc. Videomate DVB-T200
         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- 
ParErr- Stepping- SERR- FastB2B- DisINTx-
         Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium 
 >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
         Latency: 64 (21000ns min, 8000ns max)
         Interrupt: pin A routed to IRQ 17
         Region 0: Memory at febffc00 (32-bit, non-prefetchable) [size=1K]
         Capabilities: [40] Power Management version 1
                 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA 
PME(D0-,D1-,D2-,D3hot-,D3cold-)
                 Status: D0 PME-Enable- DSel=0 DScale=1 PME-
         Kernel modules: saa7134


When assigned to pciback/pcifront, from within the domU I see this


00:00.0 Multimedia controller: Philips Semiconductors SAA7130 Video 
Broadcast Decoder (rev 01)
         Subsystem: Compro Technology, Inc. Videomate DVB-T200
         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- 
ParErr- Stepping- SERR- FastB2B-
         Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium 
 >TAbort- <TAbort- <MAbort- >SERR- <PERR-
         Latency: 64 (21000ns min, 8000ns max)
         Interrupt: pin A routed to IRQ 17
         Region 0: Memory at febffc00 (32-bit, non-prefetchable) [size=1K]
         Capabilities: [40] Power Management version 1
                 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA 
PME(D0-,D1-,D2-,D3hot-,D3cold-)
                 Status: D0 PME-Enable- DSel=0 DScale=1 PME-



> It looks like fec00000 is past the end of the device's mmio region.


Well 0xFEBFFC00 + 1K = 0xFEC00000, is this some sort of off-by-one 
error, where it should only extend to 0xFEBFFFFF?

Is it worth me attempting a change in the driver code from

        dev->lmmio = ioremap(pci_resource_start(pci_dev,0), 0x1000);

to

        dev->lmmio = ioremap(pci_resource_start(pci_dev,0), 0xFFF);

Re: MMIO ioremap() error with PCI passthrough

[Keir Fraser]

On 1/7/08 09:58, "Andy Burns" <lists.xensource.com@adslpipe.co.uk> wrote:

>> It looks like fec00000 is past the end of the device's mmio region.
> 
> 
> Well 0xFEBFFC00 + 1K = 0xFEC00000, is this some sort of off-by-one
> error, where it should only extend to 0xFEBFFFFF?
> 
> Is it worth me attempting a change in the driver code from
> 
>         dev->lmmio = ioremap(pci_resource_start(pci_dev,0), 0x1000);
> 
> to
> 
>         dev->lmmio = ioremap(pci_resource_start(pci_dev,0), 0xFFF);

It shouldn't be necessary, since the range specified to ioremap is by base
address and length and covers the inclusive range 'base' to 'base+length-1'.
It's rather odd that the kernel tries to map FEC00000 at all. If you're
happy changing kernel code, you might try adding printk() instrumentation to
__ioremap() and __direct_remap_pfn_range() in arch/i386/mm/ioremap-xen.c.
The functions look like they should map the correct range of pages (in this
case only the page covering FEBFF000-FEBFFFFF) but perhaps there is an
off-by-one error lurking. Or perhaps the mapping attempt of FEC00000 comes
from somewhere else in the kernel.

 -- Keir

Re: MMIO ioremap() error with PCI passthrough

[Andy Burns]

On 01/07/2008 10:45, Keir Fraser wrote:

> If you're
> happy changing kernel code, you might try adding printk() instrumentation to
> __ioremap() 

printk(KERN_DEBUG "REMAP: phys=0x%lx, len=%lu\n",
        phys_addr, size);

and __direct_remap_pfn_range() in arch/i386/mm/ioremap-xen.c.

printk(KERN_DEBUG "REMAPPFN: addr=0x%lx, mfn=0x%lx, size=%lu\n",
        address, mfn, size);

> The functions look like they should map the correct range of pages (in this
> case only the page covering FEBFF000-FEBFFFFF)

Built and installed kernel/modules/initrd, rebooted, when it loads the 
saa7134 driver I see this

REMAP: phys=0xfebffc00, len=4096
REMAPPFN: addr=0xffffc20000038000, mfn=0xfebff, size=8192

So the 1K mapping has been rounded up to 4K (x86_64 page size?) before 
passing to __ioremap() and then rounded up again to 8K by the time it 
gets passed to __direct_remap_pfn_range() is that right?

I can manage to build a kernel, but I'm out of my depth in terms of 
knowing what would be good or bad with physical addresses and page 
mappings ...

Re: MMIO ioremap() error with PCI passthrough

[Keir Fraser]

On 1/7/08 14:16, "Andy Burns" <lists.xensource.com@adslpipe.co.uk> wrote:

>> The functions look like they should map the correct range of pages (in this
>> case only the page covering FEBFF000-FEBFFFFF)
> 
> Built and installed kernel/modules/initrd, rebooted, when it loads the
> saa7134 driver I see this
> 
> REMAP: phys=0xfebffc00, len=4096
> REMAPPFN: addr=0xffffc20000038000, mfn=0xfebff, size=8192
> 
> So the 1K mapping has been rounded up to 4K (x86_64 page size?) before
> passing to __ioremap() and then rounded up again to 8K by the time it
> gets passed to __direct_remap_pfn_range() is that right?

Well, your analysis is correct, and the size argument to __ioremap() is
bogus. It shouldn't have been rounded up to 4096 without also rounding down
the base address. I don't think this would happen with out linux-2.6.18-xen
tree. In there, ioremap() is defined in include/asm-x86_64/mach-xen/io.h as
a thin wrapper around __ioremap() which does not modify the size parameter.

So, could be a bug specific to the FC8 kernel. I don't have its sources to
hand to pinpoint where the size is getting changed from 0x400 to 0x1000. You
should be able to dig up that detail pretty easily though.

 -- Keir

Re: MMIO ioremap() error with PCI passthrough

[Andy Burns]

On 01/07/2008 14:31, Keir Fraser wrote:

> Well, your analysis is correct, and the size argument to __ioremap() is
> bogus. It shouldn't have been rounded up to 4096 without also rounding down
> the base address. 

Just been having a look and a think, the size isn't being rounded up 
anywhere, the size is actually passed in as 4K from the driver

      dev->lmmio = ioremap(pci_resource_start(pci_dev,0), 0x1000);

Perhaps the rounding up to 8K is happening because the address isn't 
aligned on a page boundary, so is split over two pages?

So I assume it is the driver at fault for mapping a larger resource area 
than the card actually supports?

I'll try changing the size argument to 0x400 and rebuilding the saa7134 
module.

> So, could be a bug specific to the FC8 kernel. 

Actually centos5.2 is the domU, FC8 is the dom0, since it's x86_64, I 
assume it's the Fedora/Redhat forward-port effort, rather than the 
paravirt_ops effort.

Re: MMIO ioremap() error with PCI passthrough

[Andy Burns]

On 01/07/2008 16:44, Andy Burns wrote:

> I'll try changing the size argument to 0x400 and rebuilding the saa7134 
> module.

OK, changing the ioremap() in the saa7134 driver to only request 1K 
allowed the driver to load :-)

saa7130/34: v4l2 driver version 0.2.14 loaded
PCI: Enabling device 0000:00:00.0 (0000 -> 0002)
saa7130[0]: found at 0000:00:00.0, rev: 1, irq: 17, latency: 64, mmio: 
0xfebffc00
saa7130[0]: subsystem: 185b:c901, board: Compro Videomate DVB-T200 
[card=71,auto
detected]
saa7130[0]: board init: gpio is 843f00
input: saa7134 IR (Compro Videomate DV as /class/input/input1
saa7130[0]: i2c eeprom 00: 5b 18 01 c9 54 20 1c 00 43 43 a9 1c 55 d2 b2 92
saa7130[0]: i2c eeprom 10: 00 ff 86 0f ff 20 ff ff ff ff ff ff ff ff ff ff
saa7130[0]: i2c eeprom 20: 01 40 01 03 03 ff 03 01 08 ff 00 88 ff ff ff ff
saa7130[0]: i2c eeprom 30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
saa7130[0]: i2c eeprom 40: ff d0 00 c2 86 10 ff ff ff ff ff ff ff ff ff ff
saa7130[0]: i2c eeprom 50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff cb
saa7130[0]: i2c eeprom 60: 30 26 ff ff ff ff ff ff ff ff ff ff ff ff ff ff
saa7130[0]: i2c eeprom 70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
saa7130[0]: registered device video0 [v4l2]
saa7130[0]: registered device vbi0

 From my extra printk() debugging the driver now requests 1K, and within 
__direct_remap_pfn_range() the size has been rounded up to 4K, 
presumably this *is* correct as the mfn seems to has been rounded down?

REMAP: phys=0xfebffc00, len=1024
REMAPPFN: addr=0xffffc2000002e000, mfn=0xfebff, size=4096

I checked in 2.6.26 git to see if the saa7134 driver had been altered so 
the size had been reduced in more recent kernels, but it has not.  I'll 
check with the V4L developers whether it should be hardcoded to 1K 
instead of 4K, or whether it needs to be sized dynamically for different 
cards.

However about 40 seconds after loading the module, I get the following 
error :-(

irq 17: nobody cared (try booting with the "irqpoll" option)

Call Trace:
  <IRQ>  [<ffffffff802af539>] __report_bad_irq+0x30/0x7d
  [<ffffffff802af770>] note_interrupt+0x1ea/0x22b
  [<ffffffff802aec76>] __do_IRQ+0xbd/0x103
  [<ffffffff8028d06f>] _local_bh_enable+0x61/0xc5
  [<ffffffff8026db3a>] do_IRQ+0xe7/0xf5
  [<ffffffff8039f151>] evtchn_do_upcall+0x86/0xe0
  [<ffffffff802608ce>] do_hypervisor_callback+0x1e/0x2c
  <EOI>  [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
  [<ffffffff8026f12b>] raw_safe_halt+0x84/0xa8
  [<ffffffff8026c675>] xen_idle+0x38/0x4a
  [<ffffffff8024a9bb>] cpu_idle+0x97/0xba
  [<ffffffff805f6b09>] start_kernel+0x21f/0x224
  [<ffffffff805f61e5>] _sinittext+0x1e5/0x1eb

handlers:
[<ffffffff88142ff6>] (saa7134_irq+0x0/0x284 [saa7134])
Disabling IRQ #17

mythbe kernel: Disabling IRQ #17

I tried adding irqpoll to the relevent stanza in /etc/grub.conf but the 
error still occurs.

Is this likely that the interrupt handler for the card isn't being wired 
up by pcifront, or perhaps a stray interrupt generated by the card?

Re: MMIO ioremap() error with PCI passthrough

[Keir Fraser]

On 1/7/08 17:42, "Andy Burns" <lists.xensource.com@adslpipe.co.uk> wrote:

>  From my extra printk() debugging the driver now requests 1K, and within
> __direct_remap_pfn_range() the size has been rounded up to 4K,
> presumably this *is* correct as the mfn seems to has been rounded down?

Yes, you can only map memory at page granularity, so this is the smallest
mapping that contains the required resource. This is valid.

> I checked in 2.6.26 git to see if the saa7134 driver had been altered so
> the size had been reduced in more recent kernels, but it has not.  I'll
> check with the V4L developers whether it should be hardcoded to 1K
> instead of 4K, or whether it needs to be sized dynamically for different
> cards.

Good idea. This kind of thing can be an honest mistake which would normally
never cause a problem running natively.

> However about 40 seconds after loading the module, I get the following
> error :-(

Does the card work at all up to this point (first 40s)? Any sign of life
after the message prints?

This would indicate either that Xen/pciback has got irq routing wrong for
this line (so the wrong interrupts are being delivered to the driver) or
that for some other reason the driver has decided that there was no work to
be done in response to the interrupt. Could the irq line for this device be
a shared line? You could try disabling the spurious irq logic in
kernel/irq/spurious.c, and then see if the device works and also what the
irq rate looks like in /proc/interrupts.

 -- Keir

> irq 17: nobody cared (try booting with the "irqpoll" option)
> 
> Call Trace:
>   <IRQ>  [<ffffffff802af539>] __report_bad_irq+0x30/0x7d
>   [<ffffffff802af770>] note_interrupt+0x1ea/0x22b
>   [<ffffffff802aec76>] __do_IRQ+0xbd/0x103
>   [<ffffffff8028d06f>] _local_bh_enable+0x61/0xc5
>   [<ffffffff8026db3a>] do_IRQ+0xe7/0xf5
>   [<ffffffff8039f151>] evtchn_do_upcall+0x86/0xe0
>   [<ffffffff802608ce>] do_hypervisor_callback+0x1e/0x2c
>   <EOI>  [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
>   [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
>   [<ffffffff8026f12b>] raw_safe_halt+0x84/0xa8
>   [<ffffffff8026c675>] xen_idle+0x38/0x4a
>   [<ffffffff8024a9bb>] cpu_idle+0x97/0xba
>   [<ffffffff805f6b09>] start_kernel+0x21f/0x224
>   [<ffffffff805f61e5>] _sinittext+0x1e5/0x1eb
> 
> handlers:
> [<ffffffff88142ff6>] (saa7134_irq+0x0/0x284 [saa7134])
> Disabling IRQ #17
> 
> mythbe kernel: Disabling IRQ #17
> 
> I tried adding irqpoll to the relevent stanza in /etc/grub.conf but the
> error still occurs.
> 
> Is this likely that the interrupt handler for the card isn't being wired
> up by pcifront, or perhaps a stray interrupt generated by the card?

Re: MMIO ioremap() error with PCI passthrough

[Andy Burns]

On 01/07/2008 18:15, Keir Fraser wrote:

> Good idea. This kind of thing can be an honest mistake which would normally
> never cause a problem running natively.

I've asked the linuxtv folk if they are interested in a patch.

> Does the card work at all up to this point (first 40s)? 

Well, if I modprobe the saa7134.ko and saa7134_dvb.ko
I get the /dev/video0 and /dev/dvb/adapter0 devices created

But if I try to scan for TV channels with

# dvbscan /usr/share/dvb/dvb-t/uk-Waltham  > ~/.tzap/channels.conf

I get "Unable to query frontend status", so the device doesn't seem to 
be functioning.

> Any sign of life after the message prints?

The machine doesn't die when the stray interrupt fires, in fact I can 
rmmod the module and modprobe it again, the error repeats after another 
40 seconds.

> This would indicate either that Xen/pciback has got irq routing wrong for
> this line (so the wrong interrupts are being delivered to the driver) or
> that for some other reason the driver has decided that there was no work to
> be done in response to the interrupt. Could the irq line for this device be
> a shared line? 

I did wonder how (or if) shared interrupts would work.

Given the relatively high number of devices on the motherboard and 
PCI-X/PCI/PCIe cards I suspect it will have to be shared.

Here is a brief lspci

00:00.0 Host bridge: Intel Corporation 82X38 Express DRAM Controller 
(rev 01)
00:01.0 PCI bridge: Intel Corporation 82X38 Express Host-Primary PCI 
Express Bridge (rev 01)
00:1a.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
Controller #4 (rev 02)
00:1a.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
Controller #5 (rev 02)
00:1a.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
Controller #6 (rev 02)
00:1a.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI 
Controller #2 (rev 02)
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio 
Controller (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express 
Port 1 (rev 02)
00:1c.2 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express 
Port 3 (rev 02)
00:1c.3 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express 
Port 4 (rev 02)
00:1c.4 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express 
Port 5 (rev 02)
00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
Controller #3 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI 
Controller #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IR (ICH9R) LPC Interface 
Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 
port SATA AHCI Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller 
(rev 02)
01:00.0 VGA compatible controller: ATI Technologies Inc RV370 [Sapphire 
X550 Silent]
01:00.1 Display controller: ATI Technologies Inc RV370 secondary 
[Sapphire X550 Silent]
02:00.0 IDE interface: Marvell Technology Group Ltd. 88SE6145 SATA II 
PCI-E controller (rev a1)
03:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E 
Gigabit Ethernet Controller (rev 12)
04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E 
Gigabit Ethernet Controller (rev 12)
05:00.0 PCI bridge: NEC Corporation uPD720400 PCI Express - PCI/PCI-X 
Bridge (rev 06)
05:00.1 PCI bridge: NEC Corporation uPD720400 PCI Express - PCI/PCI-X 
Bridge (rev 06)
07:01.0 SCSI storage controller: Marvell Technology Group Ltd. 
MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
08:01.0 Multimedia controller: Philips Semiconductors SAA7130 Video 
Broadcast Decoder (rev 01)
08:03.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host 
Controller (rev c0)

Looks like the PCI slot with the tuner shares IRQ17 with the PCI-X slot 
used by my 8xSATA card.

07:01.0 SCSI storage controller: Marvell Technology Group Ltd. 
MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
         Subsystem: Marvell Technology Group Ltd. Unknown device 11ab
         Interrupt: pin A routed to IRQ 17

08:01.0 Multimedia controller: Philips Semiconductors SAA7130 Video 
Broadcast Decoder (rev 01)
         Subsystem: Compro Technology, Inc. Videomate DVB-T200
         Interrupt: pin A routed to IRQ 17

I can try to move the tuner card to the other PCI slot, but I suspect it 
will just share with something else instead, besides I planned to 
populate both PCI slots with tuners for passthrough, once I got the 
first one working.

> You could try disabling the spurious irq logic in
> kernel/irq/spurious.c, and then see if the device works and also what the
> irq rate looks like in /proc/interrupts.

I'll have a go at that.

Re: MMIO ioremap() error with PCI passthrough

[Andy Burns]

On 01/07/2008 19:50, Andy Burns wrote:

> I can try to move the tuner card to the other PCI slot, but I suspect it 
> will just share with something else instead

In the other PCI slot, the tuner shares with a USB controller and a SATA 
controller on the motherboard (instead of the PCI-X SATA card)

# lspci -vv| grep IRQ

         Interrupt: pin A routed to IRQ 16
         Interrupt: pin B routed to IRQ 21
         Interrupt: pin C routed to IRQ 18
         Interrupt: pin C routed to IRQ 18
         Interrupt: pin A routed to IRQ 22
         Interrupt: pin A routed to IRQ 23
         Interrupt: pin B routed to IRQ 19
         Interrupt: pin C routed to IRQ 18
         Interrupt: pin A routed to IRQ 23
         Interrupt: pin B routed to IRQ 22
         Interrupt: pin C routed to IRQ 18
         Interrupt: pin A routed to IRQ 11
         Interrupt: pin A routed to IRQ 16
         Interrupt: pin A routed to IRQ 19
         Interrupt: pin A routed to IRQ 18
         Interrupt: pin A routed to IRQ 17
         Interrupt: pin A routed to IRQ 16
         Interrupt: pin A routed to IRQ 15


# dmesg | grep -i apic
ACPI: APIC CFF80390, 0078 (r1 A_M_I_ OEMAPIC   4000828 MSFT       97)
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 4, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x05] address[0xfec10000] gsi_base[24])
IOAPIC[1]: apic_id 5, address 0xfec10000, GSI 24-279
Setting APIC routing to xen
ACPI: Using IOAPIC for interrupt routing

Is there any way to increase the number of "logical" interrupts to avoid 
(or give the impression of avoiding) sharing? When watching vmware 
servers boot, I remember they split devices into a huge number of 
interrupts numbering 100 upwards.

Can Xen make use of IRQ24 to 279?

Re: MMIO ioremap() error with PCI passthrough

[Keir Fraser]

On 1/7/08 20:24, "Andy Burns" <lists.xensource.com@adslpipe.co.uk> wrote:

> # dmesg | grep -i apic
> ACPI: APIC CFF80390, 0078 (r1 A_M_I_ OEMAPIC   4000828 MSFT       97)
> ACPI: Local APIC address 0xfee00000
> ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
> ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
> ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled)
> ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
> ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
> IOAPIC[0]: apic_id 4, address 0xfec00000, GSI 0-23
> ACPI: IOAPIC (id[0x05] address[0xfec10000] gsi_base[24])
> IOAPIC[1]: apic_id 5, address 0xfec10000, GSI 24-279
> Setting APIC routing to xen
> ACPI: Using IOAPIC for interrupt routing
> 
> Is there any way to increase the number of "logical" interrupts to avoid
> (or give the impression of avoiding) sharing? When watching vmware
> servers boot, I remember they split devices into a huge number of
> interrupts numbering 100 upwards.

It would be a lie. Interrupt sharing generally depends on which lines are
wired together on your mainboard. When Xen receives an interrupt on a shared
line it does not have enough information to demultiplex that interrupt to
the appropriate receiver -- it has to broadcast to all potential recipients.
Some of these may regard it as a spurious interrupt because their device did
not raise the interrupt. In our linux-2.6.18-xen tree we specifically handle
this case by disabling spurious irq detection for interrupts which Xen tells
us are shared.

> Can Xen make use of IRQ24 to 279?

I have no idea what IOAPIC[1] above really represents. Usually the GSI range
for an IOAPIC corresponds to its array of interrupt input pins. I've never
heard of a 256-pin IOAPIC!

Anyhow, the assignment of PCI devices to GSIs is determined by mainboard
wiring and firmware configuration. The OS (or hypervisor) does not determine
that relationship.

The best you could hope for would be to try out the MSI/MSI-X support in the
xen-unstable tree (soon to be Xen 3.3). These are message-based interrupts
which completely sidesteps the issue of interrupt lines and their wiring
together.

 -- Keir

Re: MMIO ioremap() error with PCI passthrough

[Andy Burns]

On 01/07/2008 20:57, Keir Fraser wrote:

> Interrupt sharing generally depends on which lines are
> wired together on your mainboard. When Xen receives an interrupt on a shared
> line it does not have enough information to demultiplex that interrupt to
> the appropriate receiver -- it has to broadcast to all potential recipients.

OK.

Sucking data at several Mb/s from a tv tuner, then squirting it to disk, 
where the tuner and disk are sharing the same interrupt doesn't sound 
ideal then ;-)

The SATA controller is presently performing an online mdraid reshape 
from 6 to 8 disks, averaging 6000 interrupts per second, so that's 
probably what is triggering the 99,900 "ignored" interrupts in 40 seconds!

> I have no idea what IOAPIC[1] above really represents. Usually the GSI range
> for an IOAPIC corresponds to its array of interrupt input pins. I've never
> heard of a 256-pin IOAPIC!

Could it relate to MSI?

> The best you could hope for would be to try out the MSI/MSI-X support in the
> xen-unstable tree (soon to be Xen 3.3). These are message-based interrupts
> which completely sidesteps the issue of interrupt lines and their wiring
> together.

Sounds good, but I gather it's only for PCIe or PCI2.3 devices, so my 
present tuners won't directly benefit, I suppose they might benefit 
indirectly by other devices using MSI and freeing up hardwired 
interrupts rather than being shared. Longer term I'll probably get a 
PCIe dual tuner.

Thanks for you help today, it'll likely be the weekend before I can look 
into this any further.

Re: MMIO ioremap() error with PCI passthrough

[Andy Burns]

On 01/07/2008 17:42, Andy Burns wrote:

> I tried adding irqpoll to the relevent stanza in /etc/grub.conf but the 
> error still occurs.

DOH!

I had added irqpoll to the domU when it should have been dom0.

Now I have irqpoll in the dom0, tuner interrupts are being sent to the 
domU, and more importantly no longer getting the "nobody cared" message.

# cat /proc/interrupts
            CPU0
  16:        440        Phys-irq  saa7130[0]
256:       6187     Dynamic-irq  timer0
257:          0     Dynamic-irq  resched0
258:          0     Dynamic-irq  callfunc0
259:        689     Dynamic-irq  xenbus
260:        296     Dynamic-irq  xencons
261:       6624     Dynamic-irq  blkif
262:         32     Dynamic-irq  blkif
263:         24     Dynamic-irq  blkif
264:        713     Dynamic-irq  eth0

Thinking about the IRQ sharing, the SATA controller was giving about 
6000 interrupts/second yesterday during expansion of a raid array, that 
is obviously quite exceptional, but sending so many unwanted interrupts 
via pciback/front is not a good thing, I'll have to do some tests later 
to see if it's a disastrously bad thing, for now I've moved the PCI card 
into the slot with less shared interrupts.

dvbscan still won't talk to the device but femon does, my second tuner 
is currently in a different machine, so I can compare and contrast.

# femon -h

FE: Philips TDA10046H DVB-T (DVBT)
status       | signal   0% | snr 100% | ber 0001fffe | unc 00000000 |

I connected the aerial to it, and despite what dvbscan/femon said, I 
thought I'd try a channel scan with mythtv anyway and it found one mux 
and has started populating the EPG, it should find several muxes (it's 
fed from the same amplifier as my other machine/tv) but I'll class that 
as "initially working" :-)

Re: MMIO ioremap() error with PCI passthrough

[Andy Burns]

On 02/07/2008 10:35, Andy Burns wrote:

> I connected the aerial to it, it found one mux 
> and has started populating the EPG, it should find several muxes (it's 
> fed from the same amplifier as my other machine/tv)

Turns out the aerial outlet in the study had been disconnected and it 
was only picking up stray signals, a quick trip into the attic and a bit 
of re-wiring and myth is firing on all cylinders under Xen.

All muxes tuned OK, and recording 3 concurrent programs from the single 
tuner (which results in about 2Mbytes of data being read from the tuner 
and written to disk) eats less than 10% CPU in domU and 0.5% in dom0, 
about 130 interrupts/sec from the tuner.

Fantastic! Thanks again for your help, looking forward to Xen3.3

Re: MMIO ioremap() error with PCI passthrough

[Andy Burns]

On 02/07/2008 13:54, Andy Burns wrote:

> All muxes tuned OK, and recording 3 concurrent programs from the single 
> tuner (which results in about 2Mbytes of data being read from the tuner 
> and written to disk) eats less than 10% CPU in domU and 0.5% in dom0, 
> about 130 interrupts/sec from the tuner.

After about an hour I got a kernel panic on the domU

Fatal DMA error! Please use 'swiotlb=force'
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at arch/x86_64/kernel/../../i386/kernel/pci-dma-xen.c:165
invalid opcode: 0000 [1] SMP
last sysfs file: /block/dm-1/range
CPU 0
Modules linked in: xfs(U) saa7134_dvb(U) dvb_pll(U) mt352(U) 
video_buf_dvb(U) dvb_core(U) nxt200x(U) tda1004x(U) autofs4(U) sunrpc(U) 
xennet(U) ip6t_REJECT(U) xt_tcpudp(U) ip6table_filter(U) ip6_tables(U) 
x_tables(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) dm_multipath(U) 
parport_pc(U) lp(U) parport(U) saa7134(U) video_buf(U) compat_ioctl32(U) 
ir_kbd_i2c(U) i2c_core(U) ir_common(U) videodev(U) pcspkr(U) 
v4l1_compat(U) v4l2_common(U) dm_snapshot(U) dm_zero(U) dm_mirror(U) 
dm_mod(U) xenblk(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U)
Pid: 10678, comm: saa7130[0] dvb Tainted: G      2.6.18-prep #6
RIP: e030:[<ffffffff802720a2>]  [<ffffffff802720a2>] dma_map_sg+0x13f/0x1ae
RSP: e02b:ffff88002ea6be00  EFLAGS: 00010282
RAX: 000000000000002f RBX: ffff8800305c7df0 RCX: ffffffff804da728
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: 0000000000000003 R08: ffffffff804da728 R09: 0000000000001f5f
R10: 000000000000002c R11: ffff88002ea6c000 R12: 0000000000000006
R13: ffff88003fded070 R14: ffff88003df7ede8 R15: 0000000000000003
FS:  00002b787a63a560(0000) GS:ffffffff805ac000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000
Process saa7130[0] dvb (pid: 10678, threadinfo ffff88002ea6a000, task 
ffff880035e32860)
Stack:  ffff880035e6a0f0  ffff880035e6a0f0  ffff88003df7ede8 
ffff88003fded000
  0000000000000080  ffffffff88133937  ffff880035e6a0c0  0000000000005e00
  ffff88003df7e000  ffffffff88144cf0
Call Trace:
  [<ffffffff88133937>] :video_buf:videobuf_dma_map+0x115/0x159
  [<ffffffff88144cf0>] :saa7134:buffer_prepare+0xbb/0x19b
  [<ffffffff80298a84>] keventd_create_kthread+0x0/0xc4
  [<ffffffff88132d43>] :video_buf:videobuf_read_start+0xa8/0x139
  [<ffffffff882f034b>] :video_buf_dvb:videobuf_dvb_thread+0x2a/0x127
  [<ffffffff882f0321>] :video_buf_dvb:videobuf_dvb_thread+0x0/0x127
  [<ffffffff802339c8>] kthread+0xfe/0x132
  [<ffffffff80260b24>] child_rip+0xa/0x12
  [<ffffffff80298a84>] keventd_create_kthread+0x0/0xc4
  [<ffffffff802338ca>] kthread+0x0/0x132
  [<ffffffff80260b1a>] child_rip+0x0/0x12

Code: 0f 0b 68 a5 df 47 80 c2 a5 00 48 8b 7b f0 48 2b 3d 21 f1 40
RIP  [<ffffffff802720a2>] dma_map_sg+0x13f/0x1ae
  RSP <ffff88002ea6be00>
  <0>Kernel panic - not syncing: Fatal exception




I presume this kernel option *is* a domU option rather than dom0? when I 
  added it to grub, the domU crashes on boot ...

----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at lib/../arch/i386/kernel/swiotlb.c:160
invalid opcode: 0000 [1] SMP
last sysfs file:
CPU 0
Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.18-prep #6
RIP: e030:[<ffffffff8033ca50>]  [<ffffffff8033ca50>] 
swiotlb_init_with_default_size+0xa0/0x19e
RSP: e02b:ffffffff805edf40  EFLAGS: 00010282
RAX: 00000000fffffff4 RBX: 0000000000000900 RCX: 000000000021d106
RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000004000000 R08: 0000000000001000 R09: ffffffff8064d9e0
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffffff805ac000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000
Process swapper (pid: 0, threadinfo ffffffff805ec000, task ffffffff804d5a80)
Stack:  00000000000002c3  0000000002040800  0000000000000000 
ffffffff80276935
  0000000000000000  ffffffff8060051b  0000000000000000  0000000002040800
  0000000000000000  0000000000000000
Call Trace:
  [<ffffffff80276935>] pci_swiotlb_init+0x9/0x2d
  [<ffffffff8060051b>] mem_init+0x5e/0x233
  [<ffffffff805f6a73>] start_kernel+0x189/0x224
  [<ffffffff805f61e5>] _sinittext+0x1e5/0x1eb

Code: 0f 0b 68 26 d3 48 80 c2 a0 00 48 83 eb 80 48 8b 05 23 cf 34
RIP  [<ffffffff8033ca50>] swiotlb_init_with_default_size+0xa0/0x19e
  RSP <ffffffff805edf40>
  <0>Kernel panic - not syncing: Fatal exception

Re: MMIO ioremap() error with PCI passthrough

[Keir Fraser]

On 1/7/08 16:44, "Andy Burns" <lists.xensource.com@adslpipe.co.uk> wrote:

> Perhaps the rounding up to 8K is happening because the address isn't
> aligned on a page boundary, so is split over two pages?

Yes, since the argument to ioremap() is 0x1000, the underlying ioremap
implementation is doing the right thing.

> So I assume it is the driver at fault for mapping a larger resource area
> than the card actually supports?
> 
> I'll try changing the size argument to 0x400 and rebuilding the saa7134
> module.

Yes, it's the driver's fault. 0x400 would be correct.

>> So, could be a bug specific to the FC8 kernel.
> 
> Actually centos5.2 is the domU, FC8 is the dom0, since it's x86_64, I
> assume it's the Fedora/Redhat forward-port effort, rather than the
> paravirt_ops effort.

Yes, it will be the FC/RH forward-port. Anyhow, the bug isn't in the generic
ioremap code after all.

 -- Keir

Re: MMIO ioremap() error with PCI passthrough

[Jeremy Fitzhardinge]

Keir Fraser wrote:
> On 1/7/08 14:16, "Andy Burns" <lists.xensource.com@adslpipe.co.uk> wrote:
>
>   
>>> The functions look like they should map the correct range of pages (in this
>>> case only the page covering FEBFF000-FEBFFFFF)
>>>       
>> Built and installed kernel/modules/initrd, rebooted, when it loads the
>> saa7134 driver I see this
>>
>> REMAP: phys=0xfebffc00, len=4096
>> REMAPPFN: addr=0xffffc20000038000, mfn=0xfebff, size=8192
>>
>> So the 1K mapping has been rounded up to 4K (x86_64 page size?) before
>> passing to __ioremap() and then rounded up again to 8K by the time it
>> gets passed to __direct_remap_pfn_range() is that right?
>>     
>
> Well, your analysis is correct, and the size argument to __ioremap() is
> bogus. It shouldn't have been rounded up to 4096 without also rounding down
> the base address. I don't think this would happen with out linux-2.6.18-xen
> tree. In there, ioremap() is defined in include/asm-x86_64/mach-xen/io.h as
> a thin wrapper around __ioremap() which does not modify the size parameter.
>
> So, could be a bug specific to the FC8 kernel. I don't have its sources to
> hand to pinpoint where the size is getting changed from 0x400 to 0x1000. You
> should be able to dig up that detail pretty easily though.

There was a kernel bug in which ioremap would add a 1 page redzone to 
the end of the mapping, and then pass that extra page to iounmap.  It 
has been fixed for a while, but since F8.

    J

Re: MMIO ioremap() error with PCI passthrough

[Andy Burns]

On 01/07/2008 09:30, Keir Fraser wrote:

> What does lspci say the resource areas are for this device? 

Within the dom0, after I've unloaded the driver and re-assigned the 
slot, I see this

08:01.0 Multimedia controller: Philips Semiconductors SAA7130 Video 
Broadcast Decoder (rev 01)
         Subsystem: Compro Technology, Inc. Videomate DVB-T200
         Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- 
ParErr- Stepping- SERR- FastB2B- DisINTx-
         Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium 
 >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
         Interrupt: pin A routed to IRQ 17
         Region 0: Memory at febffc00 (32-bit, non-prefetchable) 
[disabled] [size=1K]
         Capabilities: [40] Power Management version 1
                 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA 
PME(D0-,D1-,D2-,D3hot-,D3cold-)
                 Status: D0 PME-Enable- DSel=0 DScale=1 PME-
         Kernel driver in use: pciback
         Kernel modules: saa7134