PCI passthrough stopped working, brainache!

PCI passthrough stopped working, brainache!

[Andy Burns]

Recently had passthrough of 2xPCI DVB-T cards and 1xPCIe DVB-S2 card working,
the last know config that was *certainly* working was

dom0
xen-4.1.1-3.fc16.x86_64
kernel-3.1.0-0.rc7.git0.0.fc16.x86_64

domU
kernel-3.1.0-0.rc8.git0.0.fc16.x86_64

Since then I've updated

xen-4.1.1-6.fc16.x86_64 on dom0

kernel-3.1.0-0.rc9.git0.0.fc16.x86_64 on dom0 and domU

and updated all other packages to current F16 updates-testing,
also lots of fiddling with grub2 and systemd on the domU

Only today did I realise that only the PCIe card is now working, not
the PCI ones, and have since spent several hours trying to get back to
a working configuration :-(

I've rolled my xen packages back from 4.1.1-6 to 4.1.1-3
tried booting various combinations of 3.1.0-rc7/rc8/rc9 as dom0 and domU

made sure I still have "pci=resource_alignment" for all the relevant
PCI devices on the dom0
made sure I still have "iommu=soft" on the domU
made sure pci-back is happy with "xm pci-list-assignable-devices"
made sure devices really have been assigned with "xm pci-list mythf16"
made sure the devices and drivers show in the domU with "lspci",
"lsusb" and "lsmod"
checked "xm dmesg" on dom0 and "dmesg" on dom0 and domU that drivers
see the hardware and load firmware OK
scandvb goes through the motions of tuning, but finds no stations,
this  *feels* as though the issue is lack of DMA transfers.

How can I tell if the iommu=soft is taking effect?
Anything stupid I sound like I've forgotten?

Re: PCI passthrough stopped working, brainache!

[Andy Burns]

[Resent from subscribed address]

Recently had passthrough of 2xPCI DVB-T cards and 1xPCIe DVB-S2 card working,
the last know config that was *certainly* working was

dom0
xen-4.1.1-3.fc16.x86_64
kernel-3.1.0-0.rc7.git0.0.fc16.x86_64

domU
kernel-3.1.0-0.rc8.git0.0.fc16.x86_64

Since then I've updated

xen-4.1.1-6.fc16.x86_64 on dom0

kernel-3.1.0-0.rc9.git0.0.fc16.x86_64 on dom0 and domU

and updated all other packages to current F16 updates-testing,
also lots of fiddling with grub2 and systemd on the domU

Only today did I realise that only the PCIe card is now working, not
the PCI ones, and have since spent several hours trying to get back to
a working configuration :-(

I've rolled my xen packages back from 4.1.1-6 to 4.1.1-3
tried booting various combinations of 3.1.0-rc7/rc8/rc9 as dom0 and domU

made sure I still have "pci=resource_alignment" for all the relevant
PCI devices on the dom0
made sure I still have "iommu=soft" on the domU
made sure pci-back is happy with "xm pci-list-assignable-devices"
made sure devices really have been assigned with "xm pci-list mythf16"
made sure the devices and drivers show in the domU with "lspci",
"lsusb" and "lsmod"
checked "xm dmesg" on dom0 and "dmesg" on dom0 and domU that drivers
see the hardware and load firmware OK
scandvb goes through the motions of tuning, but finds no stations,
this  *feels* as though the issue is lack of DMA transfers.

How can I tell if the iommu=soft is taking effect?
Anything stupid I sound like I've forgotten?

Re: Re: PCI passthrough stopped working, brainache!

[Ian Campbell]

On Mon, 2011-10-10 at 23:02 +0100, Andy Burns wrote:
> [Resent from subscribed address]
> 
> Recently had passthrough of 2xPCI DVB-T cards and 1xPCIe DVB-S2 card working,
> the last know config that was *certainly* working was
> 
> dom0
> xen-4.1.1-3.fc16.x86_64
> kernel-3.1.0-0.rc7.git0.0.fc16.x86_64
> 
> domU
> kernel-3.1.0-0.rc8.git0.0.fc16.x86_64
> 
> Since then I've updated
> 
> xen-4.1.1-6.fc16.x86_64 on dom0
> 
> kernel-3.1.0-0.rc9.git0.0.fc16.x86_64 on dom0 and domU
> 
> and updated all other packages to current F16 updates-testing,
> also lots of fiddling with grub2 and systemd on the domU
> 
> Only today did I realise that only the PCIe card is now working, not
> the PCI ones, and have since spent several hours trying to get back to
> a working configuration :-(
> 
> I've rolled my xen packages back from 4.1.1-6 to 4.1.1-3
> tried booting various combinations of 3.1.0-rc7/rc8/rc9 as dom0 and domU
> 
> made sure I still have "pci=resource_alignment" for all the relevant
> PCI devices on the dom0
> made sure I still have "iommu=soft" on the domU
> made sure pci-back is happy with "xm pci-list-assignable-devices"
> made sure devices really have been assigned with "xm pci-list mythf16"
> made sure the devices and drivers show in the domU with "lspci",
> "lsusb" and "lsmod"
> checked "xm dmesg" on dom0 and "dmesg" on dom0 and domU that drivers
> see the hardware and load firmware OK
> scandvb goes through the motions of tuning, but finds no stations,
> this  *feels* as though the issue is lack of DMA transfers.
> 
> How can I tell if the iommu=soft is taking effect?
> Anything stupid I sound like I've forgotten?

It looks as if you have been pretty thorough. 

One thing which is not clear is if you also downgraded all the Xen
utilities/tools packages when you say "Xen packages" or just the
hypervisor itself. (You probably downgraded everything, but I have to
ask).

Does Fedora (yum?) log which packages which it is upgrading? Is it
possible that e.g. scandvb or the firmware package has also been
upgraded (I admit this is slightly straw-clutching).

If you don't passthrough a device are you able to drive it from dom0?

It would probably be useful to post full dmesg output for hypervisor and
both kernels.

Ian.

Re: Re: PCI passthrough stopped working, brainache!

[Konrad Rzeszutek Wilk]

> > How can I tell if the iommu=soft is taking effect?
> > Anything stupid I sound like I've forgotten?
> 
> It looks as if you have been pretty thorough. 
> 
> One thing which is not clear is if you also downgraded all the Xen
> utilities/tools packages when you say "Xen packages" or just the
> hypervisor itself. (You probably downgraded everything, but I have to
> ask).
> 
> Does Fedora (yum?) log which packages which it is upgrading? Is it
> possible that e.g. scandvb or the firmware package has also been
> upgraded (I admit this is slightly straw-clutching).
> 
> If you don't passthrough a device are you able to drive it from dom0?
> 
> It would probably be useful to post full dmesg output for hypervisor and
> both kernels.

Andy,

I am going to try to see if I can pass in some of my TV cards in
my guest too. Perhaps that might shed some light.

Re: PCI passthrough stopped working, brainache!

[Andy Burns]

On 10 October 2011 20:45, Andy Burns <burns.me.uk@gmail.com> wrote:

> Anything stupid I sound like I've forgotten?

Well, I have it partly working again, I've not tied it down precisely
yet (because I'm trying multiple things at once of course!) but I
think it's either ...

Using "com1=115200,8n1 console=com1" on the hypervisor command with
"console=hvc0" on the dom0 kernel command
which I think would be a WTF? if that turns out to be the cause.

or

It works with only two PCI cards passed at once, which could be an
interrupt sharing issue or pciback issue I suppose.

or

it depends on the order of unbinding/binding devices from
     sys/bus/pci/drivers/ohci_hcd
     sys/bus/pci/drivers/ehci_hcd
     sys/bus/pci/drivers/pciback

I'll report back as I discover more ...

Re: PCI passthrough stopped working, brainache!

[Andy Burns]

On 11 October 2011 22:25, Andy Burns <xen.lists@burns.me.uk> wrote:

> Well, I have it partly working again
> I think it's either ...
>
> Using "com1=115200,8n1 console=com1" on the hypervisor command with
> "console=hvc0" on the dom0 kernel command
> which I think would be a WTF? if that turns out to be the cause.

And the winner is  .......... WTF?

I'd enabled serial console for catching some crashers around the
3.1.0-rc5 and -rc6 releases, once Konrad's fixes made it upstream I'd
left the console serial enabled with -rc7 and -rc8, but I reverted to
VGA console with -rc9 as I was no longer seeing any crashers!

The motherboard is an X38 chipset, but doesn't have VT-d due to lack
of BIOS support (thanks ASUS) so there's no VGA passthrough going on.

Any educated guesses as to what's the cause?

Re: Re: PCI passthrough stopped working, brainache!

[Konrad Rzeszutek Wilk]

On Tue, Oct 11, 2011 at 10:49:37PM +0100, Andy Burns wrote:
> On 11 October 2011 22:25, Andy Burns <xen.lists@burns.me.uk> wrote:
> 
> > Well, I have it partly working again
> > I think it's either ...
> >
> > Using "com1=115200,8n1 console=com1" on the hypervisor command with
> > "console=hvc0" on the dom0 kernel command
> > which I think would be a WTF? if that turns out to be the cause.
> 
> And the winner is  .......... WTF?
> 
> I'd enabled serial console for catching some crashers around the
> 3.1.0-rc5 and -rc6 releases, once Konrad's fixes made it upstream I'd
> left the console serial enabled with -rc7 and -rc8, but I reverted to
> VGA console with -rc9 as I was no longer seeing any crashers!
> 
> The motherboard is an X38 chipset, but doesn't have VT-d due to lack
> of BIOS support (thanks ASUS) so there's no VGA passthrough going on.
> 
> Any educated guesses as to what's the cause?

So it works now right? Did you turn off the machine in between your
reboots - when you went back from CentOS to F16?

(Yes, I did encounter a bug where it would program an hardware interface
that would retain its state after a reboot and cause the newly built kernel
to go wrongly. Spent couple of good days trying to narrow that one down).

Re: Re: PCI passthrough stopped working, brainache!

[Andy Burns]

Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> So it works now right?

No.  When I went to bed last night I was convinced it was working,
having seen it work three times in a row where it hadn't at all for
the past 24 hours, albeit not knowing why serial console should make
it work.

I left the machines updating to latest Fedora16 branch, but this
morning it's not working again.

> Did you turn off the machine in between your
> reboots - when you went back from CentOS to F16?

I've not booted CentOS for a few weeks now, machine has certainly been
physically powered off since then,both when working and when faiing.

I do know that for the test above where I posted dmesg results as
requested by Ian, the machine was physically powered off and
passthrough  failed.

I didn't turn the machine physically off for every test, sometimes I
rebooted it from within the dom0, sometimes powered it down to S5
state (which doesn't always work) and re-woke with WOL, sometimes it
hung and  I used the reset button, sometines it hung and I'd press and
hold the power button.

> I did encounter a bug where it would program an hardware interface
> that would retain its state after a reboot and cause the newly built kernel
> to go wrongly. Spent couple of good days trying to narrow that one down

I'll make sure I always physically turn it off until I find the true
cause, back to older  hypervisor and kernel packages again for more
tests ...

I think I remember what was the test I did before the boot that
started working again , but honestly can't remember whether I rebooted
or powered off after that.

Re: Re: PCI passthrough stopped working, brainache!

[Andy Burns]

On 12 October 2011 09:01, Andy Burns <xen.lists@burns.me.uk> wrote:

> Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
>
>> So it works now right?
>
> No.
>
> I think I remember what was the test I did before the boot that
> started working again

I didn't get much time to test this today ... reverted to same Xen and
kernel versions that worked briefly last night.

I discovered that (despite what I answered earlier) the PCI tuners
don't work in dom0 under Xen, they only work if the dom0 is booted  as
baremetal.

If I reboot from dom0 from baremetal with the PCI cards working into
Xen without powering off, it doesn't "magically" leave the PCI cards
in a state that allows them to work in the domU.

The thing which *seemed* to put it into a good mood last night was
booting dom0 with serial console and the domU with the PIC cards but
without the PCIe card, but that made no difference today.

I'm beginning to follow Konrad's thoughts that there is a specific
sequence of events, that persists in hardware state across soft
reboots, occasionally ending up with functioning PCI cards.

Is the fact that the PCI cards fail in dom0 under Xen a hint?  Any
debugging I can do with the tuners from the dom0 rather than the domU
with passthrough?

Re: Re: PCI passthrough stopped working, brainache!

[Konrad Rzeszutek Wilk]

On Wed, Oct 12, 2011 at 10:36:12PM +0100, Andy Burns wrote:
> On 12 October 2011 09:01, Andy Burns <xen.lists@burns.me.uk> wrote:
> 
> > Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> >
> >> So it works now right?
> >
> > No.

<Grumble>
> >
> > I think I remember what was the test I did before the boot that
> > started working again
> 
> I didn't get much time to test this today ... reverted to same Xen and
> kernel versions that worked briefly last night.
> 
> I discovered that (despite what I answered earlier) the PCI tuners
> don't work in dom0 under Xen, they only work if the dom0 is booted  as
> baremetal.
> 
> If I reboot from dom0 from baremetal with the PCI cards working into
> Xen without powering off, it doesn't "magically" leave the PCI cards
> in a state that allows them to work in the domU.
> 
> The thing which *seemed* to put it into a good mood last night was
> booting dom0 with serial console and the domU with the PIC cards but
> without the PCIe card, but that made no difference today.
> 
> I'm beginning to follow Konrad's thoughts that there is a specific
> sequence of events, that persists in hardware state across soft
> reboots, occasionally ending up with functioning PCI cards.
> 
> Is the fact that the PCI cards fail in dom0 under Xen a hint?  Any
> debugging I can do with the tuners from the dom0 rather than the domU
> with passthrough?

That is. That would imply it is not the PCI passthrough code (good!).
It is something related to the driver (as I presume your network card
works in that box). Perhaps it is the VM_IO bug that sometimes creeps
up.. Can you give me the lsmod output please? I want to see which
drivers are loaded for this TV card and I can dig a bit in the
driver to see if there is something fishy.

I saw something about I2C, is there a knob in the driver to _not_
use I2C?

Re: Re: PCI passthrough stopped working, brainache!

[Andy Burns]

Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> Andy Burns wrote:
>
>> Is the fact that the PCI cards fail in dom0 under Xen a hint?
>
> That is. That would imply it is not the PCI passthrough code (good!).
> It is something related to the driver (as I presume your network card
> works in that box).

Yes, two onboard sky2 gigabit ports and a supermicro 64bit PCI-X JBOD
card with 8 SATA disks

> Perhaps it is the VM_IO bug that sometimes creeps
> up.. Can you give me the lsmod output please? I want to see which
> drivers are loaded for this TV card and I can dig a bit in the
> driver to see if there is something fishy.

Here is an lsmod (from within the domU, but the same modules get used
if it's in dom0 and same kernel 3.1.0-rc9 everywhere now.

Module                  Size  Used by
ds3000                 12827  2
dvb_usb_dw2102         41753  27
dvb_usb                14988  1 dvb_usb_dw2102
lockd                  70080  0
ip6t_REJECT             3992  2
nf_conntrack_ipv6       7730  2
nf_defrag_ipv6          9083  1 nf_conntrack_ipv6
xt_state                1306  2
nf_conntrack           67597  2 nf_conntrack_ipv6,xt_state
ip6table_filter         1655  1
ip6_tables             16792  1 ip6table_filter
tda1004x               14722  2
saa7134_dvb            27032  12
videobuf_dvb            5146  1 saa7134_dvb
dvb_core               87211  2 dvb_usb,videobuf_dvb
firewire_ohci          26101  0
ir_lirc_codec           4214  0
lirc_dev               12904  1 ir_lirc_codec
firewire_core          49303  1 firewire_ohci
ir_mce_kbd_decoder      4208  0
ir_sony_decoder         2109  0
ir_jvc_decoder          2218  0
ir_rc6_decoder          2682  0
ir_rc5_decoder          2138  0
rc_videomate_tv_pvr     1289  0
ir_nec_decoder          2570  0
saa7134               159679  1 saa7134_dvb
crc_itu_t               1547  1 firewire_core
rc_core                17136  12
dvb_usb,ir_lirc_codec,ir_mce_kbd_decoder,ir_sony_decoder,ir_jvc_decoder,ir_rc6_decoder,ir_rc5_decoder,rc_videomate_tv_pvr,ir_nec_decoder,saa7134
videobuf_dma_sg         8462  2 saa7134_dvb,saa7134
videobuf_core          15780  3 videobuf_dvb,saa7134,videobuf_dma_sg
v4l2_common             6905  1 saa7134
videodev               78689  2 saa7134,v4l2_common
media                  11511  1 videodev
v4l2_compat_ioctl32     7665  1 videodev
tveeprom               13045  1 saa7134
i2c_core               25728  9
ds3000,dvb_usb_dw2102,dvb_usb,tda1004x,saa7134_dvb,saa7134,v4l2_common,videodev,tveeprom
sunrpc                200831  2 lockd
xen_pcifront           12182  0
xen_netfront           16358  0
xen_blkfront           12741  4

Specifically the tuner drivers are

ds3000  and  dvb_usb_dw2102  for the DVB-S2 PCIe that works

tveeprom,  saa7134,  saa7134_dvb   and  tda1004x  for the troublesome
DVB-T PCI cards

> I saw something about I2C, is there a knob in the driver to _not_
> use I2C?

I can certainly live without all the LIRC stuff by blacklisting all
the ir_xxx_decoder stuff, I'll check modinfo of the drivers to see
what options exist.

Re: Re: PCI passthrough stopped working, brainache!

[Andy Burns]

Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> would imply it is not the PCI passthrough code (good!).
> It is something related to the driver
> Perhaps it is the VM_IO bug that sometimes creeps
> up..

I can try *NOT* using pci=resource_realignment=(blah), but I always
had to do that (or rather the old reassigndev equivalent) under
xenified 2.6.18 kernels as the BARs of the PCI cards were smaller than
the page size and I think the drivers for the two instances of the
DVB-T card used to trample all over each other without it.

Re: Re: PCI passthrough stopped working, brainache!

[Konrad Rzeszutek Wilk]

On Thu, Oct 13, 2011 at 09:28:32PM +0100, Andy Burns wrote:
> Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> 
> > would imply it is not the PCI passthrough code (good!).
> > It is something related to the driver
> > Perhaps it is the VM_IO bug that sometimes creeps
> > up..
> 
> I can try *NOT* using pci=resource_realignment=(blah), but I always
> had to do that (or rather the old reassigndev equivalent) under
> xenified 2.6.18 kernels as the BARs of the PCI cards were smaller than
> the page size and I think the drivers for the two instances of the
> DVB-T card used to trample all over each other without it.

Is there a good application you use to test this? xawtv?

Re: Re: PCI passthrough stopped working, brainache!

[Andy Burns]

On 14 October 2011 16:55, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> Is there a good application you use to test this? xawtv?

Yeah, mythtv *is* a bit too heavy for testing :-P

 FOr quick tests I just use scandvb (sometimes known as "dvbscan" or
"scan" depending on distro) but it needs some "seed" information
that's geographically sensitive to start searching for stations e.g.
for me ...

scandvb -a0 /usr/share/dvb-apps/dvb-t/uk-Waltham

I've no idea where you're located and whether you have
DVB-T/DVB-S/ATSC tuners ...

if your distro provides it then "w_scan" is easier as it will go off
and do a brute force scan by itself, you just need to tell it if
you're using terrestrial or satellite, and prefereably a hint to which
country and off it goes ...

Re: Re: PCI passthrough stopped working, brainache!

[Andy Burns]

On 13 October 2011 19:15, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> On Wed, Oct 12, 2011 at 10:36:12PM +0100, Andy Burns wrote:
>
>> I discovered that the PCI tuners don't work in dom0 under Xen,
>> they only work if the dom0 is booted  as baremetal.
>
> That would imply it is not the PCI passthrough code (good!).
> It is something related to the driver

I've been having a go at getting the card running in dom0 rather than
domU, no more success yet.

I've stopped using the PCI resource realignment for now, and enabled
debugging knobs in the tuner modules, seems the I2C transfers used to
program the tuner and receive status bits back from it are working OK,
I can see the driver queueing up DMA using a succession of transfer
buffers, and apparently interrupts signalling their completion.

Presume it's correct that within the dom0, the buffers should *NOT* be
within the SWIOTLB range?

# dmesg | grep ffff8800

[    0.000000] found SMP MP-table at [ffff8800000ff780] ff780
[    0.000000] Base memory trampoline at [ffff880000097000] 97000 size 20480
[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff88001fe43000 s81024
r8192 d21376 u110592
[    0.000000] Placing 64MB software IO TLB between ffff880015c00000 -
ffff880019c00000

... snip ...

[ 1539.146723] saa7130[0]/ts: buffer_activate [ffff88000c25ca00]
[ 1539.146726] saa7130[0]/ts: - [top]     buf=ffff88000c25ca00
next=ffff88000959c800
[ 1539.146737] saa7130[0]/core: buffer_next #2
prev=ffff8800041f8440/next=ffff88000959c840
[ 1539.146790] saa7130[0]/core: buffer_queue ffff88000c25c000
[ 1539.154693] saa7130[0]/core: buffer_finish ffff88000c25ca00
[ 1539.154698] saa7130[0]/core: buffer_next ffff88000959c800
[prev=ffff88000c25c040/next=ffff88000959c840]
[ 1539.154702] saa7130[0]/ts: buffer_activate [ffff88000959c800]
[ 1539.154705] saa7130[0]/ts: - [bottom]  buf=ffff88000959c800
next=ffff88000959c600
[ 1539.154716] saa7130[0]/core: buffer_next #2
prev=ffff88000c25c040/next=ffff88000959c640
[ 1540.156114] saa7130[0]/core: timeout on ffff88000959c800
[ 1540.156118] saa7130[0]/core: buffer_finish ffff88000959c800

I assume no meaningful transport stream contents are contained in the
buffers following the transfers, as no stations are found.

Hypervisor, dom0 (and domU when I was using it) are all 64bit, the
tuner device is 32bit, is this likely to be an issue with DMA
transfers? Any extra logging for Xen?

09:00.0 Multimedia controller: Philips Semiconductors SAA7130 Video
Broadcast Decoder (rev 01)
        Subsystem: Compro Technology, Inc. Videomate DVB-T200
        Flags: bus master, medium devsel, latency 64, IRQ 16
        Memory at febffc00 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [40] Power Management version 1
        Kernel driver in use: saa7134
        Kernel modules: saa7134

09:01.0 Multimedia controller: Philips Semiconductors SAA7130 Video
Broadcast Decoder (rev 01)
        Subsystem: Compro Technology, Inc. Videomate DVB-T200
        Flags: bus master, medium devsel, latency 64, IRQ 17
        Memory at febff800 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [40] Power Management version 1
        Kernel driver in use: saa7134
        Kernel modules: saa7134

> Perhaps it is the VM_IO bug that sometimes creeps up..

Any pointers to previous cases of that bug, or the methods to discover it?

> I saw something about I2C, is there a knob in the driver to _not_ use I2C?

Not for my tuner, I've added modprobe options to blacklist devices
that I don't need and turn on debugging for the tuner modules

# cat /etc/modprobe.d/dvb.conf

blacklist ds3000
blacklist dvb_usb_dw2102
blacklist firewire_ehci
blacklist firewire_ohci
options saa7134 disable_ir=1 video_debug=1 ts_debug=1 i2c_debug=1
i2c_scan=1 irq_debug=1 core_debug=1 gpio_tracking=1
options saa7134-dvb debug=1
options tda1004x debug=1

# lsmod
Module                  Size  Used by
lockd                  70080  0
bridge                 72368  0
stp                     1927  1 bridge
llc                     4738  2 bridge,stp
ip6t_REJECT             3992  2
nf_conntrack_ipv6       7730  2
nf_defrag_ipv6          9083  1 nf_conntrack_ipv6
xt_state                1306  2
nf_conntrack           67597  2 nf_conntrack_ipv6,xt_state
ip6table_filter         1655  1
ip6_tables             16792  1 ip6table_filter
snd_hda_codec_realtek   312621  1
raid456                54497  1
async_raid6_recov       5358  1 raid456
async_pq                4339  2 raid456,async_raid6_recov
raid6_pq               78299  2 async_raid6_recov,async_pq
async_xor               3255  3 raid456,async_raid6_recov,async_pq
xor                     4793  1 async_xor
async_memcpy            1845  2 raid456,async_raid6_recov
async_tx                2702  5
raid456,async_raid6_recov,async_pq,async_xor,async_memcpy
tda1004x               14722  2
snd_hda_intel          24072  0
snd_hda_codec          85181  2 snd_hda_codec_realtek,snd_hda_intel
snd_hwdep               6264  1 snd_hda_codec
saa7134_dvb            27032  0
videobuf_dvb            5146  1 saa7134_dvb
dvb_core               87211  1 videobuf_dvb
snd_seq                52186  0
snd_seq_device          5941  1 snd_seq
iTCO_wdt               12024  0
snd_pcm                78514  2 snd_hda_intel,snd_hda_codec
saa7134               159679  1 saa7134_dvb
rc_core                17136  1 saa7134
videobuf_dma_sg         8462  2 saa7134_dvb,saa7134
videobuf_core          15780  3 videobuf_dvb,saa7134,videobuf_dma_sg
v4l2_common             6905  1 saa7134
videodev               78689  2 saa7134,v4l2_common
media                  11511  1 videodev
v4l2_compat_ioctl32     7665  1 videodev
shpchp                 24554  0
i2c_i801                9237  0
serio_raw               4298  0
iTCO_vendor_support     2578  1 iTCO_wdt
sky2                   42923  0
snd_timer              19372  2 snd_seq,snd_pcm
snd                    63124  8
snd_hda_codec_realtek,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_seq,snd_seq_device,snd_pcm,snd_timer
soundcore               6267  1 snd
snd_page_alloc          7311  2 snd_hda_intel,snd_pcm
tveeprom               13045  1 saa7134
asus_atk0110           12395  0
x38_edac                3159  0
edac_core              40154  2 x38_edac
xen_netback            23987  0 [permanent]
xen_blkback            17924  0 [permanent]
xen_gntdev              9019  0
xen_evtchn              5032  1
sunrpc                200831  2 lockd
xenfs                   9621  1
raid1                  22676  2
ata_generic             3587  0
uas                     7775  0
pata_acpi               3419  0
usb_storage            46027  0
sata_mv                24941  8
pata_marvell            3240  0
radeon                690803  1
ttm                    54997  1 radeon
drm_kms_helper         26490  1 radeon
drm                   194532  3 radeon,ttm,drm_kms_helper
i2c_algo_bit            4958  1 radeon
i2c_core               25728  11
tda1004x,saa7134_dvb,saa7134,v4l2_common,videodev,i2c_i801,tveeprom,radeon,drm_kms_helper,drm,i2c_algo_bit
[root@xen ~]# lsmod|more
Module                  Size  Used by
lockd                  70080  0
bridge                 72368  0
stp                     1927  1 bridge
llc                     4738  2 bridge,stp
ip6t_REJECT             3992  2
nf_conntrack_ipv6       7730  2
nf_defrag_ipv6          9083  1 nf_conntrack_ipv6
xt_state                1306  2
nf_conntrack           67597  2 nf_conntrack_ipv6,xt_state
ip6table_filter         1655  1
ip6_tables             16792  1 ip6table_filter
snd_hda_codec_realtek   312621  1
raid456                54497  1
async_raid6_recov       5358  1 raid456
async_pq                4339  2 raid456,async_raid6_recov
raid6_pq               78299  2 async_raid6_recov,async_pq
async_xor               3255  3 raid456,async_raid6_recov,async_pq
xor                     4793  1 async_xor
async_memcpy            1845  2 raid456,async_raid6_recov
async_tx                2702  5
raid456,async_raid6_recov,async_pq,async_xor,async_memcpy
tda1004x               14722  2
snd_hda_intel          24072  0
snd_hda_codec          85181  2 snd_hda_codec_realtek,snd_hda_intel
snd_hwdep               6264  1 snd_hda_codec
saa7134_dvb            27032  0
videobuf_dvb            5146  1 saa7134_dvb
dvb_core               87211  1 videobuf_dvb
snd_seq                52186  0
snd_seq_device          5941  1 snd_seq
iTCO_wdt               12024  0
snd_pcm                78514  2 snd_hda_intel,snd_hda_codec
saa7134               159679  1 saa7134_dvb
rc_core                17136  1 saa7134
videobuf_dma_sg         8462  2 saa7134_dvb,saa7134
videobuf_core          15780  3 videobuf_dvb,saa7134,videobuf_dma_sg
v4l2_common             6905  1 saa7134
videodev               78689  2 saa7134,v4l2_common
media                  11511  1 videodev
v4l2_compat_ioctl32     7665  1 videodev
shpchp                 24554  0
i2c_i801                9237  0
serio_raw               4298  0
iTCO_vendor_support     2578  1 iTCO_wdt
sky2                   42923  0
snd_timer              19372  2 snd_seq,snd_pcm
snd                    63124  8
snd_hda_codec_realtek,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_seq,snd_seq_device,snd_pcm,snd_timer
soundcore               6267  1 snd
snd_page_alloc          7311  2 snd_hda_intel,snd_pcm
tveeprom               13045  1 saa7134
asus_atk0110           12395  0
x38_edac                3159  0
edac_core              40154  2 x38_edac
xen_netback            23987  0 [permanent]
xen_blkback            17924  0 [permanent]
xen_gntdev              9019  0
xen_evtchn              5032  1
sunrpc                200831  2 lockd
xenfs                   9621  1
raid1                  22676  2
ata_generic             3587  0
uas                     7775  0
pata_acpi               3419  0
usb_storage            46027  0
sata_mv                24941  8
pata_marvell            3240  0
radeon                690803  1
ttm                    54997  1 radeon
drm_kms_helper         26490  1 radeon
drm                   194532  3 radeon,ttm,drm_kms_helper
i2c_algo_bit            4958  1 radeon
i2c_core               25728  11
tda1004x,saa7134_dvb,saa7134,v4l2_common,videodev,i2c_i801,tveeprom,radeon,drm_kms_helper,drm,i2c_algo_bit

Re: Re: PCI passthrough stopped working, brainache!

[Ian Campbell]

On Sat, 2011-10-15 at 11:07 +0100, Andy Burns wrote:
> On 13 October 2011 19:15, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> 
> > On Wed, Oct 12, 2011 at 10:36:12PM +0100, Andy Burns wrote:
> >
> >> I discovered that the PCI tuners don't work in dom0 under Xen,
> >> they only work if the dom0 is booted  as baremetal.
> >
> > That would imply it is not the PCI passthrough code (good!).
> > It is something related to the driver
> 
> I've been having a go at getting the card running in dom0 rather than
> domU, no more success yet.
> 
> I've stopped using the PCI resource realignment for now, and enabled
> debugging knobs in the tuner modules, seems the I2C transfers used to
> program the tuner and receive status bits back from it are working OK,
> I can see the driver queueing up DMA using a succession of transfer
> buffers, and apparently interrupts signalling their completion.
> 
> Presume it's correct that within the dom0, the buffers should *NOT* be
> within the SWIOTLB range?

In general if a driver is correctly using the DMA API things should
never be within the swiotlb.

[...]
> Hypervisor, dom0 (and domU when I was using it) are all 64bit, the
> tuner device is 32bit, is this likely to be an issue with DMA
> transfers? Any extra logging for Xen?

I think you've got 8G of RAM so one thing which might be worth trying is
to give "mem=2G" (or perhaps 3G) on the hypervisor command line. That
ought to rule out addresses which are too high. (just a datapoint, not a
solution)

I wonder if the hypervisor's "dma_bits" option has any relevance here?
Might be worth a go?

For a domU (but not dom0, AFAICT) you can limit the machine addresses
allocated to a guest using "machine_address_size" in the cfg file. Only
seems to be supported by xend and not xl at the moment. That might be
another worthwhile experiment.

> > I saw something about I2C, is there a knob in the driver to _not_ use I2C?
> 
> Not for my tuner

In general I think these sorts cards use I2C extensively, i.e. the tuner
etc is on an i2c bus, so I wouldn't expect anything to work at all
without it.

Ian.

Re: Re: PCI passthrough stopped working, brainache!

[Andy Burns]

On 15 October 2011 11:36, Ian Campbell <Ian.Campbell@citrix.com> wrote:

> In general if a driver is correctly using the DMA API things should
> never be within the swiotlb.

OK, they just looked coincidentally similar.

> I think you've got 8G of RAM so one thing which might be worth trying is
> to give "mem=2G" (or perhaps 3G) on the hypervisor command line.

Yes I do have 8GB, will give it a try.

> I wonder if the hypervisor's "dma_bits" option has any relevance here?
> Might be worth a go?

I noticed that parameter, but couldn't see much indication of sensible
values, something a few bits less than 32 perhaps?

> In general I think these sorts cards use I2C extensively, i.e. the tuner
> etc is on an i2c bus, so I wouldn't expect anything to work at all
> without it.

Yes, that part of the card is working, w_scan can control the tuner
and detect when it latches onto valid muxes, receiving the bulk
datastream from the demodulator is the problem.

Re: Re: PCI passthrough stopped working, brainache!

[Ian Campbell]

On Sat, 2011-10-15 at 11:54 +0100, Andy Burns wrote:
> On 15 October 2011 11:36, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> 
> > In general if a driver is correctly using the DMA API things should
> > never be within the swiotlb.
> 
> OK, they just looked coincidentally similar.
> 
> > I think you've got 8G of RAM so one thing which might be worth trying is
> > to give "mem=2G" (or perhaps 3G) on the hypervisor command line.
> 
> Yes I do have 8GB, will give it a try.
> 
> > I wonder if the hypervisor's "dma_bits" option has any relevance here?
> > Might be worth a go?
> 
> I noticed that parameter, but couldn't see much indication of sensible
> values, something a few bits less than 32 perhaps?

Anything <32 would be a good guess, maybe try 30? Even 32 might be
worthwhile to try (not sure what you are getting by default).

Ian.

> > In general I think these sorts cards use I2C extensively, i.e. the tuner
> > etc is on an i2c bus, so I wouldn't expect anything to work at all
> > without it.
> 
> Yes, that part of the card is working, w_scan can control the tuner
> and detect when it latches onto valid muxes, receiving the bulk
> datastream from the demodulator is the problem.

Re: Re: PCI passthrough stopped working, brainache!

[Andy Burns]

On 15 October 2011 11:36, Ian Campbell <Ian.Campbell@citrix.com> wrote:

> I think you've got 8G of RAM so one thing which might be worth trying is
> to give "mem=2G" (or perhaps 3G) on the hypervisor command line. That
> ought to rule out addresses which are too high. (just a datapoint, not a
> solution)

A coconut for the gentleman!

Working in dom0 and (with page-alignment of the PCI BARs) in the domU,
I did check a couple of reboots and a cold start too, just in case!

So what's to look at for the real cause?

Re: Re: PCI passthrough stopped working, brainache!

[Ian Campbell]

On Sat, 2011-10-15 at 12:27 +0100, Andy Burns wrote:
> On 15 October 2011 11:36, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> 
> > I think you've got 8G of RAM so one thing which might be worth trying is
> > to give "mem=2G" (or perhaps 3G) on the hypervisor command line. That
> > ought to rule out addresses which are too high. (just a datapoint, not a
> > solution)
> 
> A coconut for the gentleman!
> 
> Working in dom0 and (with page-alignment of the PCI BARs) in the domU,
> I did check a couple of reboots and a cold start too, just in case!

Excellent. I should read the whole thread before replying.

> So what's to look at for the real cause?

Will have to have a think on Monday.

The VM_IO stuff which Konrad mentioned and perhaps the DMA mask of the
device spring to mind.

Ian.

AW: Re: Re: PCI passthrough stopped working, brainache!

[Carsten Schiers]

Interesting. Konrad, do you remember that one of my DVB cards did 
consume double CPU time 
as compared to a) old Xenified 2.6.18 b) Xenified 2.6.34 and c) mem=2G 
Dom0? Maybe these
observations are somehow connected.

It did work, though... Unfortunately, I sold these cards to have only 
one, which is 

Carsten.

-----Ursprüngliche Nachricht-----
Von: Andy Burns [mailto:xen.lists@burns.me.uk] 
Gesendet: Samstag, 15. Oktober 2011 13:28
An: Ian Campbell; Konrad Rzeszutek Wilk; xen-devel
Betreff: Re: [Xen-devel] Re: PCI passthrough stopped working, brainache!

On 15 October 2011 11:36, Ian Campbell <Ian.Campbell@citrix.com> wrote:

> I think you've got 8G of RAM so one thing which might be worth trying 
is
> to give "mem=2G" (or perhaps 3G) on the hypervisor command line. That
> ought to rule out addresses which are too high. (just a datapoint, not 
a
> solution)

A coconut for the gentleman!

Working in dom0 and (with page-alignment of the PCI BARs) in the domU,
I did check a couple of reboots and a cold start too, just in case!

So what's to look at for the real cause?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: Re: PCI passthrough stopped working, brainache!

[Konrad Rzeszutek Wilk]

On Sat, Oct 15, 2011 at 05:37:20PM +0200, Carsten Schiers wrote:
> Interesting. Konrad, do you remember that one of my DVB cards did 
> consume double CPU time 
> as compared to a) old Xenified 2.6.18 b) Xenified 2.6.34 and c) mem=2G 

Well, your DVB cards started consuming when you added more than 4GB to your
box. I don't remember the 2.6.18 figuring in this picture.

> Dom0? Maybe these
> observations are somehow connected.
> 
> It did work, though... Unfortunately, I sold these cards to have only 
> one, which is 

.. which is?

.. snip..
> A coconut for the gentleman!

Beer!
> 
> Working in dom0 and (with page-alignment of the PCI BARs) in the domU,
> I did check a couple of reboots and a cold start too, just in case!
> 
> So what's to look at for the real cause?

There can be one more test to conclude this and that is to raise right about 4GB
(ought to work) or perhaps to 5GB (it should fail there) under Xen .
Either way it sounds like a DMA issue - either we are:

 1). Not having the right dma_mask set. Check the /sys/bus/pci/devices/<BDF>/dma_mask
     and coherent_dma_mask. Both values should be the same and it ought to be 32
 
 2). The driver is not doing pci_sync_range .. which you should be able
     to easily find out. Try booting the Linux kernel in baremetal, with
     8GB and with "iommu=soft swiotlb=force". That will bounce buffer _everything_
     - which might show the same problem. It also might show problems with some
     other drivers.

 3). I can send you some patches to instrument Xen SWIOTLB to see what DMA pages
     (and where in the driver code is doing it) are using bounce buffer and not
     properly doing 2).

Re: Re: PCI passthrough stopped working, brainache!

[Andy Burns]

On 15 October 2011 12:27, Andy Burns <xen.lists@burns.me.uk> wrote:

> On 15 October 2011 11:36, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>
>> I think you've got 8G of RAM so one thing which might be worth trying is
>> to give "mem=2G"
>
> A coconut for the gentleman!

I can see that the devs have been busy getting their ducks in a row
for when the 3.2 merge window opens (3.1 is released now) so not
wanted to pester about this as it's OK for now with the workarounds.

Initially I tried mem=2G, with dom0_mem=512M and two domUs of 512M and
1G respectively, I presume this caused a bit of ballooning in dom0 as
I caught it with 400M'ish at one point and I had a little instability.

I pushed it up to mem=3G and at the same time added irqpoll for dom0
and the pci domU (because I was seeing a few "nobody cared" messages
in the domU at bootup and in the dom0 after shutting down the domU)
those changes so far have resulted in a stable setup.

Not wanting to reboot the dom0 just yet (would like to see how stable
it really is) but want to think about things to try when I *do* reboot
it,
I'm assuming the DMA problem will rear its head again if I try mem >=
4G, I haven't tried anything with dma_bits yet either, anything other
suggestions or logging that might help?

Re: Re: PCI passthrough stopped working, brainache!

[Ian Campbell]

On Mon, 2011-10-24 at 12:30 +0100, Andy Burns wrote:
> On 15 October 2011 12:27, Andy Burns <xen.lists@burns.me.uk> wrote:
> 
> > On 15 October 2011 11:36, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >
> >> I think you've got 8G of RAM so one thing which might be worth trying is
> >> to give "mem=2G"
> >
> > A coconut for the gentleman!
> 
> I can see that the devs have been busy getting their ducks in a row
> for when the 3.2 merge window opens (3.1 is released now) so not
> wanted to pester about this as it's OK for now with the workarounds.
> 
> Initially I tried mem=2G, with dom0_mem=512M and two domUs of 512M and
> 1G respectively, I presume this caused a bit of ballooning in dom0 as
> I caught it with 400M'ish at one point and I had a little instability.
> 
> I pushed it up to mem=3G and at the same time added irqpoll for dom0
> and the pci domU (because I was seeing a few "nobody cared" messages
> in the domU at bootup and in the dom0 after shutting down the domU)
> those changes so far have resulted in a stable setup.
> 
> Not wanting to reboot the dom0 just yet (would like to see how stable
> it really is) but want to think about things to try when I *do* reboot
> it,
> I'm assuming the DMA problem will rear its head again if I try mem >=
> 4G, I haven't tried anything with dma_bits yet either, anything other
> suggestions or logging that might help?

Konrad had some suggestions in
<20111020034000.GA2401@phenom.dumpdata.com> which would be worth trying.
(that mail was sent on Thursday but appears to have sat in a queue
somewhere until after you sent this mail so just checking you've seen
it).

Ian.

Re: PCI passthrough stopped working, brainache!

[Taylor, Neal E]

Andy,

Did I mess the solution for this or are you still working with 'mem=3.99999G'?

Neal Taylor
CA Technologies

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Ian Campbell
Sent: Wednesday, October 26, 2011 1:05 AM
To: Andy Burns
Cc: xen-devel
Subject: Re: [Xen-devel] Re: PCI passthrough stopped working, brainache!

On Mon, 2011-10-24 at 12:30 +0100, Andy Burns wrote:
> On 15 October 2011 12:27, Andy Burns <xen.lists@burns.me.uk> wrote:
> 
> > On 15 October 2011 11:36, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >
> >> I think you've got 8G of RAM so one thing which might be worth trying is
> >> to give "mem=2G"
> >
> > A coconut for the gentleman!
> 
> I can see that the devs have been busy getting their ducks in a row
> for when the 3.2 merge window opens (3.1 is released now) so not
> wanted to pester about this as it's OK for now with the workarounds.
> 
> Initially I tried mem=2G, with dom0_mem=512M and two domUs of 512M and
> 1G respectively, I presume this caused a bit of ballooning in dom0 as
> I caught it with 400M'ish at one point and I had a little instability.
> 
> I pushed it up to mem=3G and at the same time added irqpoll for dom0
> and the pci domU (because I was seeing a few "nobody cared" messages
> in the domU at bootup and in the dom0 after shutting down the domU)
> those changes so far have resulted in a stable setup.
> 
> Not wanting to reboot the dom0 just yet (would like to see how stable
> it really is) but want to think about things to try when I *do* reboot
> it,
> I'm assuming the DMA problem will rear its head again if I try mem >=
> 4G, I haven't tried anything with dma_bits yet either, anything other
> suggestions or logging that might help?

Konrad had some suggestions in
<20111020034000.GA2401@phenom.dumpdata.com> which would be worth trying.
(that mail was sent on Thursday but appears to have sat in a queue
somewhere until after you sent this mail so just checking you've seen
it).

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: PCI passthrough stopped working, brainache!

[Konrad Rzeszutek Wilk]

[-- Attachment #1: Type: text/plain, Size: 578 bytes --]

On Mon, Dec 05, 2011 at 04:10:39PM +0000, Taylor, Neal E wrote:
> Andy,
> 
> Did I mess the solution for this or are you still working with 'mem=3.99999G'?
> 
.. snip..
> 
> Konrad had some suggestions in
> <20111020034000.GA2401@phenom.dumpdata.com> which would be worth trying.
> (that mail was sent on Thursday but appears to have sat in a queue
> somewhere until after you sent this mail so just checking you've seen
> it).
> 

.. which resolves itself to:

http://article.gmane.org/gmane.comp.emulators.xen.devel/114063

and in 3). I mentioned a patch. See attached please

[-- Attachment #2: swiotlb-debug.patch --]
[-- Type: text/plain, Size: 10546 bytes --]

diff --git a/drivers/xen/Kconfig b/drivers/xen/Kconfig
index a59638b..d513c8d 100644
--- a/drivers/xen/Kconfig
+++ b/drivers/xen/Kconfig
@@ -105,4 +105,10 @@ config SWIOTLB_XEN
 	depends on PCI
 	select SWIOTLB
 
+config SWIOTLB_DEBUG
+	tristate "swiotlb debug facility"
+	default m
+	help
+	  Do not enable it unless you know what you are doing.
+
 endmenu
diff --git a/drivers/xen/Makefile b/drivers/xen/Makefile
index bbc1825..1dea490 100644
--- a/drivers/xen/Makefile
+++ b/drivers/xen/Makefile
@@ -16,6 +16,7 @@ obj-$(CONFIG_XENFS)			+= xenfs/
 obj-$(CONFIG_XEN_SYS_HYPERVISOR)	+= sys-hypervisor.o
 obj-$(CONFIG_XEN_PLATFORM_PCI)		+= xen-platform-pci.o
 obj-$(CONFIG_SWIOTLB_XEN)		+= swiotlb-xen.o
+obj-$(CONFIG_SWIOTLB_DEBUG)		+= dump_swiotlb.o
 obj-$(CONFIG_XEN_DOM0)			+= pci.o
 
 xen-evtchn-y				:= evtchn.o
diff --git a/drivers/xen/dump_swiotlb.c b/drivers/xen/dump_swiotlb.c
new file mode 100644
index 0000000..e0e4a0b
--- /dev/null
+++ b/drivers/xen/dump_swiotlb.c
@@ -0,0 +1,72 @@
+/*
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License v2.0 as published by
+ * the Free Software Foundation
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/module.h>
+#include <linux/string.h>
+#include <linux/types.h>
+#include <linux/init.h>
+#include <linux/stat.h>
+#include <linux/err.h>
+#include <linux/ctype.h>
+#include <linux/slab.h>
+#include <linux/limits.h>
+#include <linux/device.h>
+#include <linux/pci.h>
+#include <linux/blkdev.h>
+#include <linux/device.h>
+
+#include <linux/init.h>
+#include <linux/mm.h>
+#include <linux/fcntl.h>
+#include <linux/slab.h>
+#include <linux/kmod.h>
+#include <linux/major.h>
+#include <linux/highmem.h>
+#include <linux/blkdev.h>
+#include <linux/module.h>
+#include <linux/blkpg.h>
+#include <linux/buffer_head.h>
+#include <linux/mpage.h>
+#include <linux/mount.h>
+#include <linux/uio.h>
+#include <linux/namei.h>
+#include <asm/uaccess.h>
+
+#include <linux/pagemap.h>
+#include <linux/pagevec.h>
+
+#include <linux/swiotlb.h>
+#define DUMP_SWIOTLB_FUN  "0.1"
+
+MODULE_AUTHOR("Konrad Rzeszutek Wilk <konrad@darnok.org>");
+MODULE_DESCRIPTION("dump swiotlb");
+MODULE_LICENSE("GPL");
+MODULE_VERSION(DUMP_SWIOTLB_FUN);
+
+extern int xen_swiotlb_start_thread(void);
+extern void xen_swiotlb_stop_thread(void);
+static int __init dump_swiotlb_init(void)
+{
+	printk(KERN_INFO "Starting SWIOTLB debug thread.\n");
+	swiotlb_start_thread();
+	xen_swiotlb_start_thread();
+	return 0;
+}
+
+static void __exit dump_swiotlb_exit(void)
+{
+	swiotlb_stop_thread();
+	xen_swiotlb_stop_thread();
+}
+
+module_init(dump_swiotlb_init);
+module_exit(dump_swiotlb_exit);
diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index 6e8c15a..c833501 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -143,6 +143,63 @@ xen_swiotlb_fixup(void *buf, size_t size, unsigned long nslabs)
 	return 0;
 }
 
+#include <linux/percpu.h>
+struct xen_swiotlb_debug {
+	unsigned long alloc;
+	unsigned long dealloc;
+	char dev_name[64];
+};
+static DEFINE_PER_CPU(struct xen_swiotlb_debug, xen_tlb_debug);
+#include <linux/kthread.h>
+static int xen_swiotlb_debug_thread(void *arg)
+{
+	int cpu;
+	do {
+		set_current_state(TASK_INTERRUPTIBLE);
+		schedule_timeout_interruptible(HZ*10);
+
+		for_each_online_cpu(cpu) {
+			struct xen_swiotlb_debug *d;
+
+			d = &per_cpu(xen_tlb_debug, cpu);
+			/* Can't really happend.*/
+			if (!d)
+				continue;
+
+			if (d->dev_name[0] == 0)
+				continue;
+
+			printk(KERN_INFO "%u %s alloc coherent: %ld, free: %ld\n",
+				cpu,
+				d->dev_name ? d->dev_name : "?",
+				d->alloc, d->dealloc);
+
+			memset(d, 0, sizeof(struct xen_swiotlb_debug));
+		}
+
+	} while (!kthread_should_stop());
+	return 0;
+}
+static struct task_struct *xen_debug_thread = NULL;
+
+int xen_swiotlb_start_thread(void) {
+
+	if (xen_debug_thread)
+		return -EINVAL;
+	printk(KERN_INFO "%s: Go!\n",__func__);
+	xen_debug_thread =  kthread_run(xen_swiotlb_debug_thread, NULL, "xen_swiotlb_debug");
+	return 0;
+}
+EXPORT_SYMBOL_GPL(xen_swiotlb_start_thread);
+void xen_swiotlb_stop_thread(void) {
+
+	printk(KERN_INFO "%s: Stop!\n",__func__);
+	if (xen_debug_thread)
+		kthread_stop(xen_debug_thread);
+	xen_debug_thread = NULL;
+}
+EXPORT_SYMBOL_GPL(xen_swiotlb_stop_thread);
+
 void __init xen_swiotlb_init(int verbose)
 {
 	unsigned long bytes;
@@ -194,7 +251,14 @@ xen_swiotlb_alloc_coherent(struct device *hwdev, size_t size,
 	int order = get_order(size);
 	u64 dma_mask = DMA_BIT_MASK(32);
 	unsigned long vstart;
-
+	struct xen_swiotlb_debug *d;
+
+	preempt_disable();
+	d = &__get_cpu_var(xen_tlb_debug);
+	preempt_enable();
+	d->alloc++;
+	snprintf(d->dev_name, sizeof(d->dev_name), "%s %s",
+                dev_driver_string(hwdev), dev_name(hwdev));
 	/*
 	* Ignore region specifiers - the kernel's ideas of
 	* pseudo-phys memory layout has nothing to do with the
@@ -230,6 +294,14 @@ xen_swiotlb_free_coherent(struct device *hwdev, size_t size, void *vaddr,
 			  dma_addr_t dev_addr)
 {
 	int order = get_order(size);
+	struct xen_swiotlb_debug *d;
+
+	preempt_disable();
+	d = &__get_cpu_var(xen_tlb_debug);
+	preempt_enable();
+	d->dealloc++;
+	snprintf(d->dev_name, sizeof(d->dev_name), "%s %s",
+                dev_driver_string(hwdev), dev_name(hwdev));
 
 	if (dma_release_from_coherent(hwdev, order, vaddr))
 		return;
diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index 445702c..0d2e049 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -26,6 +26,8 @@ extern void swiotlb_init(int verbose);
 extern void swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose);
 extern unsigned long swioltb_nr_tbl(void);
 
+extern int swiotlb_start_thread(void);
+extern void swiotlb_stop_thread(void);
 /*
  * Enumeration for sync targets
  */
diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 99093b3..5446076 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -92,6 +92,74 @@ static DEFINE_SPINLOCK(io_tlb_lock);
 
 static int late_alloc;
 
+#include <linux/percpu.h>
+struct swiotlb_debug {
+	unsigned long bounce_to;
+	unsigned long bounce_from;
+	unsigned long bounce_slow;
+	unsigned long map;
+	unsigned long unmap;
+	unsigned long sync;
+	char dev_name[64];
+};
+static DEFINE_PER_CPU(struct swiotlb_debug, tlb_debug);
+#include <linux/kthread.h>
+static int swiotlb_debug_thread(void *arg)
+{
+	int cpu;
+	int size = io_tlb_nslabs;
+	do {
+		int i;
+		unsigned long filled = 0;
+		set_current_state(TASK_INTERRUPTIBLE);
+		schedule_timeout_interruptible(HZ*5);
+
+		for_each_online_cpu(cpu) {
+			struct swiotlb_debug *d = &per_cpu(tlb_debug, cpu);
+			/* Can't really happend.*/
+			if (!d)
+				continue;
+			if (d->dev_name[0] == 0)
+				continue;
+
+			printk(KERN_INFO "%d [%s] bounce: from:%ld(slow:%ld)to:%ld map:%ld unmap:%ld sync:%ld\n",
+				cpu,
+				d->dev_name ? d->dev_name : "?",
+				d->bounce_from,
+				d->bounce_slow,
+				d->bounce_to,
+				d->map, d->unmap, d->sync);
+			memset(d, 0, sizeof(struct swiotlb_debug));
+		}
+		/* Very crude calculation. */
+		for (i = 0; i < size; i++) {
+			if (io_tlb_list[i] == 0)
+				filled++;
+		}
+		printk(KERN_INFO "SWIOTLB is %ld%% full\n", (filled * 100) / size);
+
+	} while (!kthread_should_stop());
+	return 0;
+}
+static struct task_struct *debug_thread = NULL;
+
+int swiotlb_start_thread(void) {
+
+	if (debug_thread)
+		return -EINVAL;
+	printk(KERN_INFO "%s: Go!\n",__func__);
+	debug_thread =  kthread_run(swiotlb_debug_thread, NULL, "swiotlb_debug");
+}
+EXPORT_SYMBOL_GPL(swiotlb_start_thread);
+void swiotlb_stop_thread(void) {
+
+	printk(KERN_INFO "%s: Stop!\n",__func__);
+	if (debug_thread)
+		kthread_stop(debug_thread);
+	debug_thread = NULL;
+}
+EXPORT_SYMBOL_GPL(swiotlb_stop_thread);
+
 static int __init
 setup_io_tlb_npages(char *str)
 {
@@ -166,6 +234,7 @@ void __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose)
 		panic("Cannot allocate SWIOTLB overflow buffer!\n");
 	if (verbose)
 		swiotlb_print_info();
+
 }
 
 /*
@@ -336,6 +405,7 @@ void swiotlb_bounce(phys_addr_t phys, char *dma_addr, size_t size,
 		    enum dma_data_direction dir)
 {
 	unsigned long pfn = PFN_DOWN(phys);
+	struct swiotlb_debug *d = &__get_cpu_var(tlb_debug);
 
 	if (PageHighMem(pfn_to_page(pfn))) {
 		/* The buffer does not have a mapping.  Map it in and copy */
@@ -362,11 +432,16 @@ void swiotlb_bounce(phys_addr_t phys, char *dma_addr, size_t size,
 			dma_addr += sz;
 			offset = 0;
 		}
+		d->bounce_slow++;
 	} else {
-		if (dir == DMA_TO_DEVICE)
+		if (dir == DMA_TO_DEVICE) {
 			memcpy(dma_addr, phys_to_virt(phys), size);
-		else
+			d->bounce_to++;
+		}
+		else {
 			memcpy(phys_to_virt(phys), dma_addr, size);
+			d->bounce_from++;
+		}
 	}
 }
 EXPORT_SYMBOL_GPL(swiotlb_bounce);
@@ -471,7 +546,15 @@ found:
 		io_tlb_orig_addr[index+i] = phys + (i << IO_TLB_SHIFT);
 	if (dir == DMA_TO_DEVICE || dir == DMA_BIDIRECTIONAL)
 		swiotlb_bounce(phys, dma_addr, size, DMA_TO_DEVICE);
-
+	{
+		struct swiotlb_debug *d;
+		preempt_disable();
+		d = &__get_cpu_var(tlb_debug);
+		preempt_enable();
+		d->map++;
+		snprintf(d->dev_name, sizeof(d->dev_name), "%s %s",
+                	dev_driver_string(hwdev), dev_name(hwdev));
+	}
 	return dma_addr;
 }
 EXPORT_SYMBOL_GPL(swiotlb_tbl_map_single);
@@ -531,6 +614,15 @@ swiotlb_tbl_unmap_single(struct device *hwdev, char *dma_addr, size_t size,
 			io_tlb_list[i] = ++count;
 	}
 	spin_unlock_irqrestore(&io_tlb_lock, flags);
+	{
+		struct swiotlb_debug *d;
+		preempt_disable();
+		d = &__get_cpu_var(tlb_debug);
+		preempt_enable();
+		d->unmap++;
+		snprintf(d->dev_name, sizeof(d->dev_name), "%s %s",
+                	dev_driver_string(hwdev), dev_name(hwdev));
+	}
 }
 EXPORT_SYMBOL_GPL(swiotlb_tbl_unmap_single);
 
@@ -541,7 +633,13 @@ swiotlb_tbl_sync_single(struct device *hwdev, char *dma_addr, size_t size,
 {
 	int index = (dma_addr - io_tlb_start) >> IO_TLB_SHIFT;
 	phys_addr_t phys = io_tlb_orig_addr[index];
-
+	struct swiotlb_debug *d;
+	preempt_disable();
+	d = &__get_cpu_var(tlb_debug);
+	preempt_enable();
+	d->sync++;
+	snprintf(d->dev_name, sizeof(d->dev_name), "%s %s",
+               	dev_driver_string(hwdev), dev_name(hwdev));
 	phys += ((unsigned long)dma_addr & ((1 << IO_TLB_SHIFT) - 1));
 
 	switch (target) {

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: PCI passthrough stopped working, brainache!

[Taylor, Neal E]

Konrad,

Thanks for the patch but all I'm getting from the patch is:

Dec  6 19:40:30 Gridos_Node kernel: [  220.270179] SWIOTLB is 0% full
Dec  6 19:40:35 Gridos_Node kernel: [  225.270172] SWIOTLB is 0% full
Dec  6 19:40:40 Gridos_Node kernel: [  230.270179] SWIOTLB is 0% full
Dec  6 19:40:45 Gridos_Node kernel: [  235.270174] SWIOTLB is 0% full
Dec  6 19:40:50 Gridos_Node kernel: [  240.270179] SWIOTLB is 0% full
Dec  6 19:40:55 Gridos_Node kernel: [  245.270169] SWIOTLB is 0% full
Dec  6 19:41:00 Gridos_Node kernel: [  250.270178] SWIOTLB is 0% full
Dec  6 19:41:05 Gridos_Node kernel: [  255.270181] SWIOTLB is 0% full
Dec  6 19:41:10 Gridos_Node kernel: [  260.270181] SWIOTLB is 0% full
Dec  6 19:41:15 Gridos_Node kernel: [  265.270171] SWIOTLB is 0% full

I was hoping my problem was related but it looks like "not close enough." My issue is with the "failed to set dma mask" and others like it in the log segment below. The "Fusion MPT SPI Host driver 3.04.19" believes it is able to support dma, but nothing after it does. 

Dec  6 19:37:37 Gridos_Node kernel: Fusion MPT SAS Host driver 3.04.19
Dec  6 19:37:37 Gridos_Node kernel: ata_piix 0000:00:1f.1: enabling device (0005 -> 0007)
Dec  6 19:37:37 Gridos_Node kernel: ata_piix 0000:00:1f.1: can't derive routing for PCI INT A

Dec  6 19:37:37 Gridos_Node kernel: ata_piix 0000:00:1f.1: BMDMA: failed to set dma mask, falling back to PIO

Dec  6 19:37:37 Gridos_Node kernel: scsi1 : ata_piix
Dec  6 19:37:37 Gridos_Node kernel: scsi2 : ata_piix
Dec  6 19:37:37 Gridos_Node kernel: ata1: PATA max PIO4 cmd 0x1f0 ctl 0x3f6 bmdma 0xfc00 irq 14
Dec  6 19:37:37 Gridos_Node kernel: ata2: PATA max PIO4 cmd 0x170 ctl 0x376 bmdma 0xfc08 irq 15
Dec  6 19:37:37 Gridos_Node kernel: ata1.00: ATAPI: PHILIPS DVD-ROM SDR089, TD36, max UDMA/33
Dec  6 19:37:37 Gridos_Node kernel: ata1.00: configured for PIO4
Dec  6 19:37:37 Gridos_Node kernel: scsi 1:0:0:0: CD-ROM            PHILIPS  DVD-ROM SDR089   TD36 PQ: 0 ANSI: 5
Dec  6 19:37:37 Gridos_Node kernel: 0 mptspi 0000:02:05.0 alloc coherent: 57, free: 56
Dec  6 19:37:37 Gridos_Node kernel: 1 mptspi 0000:02:05.0 alloc coherent: 3, free: 0
Dec  6 19:37:37 Gridos_Node kernel: SWIOTLB is 0% full
Dec  6 19:37:37 Gridos_Node kernel: SWIOTLB is 0% full
Dec  6 19:37:37 Gridos_Node kernel: EXT3-fs: barriers not enabled
Dec  6 19:37:37 Gridos_Node kernel: kjournald starting.  Commit interval 5 seconds
Dec  6 19:37:37 Gridos_Node kernel: EXT3-fs (sda3): mounted filesystem with ordered data mode
Dec  6 19:37:37 Gridos_Node kernel: SWIOTLB is 0% full
Dec  6 19:37:37 Gridos_Node kernel: input: PC Speaker as /devices/platform/pcspkr/input/input2
Dec  6 19:37:37 Gridos_Node kernel: iTCO_vendor_support: vendor-support=0
Dec  6 19:37:37 Gridos_Node kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.06
Dec  6 19:37:37 Gridos_Node kernel: iTCO_wdt: Found a ICH5 or ICH5R TCO device (Version=1, TCOBASE=0x0860)
Dec  6 19:37:37 Gridos_Node kernel: iTCO_wdt: initialized. heartbeat=30 sec (nowayout=1)
Dec  6 19:37:37 Gridos_Node kernel: pata_acpi 0000:09:06.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23
Dec  6 19:37:37 Gridos_Node kernel: pata_acpi 0000:09:06.0: BMDMA: failed to set dma mask, falling back to PIO
Dec  6 19:37:37 Gridos_Node kernel: pata_acpi 0000:09:06.0: PCI INT A disabled
Dec  6 19:37:37 Gridos_Node kernel: FDC 0 is a National Semiconductor PC87306
Dec  6 19:37:37 Gridos_Node kernel: xen_map_pirq_gsi: returning irq 23 for gsi 23
Dec  6 19:37:38 Gridos_Node kernel: Already setup the GSI :23
Dec  6 19:37:38 Gridos_Node kernel: pata_sil680 0000:09:06.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23
Dec  6 19:37:38 Gridos_Node kernel: sil680: 133MHz clock.

Dec  6 19:37:38 Gridos_Node kernel: pata_sil680 0000:09:06.0: BMDMA: failed to set dma mask, falling back to PIO

neal

-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:konrad@darnok.org] 
Sent: Monday, December 05, 2011 1:09 PM
To: Taylor, Neal E
Cc: Ian Campbell; Andy Burns; xen-devel
Subject: Re: [Xen-devel] PCI passthrough stopped working, brainache!

On Mon, Dec 05, 2011 at 04:10:39PM +0000, Taylor, Neal E wrote:
> Andy,
> 
> Did I mess the solution for this or are you still working with 'mem=3.99999G'?
> 
.. snip..
> 
> Konrad had some suggestions in
> <20111020034000.GA2401@phenom.dumpdata.com> which would be worth trying.
> (that mail was sent on Thursday but appears to have sat in a queue 
> somewhere until after you sent this mail so just checking you've seen 
> it).
> 

.. which resolves itself to:

http://article.gmane.org/gmane.comp.emulators.xen.devel/114063

and in 3). I mentioned a patch. See attached please