Re: PCI passthrough issue

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Wed, Feb 02, 2011 at 11:24:51AM +0100, Jean Baptiste Favre wrote:
> Hello Ian,
> 
> My domU config file:
> 
> # cat /cluster/xen/xps-106.cfg
> kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'
> ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'
> builder      = 'linux'
> memory       = '398'
> vcpus        = '1'
> cpus         = '2'
> localtime    = 0
> serial       = 'pty'
> boot         = 'cdn'
> disk         = [ 'drbd:xps-106,xvda,w' ]
> on_poweroff  = 'destroy'
> on_reboot    = 'restart'
> on_crash     = 'restart'
> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
> console=hvc0 xencons=tty"
> pci = [ '04:00.0' ]
> name         = 'xps-106'
> hostname     = 'xps-106.clichy.jbfavre.org'
> 
> 
> Le 02/02/2011 10:27, Ian Campbell a écrit :
> > On Tue, 2011-02-01 at 22:04 +0000, Jean Baptiste Favre wrote:
> > > Le 01/02/2011 17:23, Ian Campbell a écrit :
> >
> > >> I assume you are not seeing "rx mapping error" in your domU dmesg? Did
> > >> you post a full guest console log at some point? Comparing the logs for
> > >> the 256MB, 398MB and 512MB guest RAM case might be useful.
> > > No sure I've ever posted that logs. But I can redo my tests :)
> >
> > yes, please do that.
> Please find attached both console startup logs with 256M and 512M:
> 256M_domU_console_logs.txt
> 512M_domU_console_logs.txt
> 
> For 512M, I saw some kernel CallTrace I can not explain. There are not
> present with 256M.
> 
> For 398M memory, I can't even start domU :
> # xm create /cluster/xen/xps-106.cfg -c
> Using config file "/cluster/xen/xps-106.cfg".
> [215739.007871] pciback 0000:04:00.0: device has been assigned to
> another domain! Over-writting the ownership, but beware.
> Started domain xps-106 (id=23)
> (XEN) mm.c:798:d23 Non-privileged (23) attempt to map I/O space 00000000
> (XEN) mm.c:4644:d23 ptwr_emulate: could not get_page_from_l1e()
> 
> As I told you, I'm still using Debian 2.6.37 kernel because I've some
> problem to compile 2.6.32.27 from Jeremy's git repository.
> I hope I can get it compiled today so I'll be able to test with that
> kernel as well.

So I've tried this on my Abit IP-35 box which has a 

04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 13)
        Subsystem: ABIT Computer Corp. Device 1085
        Flags: bus master, fast devsel, latency 0, IRQ 29
        Memory at fdefc000 (64-bit, non-prefetchable) [size=16K]
        I/O ports at be00 [size=256]
        Expansion ROM at <ignored> [disabled]
        Capabilities: [48] Power Management version 3
        Capabilities: [50] Vital Product Data
        Capabilities: [5c] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [e0] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: sky2
        Kernel modules: sky2

And when I launch this guest with a 32-bit DomU:
kernel="/mnt/lab/latest/vmlinuz"
ramdisk="/mnt/lab/latest/initramfs.cpio.gz"
extra="console=hvc0 debug iommu=soft"
memory=320
vcpus=1
cpu='2'
on_crash="preserve"
#vif = [ 'bridge=switch' ]
pci = ["04:00.0"]
vfb = [ 'vnc=1, vnclisten=0.0.0.0,vncunused=1']

And played around with the 'extra' to add 'swiotlb=force'.

The moment I had 'swiotlb=force' I could not get any DHCP
address from the NIC. If I did not have 'swiotlb=force' it would
work just fine (can ping any size, etc, this is with 320MB)


For fun, I upped the memory (320->720) and kept 'swiotlb=force' in effect.
Same effect: can't do DHCP.

I look to have a different issue than you, which is that whenever I use
swiotlb=force, things go haywire.

Fyi, this is what DomU tells me:

12:22:41 # 9 :~/ 
> dmesg |grep Memor
[    0.000000] Memory: 145640k/335872k available (3731k kernel code, 189784k reserved, 1565k data, 436k init, 0k highmem)

12:22:56 # 10 :~/ 
> uname -a
Linux (none) 2.6.38-rc2-00028-gf2a2d8b #2 SMP Wed Feb 2 12:10:25 EST 2011 i686 i686 i386 GNU/Linux
rnet driver.
[    0.921406] udevd (1126): /proc/1126/oom_adj is deprecated, please use /proc/1126/oom_score_adj instead.
[    0.984886] sky2: driver version 1.28
[    0.995595] sky2 0000:04:00.0: BAR 0: set to [mem 0xfdefc000-0xfdefffff 64bit] (PCI address [0xfdefc000-0xfdefffff])
[    0.995662] sky2 0000:04:00.0: BAR 2: set to [io  0xbe00-0xbeff] (PCI address [0xbe00-0xbeff])
[    0.995697] sky2 0000:04:00.0: enabling device (0000 -> 0003)
[    0.996440] sky2 0000:04:00.0: Xen PCI mapped GSI18 to IRQ27

(This is the #master branch from git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git)