Re: pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Mon, Apr 05, 2010 at 06:03:55PM -0400, listmail wrote:
> Per request of Konrad I am forwarding details on an issue I ran into.   

Thanks.

Samuel,

It looks to be be related to the arguments being passed in to pvgrub.

With "extra='root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a2e ro iommu=soft swiotlb=force"
it fails. If the 'iommu=soft swiotlb=force' is removed it boots. I've
asked the user to move the 'ro' before the 'root' but still had the same
issue.


> The goal here was non VT-D pci passthrough to a pv guest.  Konrad  
> provided a working solution as detailed at the end of this post.
>
> dom 0 grub
> menuentry "Xen 4.0.0-rc8 / Ubuntu 10.4 kernel 2.6.32.10-xen2 pvops" {
> insmod ext2
> set root='(hd0,2)'
> search --no-floppy --fs-uuid --set 321dedfa-be48-4b4d-bff0-0e5ecce240fd
> multiboot /boot/xen-4.0.0-rc8.gz dummy=dummy iommu=0
> module /boot/vmlinuz-2.6.32.10-xen2 dummy=dummy root=/dev/sdc2 ro  
> max_loop=64 xen-pciback.hide=(07:01.0)
> module /boot/initrd.img-2.6.32.10-xen2
> }
>
> # xm pci-list-assignable-devices
> 0000:07:01.0
>
> # dmesg |grep 07:01.0
> [    0.000000] Command line: root=/dev/sdc2 ro max_loop=64  
> xen-pciback.hide=(07:01.0)
> [    0.211959] Kernel command line: root=/dev/sdc2 ro max_loop=64  
> xen-pciback.hide=(07:01.0)
> [    0.520930] pci 0000:07:01.0: reg 10 32bit mmio: [0xf4000000-0xf7ffffff]
> [    0.558471] pciback 0000:07:01.0: seizing device
> [    0.609245] pciback 0000:07:01.0: PCI INT A -> GSI 16 (level, low) ->  
> IRQ 16
> [    0.609299] pciback 0000:07:01.0: PCI INT A disabled
> [  644.828556] pciback: vpci: 0000:07:01.0: assign to virtual slot 0
> [  731.747948] pciback: vpci: 0000:07:01.0: assign to virtual slot 0
> <note this shows up when i start the domU>
> [  731.748392] pciback 0000:07:01.0: device has been assigned to another  
> domain! Over-writting the ownership, but beware.
>
> at which point i can see card assigned:
> # xm pci-list lucidpv
> domain bus  slot func
> 0x0000 0x07 0x01 0x0
>
>
>
> domU grub
> title        Ubuntu lucid (development branch), kernel 2.6.32.10-xen2
> root        (hd0)
> kernel        /boot/vmlinuz-2.6.32.10-xen2  
> root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a2e ro iommu=soft 
> swiotlb=force
> initrd        /boot/initrd.img-2.6.32.10-xen2
>
>
> domU config
> kernel="/usr/lib/xen/boot/pv-grub-x86_64.gz"
> extra="(hd0)/boot/grub/menu.lst"
>
> name="lucidpv"
> memory=512
> maxmem=512
>
> vif=['bridge=xenbr0']
>
> disk = [ 'tap:aio:/etc/xen/images/lucidhvm.img,ioemu:xvda,w' ]
>
> pci = [ '07:01.0' ]
>
> on_reboot = 'destroy'
>
>
> domU output
> # xm create -c ubstub.cfg
> Using config file "/etc/xen/ubstub.cfg".
> Started domain lucidpv (id=2)
> Xen Minimal OS!
>  start_info: 0xa9a000(VA)
>    nr_pages: 0x20000
>  shared_inf: 0xbf44d000(MA)
>     pt_base: 0xa9d000(VA)
> nr_pt_frames: 0x9
>    mfn_list: 0x99a000(VA)
>   mod_start: 0x0(VA)
>     mod_len: 0
>       flags: 0x0
>    cmd_line: (hd0)/boot/grub/menu.lst
>  stack:      0x959980-0x979980
> MM: Init
>      _text: 0x0(VA)
>     _etext: 0x6987c(VA)
>   _erodata: 0x83000(VA)
>     _edata: 0x8bae0(VA)
> stack start: 0x959980(VA)
>       _end: 0x999f88(VA)
>  start_pfn: aa9
>    max_pfn: 20000
> Mapping memory range 0xc00000 - 0x20000000
> setting 0x0-0x83000 readonly
> skipped 0x1000
> MM: Initialise page allocator for ba3000(ba3000)-20000000(20000000)
> MM: done
> Demand map pfns at 20001000-2020001000.
> Heap resides at 2020002000-4020002000.
>  Booting command-list
>
> root  (hd0)
> Filesystem type is ext2fs, using whole disk
> kernel  /boot/vmlinuz-2.6.32.10-xen2  
> root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a
> 2e ro iommu=soft swiotlb=force
> initrd  /boot/initrd.img-2.6.32.10-xen2
>
> block error -2 for op 2
> close blk: backend=/local/domain/0/backend/vbd/2/51712 node=device/vbd/51712
> port 5 still bound!
>
>
> SOLUTION:
> Setting the following in the .cfg works:
>
> kernel = '/boot/vmlinuz-2.6.32.10-xen2'
> ramdisk= '/boot/initrd.img-2.6.32.10-xen2'
> extra='root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a2e ro iommu=soft  
> swiotlb=force'
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel