pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works

[listmail <listmail@triad.rr.com>]

Per request of Konrad I am forwarding details on an issue I ran into.  
The goal here was non VT-D pci passthrough to a pv guest.  Konrad 
provided a working solution as detailed at the end of this post.

dom 0 grub
menuentry "Xen 4.0.0-rc8 / Ubuntu 10.4 kernel 2.6.32.10-xen2 pvops" {
insmod ext2
set root='(hd0,2)'
search --no-floppy --fs-uuid --set 321dedfa-be48-4b4d-bff0-0e5ecce240fd
multiboot /boot/xen-4.0.0-rc8.gz dummy=dummy iommu=0
module /boot/vmlinuz-2.6.32.10-xen2 dummy=dummy root=/dev/sdc2 ro 
max_loop=64 xen-pciback.hide=(07:01.0)
module /boot/initrd.img-2.6.32.10-xen2
}

# xm pci-list-assignable-devices
0000:07:01.0

# dmesg |grep 07:01.0
[    0.000000] Command line: root=/dev/sdc2 ro max_loop=64 
xen-pciback.hide=(07:01.0)
[    0.211959] Kernel command line: root=/dev/sdc2 ro max_loop=64 
xen-pciback.hide=(07:01.0)
[    0.520930] pci 0000:07:01.0: reg 10 32bit mmio: [0xf4000000-0xf7ffffff]
[    0.558471] pciback 0000:07:01.0: seizing device
[    0.609245] pciback 0000:07:01.0: PCI INT A -> GSI 16 (level, low) -> 
IRQ 16
[    0.609299] pciback 0000:07:01.0: PCI INT A disabled
[  644.828556] pciback: vpci: 0000:07:01.0: assign to virtual slot 0
[  731.747948] pciback: vpci: 0000:07:01.0: assign to virtual slot 0
<note this shows up when i start the domU>
[  731.748392] pciback 0000:07:01.0: device has been assigned to another 
domain! Over-writting the ownership, but beware.

at which point i can see card assigned:
# xm pci-list lucidpv
domain bus  slot func
0x0000 0x07 0x01 0x0



domU grub
title        Ubuntu lucid (development branch), kernel 2.6.32.10-xen2
root        (hd0)
kernel        /boot/vmlinuz-2.6.32.10-xen2 
root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a2e ro iommu=soft swiotlb=force
initrd        /boot/initrd.img-2.6.32.10-xen2


domU config
kernel="/usr/lib/xen/boot/pv-grub-x86_64.gz"
extra="(hd0)/boot/grub/menu.lst"

name="lucidpv"
memory=512
maxmem=512

vif=['bridge=xenbr0']

disk = [ 'tap:aio:/etc/xen/images/lucidhvm.img,ioemu:xvda,w' ]

pci = [ '07:01.0' ]

on_reboot = 'destroy'


domU output
# xm create -c ubstub.cfg
Using config file "/etc/xen/ubstub.cfg".
Started domain lucidpv (id=2)
Xen Minimal OS!
  start_info: 0xa9a000(VA)
    nr_pages: 0x20000
  shared_inf: 0xbf44d000(MA)
     pt_base: 0xa9d000(VA)
nr_pt_frames: 0x9
    mfn_list: 0x99a000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line: (hd0)/boot/grub/menu.lst
  stack:      0x959980-0x979980
MM: Init
      _text: 0x0(VA)
     _etext: 0x6987c(VA)
   _erodata: 0x83000(VA)
     _edata: 0x8bae0(VA)
stack start: 0x959980(VA)
       _end: 0x999f88(VA)
  start_pfn: aa9
    max_pfn: 20000
Mapping memory range 0xc00000 - 0x20000000
setting 0x0-0x83000 readonly
skipped 0x1000
MM: Initialise page allocator for ba3000(ba3000)-20000000(20000000)
MM: done
Demand map pfns at 20001000-2020001000.
Heap resides at 2020002000-4020002000.
  Booting command-list

root  (hd0)
 Filesystem type is ext2fs, using whole disk
kernel  /boot/vmlinuz-2.6.32.10-xen2 
root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a
2e ro iommu=soft swiotlb=force
initrd  /boot/initrd.img-2.6.32.10-xen2

block error -2 for op 2
close blk: backend=/local/domain/0/backend/vbd/2/51712 node=device/vbd/51712
port 5 still bound!


SOLUTION:
Setting the following in the .cfg works:

kernel = '/boot/vmlinuz-2.6.32.10-xen2'
ramdisk= '/boot/initrd.img-2.6.32.10-xen2'
extra='root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a2e ro iommu=soft 
swiotlb=force'