pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works

pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works

[listmail]

Per request of Konrad I am forwarding details on an issue I ran into.  
The goal here was non VT-D pci passthrough to a pv guest.  Konrad 
provided a working solution as detailed at the end of this post.

dom 0 grub
menuentry "Xen 4.0.0-rc8 / Ubuntu 10.4 kernel 2.6.32.10-xen2 pvops" {
insmod ext2
set root='(hd0,2)'
search --no-floppy --fs-uuid --set 321dedfa-be48-4b4d-bff0-0e5ecce240fd
multiboot /boot/xen-4.0.0-rc8.gz dummy=dummy iommu=0
module /boot/vmlinuz-2.6.32.10-xen2 dummy=dummy root=/dev/sdc2 ro 
max_loop=64 xen-pciback.hide=(07:01.0)
module /boot/initrd.img-2.6.32.10-xen2
}

# xm pci-list-assignable-devices
0000:07:01.0

# dmesg |grep 07:01.0
[    0.000000] Command line: root=/dev/sdc2 ro max_loop=64 
xen-pciback.hide=(07:01.0)
[    0.211959] Kernel command line: root=/dev/sdc2 ro max_loop=64 
xen-pciback.hide=(07:01.0)
[    0.520930] pci 0000:07:01.0: reg 10 32bit mmio: [0xf4000000-0xf7ffffff]
[    0.558471] pciback 0000:07:01.0: seizing device
[    0.609245] pciback 0000:07:01.0: PCI INT A -> GSI 16 (level, low) -> 
IRQ 16
[    0.609299] pciback 0000:07:01.0: PCI INT A disabled
[  644.828556] pciback: vpci: 0000:07:01.0: assign to virtual slot 0
[  731.747948] pciback: vpci: 0000:07:01.0: assign to virtual slot 0
<note this shows up when i start the domU>
[  731.748392] pciback 0000:07:01.0: device has been assigned to another 
domain! Over-writting the ownership, but beware.

at which point i can see card assigned:
# xm pci-list lucidpv
domain bus  slot func
0x0000 0x07 0x01 0x0



domU grub
title        Ubuntu lucid (development branch), kernel 2.6.32.10-xen2
root        (hd0)
kernel        /boot/vmlinuz-2.6.32.10-xen2 
root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a2e ro iommu=soft swiotlb=force
initrd        /boot/initrd.img-2.6.32.10-xen2


domU config
kernel="/usr/lib/xen/boot/pv-grub-x86_64.gz"
extra="(hd0)/boot/grub/menu.lst"

name="lucidpv"
memory=512
maxmem=512

vif=['bridge=xenbr0']

disk = [ 'tap:aio:/etc/xen/images/lucidhvm.img,ioemu:xvda,w' ]

pci = [ '07:01.0' ]

on_reboot = 'destroy'


domU output
# xm create -c ubstub.cfg
Using config file "/etc/xen/ubstub.cfg".
Started domain lucidpv (id=2)
Xen Minimal OS!
  start_info: 0xa9a000(VA)
    nr_pages: 0x20000
  shared_inf: 0xbf44d000(MA)
     pt_base: 0xa9d000(VA)
nr_pt_frames: 0x9
    mfn_list: 0x99a000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line: (hd0)/boot/grub/menu.lst
  stack:      0x959980-0x979980
MM: Init
      _text: 0x0(VA)
     _etext: 0x6987c(VA)
   _erodata: 0x83000(VA)
     _edata: 0x8bae0(VA)
stack start: 0x959980(VA)
       _end: 0x999f88(VA)
  start_pfn: aa9
    max_pfn: 20000
Mapping memory range 0xc00000 - 0x20000000
setting 0x0-0x83000 readonly
skipped 0x1000
MM: Initialise page allocator for ba3000(ba3000)-20000000(20000000)
MM: done
Demand map pfns at 20001000-2020001000.
Heap resides at 2020002000-4020002000.
  Booting command-list

root  (hd0)
 Filesystem type is ext2fs, using whole disk
kernel  /boot/vmlinuz-2.6.32.10-xen2 
root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a
2e ro iommu=soft swiotlb=force
initrd  /boot/initrd.img-2.6.32.10-xen2

block error -2 for op 2
close blk: backend=/local/domain/0/backend/vbd/2/51712 node=device/vbd/51712
port 5 still bound!


SOLUTION:
Setting the following in the .cfg works:

kernel = '/boot/vmlinuz-2.6.32.10-xen2'
ramdisk= '/boot/initrd.img-2.6.32.10-xen2'
extra='root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a2e ro iommu=soft 
swiotlb=force'

Re: pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works

[Konrad Rzeszutek Wilk]

On Mon, Apr 05, 2010 at 06:03:55PM -0400, listmail wrote:
> Per request of Konrad I am forwarding details on an issue I ran into.   

Thanks.

Samuel,

It looks to be be related to the arguments being passed in to pvgrub.

With "extra='root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a2e ro iommu=soft swiotlb=force"
it fails. If the 'iommu=soft swiotlb=force' is removed it boots. I've
asked the user to move the 'ro' before the 'root' but still had the same
issue.


> The goal here was non VT-D pci passthrough to a pv guest.  Konrad  
> provided a working solution as detailed at the end of this post.
>
> dom 0 grub
> menuentry "Xen 4.0.0-rc8 / Ubuntu 10.4 kernel 2.6.32.10-xen2 pvops" {
> insmod ext2
> set root='(hd0,2)'
> search --no-floppy --fs-uuid --set 321dedfa-be48-4b4d-bff0-0e5ecce240fd
> multiboot /boot/xen-4.0.0-rc8.gz dummy=dummy iommu=0
> module /boot/vmlinuz-2.6.32.10-xen2 dummy=dummy root=/dev/sdc2 ro  
> max_loop=64 xen-pciback.hide=(07:01.0)
> module /boot/initrd.img-2.6.32.10-xen2
> }
>
> # xm pci-list-assignable-devices
> 0000:07:01.0
>
> # dmesg |grep 07:01.0
> [    0.000000] Command line: root=/dev/sdc2 ro max_loop=64  
> xen-pciback.hide=(07:01.0)
> [    0.211959] Kernel command line: root=/dev/sdc2 ro max_loop=64  
> xen-pciback.hide=(07:01.0)
> [    0.520930] pci 0000:07:01.0: reg 10 32bit mmio: [0xf4000000-0xf7ffffff]
> [    0.558471] pciback 0000:07:01.0: seizing device
> [    0.609245] pciback 0000:07:01.0: PCI INT A -> GSI 16 (level, low) ->  
> IRQ 16
> [    0.609299] pciback 0000:07:01.0: PCI INT A disabled
> [  644.828556] pciback: vpci: 0000:07:01.0: assign to virtual slot 0
> [  731.747948] pciback: vpci: 0000:07:01.0: assign to virtual slot 0
> <note this shows up when i start the domU>
> [  731.748392] pciback 0000:07:01.0: device has been assigned to another  
> domain! Over-writting the ownership, but beware.
>
> at which point i can see card assigned:
> # xm pci-list lucidpv
> domain bus  slot func
> 0x0000 0x07 0x01 0x0
>
>
>
> domU grub
> title        Ubuntu lucid (development branch), kernel 2.6.32.10-xen2
> root        (hd0)
> kernel        /boot/vmlinuz-2.6.32.10-xen2  
> root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a2e ro iommu=soft 
> swiotlb=force
> initrd        /boot/initrd.img-2.6.32.10-xen2
>
>
> domU config
> kernel="/usr/lib/xen/boot/pv-grub-x86_64.gz"
> extra="(hd0)/boot/grub/menu.lst"
>
> name="lucidpv"
> memory=512
> maxmem=512
>
> vif=['bridge=xenbr0']
>
> disk = [ 'tap:aio:/etc/xen/images/lucidhvm.img,ioemu:xvda,w' ]
>
> pci = [ '07:01.0' ]
>
> on_reboot = 'destroy'
>
>
> domU output
> # xm create -c ubstub.cfg
> Using config file "/etc/xen/ubstub.cfg".
> Started domain lucidpv (id=2)
> Xen Minimal OS!
>  start_info: 0xa9a000(VA)
>    nr_pages: 0x20000
>  shared_inf: 0xbf44d000(MA)
>     pt_base: 0xa9d000(VA)
> nr_pt_frames: 0x9
>    mfn_list: 0x99a000(VA)
>   mod_start: 0x0(VA)
>     mod_len: 0
>       flags: 0x0
>    cmd_line: (hd0)/boot/grub/menu.lst
>  stack:      0x959980-0x979980
> MM: Init
>      _text: 0x0(VA)
>     _etext: 0x6987c(VA)
>   _erodata: 0x83000(VA)
>     _edata: 0x8bae0(VA)
> stack start: 0x959980(VA)
>       _end: 0x999f88(VA)
>  start_pfn: aa9
>    max_pfn: 20000
> Mapping memory range 0xc00000 - 0x20000000
> setting 0x0-0x83000 readonly
> skipped 0x1000
> MM: Initialise page allocator for ba3000(ba3000)-20000000(20000000)
> MM: done
> Demand map pfns at 20001000-2020001000.
> Heap resides at 2020002000-4020002000.
>  Booting command-list
>
> root  (hd0)
> Filesystem type is ext2fs, using whole disk
> kernel  /boot/vmlinuz-2.6.32.10-xen2  
> root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a
> 2e ro iommu=soft swiotlb=force
> initrd  /boot/initrd.img-2.6.32.10-xen2
>
> block error -2 for op 2
> close blk: backend=/local/domain/0/backend/vbd/2/51712 node=device/vbd/51712
> port 5 still bound!
>
>
> SOLUTION:
> Setting the following in the .cfg works:
>
> kernel = '/boot/vmlinuz-2.6.32.10-xen2'
> ramdisk= '/boot/initrd.img-2.6.32.10-xen2'
> extra='root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a2e ro iommu=soft  
> swiotlb=force'
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works

[Samuel Thibault]

(Note: I don't know at all about what iommu=soft does, so for now I'm
just as clueless as you are, and do not have the time to investage right
in the near future).

Samuel

Re: pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works

[Konrad Rzeszutek Wilk]

On Tue, Apr 06, 2010 at 02:25:07AM +0200, Samuel Thibault wrote:
> (Note: I don't know at all about what iommu=soft does, so for now I'm

The iommu=soft enables in the kernel the Software IOMMU. This is
required for PCI passthrough devices. But the problem isn't with those
flags (I've asked the user to extract the kernel and initrd and use
those exact parameters and just do a PV boot - and it worked).

The problem is that MiniOS is doing something wrongly.. I don't know if
the problem is with arguments or if the underlaying storage is the issue
(tap:aio) with MiniOS.

> just as clueless as you are, and do not have the time to investage right
> in the near future).

Heheh. Yeah figured but in case you have a todo list thought this might
be a good thing to put there.

Re: pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works

[Samuel Thibault]

Konrad Rzeszutek Wilk, le Wed 07 Apr 2010 12:24:16 -0400, a écrit :
> On Tue, Apr 06, 2010 at 02:25:07AM +0200, Samuel Thibault wrote:
> > (Note: I don't know at all about what iommu=soft does, so for now I'm
> 
> The iommu=soft enables in the kernel the Software IOMMU. This is
> required for PCI passthrough devices. But the problem isn't with those
> flags (I've asked the user to extract the kernel and initrd and use
> those exact parameters and just do a PV boot - and it worked).
> 
> The problem is that MiniOS is doing something wrongly..

In combination with those flags.

> I don't know if the problem is with arguments

I doubt this as I do already pass arguments to my kernels fine.

> or if the underlaying storage is the issue (tap:aio) with MiniOS.

The user should try the simplest way (i.e. the provided pv-grub
configuration examples), and then from there, check which flags makes
things not work.

Samuel

Re: pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works

[listmail]

Konrad previously stated:

"If the 'iommu=soft swiotlb=force' is removed it boots"

but in actuality,  "switotlb=force" can be left in and it boots ok as well.  From what I've see it is only when having "iommu=soft" added in that exposes the issue.  The original post provides a failing example and a working example, so I'm not sure what further examples I can provide.   I understand that you have communicated that you don't have time for this in near future and I respect that, however, if there is something specific to try that would shed more light on the issue then let me know.

Richie


Samuel Thibault wrote:
> Konrad Rzeszutek Wilk, le Wed 07 Apr 2010 12:24:16 -0400, a écrit :
>   
>> On Tue, Apr 06, 2010 at 02:25:07AM +0200, Samuel Thibault wrote:
>>     
>>> (Note: I don't know at all about what iommu=soft does, so for now I'm
>>>       
>> The iommu=soft enables in the kernel the Software IOMMU. This is
>> required for PCI passthrough devices. But the problem isn't with those
>> flags (I've asked the user to extract the kernel and initrd and use
>> those exact parameters and just do a PV boot - and it worked).
>>
>> The problem is that MiniOS is doing something wrongly..
>>     
>
> In combination with those flags.
>
>   
>> I don't know if the problem is with arguments
>>     
>
> I doubt this as I do already pass arguments to my kernels fine.
>
>   
>> or if the underlaying storage is the issue (tap:aio) with MiniOS.
>>     
>
> The user should try the simplest way (i.e. the provided pv-grub
> configuration examples), and then from there, check which flags makes
> things not work.
>
> Samuel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>   

Re: pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works

[Samuel Thibault]

listmail, le Wed 07 Apr 2010 12:53:19 -0400, a écrit :
> "If the 'iommu=soft swiotlb=force' is removed it boots"
> 
> but in actuality,  "switotlb=force" can be left in and it boots ok as well. 
> From what I've see it is only when having "iommu=soft" added in that 
> exposes the issue.

Ok, so the problem really is the combination of pv-grub and iommu=soft.

> I understand that you have communicated that you don't have time for
> this in near future and I respect that, however, if there is something
> specific to try that would shed more light on the issue then let me
> know.

I have no idea off-hand.  In "I don't have time in the near future", I
mean that other people could perhaps have a closer look at the issue...

Samuel

Re: pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works

[Bruce Edge]

[-- Attachment #1.1: Type: text/plain, Size: 1230 bytes --]

On Wed, Apr 7, 2010 at 9:57 AM, Samuel Thibault <
samuel.thibault@ens-lyon.org> wrote:

> listmail, le Wed 07 Apr 2010 12:53:19 -0400, a écrit :
> > "If the 'iommu=soft swiotlb=force' is removed it boots"
> >
> > but in actuality,  "switotlb=force" can be left in and it boots ok as
> well.
> > From what I've see it is only when having "iommu=soft" added in that
> > exposes the issue.
>
> Ok, so the problem really is the combination of pv-grub and iommu=soft.
>

Any update on this problem?
I'm assuming that this is related to my inability to pass through PCI
devices to a domU when using pv-grub as such requires the use of iommu=soft.

-Bruce


>
> > I understand that you have communicated that you don't have time for
> > this in near future and I respect that, however, if there is something
> > specific to try that would shed more light on the issue then let me
> > know.
>
> I have no idea off-hand.  In "I don't have time in the near future", I
> mean that other people could perhaps have a closer look at the issue...
>
> Samuel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

[-- Attachment #1.2: Type: text/html, Size: 1979 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works

[Richie]

Are you able to pv-grub boot while specifying iommu=soft?

To clarify this specific issue; the problem was that minios would 
produce an error message (see example, specifically: port 5 still 
bound!) and fail to boot the kernel.  I suspect if that if you are able 
to boot then the kernel would receive and make use of the parameter as 
it normally would.

# xm create -c ubstub.cfg
> Using config file "/etc/xen/ubstub.cfg".
> Started domain lucidpv (id=2)
> Xen Minimal OS!
>  start_info: 0xa9a000(VA)
>    nr_pages: 0x20000
>  shared_inf: 0xbf44d000(MA)
>     pt_base: 0xa9d000(VA)
> nr_pt_frames: 0x9
>    mfn_list: 0x99a000(VA)
>   mod_start: 0x0(VA)
>     mod_len: 0
>       flags: 0x0
>    cmd_line: (hd0)/boot/grub/menu.lst
>  stack:      0x959980-0x979980
> MM: Init
>      _text: 0x0(VA)
>     _etext: 0x6987c(VA)
>   _erodata: 0x83000(VA)
>     _edata: 0x8bae0(VA)
> stack start: 0x959980(VA)
>       _end: 0x999f88(VA)
>  start_pfn: aa9
>    max_pfn: 20000
> Mapping memory range 0xc00000 - 0x20000000
> setting 0x0-0x83000 readonly
> skipped 0x1000
> MM: Initialise page allocator for ba3000(ba3000)-20000000(20000000)
> MM: done
> Demand map pfns at 20001000-2020001000.
> Heap resides at 2020002000-4020002000.
>  Booting command-list
>
> root  (hd0)
> Filesystem type is ext2fs, using whole disk
> kernel  /boot/vmlinuz-2.6.32.10-xen2  
> root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a
> 2e ro iommu=soft swiotlb=force
> initrd  /boot/initrd.img-2.6.32.10-xen2
>
> block error -2 for op 2
> close blk: backend=/local/domain/0/backend/vbd/2/51712 node=device/vbd/51712
> port 5 still bound!
>


Bruce Edge wrote:
> On Wed, Apr 7, 2010 at 9:57 AM, Samuel Thibault 
> <samuel.thibault@ens-lyon.org <mailto:samuel.thibault@ens-lyon.org>> 
> wrote:
>
>     listmail, le Wed 07 Apr 2010 12:53:19 -0400, a écrit :
>     > "If the 'iommu=soft swiotlb=force' is removed it boots"
>     >
>     > but in actuality,  "switotlb=force" can be left in and it boots
>     ok as well.
>     > From what I've see it is only when having "iommu=soft" added in that
>     > exposes the issue.
>
>     Ok, so the problem really is the combination of pv-grub and
>     iommu=soft.
>
>
> Any update on this problem?
> I'm assuming that this is related to my inability to pass through PCI 
> devices to a domU when using pv-grub as such requires the use of 
> iommu=soft.
>
> -Bruce
>  
>
>
>     > I understand that you have communicated that you don't have time for
>     > this in near future and I respect that, however, if there is
>     something
>     > specific to try that would shed more light on the issue then let me
>     > know.
>
>     I have no idea off-hand.  In "I don't have time in the near future", I
>     mean that other people could perhaps have a closer look at the
>     issue...
>
>     Samuel
>
>     _______________________________________________
>     Xen-devel mailing list
>     Xen-devel@lists.xensource.com <mailto:Xen-devel@lists.xensource.com>
>     http://lists.xensource.com/xen-devel
>
>

Re: pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works

[Bruce Edge]

[-- Attachment #1.1: Type: text/plain, Size: 4305 bytes --]

On Tue, Jul 6, 2010 at 9:24 AM, Richie <listmail@triad.rr.com> wrote:

> Are you able to pv-grub boot while specifying iommu=soft?
>

The 2 options I'm playing around with are:  iommu=soft and swiotlb=force

With both:
With just iommu=soft:
close network: backend at /local/domain/0/backend/vif/29/0
port 4 still bound!
<hang>

With just iommu=soft:
close network: backend at /local/domain/0/backend/vif/29/0
port 4 still bound!
<hang>

With just swiotlb=force:
Has problems mounting rootfs. Fails inconsistenly, but always in some
reference to mounting root.

With neither option:
Boots ok, but lspci shows no devices even though dom0 shows 2 PCI devices:

   (image
        (linux
            (kernel /usr/lib/xen/boot/pv-grub-x86_64.gz)
            (args '(nd)/import/tools/pxe/kaan-01-dpm/menu.lst')
            (superpages 0)
            (videoram 4)
            (pci
                ((0x0000 0x04 0x00 0x0 0x100 ())
                    (0x0000 0x04 0x00 0x1 0x100 ())
                )
            )
            (serial pty)
            (nomigrate 0)
            (tsc_mode 0)
            (notes)
        )
    )

-Bruce


>
> To clarify this specific issue; the problem was that minios would produce
> an error message (see example, specifically: port 5 still bound!) and fail
> to boot the kernel.  I suspect if that if you are able to boot then the
> kernel would receive and make use of the parameter as it normally would.
>
>
> # xm create -c ubstub.cfg
>
>> Using config file "/etc/xen/ubstub.cfg".
>> Started domain lucidpv (id=2)
>> Xen Minimal OS!
>>  start_info: 0xa9a000(VA)
>>   nr_pages: 0x20000
>>  shared_inf: 0xbf44d000(MA)
>>    pt_base: 0xa9d000(VA)
>> nr_pt_frames: 0x9
>>   mfn_list: 0x99a000(VA)
>>  mod_start: 0x0(VA)
>>    mod_len: 0
>>      flags: 0x0
>>   cmd_line: (hd0)/boot/grub/menu.lst
>>  stack:      0x959980-0x979980
>> MM: Init
>>     _text: 0x0(VA)
>>    _etext: 0x6987c(VA)
>>  _erodata: 0x83000(VA)
>>    _edata: 0x8bae0(VA)
>> stack start: 0x959980(VA)
>>      _end: 0x999f88(VA)
>>  start_pfn: aa9
>>   max_pfn: 20000
>> Mapping memory range 0xc00000 - 0x20000000
>> setting 0x0-0x83000 readonly
>> skipped 0x1000
>> MM: Initialise page allocator for ba3000(ba3000)-20000000(20000000)
>> MM: done
>> Demand map pfns at 20001000-2020001000.
>> Heap resides at 2020002000-4020002000.
>>  Booting command-list
>>
>> root  (hd0)
>> Filesystem type is ext2fs, using whole disk
>> kernel  /boot/vmlinuz-2.6.32.10-xen2
>>  root=UUID=2bb30c38-70fc-4e9d-ae69-52db68589a
>> 2e ro iommu=soft swiotlb=force
>> initrd  /boot/initrd.img-2.6.32.10-xen2
>>
>> block error -2 for op 2
>> close blk: backend=/local/domain/0/backend/vbd/2/51712
>> node=device/vbd/51712
>> port 5 still bound!
>>
>>
>
> Bruce Edge wrote:
>
>> On Wed, Apr 7, 2010 at 9:57 AM, Samuel Thibault <
>> samuel.thibault@ens-lyon.org <mailto:samuel.thibault@ens-lyon.org>>
>> wrote:
>>
>>    listmail, le Wed 07 Apr 2010 12:53:19 -0400, a écrit :
>>    > "If the 'iommu=soft swiotlb=force' is removed it boots"
>>    >
>>    > but in actuality,  "switotlb=force" can be left in and it boots
>>    ok as well.
>>    > From what I've see it is only when having "iommu=soft" added in that
>>    > exposes the issue.
>>
>>    Ok, so the problem really is the combination of pv-grub and
>>    iommu=soft.
>>
>>
>> Any update on this problem?
>> I'm assuming that this is related to my inability to pass through PCI
>> devices to a domU when using pv-grub as such requires the use of iommu=soft.
>>
>> -Bruce
>>
>>
>>    > I understand that you have communicated that you don't have time for
>>    > this in near future and I respect that, however, if there is
>>    something
>>    > specific to try that would shed more light on the issue then let me
>>    > know.
>>
>>    I have no idea off-hand.  In "I don't have time in the near future", I
>>    mean that other people could perhaps have a closer look at the
>>    issue...
>>
>>    Samuel
>>
>>    _______________________________________________
>>    Xen-devel mailing list
>>    Xen-devel@lists.xensource.com <mailto:Xen-devel@lists.xensource.com>
>>
>>    http://lists.xensource.com/xen-devel
>>
>>
>>
>

[-- Attachment #1.2: Type: text/html, Size: 6023 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works

[Konrad Rzeszutek Wilk]

On Tue, Jul 06, 2010 at 12:24:15PM -0400, Richie wrote:
> Are you able to pv-grub boot while specifying iommu=soft?

No.
> 
> To clarify this specific issue; the problem was that minios would
> produce an error message (see example, specifically: port 5 still
> bound!) and fail to boot the kernel.  I suspect if that if you are

I get another error, wherein Linux kernel does boot but up to
initializing the SWIOTLB, and then it crashes due to Xen not allowing
SWIOTLB to swizzle the pages. This is what Xen says:

(XEN) mm.c:3847:d10 Bad page 00000000001beab4: ed=ffff830151210000(10), 
sd=ffff830151210000, caf=8000000000000002, taf=7400000000000001

Also experienced by another user:
http://lists.xensource.com/archives/html/xen-devel/2010-04/msg00847.html

Haven't dug deeper in the hypervisor to figure out the caf/taf business.

Re: pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works. MiniOS screws around with page flags.

[Konrad Rzeszutek Wilk]

On Tue, Jul 06, 2010 at 09:06:26AM -0700, Bruce Edge wrote:
> On Wed, Apr 7, 2010 at 9:57 AM, Samuel Thibault <
> samuel.thibault@ens-lyon.org> wrote:
> 
> > listmail, le Wed 07 Apr 2010 12:53:19 -0400, a écrit :
> > > "If the 'iommu=soft swiotlb=force' is removed it boots"
> > >
> > > but in actuality,  "switotlb=force" can be left in and it boots ok as
> > well.
> > > From what I've see it is only when having "iommu=soft" added in that
> > > exposes the issue.
> >
> > Ok, so the problem really is the combination of pv-grub and iommu=soft.
> >
> 
> Any update on this problem?

Thanks for reminding me. I fixed one problem (where the 'vfb' would hang
pv-grub) but have't addressed this one.

Here is what I get:


Bootstrapping...
Xen Minimal OS!
  start_info: 0xb9d000(VA)
    nr_pages: 0x40000
  shared_inf: 0xb45ff000(MA)
     pt_base: 0xba0000(VA)
nr_pt_frames: 0xb
    mfn_list: 0x99d000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line: (hd0,0)/grub/menu.lst
  stack:      0x95c980-0x97c980
MM: Init
      _text: 0x0(VA)
     _etext: 0x6bf94(VA)
   _erodata: 0x86000(VA)
     _edata: 0x8eae0(VA)
stack start: 0x95c980(VA)
       _end: 0x99cf88(VA)
  start_pfn: bae
    max_pfn: 40000
Mapping memory range 0x1000000 - 0x40000000
setting 0x0-0x86000 readonly
skipped 0x1000
MM: Initialise page allocator for da6000(da6000)-40000000(40000000)
MM: done
Demand map pfns at 40001000-2040001000.
Heap resides at 2040002000-4040002000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x40001000.
Initialising scheduler
Thread "Idle": pointer: 0x2040002050, stack: 0xfc0000
Initialising xenbus
Thread "xenstore": pointer: 0x2040002800, stack: 0xfd0000
Dummy main: start_info=0x97ca80
Thread "main": pointer: 0x2040002fb0, stack: 0xfe0000
Thread "pcifront": pointer: 0x2040003760, stack: 0xff0000
pcifront_watches: waiting for backend path to appear device/pci/0/backend
vbd 51712 is hd0
******************* BLKFRONT for device/vbd/51712 **********


backend at /local/domain/0/backend/vbd/6/51712
[ 9955.307884] blkback: ring-ref 2047, event-channel 4, protocol 1 (x86_64-abi)
Failed to read /local/domain/0/backend/vbd/6/51712/feature-flush-cache.
41943040 sectors of 512 bytes
**************************
Thread "kbdfront": pointer: 0x2040004590, stack: 0x1000000
******************* FBFRONT for device/vfb/0 **********


******************* KBDFRONT for device/vkbd/0 **********


backend at /local/domain/0/backend/vkbd/6/0
/local/domain/0/backend/vkbd/6/0 connected
************************** KBDFRONT
Thread "kbdfront" exited.
backend at /local/domain/0/backend/vfb/6/0
/local/domain/0/backend/vfb/6/0 connected
************************** FBFRONT
[ 9966.656265] vif6.0: no IPv6 routers present
[ 9971.301098] switch: port 2(vif6.0) entering forwarding state
Thread "kbdfront close": pointer: 0x2040004590, stack: 0x1000000
close fb: backend at /local/domain/0/backend/vfb/6/0
close kbd: backend at /local/domain/0/backend/vkbd/6/0
Thread "kbdfront close" exited.
close blk: backend=/local/domain/0/backend/vbd/6/51712 node=device/vbd/51712
[ 9974.405828] frontend_changed: backend/vbd/6/51712: prepare for reconnect
go!
mapping kernel into physical memory
about to get started...
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.34NEB (root@vs03.dumpdata.com) (gcc version 4.4.4 20100503 (Red Hat 4.4.4-2) (GCC) ) #2 SMP Fri Jun 25 15:01:02 EDT 2010
[    0.000000] Command line: ro root=/dev/mapper/vg_vs03-lv_root rd_LVM_LV=vg_vs03/lv_root rd_LVM_LV=vg_vs03/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us debug earlyprintk=xen console=hvc0 iommu=swiotlb swiotlb=force
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000000040000000 (usable)
[    0.000000] bootconsole [xenboot0] enabled
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved)
[    0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x40000 max_arch_pfn = 0x400000000
[    0.000000] e820 update range: 0000000000001000 - 0000000000010000 (usable) ==> (reserved)
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] modified physical RAM map:
[    0.000000]  modified: 0000000000000000 - 0000000000010000 (reserved)
[    0.000000]  modified: 0000000000010000 - 00000000000a0000 (usable)
[    0.000000]  modified: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  modified: 0000000000100000 - 0000000040000000 (usable)
[    0.000000] initial memory mapped : 0 - 20000000
[    0.000000] init_memory_mapping: 0000000000000000-0000000040000000
[    0.000000]  0000000000 - 0040000000 page 4k
[    0.000000] kernel direct mapping tables up to 40000000 @ 100000-302000
[    0.000000] RAMDISK: 01a23000 - 024c0000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at 0000000000000000-0000000040000000
[    0.000000] Initmem setup node 0 0000000000000000-0000000040000000
[    0.000000]   NODE_DATA [0000000001a02180 - 0000000001a0717f]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   empty
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x000000a0
[    0.000000]     0: 0x00000100 -> 0x00040000
[    0.000000] On node 0 totalpages: 262032
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 3928 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 3528 pages used for memmap
[    0.000000]   DMA32 zone: 254520 pages, LIFO batch:31
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] nr_irqs_gsi: 16
[    0.000000] early_res array is doubled to 64 at [16000 - 167ff]
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[    0.000000] Allocating PCI resources starting at 40000000 (gap: 40000000:c0000000)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.1-100622 (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:4 nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff8800026da000 s84776 r8192 d21720 u114688
[    0.000000] pcpu-alloc: s84776 r8192 d21720 u114688 alloc=28*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] trying to map vcpu_info 0 at ffff8800026e5020, mfn 1425c9, offset 32
[    0.000000] cpu 0 using vcpu_info at ffff8800026e5020
[    0.000000] trying to map vcpu_info 1 at ffff880002701020, mfn 1425ad, offset 32
[    0.000000] cpu 1 using vcpu_info at ffff880002701020
[    0.000000] trying to map vcpu_info 2 at ffff88000271d020, mfn 142591, offset 32
[    0.000000] cpu 2 using vcpu_info at ffff88000271d020
[    0.000000] trying to map vcpu_info 3 at ffff880002739020, mfn 142575, offset 32
[    0.000000] cpu 3 using vcpu_info at ffff880002739020
[    0.000000] Xen: using vcpu_info placement
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 258448
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: ro root=/dev/mapper/vg_vs03-lv_root rd_LVM_LV=vg_vs03/lv_root rd_LVM_LV=vg_vs03/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us debug earlyprintk=xen console=hvc0 iommu=swiotlb swiotlb=force
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
(XEN) mm.c:4137:d6 Bad page 00000000001331a9: ed=ffff83033ffc0000(6), sd=ffff83033ffc0000, caf=8000000000000002, taf=7400000000000001
(XEN) mm.c:4137:d6 Bad page 00000000001331a9: ed=ffff83033ffc0000(6), sd=ffff83033ffc0000, caf=8000000000000002, taf=7400000000000001
(XEN) mm.c:4137:d6 Bad page 00000000001331a9: ed=ffff83033ffc0000(6), sd=ffff83033ffc0000, caf=8000000000000002, taf=7400000000000001
(XEN) mm.c:4137:d6 Bad page 00000000001331a9: ed=ffff83033ffc0000(6), sd=ffff83033ffc0000, caf=8000000000000002, taf=7400000000000001
(XEN) mm.c:4137:d6 Bad page 00000000001331a9: ed=ffff83033ffc0000(6), sd=ffff83033ffc0000, caf=8000000000000002, taf=7400000000000001
(XEN) mm.c:4137:d6 Bad page 00000000001331a9: ed=ffff83033ffc0000(6), sd=ffff83033ffc0000, caf=8000000000000002, taf=7400000000000001
[    0.000000] Kernel panic - not syncing: DMA(-12): Failed to exchange pages allocated for DMA with Xen! We either don\047t have the permission or you do not have enoughfree memory under 4GB!
[    0.000000] 
[    0.000000] Pid: 0, comm: swapper Not tainted 2.6.34NEB #2
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff8145373c>] panic+0x73/0xee
[    0.000000]  [<ffffffff810052c2>] ? xen_create_contiguous_region+0xd3/0xee
[    0.000000]  [<ffffffff818c6946>] xen_swiotlb_init+0x10a/0x142
[    0.000000]  [<ffffffff8145cf35>] ? _etext+0x0/0x3
[    0.000000]  [<ffffffff818a5763>] pci_xen_swiotlb_init+0x17/0x24
[    0.000000]  [<ffffffff818a808f>] pci_iommu_alloc+0x2f/0x36
[    0.000000]  [<ffffffff818b5c05>] mem_init+0x19/0xec
[    0.000000]  [<ffffffff818a1b7c>] start_kernel+0x1ce/0x3ae
[    0.000000]  [<ffffffff818a12c8>] x86_64_start_reservations+0xb3/0xb7
[    0.000000]  [<ffffffff818a437b>] xen_start_kernel+0x54b/0x54f
[    0.000000] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[    0.000000] IP: [<ffffffff8122b7c0>] xen_evtchn_do_upcall+0xe5/0x19a
[    0.000000] PGD 0 
[    0.000000] Oops: 0000 [#1] SMP 
[    0.000000] last sysfs file: 
[    0.000000] CPU 0 
[    0.000000] Modules linked in:
[    0.000000] 
[    0.000000] Pid: 0, comm: swapper Not tainted 2.6.34NEB #2 /
[    0.000000] RIP: e030:[<ffffffff8122b7c0>]  [<ffffffff8122b7c0>] xen_evtchn_do_upcall+0xe5/0x19a
[    0.000000] RSP: e02b:ffff8800026ddf28  EFLAGS: 00010002
[    0.000000] RAX: 0000000000000000 RBX: 00000000000114e0 RCX: 0000000000000002
[    0.000000] RDX: 0000000000000100 RSI: 0000000000000000 RDI: 0000000000000000
[    0.000000] RBP: ffff8800026ddfa8 R08: 0000000000000000 R09: ffffffff81801e38
[    0.000000] R10: 0000000000000006 R11: 0000000000000000 R12: ffffffffffffffff
[    0.000000] R13: 0000000000000000 R14: ffffffffff57a000 R15: 00000000000114e0
[    0.000000] FS:  0000000000000000(0000) GS:ffff8800026da000(0000) knlGS:0000000000000000
[    0.000000] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[    0.000000] CR2: 0000000000000008 CR3: 0000000001811000 CR4: 0000000000002660
[    0.000000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    0.000000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[    0.000000] Process swapper (pid: 0, threadinfo ffffffff81800000, task ffffffff81819020)
[    0.000000] Stack:
[    0.000000]  0000000000000000 0000000000000000 0000000000000140 0000000000000000
[    0.000000] <0> ffff8800026e5028 0000000000000000 0000000000000000 0000000000000100
[    0.000000] <0> ffff8800026e5020 0000000000000000 0000000000000000 0000000000000001
[    0.000000] Call Trace:
[    0.000000]  <IRQ> 
[    0.000000]  [<ffffffff8100a92e>] xen_do_hypervisor_callback+0x1e/0x30
[    0.000000]  <EOI> 
[    0.000000]  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1006
[    0.000000]  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1006
[    0.000000]  [<ffffffff81041f74>] ? release_console_sem+0x18f/0x1c7
[    0.000000]  [<ffffffff8100647d>] ? xen_force_evtchn_callback+0xd/0xf
[    0.000000]  [<ffffffff81006b52>] ? check_events+0x12/0x20
[    0.000000]  [<ffffffff81006af9>] ? xen_irq_enable_direct_end+0x0/0x7
[    0.000000]  [<ffffffff814537b0>] ? panic+0xe7/0xee
[    0.000000]  [<ffffffff810052c2>] ? xen_create_contiguous_region+0xd3/0xee
[    0.000000]  [<ffffffff818c6946>] ? xen_swiotlb_init+0x10a/0x142
[    0.000000]  [<ffffffff8145cf35>] ? _etext+0x0/0x3
[    0.000000]  [<ffffffff818a5763>] ? pci_xen_swiotlb_init+0x17/0x24
[    0.000000]  [<ffffffff818a808f>] ? pci_iommu_alloc+0x2f/0x36
[    0.000000]  [<ffffffff818b5c05>] ? mem_init+0x19/0xec
[    0.000000]  [<ffffffff818a1b7c>] ? start_kernel+0x1ce/0x3ae
[    0.000000]  [<ffffffff818a12c8>] ? x86_64_start_reservations+0xb3/0xb7
[    0.000000]  [<ffffffff818a437b>] ? xen_start_kernel+0x54b/0x54f
[    0.000000] Code: 00 00 00 89 55 ac 48 8d 90 40 01 00 00 48 05 00 01 00 00 48 89 4d 98 48 89 45 b8 48 89 55 90 eb 2c 48 0f bc c9 03 4d ac 48 63 c9 <8b> 14 88 83 fa ff 74 f8 89 d7 89 55 88 e8 59 92 e5 ff 48 85 c0 
[    0.000000] RIP  [<ffffffff8122b7c0>] xen_evtchn_do_upcall+0xe5/0x19a
[    0.000000]  RSP <ffff8800026ddf28>
[    0.000000] CR2: 0000000000000008
[    0.000000] ---[ end trace 4eaa2a86a8e2da22 ]---
[    0.000000] Kernel panic - not syncing: Fatal exception in interrupt
[    0.000000] Pid: 0, comm: swapper Tainted: G      D    2.6.34NEB #2
[    0.000000] Call Trace:
[    0.000000]  <IRQ>  [<ffffffff8145373c>] panic+0x73/0xee
[    0.000000]  [<ffffffff814565c0>] oops_end+0xb7/0xc7
[    0.000000]  [<ffffffff8102d91b>] no_context+0x1f7/0x206
[    0.000000]  [<ffffffff8102daa9>] __bad_area_nosemaphore+0x17f/0x1a2
[    0.000000]  [<ffffffff8102dada>] bad_area_nosemaphore+0xe/0x10
[    0.000000]  [<ffffffff81458617>] do_page_fault+0x18b/0x2a3
[    0.000000]  [<ffffffff81455a75>] page_fault+0x25/0x30
[    0.000000]  [<ffffffff8122b7c0>] ? xen_evtchn_do_upcall+0xe5/0x19a
[    0.000000]  [<ffffffff8122b745>] ? xen_evtchn_do_upcall+0x6a/0x19a
[    0.000000]  [<ffffffff8100a92e>] xen_do_hypervisor_callback+0x1e/0x30
[    0.000000]  <EOI>  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1006
[    0.000000]  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1006
[    0.000000]  [<ffffffff81041f74>] ? release_console_sem+0x18f/0x1c7
[    0.000000]  [<ffffffff8100647d>] ? xen_force_evtchn_callback+0xd/0xf
[    0.000000]  [<ffffffff81006b52>] ? check_events+0x12/0x20
[    0.000000]  [<ffffffff81006af9>] ? xen_irq_enable_direct_end+0x0/0x7
[    0.000000]  [<ffffffff814537b0>] ? panic+0xe7/0xee
[    0.000000]  [<ffffffff810052c2>] ? xen_create_contiguous_region+0xd3/0xee
[    0.000000]  [<ffffffff818c6946>] ? xen_swiotlb_init+0x10a/0x142
[    0.000000]  [<ffffffff8145cf35>] ? _etext+0x0/0x3
[    0.000000]  [<ffffffff818a5763>] ? pci_xen_swiotlb_init+0x17/0x24
[    0.000000]  [<ffffffff818a808f>] ? pci_iommu_alloc+0x2f/0x36
[    0.000000]  [<ffffffff818b5c05>] ? mem_init+0x19/0xec
[    0.000000]  [<ffffffff818a1b7c>] ? start_kernel+0x1ce/0x3ae
[    0.000000]  [<ffffffff818a12c8>] ? x86_64_start_reservations+0xb3/0xb7
[    0.000000]  [<ffffffff818a437b>] ? xen_start_kernel+0x54b/0x54f


I wonder if this is a similar problem that another user of Xen has
reported when they use blktap with pv-grub - it puts some weird flags
on the I/O pages so during shutdown the guest is in a weird state..

Re: pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works. MiniOS screws around with page flags.

[Richie]

Konrad Rzeszutek Wilk wrote:
> [    0.000000] Kernel command line: ro root=/dev/mapper/vg_vs03-lv_root rd_LVM_LV=vg_vs03/lv_root rd_LVM_LV=vg_vs03/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us debug earlyprintk=xen console=hvc0 iommu=swiotlb swiotlb=force
> [    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
>   
shouldn't that be iommu=soft or is it another valid spec.?

Re: pv-grub boot hangs when iommu=soft. Booting kernel directly with the flags works. MiniOS screws around with page flags.

[Konrad Rzeszutek Wilk]

On Tue, Jul 06, 2010 at 05:36:33PM -0400, Richie wrote:
> Konrad Rzeszutek Wilk wrote:
> >[    0.000000] Kernel command line: ro root=/dev/mapper/vg_vs03-lv_root rd_LVM_LV=vg_vs03/lv_root rd_LVM_LV=vg_vs03/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us debug earlyprintk=xen console=hvc0 iommu=swiotlb swiotlb=force
> >[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
> shouldn't that be iommu=soft or is it another valid spec.?

It should be as iommu=soft. If you use the override (swiotlb=force) it
can also automatically turn SWIOTLB on.