Re: [Xen-users] HVM PCI Passthrough: Code 12: Undersized PCI MMIO region?

Re: [Xen-users] HVM PCI Passthrough: Code 12: Undersized PCI MMIO region?

[Pasi Kärkkäinen]

On Tue, Dec 02, 2014 at 10:08:35PM -0500, Stephen Oberholtzer wrote:
> All,
> 

Hello,

> Please bear with me.  I don't fully understand all of the mechanisms
> involved here, so this message contains a large quantity of data,
> including some links to even larger quantities of data.
> The short version is: I think something isn't reserving enough (GPFN)
> address space for MMIO.
>

This issue probably should be taken to xen-devel mailinglist.. CC added..


> I'm trying to do a three-part PCI passthrough to a HVM domU running Windows 8.1:
> 00:14.0, An xHCI (USB 3.0) controller
> 01:00.0, Function 0 of an ATI card (VGA controller, non-primary passthrough)
> 01:00.1, Function 1 of the same ATI card (HDMI audio)
> 
> The USB controller seems to work fine.  USB mouse/keyboard are
> functional, and I plugged in a few Mass Storage devices and they work
> fine.
> The HDMI audio controller is harder to vet, because I don't have
> anything that will process audio without an associated video stream.
> However, I don't get any complains from the software.
> 
> For the the VGA controller (01:00.0), Windows reports "Code 12" and
> complains about insufficient resources being available for the device.
> 
> I tried to gather all of the information I could on the memory layouts
> involved, from three sources:
> (1) From dom0 and Xen itself
> (2) From domU running Windows, using a program called "SIV" (System
> Information Viewer)
> (3) From domU running Linux (by booting a Knoppix live DVD image)
> 
> ---
> 
> Information from dom0:
> http://www.klozoff.org/xen/dom0-xl-info.txt - 'xl info'
> http://www.klozoff.org/xen/dom0-xl-dmesg.txt - 'xl dmesg'
> http://www.klozoff.org/xen/dom0-proc_iomem.txt - 'cat /proc/iomem'
> http://www.klozoff.org/xen/dom0-lspci-output.txt - 'lspci -vvv' for
> the three devices I'm forwarding.
> 
> Information from domU (Windows):
> http://www.klozoff.org/xen/domU-device-manager-info.txt - Text
> transcribed from Windows' "Device Manager"
> http://www.klozoff.org/xen/domU-siv-usb-controller-info.txt -
> Information on 00:14.0 from SIV
> http://www.klozoff.org/xen/domU-siv-video-card-info.txt - Information
> on 01:00.0 from SIV
> http://www.klozoff.org/xen/domU-siv-hdmi-audio-info.txt - Information
> on 01:00.1 from SIV
> 
> Information from domU (Linux/Knoppix):
> http://www.klozoff.org/xen/domU-knoppix-proc-iomem.txt - 'cat /proc/iomem'
> http://www.klozoff.org/xen/domU-knoppix-lspci-output.txt - 'lspci
> -vvv' for the three devices.
> 
> 
> ---
> 
> Of particular interest to me is domU-knoppix-lspci-output.txt, which
> for the VGA controller, says:
> 
>     Region 0: Memory at <ignored> (64-bit, prefetchable)
>     Region 2: Memory at f3040000 (64-bit, non-prefetchable) [size=256K]
> 
> In dom0, this showed as:
> 
>     Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M]
>     Region 2: Memory at efc00000 (64-bit, non-prefetchable) [size=256K]
> 
> 
> I looked closer by running the following script:
> 
> perl -lne '/^([0-9a-f]+)-([0-9a-f]+)/ && print $_, " (", ((hex($2) -
> hex($1)) >> 20), " MB)"' /proc/iomem
> 
> In dom0, I see this:
> 
> 00100000-99f53fff : System RAM (2462 MB)
> bf200000-feafffff : PCI Bus 0000:00 (1016 MB)   <- Region 0 is in this range
> 100000000-83edfffff : System RAM (29677 MB)
> 
> 
> In domU, I see this:
> 00100000-effecfff : System RAM (3838 MB)
> f0000000-fbffffff : PCI Bus 0000:00 (191 MB)    <- Definitely not big
> enough to hold the 256MB we need
> 100000000-20f7fffff : System RAM (4343 MB)
> 
> Now, I don't fully understand how this stuff all works, but here's
> what appears to be happening: Xen isn't reserving enough contiguous
> GPFN (virtual machine address space) for all of the devices on the
> virtual PCI bus -- in particular, for what is probably a big honkin'
> window into the card's onboard RAM.
> 
> So my two questions are:
> (1) Is my guess correct?  Or at least close?
> (2) What the heck do I do about it? I don't see any options in
> xl.cfg(5) for specifying the size of the MMIO region for the virtual
> PCI bus, or anything similar to that.
> 
> 
> NOTE: If I assign 2048MB or less RAM to this VM, the VM crashes at
> bootup, and has a nasty tendency to take the host with it (lockup).
> 
> -- 
> -- Stevie-O
> Real programmers use COPY CON PROGRAM.EXE
> 


-- Pasi


> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xen.org
> http://lists.xen.org/xen-users

Re: [Xen-users] HVM PCI Passthrough: Code 12: Undersized PCI MMIO region?

[Don Slutz]

On 01/06/15 08:00, Pasi Kärkkäinen wrote:
> On Tue, Dec 02, 2014 at 10:08:35PM -0500, Stephen Oberholtzer wrote:
>> All,
>>
> 
> Hello,
> 
>> Please bear with me.  I don't fully understand all of the mechanisms
>> involved here, so this message contains a large quantity of data,
>> including some links to even larger quantities of data.
>> The short version is: I think something isn't reserving enough (GPFN)
>> address space for MMIO.
>>

...

>>
>> Now, I don't fully understand how this stuff all works, but here's
>> what appears to be happening: Xen isn't reserving enough contiguous
>> GPFN (virtual machine address space) for all of the devices on the
>> virtual PCI bus -- in particular, for what is probably a big honkin'
>> window into the card's onboard RAM.
>>
>> So my two questions are:
>> (1) Is my guess correct?  Or at least close?

I think so.

>> (2) What the heck do I do about it? I don't see any options in
>> xl.cfg(5) for specifying the size of the MMIO region for the virtual
>> PCI bus, or anything similar to that.
>>

There is a new feature in xen 4.5 (soon to be released) that you can
add to the config file (xl.cfg):

   Memory layout
    mmio_hole=MBYTES
        Specifies the size the MMIO hole below 4GiB will be. Only valid for
        device_model_version = "qemu-xen".

        Cannot be smaller than 256. Cannot be larger than 3840.

        Known good large value is 3072.


Subject: [Xen-devel] Xen Project 4.5 RC4 is Ready Today; Test Day is
Wednesday
Date: Mon, 15 Dec 2014 14:33:33 -0500
From: Russ Pavlicek <russell.pavlicek@xenproject.org>
To: xen-devel@lists.xen.org, xen-users@lists.xen.org
<xen-users@lists.xen.org>, xen-announce@lists.xenproject.org,
xs-devel@lists.xenserver.org, mirageos-devel@lists.xenproject.org,
xen-api@lists.xen.org

Folks,

This Wednesday, December 17, is our fourth and FINAL Test Day
for the 4.5 release cycle (barring any changes which may result from
Wednesday's Test Day). Release Candidate 4 is available for
assessment today.

If you've held off testing the new release until it matures, delay no
longer!  Test and state your concerns now or the next release you
see could be the official one

Information about testing this release can be found here:
http://wiki.xenproject.org/wiki/Xen_4.5_RC4_test_instructions

To learn more about Test Days, including the scheduled date
for the final release, check out:
http://wiki.xenproject.org/wiki/Xen_Project_Test_Days

   -Don Slutz

>>
>> NOTE: If I assign 2048MB or less RAM to this VM, the VM crashes at
>> bootup, and has a nasty tendency to take the host with it (lockup).
>>
>> -- 
>> -- Stevie-O
>> Real programmers use COPY CON PROGRAM.EXE
>>
> 
> 
> -- Pasi
> 
> 
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xen.org
>> http://lists.xen.org/xen-users
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [Xen-users] HVM PCI Passthrough: Code 12: Undersized PCI MMIO region?

[Jan Beulich]

>>> On 06.01.15 at 14:00, <pasik@iki.fi> wrote:
> On Tue, Dec 02, 2014 at 10:08:35PM -0500, Stephen Oberholtzer wrote:
>> (1) Is my guess correct?  Or at least close?

Depends on whether you assign the devices at boot time of the guest,
or via hotplug. In the latter case, I suppose only the MMIO hole size
setting as pointed out by Don will help you, while in the former case I'd
expect things to work (and if not, using a debug hypervisor with high
enough log level would make hvmloader send information to the
hypervisor log that ought to help diagnosing the issue).

Jan