Re: Does xen-4.2.0 support VGA passthrough with the virtual machine created by xl command?

Re: Does xen-4.2.0 support VGA passthrough with the virtual machine created by xl command?

[Dr. Greg Wettstein]

On Nov 14,  3:40pm, "Dr. Greg Wettstein" wrote:
} Subject: Re: [Xen-devel] Does xen-4.2.0 support VGA passthrough with the v

Good morning, hope the day is going well for everyone.

> On Nov 13, 10:02am, Ian Campbell wrote:
> } Subject: Re: [Xen-devel] Does xen-4.2.0 support VGA passthrough with the v
> 
> Good afternoon, hope the week is going well for everyone.
> 
> > On Tue, 2012-11-13 at 06:30 +0000, Qian Hu wrote:
> > 
> > > With spice tool, I have to create a VM by xl command, and now I am
> > > wondering if it supports VGA passghrouth?
> 
> > This list is for the development of Xen. You';d probably have more
> > luck with these sorts of support requests on the xen-users list.
> 
> That would normally be the case but I'm suspicious there are issues
> with VGA passthrough in 4.2.0.

I just wanted to follow up to the list on the status of passthrough
issues.

We reverted our test machine back to the 2.6.32.45 kernel which we had
been using in production.  That kernel was based on Jeremy's GIT
tree.  Using xm and the updated ATI patches which I referenced in my
original mail passthrough works as it should.

Passthrough does not work with xl.  Windows started but fell into its
text mode rescue screen and registered a crash dump.  It flashed the
screen back and forth between a stipled blue/grey and totally black
screen a few times and then locked the physical machine up solidly.

On the next boot I thought about it but declined to register the crash
dump with Microsoft.... :-)

We then went back and tested the 3.4.18 kernel and with both xm and xl
the guest faults on the first attempt to do an I/O port access.  All
factors (windows image, hardware, xen guest config) are held identical
so the difference would seem to be linked to the PCI passthrough
implementation between the two kernels.  I've copied Konrad on the
note since he would seem to be the person most familiar with this
area.

I'm including below a diff between a successful qemu-dm passthrough
session (2.6.32.45) and an unsuccessful session (3.4.18).  It would
appear 3.4.18 is getting the both the I/O port and memory ranges
wrong.

Let me know if I can forward any additional information or run any
additional tests.

Have a good weekend.

Greg

qemu-dm log diff: ---------------------------------------------------------
1c1
< domid: 3
---
> domid: 1
3,5c3,5
< Watching /local/domain/0/device-model/3/logdirty/cmd
< Watching /local/domain/0/device-model/3/command
< Watching /local/domain/3/cpu
---
> Watching /local/domain/0/device-model/1/logdirty/cmd
> Watching /local/domain/0/device-model/1/command
> Watching /local/domain/1/cpu
9c9
< Guest uuid = 7fcefb13-d1ef-105b-e38c-1e1454411e80
---
> Guest uuid = eab6bbbb-4819-b970-a83c-03288a1541ad
14c14
< xs_read(/local/domain/0/device-model/3/xen_extended_power_mgmt): read error
---
> xs_read(/local/domain/0/device-model/1/xen_extended_power_mgmt): read error
18,20c18,20
< xs_read(/local/domain/3/log-throttling): read error
< qemu: ignoring not-understood drive `/local/domain/3/log-throttling'
< medium change watch on `/local/domain/3/log-throttling' - unknown device, ignored
---
> xs_read(/local/domain/1/log-throttling): read error
> qemu: ignoring not-understood drive `/local/domain/1/log-throttling'
> medium change watch on `/local/domain/1/log-throttling' - unknown device, ignored
69,106c69,77
< pt_iomem_map: e_phys=f1020000 maddr=c1a00000 type=0 len=131072 index=2 first_map=1
< pt_iomem_map: e_phys=f1060000 maddr=c1b22000 type=0 len=4096 index=0 first_map=1
< pt_ioport_map: e_phys=c600 pio_base=3000 len=256 index=4 first_map=1
< pt_ioport_map: e_phys=c600 pio_base=3000 len=256 index=4 first_map=0
< ati_gfx_init: guest_pio_bar = 0xc600, host_pio_bar = 0x3000, pio_size=0x100 guest_mmio_bar1=0xe0000000, guest_mmio_bar2=0x0
< platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
< platform_fixed_ioport: changed ro/rw state of ROM memory area. now is ro state.
< pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
< pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
< pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
< pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
< pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
< pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
< pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
< pt_iomem_map: e_phys=ffffffff maddr=b0000000 type=8 len=268435456 index=0 first_map=0
< pt_iomem_map: e_phys=ffffffff maddr=c1a00000 type=0 len=131072 index=2 first_map=0
< pt_iomem_map: e_phys=e0000000 maddr=b0000000 type=8 len=268435456 index=0 first_map=0
< pt_iomem_map: e_phys=f1020000 maddr=c1a00000 type=0 len=131072 index=2 first_map=0
< pt_ioport_map: e_phys=c600 pio_base=3000 len=256 index=4 first_map=0
< pt_iomem_map: e_phys=ffffffff maddr=c1b22000 type=0 len=4096 index=0 first_map=0
< pt_iomem_map: e_phys=f1060000 maddr=c1b22000 type=0 len=4096 index=0 first_map=0
< pt_iomem_map: e_phys=ffffffff maddr=b0000000 type=8 len=268435456 index=0 first_map=0
< pt_iomem_map: e_phys=ffffffff maddr=c1a00000 type=0 len=131072 index=2 first_map=0
< pt_iomem_map: e_phys=e0000000 maddr=b0000000 type=8 len=268435456 index=0 first_map=0
< pt_iomem_map: e_phys=f1020000 maddr=c1a00000 type=0 len=131072 index=2 first_map=0
< pt_ioport_map: e_phys=c600 pio_base=3000 len=256 index=4 first_map=0
< pt_ioport_map: e_phys=c600 pio_base=3000 len=256 index=4 first_map=0
< pt_msgctrl_reg_write: guest enabling MSI, disable MSI-INTx translation
< pci_intx: intx=1
< pt_msi_disable: Unmap msi with pirq 37
< pt_msgctrl_reg_write: setup msi for dev 30
< pt_msi_setup: msi mapped with pirq 37
< pt_msi_update: Update msi with pirq 37 gvec b0 gflags 0
< pt_iomem_map: e_phys=ffffffff maddr=c1b22000 type=0 len=4096 index=0 first_map=0
< pt_iomem_map: e_phys=f1060000 maddr=c1b22000 type=0 len=4096 index=0 first_map=0
< pt_iomem_map: e_phys=ffffffff maddr=c1b22000 type=0 len=4096 index=0 first_map=0
< shutdown requested in cpu_handle_ioreq
< Issued domain 3 poweroff
---
> pt_iomem_map: e_phys=f1000000 maddr=c1a00000 type=0 len=131072 index=2 first_map=1
> pt_iomem_map: e_phys=f1040000 maddr=c1b22000 type=0 len=4096 index=0 first_map=1
> pt_ioport_map: e_phys=c700 pio_base=3000 len=256 index=4 first_map=1
> pt_ioport_map: e_phys=c700 pio_base=3000 len=256 index=4 first_map=0
> ati_gfx_init: guest_pio_bar = 0xc700, host_pio_bar = 0x3000, pio_size=0x100 guest_mmio_bar1=0xe0000000, guest_mmio_bar2=0x0
> ati_io_regs_read: Requested read of c74c/51020, mapped: 304c/12364
> ati_hw_in: port I/O: 304c, base: 3000, size: 100
> ati_hw_in: ioperm successful
> ati_hw_in: Read: 0
---------------------------------------------------------------------------

}-- End of excerpt from "Dr. Greg Wettstein"

As always,
Dr. G.W. Wettstein, Ph.D.   Enjellic Systems Development, LLC.
4206 N. 19th Ave.           Specializing in information infra-structure
Fargo, ND  58102            development.
PH: 701-281-1686
FAX: 701-281-3949           EMAIL: greg@enjellic.com
------------------------------------------------------------------------------
"Boy, it must not take much to make a phone work.  Looking at
 everthing else here it must be the same way with the INTERNET."
                                -- Francis 'Fritz' Wettstein

Re: Does xen-4.2.0 support VGA passthrough with the virtual machine created by xl command?

[Konrad Rzeszutek Wilk]

On Sat, Nov 17, 2012 at 09:17:36AM -0600, Dr. Greg Wettstein wrote:
> On Nov 14,  3:40pm, "Dr. Greg Wettstein" wrote:
> } Subject: Re: [Xen-devel] Does xen-4.2.0 support VGA passthrough with the v
> 
> Good morning, hope the day is going well for everyone.

Heya!
> 
> > On Nov 13, 10:02am, Ian Campbell wrote:
> > } Subject: Re: [Xen-devel] Does xen-4.2.0 support VGA passthrough with the v
> > 
> > Good afternoon, hope the week is going well for everyone.
> > 
> > > On Tue, 2012-11-13 at 06:30 +0000, Qian Hu wrote:
> > > 
> > > > With spice tool, I have to create a VM by xl command, and now I am
> > > > wondering if it supports VGA passghrouth?
> > 
> > > This list is for the development of Xen. You';d probably have more
> > > luck with these sorts of support requests on the xen-users list.
> > 
> > That would normally be the case but I'm suspicious there are issues
> > with VGA passthrough in 4.2.0.
> 
> I just wanted to follow up to the list on the status of passthrough
> issues.
> 
> We reverted our test machine back to the 2.6.32.45 kernel which we had
> been using in production.  That kernel was based on Jeremy's GIT
> tree.  Using xm and the updated ATI patches which I referenced in my
> original mail passthrough works as it should.
> 
> Passthrough does not work with xl.  Windows started but fell into its
> text mode rescue screen and registered a crash dump.  It flashed the
> screen back and forth between a stipled blue/grey and totally black
> screen a few times and then locked the physical machine up solidly.
> 
> On the next boot I thought about it but declined to register the crash
> dump with Microsoft.... :-)

Ha!
> 
> We then went back and tested the 3.4.18 kernel and with both xm and xl
> the guest faults on the first attempt to do an I/O port access.  All
> factors (windows image, hardware, xen guest config) are held identical
> so the difference would seem to be linked to the PCI passthrough
> implementation between the two kernels.  I've copied Konrad on the
> note since he would seem to be the person most familiar with this
> area.
> 
> I'm including below a diff between a successful qemu-dm passthrough
> session (2.6.32.45) and an unsuccessful session (3.4.18).  It would
> appear 3.4.18 is getting the both the I/O port and memory ranges
> wrong.

Hm, can you also provide the 'lspci -vvv' with the 2.6.32.45 and 3.4.18 kernel?

I am specifically looking to see if there are any:

[from git commit c341ca45ce56143804ef5a8f4db753e554e640b4]
Author: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date:   Tue Sep 25 16:48:24 2012 -0400

   
	.. snip.. 
    -       Region 0: [virtual] Memory at fe8c0000 (32-bit, non-prefetchable) [size=128K]
    -       Region 1: [virtual] Memory at fe800000 (32-bit, non-prefetchable) [size=512K]
    +       Region 0: Memory at fe8c0000 (32-bit, non-prefetchable) [size=128K]
    +       Region 1: Memory at fe800000 (32-bit, non-prefetchable) [size=512K]
            Region 2: I/O ports at c000 [size=32]
    -       Region 3: [virtual] Memory at fe8e0000 (32-bit, non-prefetchable) [size=16K]
    +       Region 3: Memory at fe8e0000 (32-bit, non-prefetchable) [size=16K]
    
    The [virtual] means that lspci read those entries from SysFS but when
    it read them from the device it got a different value (0xfffffff).
   

Any of those '[virtual]' ones? And how are you reserving the PCI devices?
Are you using xen-pciback.hide on the Linux command line?

Does 'xm dmesg' give you an logs? Do the logs have any warnings?
 
> 
> Let me know if I can forward any additional information or run any
> additional tests.
> 
> Have a good weekend.
> 
> Greg
> 
> qemu-dm log diff: ---------------------------------------------------------
> 1c1
> < domid: 3
> ---
> > domid: 1
> 3,5c3,5
> < Watching /local/domain/0/device-model/3/logdirty/cmd
> < Watching /local/domain/0/device-model/3/command
> < Watching /local/domain/3/cpu
> ---
> > Watching /local/domain/0/device-model/1/logdirty/cmd
> > Watching /local/domain/0/device-model/1/command
> > Watching /local/domain/1/cpu
> 9c9
> < Guest uuid = 7fcefb13-d1ef-105b-e38c-1e1454411e80
> ---
> > Guest uuid = eab6bbbb-4819-b970-a83c-03288a1541ad
> 14c14
> < xs_read(/local/domain/0/device-model/3/xen_extended_power_mgmt): read error
> ---
> > xs_read(/local/domain/0/device-model/1/xen_extended_power_mgmt): read error
> 18,20c18,20
> < xs_read(/local/domain/3/log-throttling): read error
> < qemu: ignoring not-understood drive `/local/domain/3/log-throttling'
> < medium change watch on `/local/domain/3/log-throttling' - unknown device, ignored
> ---
> > xs_read(/local/domain/1/log-throttling): read error
> > qemu: ignoring not-understood drive `/local/domain/1/log-throttling'
> > medium change watch on `/local/domain/1/log-throttling' - unknown device, ignored
> 69,106c69,77
> < pt_iomem_map: e_phys=f1020000 maddr=c1a00000 type=0 len=131072 index=2 first_map=1
> < pt_iomem_map: e_phys=f1060000 maddr=c1b22000 type=0 len=4096 index=0 first_map=1
> < pt_ioport_map: e_phys=c600 pio_base=3000 len=256 index=4 first_map=1
> < pt_ioport_map: e_phys=c600 pio_base=3000 len=256 index=4 first_map=0
> < ati_gfx_init: guest_pio_bar = 0xc600, host_pio_bar = 0x3000, pio_size=0x100 guest_mmio_bar1=0xe0000000, guest_mmio_bar2=0x0
> < platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
> < platform_fixed_ioport: changed ro/rw state of ROM memory area. now is ro state.
> < pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
> < pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
> < pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
> < pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
> < pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
> < pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
> < pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
> < pt_iomem_map: e_phys=ffffffff maddr=b0000000 type=8 len=268435456 index=0 first_map=0
> < pt_iomem_map: e_phys=ffffffff maddr=c1a00000 type=0 len=131072 index=2 first_map=0
> < pt_iomem_map: e_phys=e0000000 maddr=b0000000 type=8 len=268435456 index=0 first_map=0
> < pt_iomem_map: e_phys=f1020000 maddr=c1a00000 type=0 len=131072 index=2 first_map=0
> < pt_ioport_map: e_phys=c600 pio_base=3000 len=256 index=4 first_map=0
> < pt_iomem_map: e_phys=ffffffff maddr=c1b22000 type=0 len=4096 index=0 first_map=0
> < pt_iomem_map: e_phys=f1060000 maddr=c1b22000 type=0 len=4096 index=0 first_map=0
> < pt_iomem_map: e_phys=ffffffff maddr=b0000000 type=8 len=268435456 index=0 first_map=0
> < pt_iomem_map: e_phys=ffffffff maddr=c1a00000 type=0 len=131072 index=2 first_map=0
> < pt_iomem_map: e_phys=e0000000 maddr=b0000000 type=8 len=268435456 index=0 first_map=0
> < pt_iomem_map: e_phys=f1020000 maddr=c1a00000 type=0 len=131072 index=2 first_map=0
> < pt_ioport_map: e_phys=c600 pio_base=3000 len=256 index=4 first_map=0
> < pt_ioport_map: e_phys=c600 pio_base=3000 len=256 index=4 first_map=0
> < pt_msgctrl_reg_write: guest enabling MSI, disable MSI-INTx translation
> < pci_intx: intx=1
> < pt_msi_disable: Unmap msi with pirq 37
> < pt_msgctrl_reg_write: setup msi for dev 30
> < pt_msi_setup: msi mapped with pirq 37
> < pt_msi_update: Update msi with pirq 37 gvec b0 gflags 0
> < pt_iomem_map: e_phys=ffffffff maddr=c1b22000 type=0 len=4096 index=0 first_map=0
> < pt_iomem_map: e_phys=f1060000 maddr=c1b22000 type=0 len=4096 index=0 first_map=0
> < pt_iomem_map: e_phys=ffffffff maddr=c1b22000 type=0 len=4096 index=0 first_map=0
> < shutdown requested in cpu_handle_ioreq
> < Issued domain 3 poweroff
> ---
> > pt_iomem_map: e_phys=f1000000 maddr=c1a00000 type=0 len=131072 index=2 first_map=1
> > pt_iomem_map: e_phys=f1040000 maddr=c1b22000 type=0 len=4096 index=0 first_map=1
> > pt_ioport_map: e_phys=c700 pio_base=3000 len=256 index=4 first_map=1
> > pt_ioport_map: e_phys=c700 pio_base=3000 len=256 index=4 first_map=0
> > ati_gfx_init: guest_pio_bar = 0xc700, host_pio_bar = 0x3000, pio_size=0x100 guest_mmio_bar1=0xe0000000, guest_mmio_bar2=0x0
> > ati_io_regs_read: Requested read of c74c/51020, mapped: 304c/12364
> > ati_hw_in: port I/O: 304c, base: 3000, size: 100
> > ati_hw_in: ioperm successful
> > ati_hw_in: Read: 0
> ---------------------------------------------------------------------------
> 
> }-- End of excerpt from "Dr. Greg Wettstein"
> 
> As always,
> Dr. G.W. Wettstein, Ph.D.   Enjellic Systems Development, LLC.
> 4206 N. 19th Ave.           Specializing in information infra-structure
> Fargo, ND  58102            development.
> PH: 701-281-1686
> FAX: 701-281-3949           EMAIL: greg@enjellic.com
> ------------------------------------------------------------------------------
> "Boy, it must not take much to make a phone work.  Looking at
>  everthing else here it must be the same way with the INTERNET."
>                                 -- Francis 'Fritz' Wettstein

Re: Q77 IGD instantly crashes on xen-pciback bind.

[Dr. Greg Wettstein]

On Nov 21,  2:57pm, "Dr. Greg Wettstein" wrote:
} Subject: [Xen-devel] Q77 IGD instantly crashes on xen-pciback bind.

> Hi, hope the week is ending well for everyone.

As I was walking out the door I remembered I had been delinquent with
information.  The dom0 kernel is 32-bit 3.14.22 straight from
kernel.org under a 64-bit hypervisor compiled from 4.4.1 sources.

The machine hang is reproducible in the absence of even starting a
guest so the problem appears to be specific to the process of
preparing the device for export through xen-pciback.

Greg

}-- End of excerpt from "Dr. Greg Wettstein"

As always,
Dr. G.W. Wettstein, Ph.D.   Enjellic Systems Development, LLC.
4206 N. 19th Ave.           Specializing in information infra-structure
Fargo, ND  58102            development.
PH: 701-281-1686
FAX: 701-281-3949           EMAIL: greg@enjellic.com
------------------------------------------------------------------------------
"Fools ignore complexity.  Pragmatists suffer it.  Some can avoid it.
Geniuses remove it.
                                -- Perliss' Programming Proverb #58
                                   SIGPLAN National, Sept. 1982

Re: Q77 IGD instantly crashes on xen-pciback bind.

[Pasi Kärkkäinen]

On Fri, Nov 21, 2014 at 03:02:33PM -0600, Dr. Greg Wettstein wrote:
> On Nov 21,  2:57pm, "Dr. Greg Wettstein" wrote:
> } Subject: [Xen-devel] Q77 IGD instantly crashes on xen-pciback bind.
> 
> > Hi, hope the week is ending well for everyone.
> 
> As I was walking out the door I remembered I had been delinquent with
> information.  The dom0 kernel is 32-bit 3.14.22 straight from
> kernel.org under a 64-bit hypervisor compiled from 4.4.1 sources.
>

Wow, quite an old thread :)
 
So you're still seeing the same problem with recent Xen/Linux versions.. 


> The machine hang is reproducible in the absence of even starting a
> guest so the problem appears to be specific to the process of
> preparing the device for export through xen-pciback.
> 

OK.. Maybe Konrad has some thoughts about this.


-- Pasi


> Greg
> 
> }-- End of excerpt from "Dr. Greg Wettstein"
> 
> As always,
> Dr. G.W. Wettstein, Ph.D.   Enjellic Systems Development, LLC.
> 4206 N. 19th Ave.           Specializing in information infra-structure
> Fargo, ND  58102            development.
> PH: 701-281-1686
> FAX: 701-281-3949           EMAIL: greg@enjellic.com
> ------------------------------------------------------------------------------
> "Fools ignore complexity.  Pragmatists suffer it.  Some can avoid it.
> Geniuses remove it.
>                                 -- Perliss' Programming Proverb #58
>                                    SIGPLAN National, Sept. 1982
> 

Re: Q77 IGD instantly crashes on xen-pciback bind.

[Pasi Kärkkäinen]

On Sun, Nov 23, 2014 at 04:05:48PM +0200, Pasi Kärkkäinen wrote:
> On Fri, Nov 21, 2014 at 03:02:33PM -0600, Dr. Greg Wettstein wrote:
> > On Nov 21,  2:57pm, "Dr. Greg Wettstein" wrote:
> > } Subject: [Xen-devel] Q77 IGD instantly crashes on xen-pciback bind.
> > 
> > > Hi, hope the week is ending well for everyone.
> > 
> > As I was walking out the door I remembered I had been delinquent with
> > information.  The dom0 kernel is 32-bit 3.14.22 straight from
> > kernel.org under a 64-bit hypervisor compiled from 4.4.1 sources.
> >
> 
> Wow, quite an old thread :)
> 


Oh, there's also the new thread about this, I guess it's better to continue there :)

http://lists.xenproject.org/archives/html/xen-devel/2014-11/msg02123.html



-- Pasi

 
> So you're still seeing the same problem with recent Xen/Linux versions.. 
> 
> 
> > The machine hang is reproducible in the absence of even starting a
> > guest so the problem appears to be specific to the process of
> > preparing the device for export through xen-pciback.
> > 
> 
> OK.. Maybe Konrad has some thoughts about this.
> 
> 
> -- Pasi
> 
> 
> > Greg
> > 
> > }-- End of excerpt from "Dr. Greg Wettstein"
> > 
> > As always,
> > Dr. G.W. Wettstein, Ph.D.   Enjellic Systems Development, LLC.
> > 4206 N. 19th Ave.           Specializing in information infra-structure
> > Fargo, ND  58102            development.
> > PH: 701-281-1686
> > FAX: 701-281-3949           EMAIL: greg@enjellic.com
> > ------------------------------------------------------------------------------
> > "Fools ignore complexity.  Pragmatists suffer it.  Some can avoid it.
> > Geniuses remove it.
> >                                 -- Perliss' Programming Proverb #58
> >                                    SIGPLAN National, Sept. 1982
> > 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Does xen-4.2.0 support VGA passthrough with the virtual machine created by xl command?

[Dr. Greg Wettstein]

On Nov 28,  4:04pm, Konrad Rzeszutek Wilk wrote:
} Subject: Re: [Xen-devel] Does xen-4.2.0 support VGA passthrough with the v

> On Sat, Nov 17, 2012 at 09:17:36AM -0600, Dr. Greg Wettstein wrote:
> > On Nov 14,  3:40pm, "Dr. Greg Wettstein" wrote:
> > } Subject: Re: [Xen-devel] Does xen-4.2.0 support VGA passthrough with the v
> > 
> > Good morning, hope the day is going well for everyone.

> Heya!

Good evening, thanks for taking the time to respond.

> > We then went back and tested the 3.4.18 kernel and with both xm and xl
> > the guest faults on the first attempt to do an I/O port access.  All
> > factors (windows image, hardware, xen guest config) are held identical
> > so the difference would seem to be linked to the PCI passthrough
> > implementation between the two kernels.  I've copied Konrad on the
> > note since he would seem to be the person most familiar with this
> > area.
> > 
> > I'm including below a diff between a successful qemu-dm passthrough
> > session (2.6.32.45) and an unsuccessful session (3.4.18).  It would
> > appear 3.4.18 is getting the both the I/O port and memory ranges
> > wrong.

I was going to get an update back to everyone but got swamped by the
holiday weekend and a series of hardware failures I had to chase
after.

I took advantage of some time over the holiday weekend to chase down
the passthrough problems and now have it working well on 4.2.0 on all
kernels up to 3.4.19 using XM.  The original ATI patches have a bug in
them which causes qemu-dm to core dump on kernels somewhere after
2.6.32.x.

The original patches were bracketing the inb/outb instructions used in
ati_hw_read()/ati_hw_write with an ioperm() call.  The fix was a
straight forward replacement of the ioperm() call with a call to
iopl(3).

I seem to vaguely remember something about the kernel not properly
enforcing access controlls on in/out instructions but don't remember
if that was with a pvops or standard kernel.  In any event the kernel
behavior changed after 2.6.32.x which triggered the breakage.

I will post an updated version of the ATI patches under separate cover
in case anytone else is using them.

> Hm, can you also provide the 'lspci -vvv' with the 2.6.32.45 and
> 3.4.18 kernel?
>
> I am specifically looking to see if there are any:
> 
> [from git commit c341ca45ce56143804ef5a8f4db753e554e640b4]
> Author: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> Date:   Tue Sep 25 16:48:24 2012 -0400
> 
>    
> 	.. snip.. 
>     -       Region 0: [virtual] Memory at fe8c0000 (32-bit, non-prefetchable) [size=128K]
>     -       Region 1: [virtual] Memory at fe800000 (32-bit, non-prefetchable) [size=512K]
>     +       Region 0: Memory at fe8c0000 (32-bit, non-prefetchable) [size=128K]
>     +       Region 1: Memory at fe800000 (32-bit, non-prefetchable) [size=512K]
>             Region 2: I/O ports at c000 [size=32]
>     -       Region 3: [virtual] Memory at fe8e0000 (32-bit, non-prefetchable) [size=16K]
>     +       Region 3: Memory at fe8e0000 (32-bit, non-prefetchable) [size=16K]

>     The [virtual] means that lspci read those entries from SysFS but when
>     it read them from the device it got a different value (0xfffffff).
>
> Any of those '[virtual]' ones? And how are you reserving the PCI devices?
> Are you using xen-pciback.hide on the Linux command line?
> 
> Does 'xm dmesg' give you an logs? Do the logs have any warnings?

We were under some time constraints to get Windows access back working
with a 'modern' kernel so once things were working reliably with xm I
didn't get a chance to fiddle with xl.  Given the behavior I saw on
2.6.32.x with xl I'm suspicious it may not work.  I'm hoping to get
back and do some testing early next week.

I just checked the machine (which is running a Windows session as I
write this) and dont see any [virtual] references.  This is on 3.4.18
but I also haven't had the chance to check xl on that kernel.  At
least a couple of hundred Windows 7 boots have been done with xm so
4.2.0 seems solid with that control plane.

With respect to reservation of the PCI device we have a script which
unplugs the device and re-plugs it after the Window session
completes.  The machines are Linux/Windows dual-use so the cards need
to be active for the Linux sessions.

The script can be picked up at the following location:

	ftp://ftp.enjellic.com/pub/xen/run-passthrough

I will give xl a try later in the weekend with the updated qemu-dm and
I will report back the results from a more throroughly controlled test
environment.

Thanks for the input, have a good weekend.

Greg

}-- End of excerpt from Konrad Rzeszutek Wilk

As always,
Dr. G.W. Wettstein, Ph.D.   Enjellic Systems Development, LLC.
4206 N. 19th Ave.           Specializing in information infra-structure
Fargo, ND  58102            development.
PH: 701-281-1686
FAX: 701-281-3949           EMAIL: greg@enjellic.com
------------------------------------------------------------------------------
"Not only is this incomprehensible, but the ink is ugly and the paper
 is from the wrong kind of tree."
                                -- Professor W.

Re: Does xen-4.2.0 support VGA passthrough with the virtual machine created by xl command?

[Pasi Kärkkäinen]

On Thu, Nov 29, 2012 at 07:25:01PM -0600, Dr. Greg Wettstein wrote:
> 
> I was going to get an update back to everyone but got swamped by the
> holiday weekend and a series of hardware failures I had to chase
> after.
> 
> I took advantage of some time over the holiday weekend to chase down
> the passthrough problems and now have it working well on 4.2.0 on all
> kernels up to 3.4.19 using XM.  The original ATI patches have a bug in
> them which causes qemu-dm to core dump on kernels somewhere after
> 2.6.32.x.
> 
> The original patches were bracketing the inb/outb instructions used in
> ati_hw_read()/ati_hw_write with an ioperm() call.  The fix was a
> straight forward replacement of the ioperm() call with a call to
> iopl(3).
> 
> I seem to vaguely remember something about the kernel not properly
> enforcing access controlls on in/out instructions but don't remember
> if that was with a pvops or standard kernel.  In any event the kernel
> behavior changed after 2.6.32.x which triggered the breakage.
> 
> I will post an updated version of the ATI patches under separate cover
> in case anytone else is using them.
> 

Yes please. Thanks!


> 
> We were under some time constraints to get Windows access back working
> with a 'modern' kernel so once things were working reliably with xm I
> didn't get a chance to fiddle with xl.  Given the behavior I saw on
> 2.6.32.x with xl I'm suspicious it may not work.  I'm hoping to get
> back and do some testing early next week.
> 
> I just checked the machine (which is running a Windows session as I
> write this) and dont see any [virtual] references.  This is on 3.4.18
> but I also haven't had the chance to check xl on that kernel.  At
> least a couple of hundred Windows 7 boots have been done with xm so
> 4.2.0 seems solid with that control plane.
> 
> With respect to reservation of the PCI device we have a script which
> unplugs the device and re-plugs it after the Window session
> completes.  The machines are Linux/Windows dual-use so the cards need
> to be active for the Linux sessions.
> 
> The script can be picked up at the following location:
> 
> 	ftp://ftp.enjellic.com/pub/xen/run-passthrough
> 
> I will give xl a try later in the weekend with the updated qemu-dm and
> I will report back the results from a more throroughly controlled test
> environment.
> 
> Thanks for the input, have a good weekend.
> 

Thanks!

-- Pasi

Re: Does xen-4.2.0 support VGA passthrough with the virtual machine created by xl command?

[Dr. Greg Wettstein]

On Nov 13, 10:02am, Ian Campbell wrote:
} Subject: Re: [Xen-devel] Does xen-4.2.0 support VGA passthrough with the v

Good afternoon, hope the week is going well for everyone.

> On Tue, 2012-11-13 at 06:30 +0000, Qian Hu wrote:
> 
> > With spice tool, I have to create a VM by xl command, and now I am
> > wondering if it supports VGA passghrouth?

> This list is for the development of Xen. You';d probably have more
> luck with these sorts of support requests on the xen-users list.

That would normally be the case but I'm suspicious there are issues
with VGA passthrough in 4.2.0.

I worked through all the vagaries of getting VGA passthrough going
with ATI cards about two years ago using the patches which are
available through the xen-vel archives.  We have run literally
thousands of Windows 7 passthrough sessions since then under 4.0.x and
4.1.x.

We've just started validating 4.2.0 and passthrough reliably generates
segmentation faults in qemu-dm on the first attempt to do an I/O port
read.  The fault occurs with an identical configuration file using
either xm or xl.  Testing is being done with a 3.4.18 dom0 kernel.

I'm thinking we may have initially hit this in 4.1.0.  Segmentation
faults in the device emulator were being experienced but these seemed
to go away when we the dom0 kernel was switched back to a 2.6.32.x
kernel which we were using from Jeremy's GIT tree.

We need to make the jump to 3.4.x so I'm going to see if I can hunt
the issue down.

For those who may want to assist in trying to get all of this working
a bit better I have a 4.2.0 port of the ATI patches available at the
following location:

	ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch

You will minimally need the qemu-dm binary from a build with this
patch applied.

I have also made available a helper script which we use to automate
the binding/rebinding of the VGA card and the USB controller passed
through to provide mouse/keyboard support.   The script can be picked
up from the following location:

	ftp://ftp.enjellic.com/pub/xen/run-passthrough

The script is straight forward but will need a bit of tweaking for
items such as the PCI configuration being passwed through.

The script uses vbetool to issue a VGA post reset after the passthough
session is completed.  We run all this from a VGA text mode console
(yes we are old fashioned) so the results are 'unspecified' for trying
do this from anything resembling a remotely sophisticated graphics
session.

This all tends to not be for the faint of heart.  I recommend having a
network login session running from a separate machine in case things
decide to go south.  A 'sync && reboot' always being a bit more
palatable then yanking the power cord.

The VGA cards we have been using were some monstrous thing which was
in vogue 2-3 years ago.  The fundamentals are as follows:

---------------------------------------------------------------------------
01:00.0 VGA compatible controller: ATI Technologies Inc Device 6898
(prog-if 00
[VGA controller])
        Subsystem: ASUSTeK Computer Inc. Device 0346
        Flags: fast devsel, IRQ 16
        Memory at b0000000 (64-bit, prefetchable) [size=256M]
        Memory at c1a00000 (64-bit, non-prefetchable) [size=128K]
        I/O ports at 3000 [size=256]
        Expansion ROM at c1a40000 [disabled] [size=128K]
        Capabilities: <access denied>
---------------------------------------------------------------------------

I will continue to hunt to see if I can further isolate the problem.
If anyone else wants to experiment a bit please do so and let me know
the results from 4.2.0.

For the record we are using Intel S3420GP motherboards.  We found the
VT-D implementation to be solid on that motherboard.

> Ian.

Hope the above is helpful.

Have a good evening.

Greg

}-- End of excerpt from Ian Campbell

As always,
Dr. G.W. Wettstein, Ph.D.   Enjellic Systems Development, LLC.
4206 N. 19th Ave.           Specializing in information infra-structure
Fargo, ND  58102            development.
PH: 701-281-1686
FAX: 701-281-3949           EMAIL: greg@enjellic.com
------------------------------------------------------------------------------
"We have more to fear from the bungling of the incompetent than from
 the machinations of the wicked."
                                -- Slashdot

Does xen-4.2.0 support VGA passthrough with the virtual machine created by xl command?

[Qian Hu]

[-- Attachment #1.1: Type: text/plain, Size: 471 bytes --]

Hi, everyone!

I am working on xen-4.2.0 with spice remote connect.  My host is Fedora 14
and guest is win7.

I have installed spice package and now I can connect to guest by spice
client.

For better graphics experience, I want to try the  VGA passthrough.

With spice tool, I have to create a VM by xl command, and now I am
wondering if it supports VGA passghrouth?

If so, how should I  test it ?

The device model in my config file is "qemu-xen".

Thank you!

huqian

[-- Attachment #1.2: Type: text/html, Size: 2707 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Does xen-4.2.0 support VGA passthrough with the virtual machine created by xl command?

[Ian Campbell]

On Tue, 2012-11-13 at 06:30 +0000, Qian Hu wrote:

> With spice tool, I have to create a VM by xl command, and now I am
> wondering if it supports VGA passghrouth?

This list is for the development of Xen. You';d probably have more luck
with these sorts of support requests on the xen-users list.

Ian.