ioperm problem

ioperm problem

[Pavel Matěja]

Hi,
I'm trying to port AMD VGA passthru patch to the latest XEN and vanila kernel 
and I got SIGSEGV in

static void ati_hw_out(uint16_t hport, uint32_t data)
{
    ioperm(gfx_info.host_pio_base, gfx_info.pio_size, 1);    
    asm volatile ("out %1, %0"::"Nd"(hport),"a"(data));
    ioperm(gfx_info.host_pio_base, gfx_info.pio_size, 0);
}

I tried old 2.6.32 XEN kernel and there is no such problem.
It looks related to arch/x86/kernel/ioport.c but I'm not sure.
Is anyone here familiar with that code?
-- 
Pavel Mateja

Re: ioperm problem

[Konrad Rzeszutek Wilk]

On Sun, Nov 13, 2011 at 10:19:06PM +0100, Pavel Matěja wrote:
> Hi,
> I'm trying to port AMD VGA passthru patch to the latest XEN and vanila kernel 
> and I got SIGSEGV in
> 
> static void ati_hw_out(uint16_t hport, uint32_t data)
> {
>     ioperm(gfx_info.host_pio_base, gfx_info.pio_size, 1);    
>     asm volatile ("out %1, %0"::"Nd"(hport),"a"(data));
>     ioperm(gfx_info.host_pio_base, gfx_info.pio_size, 0);
> }

Does it work under baremetal?

What is the host_pio_base?

Is the host_pio_base part of the permitted IO ports? (you can
see that if you run 'xl debug-keys q' and it should show you something
like this:

(XEN) Rangesets belonging to domain 1:
(XEN)     I/O Ports  { b400-b41f, b800-b81f }
(XEN)     Interrupts { 18-19, 54-55 }
(XEN)     I/O Memory { fe940-fe9ff }
(XEN) Memory pages belonging to domain 1:

(you can get that from xm dmesg).

As you can see, the b400->b41f are allowed in the domain 1. Is your
host_pio_base in there?


> 
> I tried old 2.6.32 XEN kernel and there is no such problem.
> It looks related to arch/x86/kernel/ioport.c but I'm not sure.
> Is anyone here familiar with that code?

Yes, and I think I saw somebody ask me about that too.

Lets rope them in this converstation - they got it to work
but my memory is foggy at what was required.

Ben, Thomas,

I remember you guys had a tough time with vd86 which did something similar
and it ultimately was due to to /dev/mem not passing in VM_IO. But the
ioperm/outb sounds familiar too. Was there a missing hypercall when
forking/copying the ioperm bitmap?

Re: ioperm problem

[Ben Guthro]

[-- Attachment #1: Type: text/plain, Size: 2363 bytes --]

Attached is our patch to work around issues with the ioports with some 
older nVidia cards.

This, admittedly is a bit of a hack, and not exactly something that I 
would see upstream wanting to carry, for a variety of reasons. It really 
should all be predicated on whether the kernel is the initial domain, etc.

This opens up the legacy VGA ports in the VGA arbiter code.

Konrad - I think that you had suggested an alternate way of doing this, 
IIRC, but I can't seem to find it in my inbox. Due to competing demands, 
and my nVidia hardware disappearing, I never went back to rework this code.

As written, however, it did solve the problem at hand - however hacky it 
may be.

/btg

On 11/16/2011 09:57 AM, Konrad Rzeszutek Wilk wrote:
> On Sun, Nov 13, 2011 at 10:19:06PM +0100, Pavel Matěja wrote:
>> Hi,
>> I'm trying to port AMD VGA passthru patch to the latest XEN and vanila kernel
>> and I got SIGSEGV in
>>
>> static void ati_hw_out(uint16_t hport, uint32_t data)
>> {
>>      ioperm(gfx_info.host_pio_base, gfx_info.pio_size, 1);
>>      asm volatile ("out %1, %0"::"Nd"(hport),"a"(data));
>>      ioperm(gfx_info.host_pio_base, gfx_info.pio_size, 0);
>> }
>
> Does it work under baremetal?
>
> What is the host_pio_base?
>
> Is the host_pio_base part of the permitted IO ports? (you can
> see that if you run 'xl debug-keys q' and it should show you something
> like this:
>
> (XEN) Rangesets belonging to domain 1:
> (XEN)     I/O Ports  { b400-b41f, b800-b81f }
> (XEN)     Interrupts { 18-19, 54-55 }
> (XEN)     I/O Memory { fe940-fe9ff }
> (XEN) Memory pages belonging to domain 1:
>
> (you can get that from xm dmesg).
>
> As you can see, the b400->b41f are allowed in the domain 1. Is your
> host_pio_base in there?
>
>
>>
>> I tried old 2.6.32 XEN kernel and there is no such problem.
>> It looks related to arch/x86/kernel/ioport.c but I'm not sure.
>> Is anyone here familiar with that code?
>
> Yes, and I think I saw somebody ask me about that too.
>
> Lets rope them in this converstation - they got it to work
> but my memory is foggy at what was required.
>
> Ben, Thomas,
>
> I remember you guys had a tough time with vd86 which did something similar
> and it ultimately was due to to /dev/mem not passing in VM_IO. But the
> ioperm/outb sounds familiar too. Was there a missing hypercall when
> forking/copying the ioperm bitmap?

[-- Attachment #2: vga-ioport-enable.patch --]
[-- Type: text/x-patch, Size: 1706 bytes --]

diff --git a/drivers/gpu/vga/vgaarb.c b/drivers/gpu/vga/vgaarb.c
index ace2b16..d488426 100644
--- a/drivers/gpu/vga/vgaarb.c
+++ b/drivers/gpu/vga/vgaarb.c
@@ -421,6 +421,50 @@ bail:
 }
 EXPORT_SYMBOL(vga_put);
 
+#ifdef CONFIG_XEN
+#include <xen/xen.h>
+#include <xen/interface/physdev.h>
+#include <xen/interface/xen.h>
+#include <asm/xen/hypervisor.h>
+#include <asm/xen/hypercall.h>
+
+#define IO_BITMAP_BITS 65536
+#define IO_BITMAP_BYTES (IO_BITMAP_BITS/8)
+
+#define VGA_START_PORT 0x3B0
+#define VGA_END_PORT 0x3DF
+
+static void
+vga_ioport_map(struct pci_dev *pdev)
+{
+	unsigned int i, j;
+	struct physdev_set_iobitmap op;
+	uint8_t *bitmap;
+
+	bitmap = kmalloc(IO_BITMAP_BYTES, GFP_KERNEL);
+	memset(bitmap, 0xff, IO_BITMAP_BYTES);
+
+	/* PCI io bars */
+	for (i = 0; i < PCI_NUM_RESOURCES; i++)
+		if (pci_resource_flags(pdev, i) & IORESOURCE_IO)
+			for (j = pci_resource_start(pdev, i);
+				j <= pci_resource_end(pdev, i); j++)
+				__clear_bit(j, (unsigned long*)bitmap);
+
+	/* legacy vga */
+	for (i = VGA_START_PORT; i <= VGA_END_PORT; i++)
+		__clear_bit(i, (unsigned long*)bitmap);
+
+	op.bitmap = bitmap;
+	op.nr_ports = IO_BITMAP_BITS;
+	if(HYPERVISOR_physdev_op(PHYSDEVOP_set_iobitmap, &op) != 0)
+		pr_info("PHYSDEVOP_set_iobitmap failed\n");
+
+}
+#else
+#define vga_ioport_map(x)
+#endif
+
 /*
  * Currently, we assume that the "initial" setup of the system is
  * not sane, that is we come up with conflicting devices and let
@@ -509,6 +553,7 @@ static bool vga_arbiter_add_pci_device(struct pci_dev *pdev)
 		vga_iostate_to_str(vgadev->owns),
 		vga_iostate_to_str(vgadev->locks));
 
+	vga_ioport_map(pdev);
 	spin_unlock_irqrestore(&vga_lock, flags);
 	return true;
 fail:

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: ioperm problem

[Konrad Rzeszutek Wilk]

On Thu, Nov 17, 2011 at 08:56:34AM -0500, Ben Guthro wrote:
> Attached is our patch to work around issues with the ioports with
> some older nVidia cards.
> 
> This, admittedly is a bit of a hack, and not exactly something that
> I would see upstream wanting to carry, for a variety of reasons. It
> really should all be predicated on whether the kernel is the initial
> domain, etc.
> 
> This opens up the legacy VGA ports in the VGA arbiter code.

Was there a userspace program that did this? As in it would
call ioperm?

> 
> Konrad - I think that you had suggested an alternate way of doing
> this, IIRC, but I can't seem to find it in my inbox. Due to
> competing demands, and my nVidia hardware disappearing, I never went
> back to rework this code.

Ah, I might have some code that I just uncovered from Jeremy's old
git tree.

I've put it up on devel/ioperm - it nicely wraps the ioperm
call to go the native or paravirt.

> 
> As written, however, it did solve the problem at hand - however
> hacky it may be.

<nods>

Pavel,

Can you try to merge the #devel/ioperm patches in your tree and see
if they fix your problem?

The git tree is at git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
> 
> /btg
> 
> On 11/16/2011 09:57 AM, Konrad Rzeszutek Wilk wrote:
> >On Sun, Nov 13, 2011 at 10:19:06PM +0100, Pavel Matěja wrote:
> >>Hi,
> >>I'm trying to port AMD VGA passthru patch to the latest XEN and vanila kernel
> >>and I got SIGSEGV in
> >>
> >>static void ati_hw_out(uint16_t hport, uint32_t data)
> >>{
> >>     ioperm(gfx_info.host_pio_base, gfx_info.pio_size, 1);
> >>     asm volatile ("out %1, %0"::"Nd"(hport),"a"(data));
> >>     ioperm(gfx_info.host_pio_base, gfx_info.pio_size, 0);
> >>}
> >
> >Does it work under baremetal?
> >
> >What is the host_pio_base?
> >
> >Is the host_pio_base part of the permitted IO ports? (you can
> >see that if you run 'xl debug-keys q' and it should show you something
> >like this:
> >
> >(XEN) Rangesets belonging to domain 1:
> >(XEN)     I/O Ports  { b400-b41f, b800-b81f }
> >(XEN)     Interrupts { 18-19, 54-55 }
> >(XEN)     I/O Memory { fe940-fe9ff }
> >(XEN) Memory pages belonging to domain 1:
> >
> >(you can get that from xm dmesg).
> >
> >As you can see, the b400->b41f are allowed in the domain 1. Is your
> >host_pio_base in there?
> >
> >
> >>
> >>I tried old 2.6.32 XEN kernel and there is no such problem.
> >>It looks related to arch/x86/kernel/ioport.c but I'm not sure.
> >>Is anyone here familiar with that code?
> >
> >Yes, and I think I saw somebody ask me about that too.
> >
> >Lets rope them in this converstation - they got it to work
> >but my memory is foggy at what was required.
> >
> >Ben, Thomas,
> >
> >I remember you guys had a tough time with vd86 which did something similar
> >and it ultimately was due to to /dev/mem not passing in VM_IO. But the
> >ioperm/outb sounds familiar too. Was there a missing hypercall when
> >forking/copying the ioperm bitmap?

> diff --git a/drivers/gpu/vga/vgaarb.c b/drivers/gpu/vga/vgaarb.c
> index ace2b16..d488426 100644
> --- a/drivers/gpu/vga/vgaarb.c
> +++ b/drivers/gpu/vga/vgaarb.c
> @@ -421,6 +421,50 @@ bail:
>  }
>  EXPORT_SYMBOL(vga_put);
>  
> +#ifdef CONFIG_XEN
> +#include <xen/xen.h>
> +#include <xen/interface/physdev.h>
> +#include <xen/interface/xen.h>
> +#include <asm/xen/hypervisor.h>
> +#include <asm/xen/hypercall.h>
> +
> +#define IO_BITMAP_BITS 65536
> +#define IO_BITMAP_BYTES (IO_BITMAP_BITS/8)
> +
> +#define VGA_START_PORT 0x3B0
> +#define VGA_END_PORT 0x3DF
> +
> +static void
> +vga_ioport_map(struct pci_dev *pdev)
> +{
> +	unsigned int i, j;
> +	struct physdev_set_iobitmap op;
> +	uint8_t *bitmap;
> +
> +	bitmap = kmalloc(IO_BITMAP_BYTES, GFP_KERNEL);
> +	memset(bitmap, 0xff, IO_BITMAP_BYTES);
> +
> +	/* PCI io bars */
> +	for (i = 0; i < PCI_NUM_RESOURCES; i++)
> +		if (pci_resource_flags(pdev, i) & IORESOURCE_IO)
> +			for (j = pci_resource_start(pdev, i);
> +				j <= pci_resource_end(pdev, i); j++)
> +				__clear_bit(j, (unsigned long*)bitmap);
> +
> +	/* legacy vga */
> +	for (i = VGA_START_PORT; i <= VGA_END_PORT; i++)
> +		__clear_bit(i, (unsigned long*)bitmap);
> +
> +	op.bitmap = bitmap;
> +	op.nr_ports = IO_BITMAP_BITS;
> +	if(HYPERVISOR_physdev_op(PHYSDEVOP_set_iobitmap, &op) != 0)
> +		pr_info("PHYSDEVOP_set_iobitmap failed\n");
> +
> +}
> +#else
> +#define vga_ioport_map(x)
> +#endif
> +
>  /*
>   * Currently, we assume that the "initial" setup of the system is
>   * not sane, that is we come up with conflicting devices and let
> @@ -509,6 +553,7 @@ static bool vga_arbiter_add_pci_device(struct pci_dev *pdev)
>  		vga_iostate_to_str(vgadev->owns),
>  		vga_iostate_to_str(vgadev->locks));
>  
> +	vga_ioport_map(pdev);
>  	spin_unlock_irqrestore(&vga_lock, flags);
>  	return true;
>  fail:

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: ioperm problem

[Alan Cox]

> > This opens up the legacy VGA ports in the VGA arbiter code.
> 
> Was there a userspace program that did this? As in it would
> call ioperm?

Not necessarily there is also a priviledged interface via the console for
historical reasons

For the legacy VGA ports the proper approach is almost certainly to scan
the PCI bus routing data and work out which device they are routed to and
then attach them to that physical device.

That may be a dynamic configuration but you'll see the virtualisation of
the PCI config changes any guest tries to do if so.

Alan

Re: ioperm problem

[Jeremy Fitzhardinge]

On 11/17/2011 09:30 AM, Konrad Rzeszutek Wilk wrote:
> On Thu, Nov 17, 2011 at 08:56:34AM -0500, Ben Guthro wrote:
>> Attached is our patch to work around issues with the ioports with
>> some older nVidia cards.
>>
>> This, admittedly is a bit of a hack, and not exactly something that
>> I would see upstream wanting to carry, for a variety of reasons. It
>> really should all be predicated on whether the kernel is the initial
>> domain, etc.
>>
>> This opens up the legacy VGA ports in the VGA arbiter code.
> Was there a userspace program that did this? As in it would
> call ioperm?
>
>> Konrad - I think that you had suggested an alternate way of doing
>> this, IIRC, but I can't seem to find it in my inbox. Due to
>> competing demands, and my nVidia hardware disappearing, I never went
>> back to rework this code.
> Ah, I might have some code that I just uncovered from Jeremy's old
> git tree.
>
> I've put it up on devel/ioperm - it nicely wraps the ioperm
> call to go the native or paravirt.

Yeah, I'd sort of let that sit, since there's nothing in a "modern"
system which should require it.  Allowing usermode to poke at ioports
from within a domain is inherently suspect, after all.

    J

Re: ioperm problem

[Konrad Rzeszutek Wilk]

On Thu, Nov 17, 2011 at 10:05:57AM -0800, Jeremy Fitzhardinge wrote:
> On 11/17/2011 09:30 AM, Konrad Rzeszutek Wilk wrote:
> > On Thu, Nov 17, 2011 at 08:56:34AM -0500, Ben Guthro wrote:
> >> Attached is our patch to work around issues with the ioports with
> >> some older nVidia cards.
> >>
> >> This, admittedly is a bit of a hack, and not exactly something that
> >> I would see upstream wanting to carry, for a variety of reasons. It
> >> really should all be predicated on whether the kernel is the initial
> >> domain, etc.
> >>
> >> This opens up the legacy VGA ports in the VGA arbiter code.
> > Was there a userspace program that did this? As in it would
> > call ioperm?
> >
> >> Konrad - I think that you had suggested an alternate way of doing
> >> this, IIRC, but I can't seem to find it in my inbox. Due to
> >> competing demands, and my nVidia hardware disappearing, I never went
> >> back to rework this code.
> > Ah, I might have some code that I just uncovered from Jeremy's old
> > git tree.
> >
> > I've put it up on devel/ioperm - it nicely wraps the ioperm
> > call to go the native or paravirt.
> 
> Yeah, I'd sort of let that sit, since there's nothing in a "modern"
> system which should require it.  Allowing usermode to poke at ioports
> from within a domain is inherently suspect, after all.

There are three different cases here:
 - Nvidia and VBE tool. They seem to do a lot of poking and Ben
   needed some way of making it work. Naturally the nouveau driver
   is not doing this.. This is in Dom0.

 - DVB cards. I *think* that the problem some folks have with passing
   in the DVB cards is that the 'scan' tools used to jump frequencies
   poke at those IO-ports. But I am not 100% sure about it, and need
   to look in more details at the drivers itself. The person reporting
   this was running the DVB card in Dom0.

 - Lastly the PCI passthrough of NVidia/ATI card in the guest. I think
   one person is trying to do it while running the guest in paravirtualized
   mode, in which case some help in doing those hypercall is needed.
   But if one were to do this in fully virtualized that means QEMU ends up
   trapping those I/O port accesses and it itself pokes them in the PCI card.
   But since QEMU is running in dom0 and dom0 is PV, that leads back to the
   VGA I/O ports not being allowed to poke as we don't have the ioperm hypercall.

This all sounds right, but in practice it might be something entirely
different..

Re: ioperm problem

[Ben Guthro]

Looking back through the bug database, the problem started in uvesafb,
while running in the nVidia closed source driver
The user reported very long boots, and I found the following repeated
over, and over in the syslog:

Sep 8 20:09:46 Qosmio-X505-1A033757W kernel: [ 29.677547] uvesafb:
Getting mode info block for mode 0x106 failed (eax=0x4f01, err=1)
Sep 8 20:09:46 Qosmio-X505-1A033757W kernel: [ 29.685667] v86d[363]
general protection ip:7f0acfdbdb43 sp:7fff100d3748 error:0 in
libx86.so.1[7f0acfdb6000+1f000]

This continues every 5s until:

Sep 8 20:09:46 Qosmio-X505-1A033757W kernel: [ 84.677745] uvesafb:
Getting mode info block for mode 0x161 failed (eax=0x4f01, err=1)
Sep 8 20:09:46 Qosmio-X505-1A033757W kernel: [ 84.692267] uvesafb:
VBIOS/hardware doesn't support DDC transfers
Sep 8 20:09:46 Qosmio-X505-1A033757W kernel: [ 84.692275] uvesafb: no
monitor limits have been set, default refresh rate will be used
Sep 8 20:09:46 Qosmio-X505-1A033757W kernel: [ 84.692369] v86d[396]
general protection ip:7ff77908bb43 sp:7fffa8b7e708 error:0 in
libx86.so.1[7ff779084000+1f000]


Chasing this down, I found it all started from drivers/video/uvesafb.c: 506:

uvesafb_reset(task);
task->t.regs.eax = 0x4f01;
task->t.regs.ecx = (u32) *mode;
task->t.flags = TF_BUF_RET | TF_BUF_ESDI;
task->t.buf_len = sizeof(struct vbe_mode_ib);
task->buf = par->vbe_modes + off;

err = uvesafb_exec(task);
if (err || (task->t.regs.eax & 0xffff) != 0x004f) {
printk(KERN_WARNING "uvesafb: Getting mode info block "
"for mode 0x%x failed (eax=0x%x, err=%d)\n",
*mode, (u32)task->t.regs.eax, err);
mode++;
par->vbe_modes_cnt--;
continue;
}

(the same code also exists in testvbe.c - line 55)

This executes in the v86d userspace process
v86d/v86_common.c:
int v86_task(struct uvesafb_task *tsk, u8 *buf)


uvesafb was executing the video BIOS code, as mapped in /dev/mem


However, it was unable to access the ioport, causing a GPF.

uvesafb would loop on that, with a sleep(5)

The end result was a colossally long boot process seen by the end user.



The (albeit hacky) solution to open up the VGA ioports resolved the
seemed hang on this older nVidia card.



Hope this explanation make some semblance of sense.


/btg


On Thu, Nov 17, 2011 at 1:05 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
>
> On 11/17/2011 09:30 AM, Konrad Rzeszutek Wilk wrote:
> > On Thu, Nov 17, 2011 at 08:56:34AM -0500, Ben Guthro wrote:
> >> Attached is our patch to work around issues with the ioports with
> >> some older nVidia cards.
> >>
> >> This, admittedly is a bit of a hack, and not exactly something that
> >> I would see upstream wanting to carry, for a variety of reasons. It
> >> really should all be predicated on whether the kernel is the initial
> >> domain, etc.
> >>
> >> This opens up the legacy VGA ports in the VGA arbiter code.
> > Was there a userspace program that did this? As in it would
> > call ioperm?
> >
> >> Konrad - I think that you had suggested an alternate way of doing
> >> this, IIRC, but I can't seem to find it in my inbox. Due to
> >> competing demands, and my nVidia hardware disappearing, I never went
> >> back to rework this code.
> > Ah, I might have some code that I just uncovered from Jeremy's old
> > git tree.
> >
> > I've put it up on devel/ioperm - it nicely wraps the ioperm
> > call to go the native or paravirt.
>
> Yeah, I'd sort of let that sit, since there's nothing in a "modern"
> system which should require it.  Allowing usermode to poke at ioports
> from within a domain is inherently suspect, after all.
>
>    J
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: ioperm problem

[Pavel Matěja]

On Thu 17. of November 2011 18:30:10 Konrad Rzeszutek Wilk wrote:
> Pavel,
> 
> Can you try to merge the #devel/ioperm patches in your tree and see
> if they fix your problem?
> 
> The git tree is at
> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git

Hi Konrad,
yes, it fixed the problem. I can see BSOD during Windows boot now.
Thanks a lot
-- 
Pavel Mateja