[Qemu-devel] How to map PCI memory into the VM without trapping

[Qemu-devel] How to map PCI memory into the VM without trapping

[Frank Berreth]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

Hello,

I am using a modified patch for the ivshmem device that was posted here
recently and discovered some performance issues with it. Essentially I
measure write performance into the PCI memory that is mapped into the VM as
really slow (about 60mb/sec). I assume that is because there is trapping on
each memory access (there are no handlers installed for handling read/write
access to that memory though). The device shows up in the guest as a UIO
device but I also tried accessing other PCI memory like the vga memory (16mb
from vga-pci.c) and it has similar performance (through
/sys/device/.../resource...).

My question is how I can map that memory into the VM so it doesn't get
trapped on in the normal use case. The mappings (registering PCI bar etc) is
done as in the patch submitted recently).

Thanks,
Frank.

[-- Attachment #2: Type: text/html, Size: 928 bytes --]

[Qemu-devel] Re: How to map PCI memory into the VM without trapping

[Frank Berreth]

[-- Attachment #1: Type: text/plain, Size: 1770 bytes --]

Hi there,

this is just an update if you are interested in the outcome. I turns out
that my initial assumption that there would be page faults/trapping on the
memory pages was false. The reason the throughput is so low is because the
memory was mapped non-cached. The VGA driver and the ivshmem driver use
pci_ioremap_bar which will *always* map the PCI bar non-cached (even the
resourceX_wc).
Changing the driver(s) to use ioremap_cache or ioremap_wc speeds up things
quite a bit. I don't know if VGA framebuffer was always mapped this way --
it appears on a real system that usually graphics memory is mapped WC.
Mapping it UC would cause quite a performance degradation. This could be the
reason for the reported VGA performance drop in another email thread. IMHO,
since QEMU emulates VGA, this could be mapped WB.

Thanks,
Frank.

On Mon, May 3, 2010 at 2:07 PM, Frank Berreth <frankber@google.com> wrote:

> Hello,
>
> I am using a modified patch for the ivshmem device that was posted here
> recently and discovered some performance issues with it. Essentially I
> measure write performance into the PCI memory that is mapped into the VM as
> really slow (about 60mb/sec). I assume that is because there is trapping on
> each memory access (there are no handlers installed for handling read/write
> access to that memory though). The device shows up in the guest as a UIO
> device but I also tried accessing other PCI memory like the vga memory (16mb
> from vga-pci.c) and it has similar performance (through
> /sys/device/.../resource...).
>
> My question is how I can map that memory into the VM so it doesn't get
> trapped on in the normal use case. The mappings (registering PCI bar etc) is
> done as in the patch submitted recently).
>
> Thanks,
> Frank.
>
>

[-- Attachment #2: Type: text/html, Size: 2181 bytes --]

Re: [Qemu-devel] Re: How to map PCI memory into the VM without trapping

[Cam Macdonell]

On Wed, May 5, 2010 at 5:20 PM, Frank Berreth <frankber@google.com> wrote:
> Hi there,
> this is just an update if you are interested in the outcome. I turns out
> that my initial assumption that there would be page faults/trapping on the
> memory pages was false. The reason the throughput is so low is because the
> memory was mapped non-cached. The VGA driver and the ivshmem driver use
> pci_ioremap_bar which will *always* map the PCI bar non-cached (even the
> resourceX_wc).
> Changing the driver(s) to use ioremap_cache or ioremap_wc speeds up things
> quite a bit. I don't know if VGA framebuffer was always mapped this way --
> it appears on a real system that usually graphics memory is mapped WC.
> Mapping it UC would cause quite a performance degradation. This could be the
> reason for the reported VGA performance drop in another email thread. IMHO,
> since QEMU emulates VGA, this could be mapped WB.
> Thanks,
> Frank.
>

Hi Frank,

Thanks for the note.  I'll make the change to cached and see if that
helps with some of the tests I'm running.

Cheers,
Cam