[Qemu-devel] bypassing pci_dma_read() and pci_dma_write() ?

[Luigi Rizzo <rizzo@iet.unipi.it>]

Hi,
with a bunch of e1000 improvements we are at a point where we are
doing over 1Mpps (short frames) and 7-8Gbit/s (1500 byte frames)
between two guests, and two things that are high in the "perf top"
stats are phys_page_find() and related memory copies.

Both are triggered by the pci_dma_read() and pci_dma_write(),
which on e1000 (and presumably other frontends) are called on
every single descriptor and every single buffer.

I have then tried to access the guest memory without going every
time through the page lookup.

For the tx and rx rings i have a partial workaround, which tracks changes
to the base address of the ring, converts it to a host virtual address

     ....
    +#ifdef MAP_RING
    +    base = tx_desc_base(s);
    +    if (base != s->txring_phi) {
    +       hwaddr desclen = s->mac_reg[TDLEN];
    +       s->txring_phi = base;
    +       s->txring = address_space_map(pci_dma_context(&s->dev)->as,
    +                       base, &desclen, 0 /* is_write */);
    +    }
    +#endif /* MAP_RING */
     ...

and then accesses the descriptor directly into guest memory

    desc = s->txring[s->mac_reg[TDH]];

(sprinkle with memory barriers as needed).

This relies on the assumption that the ring (which is contiguous in the
guest's physical address space) is also contiguous in the host's virtual
address space.  In principle the property could be easily verified once
the ring is set up.

I have not done this for buffers because I am not sure how to verify
that the same mapping holds for all packet buffers.
One way could be the following:
on the first buffer access, make the address translation and try to
determine the boundaries of the contiguous (in virtual host memory)
region that holds the buffer. Then subsequent buffers can be easily
validated against this region.

So the problem is now the following: given a guest physical
address, is there a quick way to determine the contiguous
region of memory in the host that contains it ?

And of course, am i missing some important detail ?
Of course the above could be used conditionally if the required
conditions hold, and then revert to the pci_dma_*()
in other cases.

cheers
luigi