Is there a way to map pv guest pseudo physical address into dom0?

Is there a way to map pv guest pseudo physical address into dom0?

[Wei Liu]

Hi, all

I'm reading the code of QEMU's virtio infrastructure (hw/virtio.c) and
Linux's virtio driver (virtio_blk.c, virtio_net.c).

Virtio drivers utilize scatter gather infrastructure in the Linux
kernel. It stores the physical addresses in scatter list. In Xen's hvm
case, QEMU can access guest address space with
cpu_physical_memory_map.

However, in Xen's pv case, this "physical address" will be "pseudo
physical address". In order to retrieve theirs content, backend must
map them into its own address space. I'm wondering if there is such
infrastructure to get this job done.

Wei.

Re: Is there a way to map pv guest pseudo physical address into dom0?

[Ian Campbell]

On Mon, 2011-06-27 at 09:47 +0100, Wei Liu wrote:
> Hi, all
> 
> I'm reading the code of QEMU's virtio infrastructure (hw/virtio.c) and
> Linux's virtio driver (virtio_blk.c, virtio_net.c).
> 
> Virtio drivers utilize scatter gather infrastructure in the Linux
> kernel. It stores the physical addresses in scatter list. In Xen's hvm
> case, QEMU can access guest address space with
> cpu_physical_memory_map.
> 
> However, in Xen's pv case, this "physical address" will be "pseudo
> physical address". In order to retrieve theirs content, backend must
> map them into its own address space. I'm wondering if there is such
> infrastructure to get this job done.

I think you want one of xc_map_* provided by libxc (see xenctrl.h). (But
avoid xc_map_foreign_batch which is deprecated).

Ian.

Re: Is there a way to map pv guest pseudo physical address into dom0?

[Wei Liu]

On Mon, Jun 27, 2011 at 5:01 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> I think you want one of xc_map_* provided by libxc (see xenctrl.h). (But
> avoid xc_map_foreign_batch which is deprecated).
>

AFAICT, they take MFN as parameter, but I have pseudo physical addresses. :-(

Wei.

> Ian.
>
>
>

Re: Is there a way to map pv guest pseudo physical address into dom0?

[Ian Campbell]

On Mon, 2011-06-27 at 10:10 +0100, Wei Liu wrote:
> On Mon, Jun 27, 2011 at 5:01 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > I think you want one of xc_map_* provided by libxc (see xenctrl.h). (But
> > avoid xc_map_foreign_batch which is deprecated).
> >
> 
> AFAICT, they take MFN as parameter, but I have pseudo physical addresses. :-(

You need to arrange for the Xen specific bit of the virtio frontend to
do the p2m translation. All externally visible addresses from a PV guest
must be in MFN space.

Otherwise you only option is to pause the guest and map his p2m. Which
would obviously suck.

Ian.

Re: Is there a way to map pv guest pseudo physical address into dom0?

[Wei Liu]

On Mon, Jun 27, 2011 at 5:17 PM, Ian Campbell
<Ian.Campbell@eu.citrix.com> wrote:
> On Mon, 2011-06-27 at 10:10 +0100, Wei Liu wrote:
>> On Mon, Jun 27, 2011 at 5:01 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > I think you want one of xc_map_* provided by libxc (see xenctrl.h). (But
>> > avoid xc_map_foreign_batch which is deprecated).
>> >
>>
>> AFAICT, they take MFN as parameter, but I have pseudo physical addresses. :-(
>
> You need to arrange for the Xen specific bit of the virtio frontend to
> do the p2m translation. All externally visible addresses from a PV guest
> must be in MFN space.
>

That's not a good approach. Dirvers reside on a higher level from the
"Xen specific bit". Doing this kind of translation will break their
generality. But there seems no other way to get this job done...

> Otherwise you only option is to pause the guest and map his p2m. Which
> would obviously suck.
>

Agreed.

Wei.

> Ian.
>
>
>

Re: Is there a way to map pv guest pseudo physical address into dom0?

[Tim Deegan]

At 17:26 +0800 on 27 Jun (1309195596), Wei Liu wrote:
> > You need to arrange for the Xen specific bit of the virtio frontend to
> > do the p2m translation. All externally visible addresses from a PV guest
> > must be in MFN space.
> >
> 
> That's not a good approach. Dirvers reside on a higher level from the
> "Xen specific bit". Doing this kind of translation will break their
> generality. But there seems no other way to get this job done...

This is a general problem with operating systems: bus addresses are not
always the same as CPU memory addresses (even though in the easy case of
a simple x86 PC they often are).  This is why OSes already have
interfaces to allocate DMA-able memory and to translate between the
address that CPU will map and the one the peripheral should use.  If
your driver's not using the right address-space to program its hardware
it is broken; the Xen PV model is just an extreme case of this.

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: Is there a way to map pv guest pseudo physical address into dom0?

[Stefano Stabellini]

On Mon, 27 Jun 2011, Tim Deegan wrote:
> At 17:26 +0800 on 27 Jun (1309195596), Wei Liu wrote:
> > > You need to arrange for the Xen specific bit of the virtio frontend to
> > > do the p2m translation. All externally visible addresses from a PV guest
> > > must be in MFN space.
> > >
> > 
> > That's not a good approach. Dirvers reside on a higher level from the
> > "Xen specific bit". Doing this kind of translation will break their
> > generality. But there seems no other way to get this job done...
> 
> This is a general problem with operating systems: bus addresses are not
> always the same as CPU memory addresses (even though in the easy case of
> a simple x86 PC they often are).  This is why OSes already have
> interfaces to allocate DMA-able memory and to translate between the
> address that CPU will map and the one the peripheral should use.  If
> your driver's not using the right address-space to program its hardware
> it is broken; the Xen PV model is just an extreme case of this.

Indeed.

When passing addresses to devices, drivers should use the DMA APIs to
make sure that proper address conversions take place (see
Documentation/DMA-API-HOWTO.txt).
On Xen the swiotlb implementation takes care of doing pfn to mfn
conversions as well.
For example, to allocate a consistent buffer that is addressable by both
a device and the cpu, the correct API is dma_alloc_coherent, that
returns two values: the virtual address which you can use to access it
from the CPU and dma_handle which you pass to the device.

I can very well imagine that virtio drivers are not doing any of this
because they didn't need to, but it would be probably OK to submit a
patch to convert the drivers to use the proper APIs.

Re: Is there a way to map pv guest pseudo physical address into dom0?

[Wei Liu]

Thank you, Tim and Stefano.

On Mon, Jun 27, 2011 at 7:23 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Mon, 27 Jun 2011, Tim Deegan wrote:
>> At 17:26 +0800 on 27 Jun (1309195596), Wei Liu wrote:
>> > > You need to arrange for the Xen specific bit of the virtio frontend to
>> > > do the p2m translation. All externally visible addresses from a PV guest
>> > > must be in MFN space.
>> > >
>> >
>> > That's not a good approach. Dirvers reside on a higher level from the
>> > "Xen specific bit". Doing this kind of translation will break their
>> > generality. But there seems no other way to get this job done...
>>
>> This is a general problem with operating systems: bus addresses are not
>> always the same as CPU memory addresses (even though in the easy case of
>> a simple x86 PC they often are).  This is why OSes already have
>> interfaces to allocate DMA-able memory and to translate between the
>> address that CPU will map and the one the peripheral should use.  If
>> your driver's not using the right address-space to program its hardware
>> it is broken; the Xen PV model is just an extreme case of this.
>
> Indeed.
>
> When passing addresses to devices, drivers should use the DMA APIs to
> make sure that proper address conversions take place (see
> Documentation/DMA-API-HOWTO.txt).
> On Xen the swiotlb implementation takes care of doing pfn to mfn
> conversions as well.
> For example, to allocate a consistent buffer that is addressable by both
> a device and the cpu, the correct API is dma_alloc_coherent, that
> returns two values: the virtual address which you can use to access it
> from the CPU and dma_handle which you pass to the device.
>

This is the first time to notice swiotlb TBH. It seems to be a good
start for me.

After skimming the code in $KERNEL/drivers/xen/swiotlb-xen.c, I have
further questions.

It seems that swiotlb is used for PCI passthrough in PV, I'm wondering
if paravirt drivers ever use it.

> I can very well imagine that virtio drivers are not doing any of this
> because they didn't need to, but it would be probably OK to submit a
> patch to convert the drivers to use the proper APIs.
>

No, they are not doing this.

Do you mean that I replace every allocation/deallocation of virtio
drivers scatter buffer with DMA API?

Then, with 'swiotlb=force' in pv guest kernel command line, these
DMA-API will call xen-swiotlb and get proper addresses?

If I don't add 'swiotlb=force', which underlying implementation will
be used? And, which DMA-API will be used in hvm case? will this change
affect hvm guest?

Wei.

Re: Is there a way to map pv guest pseudo physical address into dom0?

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 3024 bytes --]

On Mon, 27 Jun 2011, Wei Liu wrote:
> On Mon, Jun 27, 2011 at 7:23 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Mon, 27 Jun 2011, Tim Deegan wrote:
> >> At 17:26 +0800 on 27 Jun (1309195596), Wei Liu wrote:
> >> > > You need to arrange for the Xen specific bit of the virtio frontend to
> >> > > do the p2m translation. All externally visible addresses from a PV guest
> >> > > must be in MFN space.
> >> > >
> >> >
> >> > That's not a good approach. Dirvers reside on a higher level from the
> >> > "Xen specific bit". Doing this kind of translation will break their
> >> > generality. But there seems no other way to get this job done...
> >>
> >> This is a general problem with operating systems: bus addresses are not
> >> always the same as CPU memory addresses (even though in the easy case of
> >> a simple x86 PC they often are).  This is why OSes already have
> >> interfaces to allocate DMA-able memory and to translate between the
> >> address that CPU will map and the one the peripheral should use.  If
> >> your driver's not using the right address-space to program its hardware
> >> it is broken; the Xen PV model is just an extreme case of this.
> >
> > Indeed.
> >
> > When passing addresses to devices, drivers should use the DMA APIs to
> > make sure that proper address conversions take place (see
> > Documentation/DMA-API-HOWTO.txt).
> > On Xen the swiotlb implementation takes care of doing pfn to mfn
> > conversions as well.
> > For example, to allocate a consistent buffer that is addressable by both
> > a device and the cpu, the correct API is dma_alloc_coherent, that
> > returns two values: the virtual address which you can use to access it
> > from the CPU and dma_handle which you pass to the device.
> >
> 
> This is the first time to notice swiotlb TBH. It seems to be a good
> start for me.
> 
> After skimming the code in $KERNEL/drivers/xen/swiotlb-xen.c, I have
> further questions.
> 
> It seems that swiotlb is used for PCI passthrough in PV, I'm wondering
> if paravirt drivers ever use it.
> 
> > I can very well imagine that virtio drivers are not doing any of this
> > because they didn't need to, but it would be probably OK to submit a
> > patch to convert the drivers to use the proper APIs.
> >
> 
> No, they are not doing this.
> 
> Do you mean that I replace every allocation/deallocation of virtio
> drivers scatter buffer with DMA API?

Yes. Konrad is the most knowledgeable about this, he might have some
suggestions.


> Then, with 'swiotlb=force' in pv guest kernel command line, these
> DMA-API will call xen-swiotlb and get proper addresses?
> 
> If I don't add 'swiotlb=force', which underlying implementation will
> be used? And, which DMA-API will be used in hvm case? will this change
> affect hvm guest?
 
The right swiotlb implementation is chosen at run time depending on your
platform. So the user doesn't need to manually add swiotlb=force to the
command line, and the changes shouldn't affect performance in the normal
cases.

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Is there a way to map pv guest pseudo physical address into dom0?

[Konrad Rzeszutek Wilk]

On Mon, Jun 27, 2011 at 09:35:24PM +0800, Wei Liu wrote:
> Thank you, Tim and Stefano.
> 
> On Mon, Jun 27, 2011 at 7:23 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Mon, 27 Jun 2011, Tim Deegan wrote:
> >> At 17:26 +0800 on 27 Jun (1309195596), Wei Liu wrote:
> >> > > You need to arrange for the Xen specific bit of the virtio frontend to
> >> > > do the p2m translation. All externally visible addresses from a PV guest
> >> > > must be in MFN space.
> >> > >
> >> >
> >> > That's not a good approach. Dirvers reside on a higher level from the
> >> > "Xen specific bit". Doing this kind of translation will break their
> >> > generality. But there seems no other way to get this job done...
> >>
> >> This is a general problem with operating systems: bus addresses are not
> >> always the same as CPU memory addresses (even though in the easy case of
> >> a simple x86 PC they often are).  This is why OSes already have
> >> interfaces to allocate DMA-able memory and to translate between the
> >> address that CPU will map and the one the peripheral should use.  If
> >> your driver's not using the right address-space to program its hardware
> >> it is broken; the Xen PV model is just an extreme case of this.
> >
> > Indeed.
> >
> > When passing addresses to devices, drivers should use the DMA APIs to
> > make sure that proper address conversions take place (see
> > Documentation/DMA-API-HOWTO.txt).
> > On Xen the swiotlb implementation takes care of doing pfn to mfn
> > conversions as well.
> > For example, to allocate a consistent buffer that is addressable by both
> > a device and the cpu, the correct API is dma_alloc_coherent, that
> > returns two values: the virtual address which you can use to access it
> > from the CPU and dma_handle which you pass to the device.
> >
> 
> This is the first time to notice swiotlb TBH. It seems to be a good
> start for me.
> 
> After skimming the code in $KERNEL/drivers/xen/swiotlb-xen.c, I have
> further questions.
> 
> It seems that swiotlb is used for PCI passthrough in PV, I'm wondering
> if paravirt drivers ever use it.

And when running in Dom0.
> 
> > I can very well imagine that virtio drivers are not doing any of this
> > because they didn't need to, but it would be probably OK to submit a
> > patch to convert the drivers to use the proper APIs.
> >
> 
> No, they are not doing this.
> 
> Do you mean that I replace every allocation/deallocation of virtio
> drivers scatter buffer with DMA API?

Use the pci_map_sg calls and follow it with pci_sync calls.
The DMA API mentions how to do it properly. Countless drivers do it.

> 
> Then, with 'swiotlb=force' in pv guest kernel command line, these
> DMA-API will call xen-swiotlb and get proper addresses?

Correct. Thought you don't need to use 'swiotlb=force'. You can
just do 'iommu=soft' and that will take care of it in PV case.

In Dom0 case there is no need for this argument.
> 
> If I don't add 'swiotlb=force', which underlying implementation will
> be used? And, which DMA-API will be used in hvm case? will this change
> affect hvm guest?

For HVM if the machine has less than 4GB it will use the no_dma implementation.
Which is to say: phys_to_virt and virt_to_phys for all DMA calls.

If you have more than 4GB (or iommu=soft) it will use the native
SWIOTLB implementation.

(And the native SWIOTLB can't properly do PFN -> MFN or vice versa).
> 
> Wei.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel