Can I specify a physical memory region for a domU

Can I specify a physical memory region for a domU

[Eric Camachat]

Hi,

I am porting our drivers to XEN's PV domU (with PV PCI passthrouth), I
have to allocate a memory block and tell the hardware to access it.
But the hardware can address 32-bit only, so I want dedicate a region
of memory that below 2GB for the domU only.
How do that in XEN?

Thanks,
/Eric

Re: Can I specify a physical memory region for a domU

[Tim Deegan]

At 17:47 -0700 on 07 Sep (1315417666), Eric Camachat wrote:
> Hi,
> 
> I am porting our drivers to XEN's PV domU (with PV PCI passthrouth), I
> have to allocate a memory block and tell the hardware to access it.
> But the hardware can address 32-bit only, so I want dedicate a region
> of memory that below 2GB for the domU only.
> How do that in XEN?

Use the XENMEM_echange memory_op hypercall.  You can specify a bit-width
for the pages you want returned. 

Of course, on a full Xen host there may be no 32-bit-clean memory left.
Xen tries to allocate the higher addresses first to avoid that, but
there's no guarantee.

Tim.

Re: Can I specify a physical memory region for a domU

[Konrad Rzeszutek Wilk]

On Wed, Sep 07, 2011 at 05:47:46PM -0700, Eric Camachat wrote:
> Hi,
> 
> I am porting our drivers to XEN's PV domU (with PV PCI passthrouth), I

Use the DMA API that Linux provides (I presume that is what you meant
by PV DomU), and use the dma_alloc_coherent to set your regions.

Also pass in 'iommu=soft' on your Linux command line to enable the
Xen SWIOTLB DMA system.

> have to allocate a memory block and tell the hardware to access it.
> But the hardware can address 32-bit only, so I want dedicate a region
> of memory that below 2GB for the domU only.

Uh, don't you mean 4GB? - 32bit is up to 4GB.
> How do that in XEN?
> 
> Thanks,
> /Eric
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Can I specify a physical memory region for a domU

[Eric Camachat]

On Thu, Sep 8, 2011 at 5:59 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Wed, Sep 07, 2011 at 05:47:46PM -0700, Eric Camachat wrote:
>> Hi,
>>
>> I am porting our drivers to XEN's PV domU (with PV PCI passthrouth), I
>
> Use the DMA API that Linux provides (I presume that is what you meant
> by PV DomU), and use the dma_alloc_coherent to set your regions.

That's what I thought before. We use a shared DMA region for multiple hardware.
Maybe I can dma_alloc_coherent for 1st and the others use the same region.
I will try it.

>
> Also pass in 'iommu=soft' on your Linux command line to enable the
> Xen SWIOTLB DMA system.
>
>> have to allocate a memory block and tell the hardware to access it.
>> But the hardware can address 32-bit only, so I want dedicate a region
>> of memory that below 2GB for the domU only.
>
> Uh, don't you mean 4GB? - 32bit is up to 4GB.

The hardware uses 32-bit addressing, but the system will crash if I
assigned above 2GB address to it.
So, 4GB from hardware spec, 2GB from my test. I am looking into that.

>> How do that in XEN?
>>
>> Thanks,
>> /Eric
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>

Re: Can I specify a physical memory region for a domU

[Konrad Rzeszutek Wilk]

On Thu, Sep 08, 2011 at 10:22:01AM -0700, Eric Camachat wrote:
> On Thu, Sep 8, 2011 at 5:59 AM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Wed, Sep 07, 2011 at 05:47:46PM -0700, Eric Camachat wrote:
> >> Hi,
> >>
> >> I am porting our drivers to XEN's PV domU (with PV PCI passthrouth), I
> >
> > Use the DMA API that Linux provides (I presume that is what you meant
> > by PV DomU), and use the dma_alloc_coherent to set your regions.
> 
> That's what I thought before. We use a shared DMA region for multiple hardware.
> Maybe I can dma_alloc_coherent for 1st and the others use the same region.

You can definitly try it. Or use the dmapool API to get a shared
pool of coherent memory.

> I will try it.
> 
> >
> > Also pass in 'iommu=soft' on your Linux command line to enable the
> > Xen SWIOTLB DMA system.
> >
> >> have to allocate a memory block and tell the hardware to access it.
> >> But the hardware can address 32-bit only, so I want dedicate a region
> >> of memory that below 2GB for the domU only.
> >
> > Uh, don't you mean 4GB? - 32bit is up to 4GB.
> 
> The hardware uses 32-bit addressing, but the system will crash if I
> assigned above 2GB address to it.

Ha! so buggy hardware.. or you are not using the XEn-SWIOTLB but something
else.
> So, 4GB from hardware spec, 2GB from my test. I am looking into that.

Make sure you set
 pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31));

on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31));
in your driver.

Re: Can I specify a physical memory region for a domU

[Eric Camachat]

On Thu, Sep 8, 2011 at 11:12 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Thu, Sep 08, 2011 at 10:22:01AM -0700, Eric Camachat wrote:
>> On Thu, Sep 8, 2011 at 5:59 AM, Konrad Rzeszutek Wilk
>> <konrad.wilk@oracle.com> wrote:
>> > On Wed, Sep 07, 2011 at 05:47:46PM -0700, Eric Camachat wrote:
>> >> Hi,
>> >>
>> >> I am porting our drivers to XEN's PV domU (with PV PCI passthrouth), I
>> >
>> > Use the DMA API that Linux provides (I presume that is what you meant
>> > by PV DomU), and use the dma_alloc_coherent to set your regions.
>>
>> That's what I thought before. We use a shared DMA region for multiple hardware.
>> Maybe I can dma_alloc_coherent for 1st and the others use the same region.
>
> You can definitly try it. Or use the dmapool API to get a shared
> pool of coherent memory.
>
>> I will try it.
>>
>> >
>> > Also pass in 'iommu=soft' on your Linux command line to enable the
>> > Xen SWIOTLB DMA system.
>> >
>> >> have to allocate a memory block and tell the hardware to access it.
>> >> But the hardware can address 32-bit only, so I want dedicate a region
>> >> of memory that below 2GB for the domU only.
>> >
>> > Uh, don't you mean 4GB? - 32bit is up to 4GB.
>>
>> The hardware uses 32-bit addressing, but the system will crash if I
>> assigned above 2GB address to it.
>
> Ha! so buggy hardware.. or you are not using the XEn-SWIOTLB but something
> else.
>> So, 4GB from hardware spec, 2GB from my test. I am looking into that.
>
> Make sure you set
>  pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31));
>
> on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31));
> in your driver.
>
A lot of work to port the driver to PV domU, hope it works.
Thanks for your help!

Re: Can I specify a physical memory region for a domU

[Konrad Rzeszutek Wilk]

> > Make sure you set
> >  pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31));
> >
> > on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31));
> > in your driver.
> >
> A lot of work to port the driver to PV domU, hope it works.

Hm? That is the normal way you would write drivers in the Linux kernel.
You use the DMA API in it to deal with the PCI devices.

Is the PV domU a Linux kernel or something else?

Re: Can I specify a physical memory region for a domU

[Eric Camachat]

On Thu, Sep 8, 2011 at 1:00 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>> > Make sure you set
>> >  pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31));
>> >
>> > on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31));
>> > in your driver.
>> >
>> A lot of work to port the driver to PV domU, hope it works.
>
> Hm? That is the normal way you would write drivers in the Linux kernel.
> You use the DMA API in it to deal with the PCI devices.
>
> Is the PV domU a Linux kernel or something else?
>

Yes, the PV domU is linux kernel.

I tested pci_alloc_consistent() verified on baremetal it worked with 8GB SDRAM.
But while I ran it in XEN, pci_alloc_consistent() cannot allocate
memory successfully for 4MB although pci_set_dma_mask(),
pci_set_consistent_dma_mask() and dma_set_mask() all succeeded.

Re: Can I specify a physical memory region for a domU

[Konrad Rzeszutek Wilk]

On Thu, Sep 08, 2011 at 05:15:06PM -0700, Eric Camachat wrote:
> On Thu, Sep 8, 2011 at 1:00 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> >> > Make sure you set
> >> >  pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31));
> >> >
> >> > on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31));
> >> > in your driver.
> >> >
> >> A lot of work to port the driver to PV domU, hope it works.
> >
> > Hm? That is the normal way you would write drivers in the Linux kernel.
> > You use the DMA API in it to deal with the PCI devices.
> >
> > Is the PV domU a Linux kernel or something else?
> >
> 
> Yes, the PV domU is linux kernel.
> 
> I tested pci_alloc_consistent() verified on baremetal it worked with 8GB SDRAM.
> But while I ran it in XEN, pci_alloc_consistent() cannot allocate
> memory successfully for 4MB although pci_set_dma_mask(),

And what is the error?

> pci_set_consistent_dma_mask() and dma_set_mask() all succeeded.

And what kernel did you use?
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Can I specify a physical memory region for a domU

[Eric Camachat]

On Thu, Sep 8, 2011 at 6:05 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Thu, Sep 08, 2011 at 05:15:06PM -0700, Eric Camachat wrote:
>> On Thu, Sep 8, 2011 at 1:00 PM, Konrad Rzeszutek Wilk
>> <konrad.wilk@oracle.com> wrote:
>> >> > Make sure you set
>> >> >  pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31));
>> >> >
>> >> > on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31));
>> >> > in your driver.
>> >> >
>> >> A lot of work to port the driver to PV domU, hope it works.
>> >
>> > Hm? That is the normal way you would write drivers in the Linux kernel.
>> > You use the DMA API in it to deal with the PCI devices.
>> >
>> > Is the PV domU a Linux kernel or something else?
>> >
>>
>> Yes, the PV domU is linux kernel.
>>
>> I tested pci_alloc_consistent() verified on baremetal it worked with 8GB SDRAM.
>> But while I ran it in XEN, pci_alloc_consistent() cannot allocate
>> memory successfully for 4MB although pci_set_dma_mask(),
>
> And what is the error?

I cannot see any error, it just returned NULL.

>
>> pci_set_consistent_dma_mask() and dma_set_mask() all succeeded.
>
> And what kernel did you use?
I am using linux-2.6.32.24.

>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>

Re: Can I specify a physical memory region for a domU

[Eric Camachat]

On Fri, Sep 9, 2011 at 9:10 AM, Eric Camachat <eric.camachat@gmail.com> wrote:
> On Thu, Sep 8, 2011 at 6:05 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
>> On Thu, Sep 08, 2011 at 05:15:06PM -0700, Eric Camachat wrote:
>>> On Thu, Sep 8, 2011 at 1:00 PM, Konrad Rzeszutek Wilk
>>> <konrad.wilk@oracle.com> wrote:
>>> >> > Make sure you set
>>> >> >  pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31));
>>> >> >
>>> >> > on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31));
>>> >> > in your driver.
>>> >> >
>>> >> A lot of work to port the driver to PV domU, hope it works.
>>> >
>>> > Hm? That is the normal way you would write drivers in the Linux kernel.
>>> > You use the DMA API in it to deal with the PCI devices.
>>> >
>>> > Is the PV domU a Linux kernel or something else?
>>> >
>>>
>>> Yes, the PV domU is linux kernel.
>>>
>>> I tested pci_alloc_consistent() verified on baremetal it worked with 8GB SDRAM.
>>> But while I ran it in XEN, pci_alloc_consistent() cannot allocate
>>> memory successfully for 4MB although pci_set_dma_mask(),
>>
>> And what is the error?
>
> I cannot see any error, it just returned NULL.

BTW, I will succeed if I allocate 2MB only.

>
>>
>>> pci_set_consistent_dma_mask() and dma_set_mask() all succeeded.
>>
>> And what kernel did you use?
> I am using linux-2.6.32.24.
>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>
>

Re: Can I specify a physical memory region for a domU

[Konrad Rzeszutek Wilk]

On Fri, Sep 09, 2011 at 09:48:03AM -0700, Eric Camachat wrote:
> On Fri, Sep 9, 2011 at 9:10 AM, Eric Camachat <eric.camachat@gmail.com> wrote:
> > On Thu, Sep 8, 2011 at 6:05 PM, Konrad Rzeszutek Wilk
> > <konrad.wilk@oracle.com> wrote:
> >> On Thu, Sep 08, 2011 at 05:15:06PM -0700, Eric Camachat wrote:
> >>> On Thu, Sep 8, 2011 at 1:00 PM, Konrad Rzeszutek Wilk
> >>> <konrad.wilk@oracle.com> wrote:
> >>> >> > Make sure you set
> >>> >> >  pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31));
> >>> >> >
> >>> >> > on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31));
> >>> >> > in your driver.
> >>> >> >
> >>> >> A lot of work to port the driver to PV domU, hope it works.
> >>> >
> >>> > Hm? That is the normal way you would write drivers in the Linux kernel.
> >>> > You use the DMA API in it to deal with the PCI devices.
> >>> >
> >>> > Is the PV domU a Linux kernel or something else?
> >>> >
> >>>
> >>> Yes, the PV domU is linux kernel.
> >>>
> >>> I tested pci_alloc_consistent() verified on baremetal it worked with 8GB SDRAM.
> >>> But while I ran it in XEN, pci_alloc_consistent() cannot allocate
> >>> memory successfully for 4MB although pci_set_dma_mask(),
> >>
> >> And what is the error?
> >
> > I cannot see any error, it just returned NULL.
> 
> BTW, I will succeed if I allocate 2MB only.

Ok, that is unsurprising. I don't know if you can do it any better on baremetal.
But I am failing to understand why you need such large swaths of contingous
memory. You can't do scatter gather? Or scatter gather on 2MB chunks?

Re: Can I specify a physical memory region for a domU

[Eric Camachat]

On Fri, Sep 9, 2011 at 11:06 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Fri, Sep 09, 2011 at 09:48:03AM -0700, Eric Camachat wrote:
>> On Fri, Sep 9, 2011 at 9:10 AM, Eric Camachat <eric.camachat@gmail.com> wrote:
>> > On Thu, Sep 8, 2011 at 6:05 PM, Konrad Rzeszutek Wilk
>> > <konrad.wilk@oracle.com> wrote:
>> >> On Thu, Sep 08, 2011 at 05:15:06PM -0700, Eric Camachat wrote:
>> >>> On Thu, Sep 8, 2011 at 1:00 PM, Konrad Rzeszutek Wilk
>> >>> <konrad.wilk@oracle.com> wrote:
>> >>> >> > Make sure you set
>> >>> >> >  pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31));
>> >>> >> >
>> >>> >> > on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31));
>> >>> >> > in your driver.
>> >>> >> >
>> >>> >> A lot of work to port the driver to PV domU, hope it works.
>> >>> >
>> >>> > Hm? That is the normal way you would write drivers in the Linux kernel.
>> >>> > You use the DMA API in it to deal with the PCI devices.
>> >>> >
>> >>> > Is the PV domU a Linux kernel or something else?
>> >>> >
>> >>>
>> >>> Yes, the PV domU is linux kernel.
>> >>>
>> >>> I tested pci_alloc_consistent() verified on baremetal it worked with 8GB SDRAM.
>> >>> But while I ran it in XEN, pci_alloc_consistent() cannot allocate
>> >>> memory successfully for 4MB although pci_set_dma_mask(),
>> >>
>> >> And what is the error?
>> >
>> > I cannot see any error, it just returned NULL.
>>
>> BTW, I will succeed if I allocate 2MB only.
>
> Ok, that is unsurprising. I don't know if you can do it any better on baremetal.
> But I am failing to understand why you need such large swaths of contingous
> memory. You can't do scatter gather? Or scatter gather on 2MB chunks?
>

I changed MAX_CONTIG_ORDER, and MAX_ORDER to 13 to fix this problem, finally.