Contiguous DMAble memory in Dom0 user space linux process

Contiguous DMAble memory in Dom0 user space linux process

[Kashmira. K.]

Hello Xen experts,

I am trying to get contiguous DMAble memory in user space application which is a processes running on xen capable Linux.

I thought I could use xen_create_contiguous_region(), but it appears that is for linux kernel modules. 

I need to get about 500MB of contiguous DMAble memory in my application. This application will run in Dom0. This memory is not to be shared between Domains.

Is it possible to use the  HYPERVISOR_grant_table_op hypercall passing GMTTABOP_map_grant_ref as command to achieve this. If so, to use this hypercall the grant reference is required. How do I get the grant reference?  Would appreciate if I can get an example code or pointer to one to clearly understand how to achieve this.

If you have any other suggestion of a better way to get contiguous memory in user space app in Dom0, I am open to that.

On a different note, where can I get detailed hypercall reference guide.  Most palaces I've searched have a list of hypercalls but not the description of each of the commands of arguments of the hypercall. 

Thanks in advance.

Regards
Kashmira

Re: Contiguous DMAble memory in Dom0 user space linux process

[Konrad Rzeszutek Wilk]

On Wed, Jan 13, 2010 at 01:03:39PM -0800, Kashmira. K. wrote:
> Hello Xen experts,
> 
> I am trying to get contiguous DMAble memory in user space application which is a processes running on xen capable Linux.

Why? Is the driver you are communicating with doing DMA of huge chunks of memory (bigger than 2MB?)

There surely must be another way to make your application do what it has
to do than allocating such a large swath of memory.


> Is it possible to use the  HYPERVISOR_grant_table_op hypercall passing GMTTABOP_map_grant_ref as command to achieve this. If so, to use this hypercall the grant reference is required. How do I get the grant reference?  Would appreciate if I can get an example code or pointer to one to clearly understand how to achieve this.

Grant reference won't get you contiguous memory. It will get you memory
shared between domains.

Re: Contiguous DMAble memory in Dom0 user space linux process

[Kashmira. K.]

Hi Konard,

Please see response inline

--- On Wed, 1/13/10, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> Subject: Re: [Xen-devel] Contiguous DMAble memory in Dom0 user space linux process
> To: "Kashmira. K." <chimnee@yahoo.com>
> Cc: xen-devel@lists.xensource.com, winterfi@cisco.com
> Date: Wednesday, January 13, 2010, 1:31 PM
> On Wed, Jan 13, 2010 at 01:03:39PM
> -0800, Kashmira. K. wrote:
> > Hello Xen experts,
> > 
> > I am trying to get contiguous DMAble memory in user
> space application which is a processes running on xen
> capable Linux.
> 
> Why? Is the driver you are communicating with doing DMA of
> huge chunks of memory (bigger than 2MB?)
> 
Actually I'm not trying to communicate with a driver. The user space application is kind of a driver that lives outside of the kernel and communicates with the device directly. 
On a linux system (non xen) the bigphysarea patch was used to get large contiguous DMA memory in  user space. 
I am trying to do the equivalant on Xen capable linux in Dom0 so the same application will work. Kind of port it from 'linux' to 'Xen capable linux'.

> There surely must be another way to make your application
> do what it has
> to do than allocating such a large swath of memory.
> 
> 
> > Is it possible to use the 
> HYPERVISOR_grant_table_op hypercall passing
> GMTTABOP_map_grant_ref as command to achieve this. If so, to
> use this hypercall the grant reference is required. How do I
> get the grant reference?  Would appreciate if I can get
> an example code or pointer to one to clearly understand how
> to achieve this.
> 
> Grant reference won't get you contiguous memory. It will
> get you memory
> shared between domains.
> 
Yes I see now. I had confused the flags options GNTMAP_device_map to be DMAble memory.
There is a HYPERVISOR_memory_op which can exchange memory for contiguous one. But then this does not help me direcltly since a hypercall can only be used in the kernel.

Thanks
Kashmira 





> 

Re: Contiguous DMAble memory in Dom0 user space linux process

[Konrad Rzeszutek Wilk]

> Actually I'm not trying to communicate with a driver. The user space application is kind of a driver that lives outside of the kernel and communicates with the device directly. 

OK, but isn't the hardware visible via the PCI bus? Does it have any
PCI BARs defined?

> On a linux system (non xen) the bigphysarea patch was used to get large contiguous DMA memory in  user space. 
> I am trying to do the equivalant on Xen capable linux in Dom0 so the same application will work. Kind of port it from 'linux' to 'Xen capable linux'.

Oh boy. I think you have two options:

1) Play with stubdomains. They are based on Mini-OS and
   allow you to create applications + MiniOS (that runs under Xen).
   An example of this is the QEMU stubdomain.

   You can modify your MiniOS to get you virtual addresses that are
   mapped directly to contingous memory areas.

2). Look at fbmem.c and write a device driver that allows the userspace
   application to mmap the driver (similar to fb_mem), which in turn sets
   up a vmalloc-ed area that is physically contigous (by using the
   xen_create_contiguous_region function to exchange PFNs).

Re: Contiguous DMAble memory in Dom0 user space linux process

[Kashmira. K.]

Hi Konard,

Thanks you for your reply. 

On the (non xen) linux system, the hardware was visible to the userspace app. The PCI bars were mapped. So I do need to get the same functionality on xen capable linux.

I'm still a newbie to the xen internals, but I was wondering if I could write a simple linux kernel driver for a xen capable linux to do the following
- Map the PCI address (BAR). 
- Get contiguous memory.
- Provide user space ioctl call to get the mapped PCI BAR
- Provide user space ioctl to get pointer to the contiguous memory.

Will that work?

Thanks
Kashmira
----- Original Message ----

From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
To: Kashmira. K. <chimnee@yahoo.com>
Cc: winterfi@cisco.com; xen-devel@lists.xensource.com
Sent: Fri, January 15, 2010 8:59:17 AM
Subject: Re: [Xen-devel] Contiguous DMAble memory in Dom0 user space linux process

> Actually I'm not trying to communicate with a driver. The user space application is kind of a driver that lives outside of the kernel and communicates with the device directly. 

OK, but isn't the hardware visible via the PCI bus? Does it have any
PCI BARs defined?

> On a linux system (non xen) the bigphysarea patch was used to get large contiguous DMA memory in  user space. 
> I am trying to do the equivalant on Xen capable linux in Dom0 so the same application will work. Kind of port it from 'linux' to 'Xen capable linux'.

Oh boy. I think you have two options:

1) Play with stubdomains. They are based on Mini-OS and
   allow you to create applications + MiniOS (that runs under Xen).
   An example of this is the QEMU stubdomain.

   You can modify your MiniOS to get you virtual addresses that are
   mapped directly to contingous memory areas.

2). Look at fbmem.c and write a device driver that allows the userspace
   application to mmap the driver (similar to fb_mem), which in turn sets
   up a vmalloc-ed area that is physically contigous (by using the
   xen_create_contiguous_region function to exchange PFNs).

Re: Contiguous DMAble memory in Dom0 user space linux process

[Konrad Rzeszutek Wilk]

On Wed, Jan 20, 2010 at 01:10:24PM -0800, Kashmira. K. wrote:
> Hi Konard,
> 
> Thanks you for your reply. 
> 
> On the (non xen) linux system, the hardware was visible to the userspace app. The PCI bars were mapped. So I do need to get the same functionality on xen capable linux.
> 
> I'm still a newbie to the xen internals, but I was wondering if I could write a simple linux kernel driver for a xen capable linux to do the following
> - Map the PCI address (BAR). 
> - Get contiguous memory.
> - Provide user space ioctl call to get the mapped PCI BAR
> - Provide user space ioctl to get pointer to the contiguous memory.
> 
> Will that work?

Sure. I would recommend you look at the 3w-xxxx.c driver for a simple
ioctl subsystem and how to get contiguous memory mapped in. For mmap functionality
hpet.c is pretty simple. Foremost, make sure you get a PDF copy of
"Understanding the Linux Kernel"  as that will answer a lot of your questions.