Driver domain on ARM questions

Driver domain on ARM questions

[Andrii Tseglytskyi]

Hi,

Can someone clarify the following questions:

- is any special support required in Xen code to implement driver
domain on ARM ?
- is it possible to have one to one memory mapping in dom0 and driver
domain simultaneously ?
- if answer to previous one is NO, how can I make DMA working in driver domain ?

I found only high level concepts:

http://events.linuxfoundation.org/sites/events/files/slides/alsf13_stabellini.pdf
http://pdub.net/proj/usenix08boston/xen_drive/resources/xensummit/slides/14_SangBumSuh.pdf

Thank you for any info, useful links or so.

Regards,
Andrii

-- 

Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com

Re: Driver domain on ARM questions

[Stefano Stabellini]

On Wed, 15 Oct 2014, Andrii Tseglytskyi wrote:
> Hi,
> 
> Can someone clarify the following questions:
> 
> - is any special support required in Xen code to implement driver
> domain on ARM ?

Not particularly, but you need to be able to assign the device to the
driver domain so the device needs to be behind an SMMU.


> - is it possible to have one to one memory mapping in dom0 and driver
> domain simultaneously ?

We would want to avoid the 1:1 mapping in driver domains, so this is why
we need the device to be behind an SMMU.


> - if answer to previous one is NO, how can I make DMA working in driver domain ?

The best and more performant option is to use an SMMU.
Otherwise you can:

- use XENMEM_exchange to make a guest buffer contiguous in machine
memory space and bounce all the dma requests on the buffer using
swiotlb-xen. It is not going to have great performances but it should
work well.

- use the 1:1 hack in the driver domain


> I found only high level concepts:
> 
> http://events.linuxfoundation.org/sites/events/files/slides/alsf13_stabellini.pdf
> http://pdub.net/proj/usenix08boston/xen_drive/resources/xensummit/slides/14_SangBumSuh.pdf
> 
> Thank you for any info, useful links or so.
> 
> Regards,
> Andrii
> 
> -- 
> 
> Andrii Tseglytskyi | Embedded Dev
> GlobalLogic
> www.globallogic.com
> 

Re: Driver domain on ARM questions

[Andrii Tseglytskyi]

Hi Stefano,

Thanks a lot for your answers.

Regards,
Andrii

On Wed, Oct 15, 2014 at 4:48 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Wed, 15 Oct 2014, Andrii Tseglytskyi wrote:
>> Hi,
>>
>> Can someone clarify the following questions:
>>
>> - is any special support required in Xen code to implement driver
>> domain on ARM ?
>
> Not particularly, but you need to be able to assign the device to the
> driver domain so the device needs to be behind an SMMU.
>
>
>> - is it possible to have one to one memory mapping in dom0 and driver
>> domain simultaneously ?
>
> We would want to avoid the 1:1 mapping in driver domains, so this is why
> we need the device to be behind an SMMU.
>
>
>> - if answer to previous one is NO, how can I make DMA working in driver domain ?
>
> The best and more performant option is to use an SMMU.
> Otherwise you can:
>
> - use XENMEM_exchange to make a guest buffer contiguous in machine
> memory space and bounce all the dma requests on the buffer using
> swiotlb-xen. It is not going to have great performances but it should
> work well.
>
> - use the 1:1 hack in the driver domain
>
>
>> I found only high level concepts:
>>
>> http://events.linuxfoundation.org/sites/events/files/slides/alsf13_stabellini.pdf
>> http://pdub.net/proj/usenix08boston/xen_drive/resources/xensummit/slides/14_SangBumSuh.pdf
>>
>> Thank you for any info, useful links or so.
>>
>> Regards,
>> Andrii
>>
>> --
>>
>> Andrii Tseglytskyi | Embedded Dev
>> GlobalLogic
>> www.globallogic.com
>>



-- 

Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com

Re: Driver domain on ARM questions

[Andrii Anisov]

[-- Attachment #1.1: Type: text/plain, Size: 2573 bytes --]

Stefano,

> Not particularly, but you need to be able to assign the device to the
> driver domain
As I understand the passthrough patch series from Julien
http://www.gossamer-threads.com/lists/xen/devel/342003 was intended to
implement similar functionality, but not fit into 4.5.

Julien, could you please clarify your plans on the mentioned series?

Andrii Anisov | Team Lead
GlobalLogic
Kyiv, 03038, Protasov Business Park, M.Grinchenka, 2/1
P +38.044.492.9695x3664  M +380505738852  S andriyanisov
www.globallogic.com
<http://www.globallogic.com/>
http://www.globallogic.com/email_disclaimer.txt

On Thu, Oct 16, 2014 at 11:36 AM, Andrii Tseglytskyi <
andrii.tseglytskyi@globallogic.com> wrote:

> Hi Stefano,
>
> Thanks a lot for your answers.
>
> Regards,
> Andrii
>
> On Wed, Oct 15, 2014 at 4:48 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Wed, 15 Oct 2014, Andrii Tseglytskyi wrote:
> >> Hi,
> >>
> >> Can someone clarify the following questions:
> >>
> >> - is any special support required in Xen code to implement driver
> >> domain on ARM ?
> >
> > Not particularly, but you need to be able to assign the device to the
> > driver domain so the device needs to be behind an SMMU.
> >
> >
> >> - is it possible to have one to one memory mapping in dom0 and driver
> >> domain simultaneously ?
> >
> > We would want to avoid the 1:1 mapping in driver domains, so this is why
> > we need the device to be behind an SMMU.
> >
> >
> >> - if answer to previous one is NO, how can I make DMA working in driver
> domain ?
> >
> > The best and more performant option is to use an SMMU.
> > Otherwise you can:
> >
> > - use XENMEM_exchange to make a guest buffer contiguous in machine
> > memory space and bounce all the dma requests on the buffer using
> > swiotlb-xen. It is not going to have great performances but it should
> > work well.
> >
> > - use the 1:1 hack in the driver domain
> >
> >
> >> I found only high level concepts:
> >>
> >>
> http://events.linuxfoundation.org/sites/events/files/slides/alsf13_stabellini.pdf
> >>
> http://pdub.net/proj/usenix08boston/xen_drive/resources/xensummit/slides/14_SangBumSuh.pdf
> >>
> >> Thank you for any info, useful links or so.
> >>
> >> Regards,
> >> Andrii
> >>
> >> --
> >>
> >> Andrii Tseglytskyi | Embedded Dev
> >> GlobalLogic
> >> www.globallogic.com
> >>
>
>
>
> --
>
> Andrii Tseglytskyi | Embedded Dev
> GlobalLogic
> www.globallogic.com
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>

[-- Attachment #1.2: Type: text/html, Size: 6287 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Driver domain on ARM questions

[Julien Grall]

Hi Andrii,

On 10/16/2014 04:20 PM, Andrii Anisov wrote:
> Stefano,
> 
>> Not particularly, but you need to be able to assign the device to the
>> driver domain
> As I understand the passthrough patch series from Julien
> http://www.gossamer-threads.com/lists/xen/devel/342003 was intended to
> implement similar functionality, but not fit into 4.5.
> 
> Julien, could you please clarify your plans on the mentioned series?

I was planning to try to push at least the interrupt passthrough to Xen
4.5, but I decided to delay until Xen 4.6. It was safer, and I found
multiple race condition in the code the last few days.

BTW, the actual series should not be used in production but only for
internal testing ;).

Except this issue, the hypervisor side is quite ready. Now the main
concern is how do we get the device tree information for the device and
how do we pass it the guest device tree?

A device may have other dependencies (such as the clock, phy...). Xen
will have to passthrough/provide PV drivers for a such things.

Regards,

-- 
Julien Grall

Re: Driver domain on ARM questions

[Andrii Anisov]

[-- Attachment #1.1: Type: text/plain, Size: 425 bytes --]

Except the devices passthrough to the actually DomU, I expect issues with
xen libs bringup in DomU.
Also it is not clear how to connect driver frontends with backends in DomU.

Andrii Anisov | Team Lead
GlobalLogic
Kyiv, 03038, Protasov Business Park, M.Grinchenka, 2/1
P +38.044.492.9695x3664  M +380505738852  S andriyanisov
www.globallogic.com
<http://www.globallogic.com/>
http://www.globallogic.com/email_disclaimer.txt

[-- Attachment #1.2: Type: text/html, Size: 2803 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Driver domain on ARM questions

[Stefano Stabellini]

On Thu, 16 Oct 2014, Andrii Anisov wrote:
> Except the devices passthrough to the actually DomU, I expect issues with xen libs bringup in DomU.
> Also it is not clear how to connect driver frontends with backends in DomU.

It should all work "out of the box". The following wiki page was written
for Xen x86 but should work as-is for Xen on ARM:

http://wiki.xen.org/wiki/Driver_Domain

Re: Driver domain on ARM questions

[Andrii Anisov]

[-- Attachment #1.1: Type: text/plain, Size: 1104 bytes --]

Julien,

Looking into this http://www.gossamer-threads.com/lists/xen/devel/342003
slides I see that dom0 could be pretty thin, with no device drivers at all.
But it is just a picture:)
I suppose that, at least block device should be in dom0 in order to not
load huge ramdisk with tools, VM images, etc.

What we actually estimate on our site is a thin dom0 with minimum number of
device drivers and minimal functionality. This domain should only start and
check other domains are alive. Driver domain should handle most of hardware
and provide correspondent backends.

BTW, the actual series should not be used in production but only for
> internal testing ;).

Our current intention is not a production, rather POC ;)

A device may have other dependencies (such as the clock, phy...).

Yep, separating the HW between different domains is really painful.

Andrii Anisov | Team Lead
GlobalLogic
Kyiv, 03038, Protasov Business Park, M.Grinchenka, 2/1
P +38.044.492.9695x3664  M +380505738852  S andriyanisov
www.globallogic.com
<http://www.globallogic.com/>
http://www.globallogic.com/email_disclaimer.txt

[-- Attachment #1.2: Type: text/html, Size: 3921 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Driver domain on ARM questions

[Andrii Anisov]

[-- Attachment #1.1: Type: text/plain, Size: 660 bytes --]

>
> It should all work "out of the box". The following wiki page was written
>
> for Xen x86 but should work as-is for Xen on ARM:
>
>
>> http://wiki.xen.org/wiki/Driver_Domain
>
> I've read the page one more time and got that backend domain for network
could be specified in configuration.
But still have doubts:

   - Network backend is a kernel driver, while most of backends we design
   are userspace.
   - Userspace  backends need xen libs and xenstored, would they run in
   domU?


Andrii Anisov | Team Lead
GlobalLogic
Kyiv, 03038, Protasov Business Park, M.Grinchenka, 2/1
P +38.044.492.9695x3664  M +380505738852  S andriyanisov
www.globallogic.com

[-- Attachment #1.2: Type: text/html, Size: 2892 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Driver domain on ARM questions

[Ian Campbell]

On Thu, 2014-10-16 at 19:01 +0300, Andrii Anisov wrote:

>       * Userspace  backends need xen libs and xenstored, would they
>         run in domU? 
> 

AFAIK the only requirement is that you mount /proc/xen so that privcmd
etc are available. There have certainly been cases of people running
userspace backends in dom0 in the past.

Ian.