Unshared IOMMU issues

[Oleksandr Tyshchenko <olekstysh@gmail.com>]

Hi, all.

As it was suggested by Julien in IRC I open this thread.

Currently, I am trying to add support for IPMMU in Xen.
It is VMSA-compatible IOMMU that integrated in the newest Renesas SoCs (ARM).
This IPMMU can't share the page table with the CPU since it uses
stage-1 page table
unlike the CPU that uses stage-2.
So, the IPMMU driver has own page table inside it and maintains them
like other "unshared IOMMU" drivers usually do.

For passing all mapping updates via IOMMU I slightly
modified P2M code (p2m_set_entry) on ARM to call iommu_map_page/iommu_unmap_page
if following (need_iommu(p2m->domain) &&
!iommu_use_hap_pt(p2m->domain)) is true.
I even optimized a bit by adding iommu_map_pages/iommu_unmap_pages API
and map_pages/unmap_pages flatform ops
for passing the whole memory block (nr pages) to the IOMMU code. But
it is not scope of this thread.

I faced several generic problems that had prevented me from making
IPMMU driver (but it might be another "unshared IOMMU" driver) happy
inside XEN on ARM.
Most of them I have already resolved somehow just to see that it
worked out well for me, but I am still have doubts about how to do it
in a right way.

So, for allowing P2M core to update IOMMU mapping from the first
"set_entry" and for "unshared IOMMU" driver to be ready to handle
IOMMU mapping updates
I do two things:
1. I always allocate IOMMU page table in iommu_domain_init() for every
domain even this domain won't have any assigned devices in future.
The main reason why I do so is not to skip any IOMMU mapping updates
from P2M code (RAM, MMIOs, etc). The IOMMU driver has to be ready for
processing
IOMMU mapping updates from the *very beginning*.
Of course, the IOMMU page table will be completely deleted in iommu_teardown().
But, anyway allocating IOMMU page table if it won't be really used in
domain looks not good.
Although there is an arch_iommu_populate_page_table() solution that
could help in a such situation,
but it does not look suitable for ARM because we have no way to
translate a MFN to a GFN as Julien had noticed me in IRC.

2. Another action I do is to explicitly set need_iommu flag during
arch_iommu_domain_init() call in ARM part if following
 (iommu_enabled && !is_hardware_domain(d) && !iommu_use_hap_pt(d)) is true.
I do that since in case of domU need_iommu flag is set during device
assignment, but it is too late. For dom0 we force need_iommu flag.
There are many mapping updates to P2M by the time the first device
will have been assigned.
I see the way how this action can be dropped. For example, don't rely
on need_iommu flag before updating IOMMU mapping from P2M,
check for iommu_enabled flag instead.

I think, but I am not 100% sure that we could avoid actions above if
we would have knowledge about device assignment for particular
domain before making any updates in P2M.

Could you please suggest me a right way in resolving such problems?

-- 
Regards,

Oleksandr Tyshchenko

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel