Trapping I/O accesses of a driver domain

Trapping I/O accesses of a driver domain

[Olatunji Ruwase]

Hi,
 I am a graduate student working on dynamic correctness checking of kernel
 mode device drivers. I want to detect/trap accesses from a Linux driver
 to device accessible locations (e.g ioremap'd, dma_* locations), and
 I am exploring the possibility of using Xen for this. I am using x86 PV
 Xen-3.3 with a dom0 and driver domU both running linux-2.6.18-xen. For
 various reasons HVM Xen is not suitable for my work.

 The idea is to use page faults to detect the I/O accesses of the driver
 by marking the affected pages not present in the page tables. For
 ioremap'd pages, this seems pretty straightforward since the ptes are
 marked with _PAGE_IO before they are passed to Xen. And so it seems
 modifying do_mmu_update () to detect and mark such ptes not present should
 work. Is this a reasonable approach ?.

 Detecting accesses to dma mapped (dma_alloc_coherent, dma_map_single)
 locations seems more difficult because, as far as I can tell there is no
 hypercall informing Xen that the locations are used for I/O. I am probably
 misunderstanding how this works and would appreciate clarifications.

 Thank you,

tunji

Re: Trapping I/O accesses of a driver domain

[Konrad Rzeszutek Wilk]

On Thu, Nov 10, 2011 at 05:57:27PM -0500, Olatunji Ruwase wrote:
> Hi,
>  I am a graduate student working on dynamic correctness checking of kernel
>  mode device drivers. I want to detect/trap accesses from a Linux driver
>  to device accessible locations (e.g ioremap'd, dma_* locations), and
>  I am exploring the possibility of using Xen for this. I am using x86 PV
>  Xen-3.3 with a dom0 and driver domU both running linux-2.6.18-xen. For
>  various reasons HVM Xen is not suitable for my work.

Um, why not use something more recent. Like Ubuntu or Fedora Core 16?
> 
>  The idea is to use page faults to detect the I/O accesses of the driver
>  by marking the affected pages not present in the page tables. For
>  ioremap'd pages, this seems pretty straightforward since the ptes are
>  marked with _PAGE_IO before they are passed to Xen. And so it seems

Not all the time and it is not a requirement.
>  modifying do_mmu_update () to detect and mark such ptes not present should
>  work. Is this a reasonable approach ?.

What about just checking the MFNs against the ones in the E820 that
are in the PCI gap space?
> 
>  Detecting accesses to dma mapped (dma_alloc_coherent, dma_map_single)
>  locations seems more difficult because, as far as I can tell there is no
>  hypercall informing Xen that the locations are used for I/O. I am probably

Right.
>  misunderstanding how this works and would appreciate clarifications.
> 
>  Thank you,
> 
> tunji
> 
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Trapping I/O accesses of a driver domain

[Olatunji Ruwase]

>> Xen-3.3 with a dom0 and driver domU both running linux-2.6.18-xen. For
>> various reasons HVM Xen is not suitable for my work.
>
> Um, why not use something more recent. Like Ubuntu or Fedora Core 16?
>
 My work is based on simulated hardware logging and a significantly
 modified FC5, porting the kernel modifications to FC6 is significantly
 than to more recent kernels like FC16.

>> ioremap'd pages, this seems pretty straightforward since the ptes are
>> marked with _PAGE_IO before they are passed to Xen. And so it seems
>
> Not all the time and it is not a requirement.
>
 I am happy to modify the 2.16.8-xen to cover the outstanding cases,
 except this is a fundamentally flawed approach. Can you elaborate the
 ioremap scenarios for pte are not marked _PAGE_IO. Are the requirements
 documented?

>> modifying do_mmu_update () to detect and mark such ptes not present
>> should work. Is this a reasonable approach ?.
>
> What about just checking the MFNs against the ones in the E820 that
> are in the PCI gap space?
>>
  I m not familiar with E820, but will explore it, thanks.

>>  hypercall informing Xen that the locations are used for I/O. I am
>> probably
>
> Right.
>

 Thanks for the response.

tunji

Re: Trapping I/O accesses of a driver domain

[Konrad Rzeszutek Wilk]

On Thu, Nov 10, 2011 at 10:25:56PM -0500, Olatunji Ruwase wrote:
> >> Xen-3.3 with a dom0 and driver domU both running linux-2.6.18-xen. For
> >> various reasons HVM Xen is not suitable for my work.
> >
> > Um, why not use something more recent. Like Ubuntu or Fedora Core 16?
> >
>  My work is based on simulated hardware logging and a significantly
>  modified FC5, porting the kernel modifications to FC6 is significantly
>  than to more recent kernels like FC16.

You could do this on real hardware. Say get an machine with IOMMU
(like a TA890FXE) and use the AMD VI to trap you on all the IOMMU
(so DMA) operations. ..

Thought it might be worth reading first the AMD VI spec whether you can
trap on all DMA operations.
> 
> >> ioremap'd pages, this seems pretty straightforward since the ptes are
> >> marked with _PAGE_IO before they are passed to Xen. And so it seems
> >
> > Not all the time and it is not a requirement.
> >
>  I am happy to modify the 2.16.8-xen to cover the outstanding cases,
>  except this is a fundamentally flawed approach. Can you elaborate the

Huh? What is the flawed approach?

>  ioremap scenarios for pte are not marked _PAGE_IO. Are the requirements
>  documented?

The _PAGE_IO is a Linux kernel concept used to figure if the PTE contains
the MFN or  PFN value. I don't think the hypervisor cares about it.

> 
> >> modifying do_mmu_update () to detect and mark such ptes not present
> >> should work. Is this a reasonable approach ?.
> >
> > What about just checking the MFNs against the ones in the E820 that
> > are in the PCI gap space?
> >>
>   I m not familiar with E820, but will explore it, thanks.

So it sounds like you are concentrating on making this work in the dom0, domU,
not in the hypervisor. In which case you can ignore the E820.

> 
> >>  hypercall informing Xen that the locations are used for I/O. I am
> >> probably
> >
> > Right.
> >
> 
>  Thanks for the response.
> 
> tunji
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Trapping I/O accesses of a driver domain

[Tim Deegan]

At 10:26 -0500 on 11 Nov (1321007179), Konrad Rzeszutek Wilk wrote:
> On Thu, Nov 10, 2011 at 10:25:56PM -0500, Olatunji Ruwase wrote:
> > >> Xen-3.3 with a dom0 and driver domU both running linux-2.6.18-xen. For
> > >> various reasons HVM Xen is not suitable for my work.
> > >
> > > Um, why not use something more recent. Like Ubuntu or Fedora Core 16?
> > >
> >  My work is based on simulated hardware logging and a significantly
> >  modified FC5, porting the kernel modifications to FC6 is significantly
> >  than to more recent kernels like FC16.
> 
> You could do this on real hardware. Say get an machine with IOMMU
> (like a TA890FXE) and use the AMD VI to trap you on all the IOMMU
> (so DMA) operations. ..
> 
> Thought it might be worth reading first the AMD VI spec whether you can
> trap on all DMA operations.

Unfortunately, AFAIK you can't currently restart operations that the
IOMMU has disallowed, so actually capturing the DMA and having the
peripheral keep working would be tricky.

Tim.

Re: Trapping I/O accesses of a driver domain

[Olatunji Ruwase]

>> I am happy to modify the 2.16.8-xen to cover the outstanding cases,
>> except this is a fundamentally flawed approach. Can you elaborate the
>
> Huh? What is the flawed approach?
>
Pardon the typo, that was meant to ask if the following idea was flawed.
Enable trapping on accesses to ioremap'd pages by (1) mark their PTEs with
_PAGE_IO before they are passed to HYPERVISOR_mmu_update(), (2) in xen
(do_mmu_update()) mark pages for which _PAGE_IO is set not present.
It seemed to me that 2.6.18-xen always does (1), but you clarified that it
was not the case.

So I wanted to know in which scenarios could an ioremap'd PTE be passed to
xen without having _PAGE_IO set. And conversely, in which scenarios could
a non-ioremap'd page PTE be passed to xen with _PAGE_IO set. However,
given your comment about xen being unware of _PAGE_IO, the converse case
probably does not matter. With knowledge of these scenerios, then perhaps
I could modify both 2.6.18-xen and xen and use _PAGE_IO markings to
achieve my goal of causing traps on ioremap'd page accesses.

> So it sounds like you are concentrating on making this work in the dom0,
> domU, not in the hypervisor. In which case you can ignore the E820.
>
I would prefer modifying only the hypervisor if possible, so your
suggestion of checking against the PCI gap space in E820 sounds relevant.
In fact it seems that the machine address(mfn) argument passed to
ioremap*() should fall into the PCI gap space. I will investigate this
assumption.

Trapping I/O accesses of a driver domain

[Olatunji Ruwase]

Hi,
 I am a graduate student working on dynamic correctness checking of kernel
 mode device drivers. I want to detect/trap accesses from a Linux driver
 to device accessible locations (e.g ioremap'd, dma_* locations), and
 I am exploring the possibility of using Xen for this. I am using x86 PV
 Xen-3.3 with a dom0 and driver domU both running linux-2.6.18-xen. For
 various reasons HVM Xen is not suitable for my work.

 The idea is to use page faults to detect the I/O accesses of the driver
 by marking the affected pages not present in the page tables. For
 ioremap'd pages, this seems pretty straightforward since the ptes are
 marked with _PAGE_IO before they are passed to Xen. And so it seems
 modifying do_mmu_update () to detect and mark such ptes not present should
 work. Is this a reasonable approach ?.

 Detecting accesses to dma mapped (dma_alloc_coherent, dma_map_single)
 locations seems more difficult because, as far as I can tell there is no
 hypercall informing Xen that the locations are used for I/O. I am probably
 misunderstanding how this works and would appreciate clarifications.

 Thank you,

tunji