DMA understanding

DMA understanding

[Abhinav Srivastava]

Hi there,

I am trying to understand how an HVM guest domain performs its DMA operations, and how this DMA operations are intercepted by the Xen. I wanted to understand both the code path with and without Vt-d support (for intel processors). On looking inside the Xen code, I found that iommu code is inside the vmx/vtd/ directory only. By seeing the code, my understanding is that when Vt-d is enabled, iommu.c and dmar.c inside the vtd directory is the place to look for DMA operations. However, I do not understand which code path inside the hypervisor is getting used in case of Vt-d is disabled?  How does Xen intercept guest DMA operations in this case? I am using Xen 3.3 version for my project (I admit that it is very old version).

Any help would be appreciated.

Thanks,
Abhinav

Re: DMA understanding

[Konrad Rzeszutek Wilk]

On Tue, Jun 29, 2010 at 12:10:48AM +0530, Abhinav Srivastava wrote:
> 
> Hi there,
> 
> I am trying to understand how an HVM guest domain performs its DMA operations, and how this DMA operations are intercepted by the Xen. I wanted to understand both the code path with and without Vt-d support (for intel processors). On looking inside the Xen code, I found that iommu code is inside the vmx/vtd/ directory only. By seeing the code, my understanding is that when Vt-d is enabled, iommu.c and dmar.c inside the vtd directory is the place to look for DMA operations. However, I do not understand which code path inside the hypervisor is getting used in case of Vt-d is disabled?  How does Xen intercept guest DMA operations in this case? I am using Xen 3.3 version for my project (I admit that it is very old version).

Lets start without the Intel VT-d or AMD Vi in the picture.

When the QEMU boots up an HVM guest, it emulates everything the guest
sees or does. Which means that when the guest decides to use the
IDE controller to do DMA operations, QEMU decodes that operation
(look in hw/ide.c, search for 'WIN_READDMA') and it follows it
through by setting up a callback mechanism that ends up fetching
the data from wherever the guest disk and then triggering an interrupt
so that the guest noticies that the DMA finished.

So in essence the hypervisor does not deal with guest DMA at all.

When you insert an Intel VT-d or AMD Vi chipset you have the option
of passing in a native PCI device to the guest. If you don't pass
in a PCI device then you are still using the old mechanism.

Re: DMA understanding

[Abhinav Srivastava]

Hi Konrad,

Thanks for your reply. You reply was very helpful in understanding the 
DMA mechanism. The goal of my project is to intercept all DMA requests issued by guest HVM domains and check for the memory regions (guest physical address) mentioned in those requests. This will help detect malicious DMA writes specified by malicious drivers. I am trying 
to achieve this without VT-d support on intel processors.

I have some follow up questions:

1. When a guest HVM domain requests DMA operations, it specifies guest physical addresses. Who converts guest physical to host physical addresses? Does this conversion happen in Dom0 or hypervisor? Which code path should I be looking at?

2. I am looking at the place where I can hook into so that I could intercept all DMA requests issued by the HVM guest and verify the addresses? Is there any place where all DMA requests come and then routed to specific devices in QEMU emulation code? If I could hook at the common place, it would be easier to intercept rather putting the check 
in each device specific files.

I really appreciate for your time.

Thanks,
Abhinav

--- On Wed, 30/6/10, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> Subject: Re: [Xen-devel] DMA understanding
> To: "Abhinav Srivastava" <abhinavs_iitkgp@yahoo.co.in>
> Cc: xen-devel@lists.xensource.com
> Date: Wednesday, 30 June, 2010, 9:32 PM
> On Tue, Jun 29, 2010 at 12:10:48AM
> +0530, Abhinav Srivastava wrote:
> > 
> > Hi there,
> > 
> > I am trying to understand how an HVM guest domain
> performs its DMA operations, and how this DMA operations are
> intercepted by the Xen. I wanted to understand both the code
> path with and without Vt-d support (for intel processors).
> On looking inside the Xen code, I found that iommu code is
> inside the vmx/vtd/ directory only. By seeing the code, my
> understanding is that when Vt-d is enabled, iommu.c and
> dmar.c inside the vtd directory is the place to look for DMA
> operations. However, I do not understand which code path
> inside the hypervisor is getting used in case of Vt-d is
> disabled?  How does Xen intercept guest DMA operations
> in this case? I am using Xen 3.3 version for my project (I
> admit that it is very old version).
> 
> Lets start without the Intel VT-d or AMD Vi in the
> picture.
> 
> When the QEMU boots up an HVM guest, it emulates everything
> the guest
> sees or does. Which means that when the guest decides to
> use the
> IDE controller to do DMA operations, QEMU decodes that
> operation
> (look in hw/ide.c, search for 'WIN_READDMA') and it follows
> it
> through by setting up a callback mechanism that ends up
> fetching
> the data from wherever the guest disk and then triggering
> an interrupt
> so that the guest noticies that the DMA finished.
> 
> So in essence the hypervisor does not deal with guest DMA
> at all.
> 
> When you insert an Intel VT-d or AMD Vi chipset you have
> the option
> of passing in a native PCI device to the guest. If you
> don't pass
> in a PCI device then you are still using the old
> mechanism.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: QEMU and hypervisor DMA understanding. Want to track DMA operations on QEMU devices.

[Konrad Rzeszutek Wilk]

On Thu, Jul 01, 2010 at 11:38:31AM +0530, Abhinav Srivastava wrote:
> Hi Konrad,
> 
> Thanks for your reply. You reply was very helpful in understanding the 
> DMA mechanism. The goal of my project is to intercept all DMA requests issued by guest HVM domains and check for the memory regions (guest physical address) mentioned in those requests. This will help detect malicious DMA writes specified by malicious drivers. I am trying 
> to achieve this without VT-d support on intel processors.
> 
> I have some follow up questions:
> 
> 1. When a guest HVM domain requests DMA operations, it specifies guest physical addresses. Who converts guest physical to host physical addresses? Does this conversion happen in Dom0 or hypervisor? Which code path should I be looking at?

Hypervisor. Shadow page table. George might have a document tucked away
explaining how the shadow page table works.

> 
> 2. I am looking at the place where I can hook into so that I could intercept all DMA requests issued by the HVM guest and verify the addresses? Is there any place where all DMA requests come and then routed to specific devices in QEMU emulation code? If I could hook at the common place, it would be easier to intercept rather putting the check 
> in each device specific files.

Ian might know this better..
> 
> I really appreciate for your time.
> 
> Thanks,
> Abhinav
> 
> --- On Wed, 30/6/10, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> 
> > From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> > Subject: Re: [Xen-devel] DMA understanding
> > To: "Abhinav Srivastava" <abhinavs_iitkgp@yahoo.co.in>
> > Cc: xen-devel@lists.xensource.com
> > Date: Wednesday, 30 June, 2010, 9:32 PM
> > On Tue, Jun 29, 2010 at 12:10:48AM
> > +0530, Abhinav Srivastava wrote:
> > > 
> > > Hi there,
> > > 
> > > I am trying to understand how an HVM guest domain
> > performs its DMA operations, and how this DMA operations are
> > intercepted by the Xen. I wanted to understand both the code
> > path with and without Vt-d support (for intel processors).
> > On looking inside the Xen code, I found that iommu code is
> > inside the vmx/vtd/ directory only. By seeing the code, my
> > understanding is that when Vt-d is enabled, iommu.c and
> > dmar.c inside the vtd directory is the place to look for DMA
> > operations. However, I do not understand which code path
> > inside the hypervisor is getting used in case of Vt-d is
> > disabled?  How does Xen intercept guest DMA operations
> > in this case? I am using Xen 3.3 version for my project (I
> > admit that it is very old version).
> > 
> > Lets start without the Intel VT-d or AMD Vi in the
> > picture.
> > 
> > When the QEMU boots up an HVM guest, it emulates everything
> > the guest
> > sees or does. Which means that when the guest decides to
> > use the
> > IDE controller to do DMA operations, QEMU decodes that
> > operation
> > (look in hw/ide.c, search for 'WIN_READDMA') and it follows
> > it
> > through by setting up a callback mechanism that ends up
> > fetching
> > the data from wherever the guest disk and then triggering
> > an interrupt
> > so that the guest noticies that the DMA finished.
> > 
> > So in essence the hypervisor does not deal with guest DMA
> > at all.
> > 
> > When you insert an Intel VT-d or AMD Vi chipset you have
> > the option
> > of passing in a native PCI device to the guest. If you
> > don't pass
> > in a PCI device then you are still using the old
> > mechanism.
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> > 
> 
> 

Re: QEMU and hypervisor DMA understanding. Want to track DMA operations on QEMU devices.

[Tim Deegan]

At 15:51 +0100 on 01 Jul (1277999480), Konrad Rzeszutek Wilk wrote:
> On Thu, Jul 01, 2010 at 11:38:31AM +0530, Abhinav Srivastava wrote:
> > Hi Konrad,
> > 
> > Thanks for your reply. You reply was very helpful in understanding the 
> > DMA mechanism. The goal of my project is to intercept all DMA requests issued by guest HVM domains and check for the memory regions (guest physical address) mentioned in those requests. This will help detect malicious DMA writes specified by malicious drivers. I am trying 
> > to achieve this without VT-d support on intel processors.
> > 
> > I have some follow up questions:
> > 
> > 1. When a guest HVM domain requests DMA operations, it specifies guest physical addresses. Who converts guest physical to host physical addresses? Does this conversion happen in Dom0 or hypervisor? Which code path should I be looking at?
> 
> Hypervisor. Shadow page table. George might have a document tucked away
> explaining how the shadow page table works.

No need to go into shadow pagetables.  Xen sets up and maintains the
guest-physical-to-host-physical mapping, called the p2m table.  DMA is
done by qemu mapping the guest's memory and writing to it; qemu makes
the mapping hypercall with a pfn and the hypervitor translates to an mfn
for it.

> > 
> > 2. I am looking at the place where I can hook into so that I could intercept all DMA requests issued by the HVM guest and verify the addresses? Is there any place where all DMA requests come and then routed to specific devices in QEMU emulation code? If I could hook at the common place, it would be easier to intercept rather putting the check 
> > in each device specific files.
> 

You definitely want to do this in qemu rather than Xen.  Wemu has
central accessors for guest memory that you can hook.  It used to be
called cpu_physical_memory_rw(), but I haven't looked at it in a while. 

Tim.

> Ian might know this better..
> > 
> > I really appreciate for your time.
> > 
> > Thanks,
> > Abhinav
> > 
> > --- On Wed, 30/6/10, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > 
> > > From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> > > Subject: Re: [Xen-devel] DMA understanding
> > > To: "Abhinav Srivastava" <abhinavs_iitkgp@yahoo.co.in>
> > > Cc: xen-devel@lists.xensource.com
> > > Date: Wednesday, 30 June, 2010, 9:32 PM
> > > On Tue, Jun 29, 2010 at 12:10:48AM
> > > +0530, Abhinav Srivastava wrote:
> > > > 
> > > > Hi there,
> > > > 
> > > > I am trying to understand how an HVM guest domain
> > > performs its DMA operations, and how this DMA operations are
> > > intercepted by the Xen. I wanted to understand both the code
> > > path with and without Vt-d support (for intel processors).
> > > On looking inside the Xen code, I found that iommu code is
> > > inside the vmx/vtd/ directory only. By seeing the code, my
> > > understanding is that when Vt-d is enabled, iommu.c and
> > > dmar.c inside the vtd directory is the place to look for DMA
> > > operations. However, I do not understand which code path
> > > inside the hypervisor is getting used in case of Vt-d is
> > > disabled?  How does Xen intercept guest DMA operations
> > > in this case? I am using Xen 3.3 version for my project (I
> > > admit that it is very old version).
> > > 
> > > Lets start without the Intel VT-d or AMD Vi in the
> > > picture.
> > > 
> > > When the QEMU boots up an HVM guest, it emulates everything
> > > the guest
> > > sees or does. Which means that when the guest decides to
> > > use the
> > > IDE controller to do DMA operations, QEMU decodes that
> > > operation
> > > (look in hw/ide.c, search for 'WIN_READDMA') and it follows
> > > it
> > > through by setting up a callback mechanism that ends up
> > > fetching
> > > the data from wherever the guest disk and then triggering
> > > an interrupt
> > > so that the guest noticies that the DMA finished.
> > > 
> > > So in essence the hypervisor does not deal with guest DMA
> > > at all.
> > > 
> > > When you insert an Intel VT-d or AMD Vi chipset you have
> > > the option
> > > of passing in a native PCI device to the guest. If you
> > > don't pass
> > > in a PCI device then you are still using the old
> > > mechanism.
> > > 
> > > 
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> > > 
> > 
> > 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: QEMU and hypervisor DMA understanding. Want to track DMA operations on QEMU devices.

[Abhinav Srivastava]

Hi Tim and Konrad,

Thanks for your reply and pointers. As per Tim's reply, I will look into the Qemu code and try to hook where the translation happens. BTW, what is the entry point into the qemu code on the DMA code path? Is it cpu_physical_memory_rw() or some other function? And, when the term "physical memory" is used in the qemu code, it is the guest physical memory or the host physical memory?

Thanks & Regards,
Abhinav

--- On Thu, 1/7/10, Tim Deegan <Tim.Deegan@citrix.com> wrote:

> From: Tim Deegan <Tim.Deegan@citrix.com>
> Subject: Re: [Xen-devel] QEMU and hypervisor DMA understanding. Want to track DMA operations on QEMU devices.
> To: "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com>
> Cc: "George Dunlap" <George.Dunlap@eu.citrix.com>, "Ian Campbell" <Ian.Campbell@eu.citrix.com>, "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>, "Abhinav Srivastava" <abhinavs_iitkgp@yahoo.co.in>
> Date: Thursday, 1 July, 2010, 8:47 PM
> At 15:51 +0100 on 01 Jul
> (1277999480), Konrad Rzeszutek Wilk wrote:
> > On Thu, Jul 01, 2010 at 11:38:31AM +0530, Abhinav
> Srivastava wrote:
> > > Hi Konrad,
> > > 
> > > Thanks for your reply. You reply was very helpful
> in understanding the 
> > > DMA mechanism. The goal of my project is to
> intercept all DMA requests issued by guest HVM domains and
> check for the memory regions (guest physical address)
> mentioned in those requests. This will help detect malicious
> DMA writes specified by malicious drivers. I am trying 
> > > to achieve this without VT-d support on intel
> processors.
> > > 
> > > I have some follow up questions:
> > > 
> > > 1. When a guest HVM domain requests DMA
> operations, it specifies guest physical addresses. Who
> converts guest physical to host physical addresses? Does
> this conversion happen in Dom0 or hypervisor? Which code
> path should I be looking at?
> > 
> > Hypervisor. Shadow page table. George might have a
> document tucked away
> > explaining how the shadow page table works.
> 
> No need to go into shadow pagetables.  Xen sets up and
> maintains the
> guest-physical-to-host-physical mapping, called the p2m
> table.  DMA is
> done by qemu mapping the guest's memory and writing to it;
> qemu makes
> the mapping hypercall with a pfn and the hypervitor
> translates to an mfn
> for it.
> 
> > > 
> > > 2. I am looking at the place where I can hook
> into so that I could intercept all DMA requests issued by
> the HVM guest and verify the addresses? Is there any place
> where all DMA requests come and then routed to specific
> devices in QEMU emulation code? If I could hook at the
> common place, it would be easier to intercept rather putting
> the check 
> > > in each device specific files.
> > 
> 
> You definitely want to do this in qemu rather than
> Xen.  Wemu has
> central accessors for guest memory that you can hook. 
> It used to be
> called cpu_physical_memory_rw(), but I haven't looked at it
> in a while. 
> 
> Tim.
> 
> > Ian might know this better..
> > > 
> > > I really appreciate for your time.
> > > 
> > > Thanks,
> > > Abhinav
> > > 
> > > --- On Wed, 30/6/10, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> wrote:
> > > 
> > > > From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> > > > Subject: Re: [Xen-devel] DMA understanding
> > > > To: "Abhinav Srivastava" <abhinavs_iitkgp@yahoo.co.in>
> > > > Cc: xen-devel@lists.xensource.com
> > > > Date: Wednesday, 30 June, 2010, 9:32 PM
> > > > On Tue, Jun 29, 2010 at 12:10:48AM
> > > > +0530, Abhinav Srivastava wrote:
> > > > > 
> > > > > Hi there,
> > > > > 
> > > > > I am trying to understand how an HVM
> guest domain
> > > > performs its DMA operations, and how this
> DMA operations are
> > > > intercepted by the Xen. I wanted to
> understand both the code
> > > > path with and without Vt-d support (for
> intel processors).
> > > > On looking inside the Xen code, I found that
> iommu code is
> > > > inside the vmx/vtd/ directory only. By
> seeing the code, my
> > > > understanding is that when Vt-d is enabled,
> iommu.c and
> > > > dmar.c inside the vtd directory is the place
> to look for DMA
> > > > operations. However, I do not understand
> which code path
> > > > inside the hypervisor is getting used in
> case of Vt-d is
> > > > disabled?  How does Xen intercept guest DMA
> operations
> > > > in this case? I am using Xen 3.3 version for
> my project (I
> > > > admit that it is very old version).
> > > > 
> > > > Lets start without the Intel VT-d or AMD Vi
> in the
> > > > picture.
> > > > 
> > > > When the QEMU boots up an HVM guest, it
> emulates everything
> > > > the guest
> > > > sees or does. Which means that when the
> guest decides to
> > > > use the
> > > > IDE controller to do DMA operations, QEMU
> decodes that
> > > > operation
> > > > (look in hw/ide.c, search for 'WIN_READDMA')
> and it follows
> > > > it
> > > > through by setting up a callback mechanism
> that ends up
> > > > fetching
> > > > the data from wherever the guest disk and
> then triggering
> > > > an interrupt
> > > > so that the guest noticies that the DMA
> finished.
> > > > 
> > > > So in essence the hypervisor does not deal
> with guest DMA
> > > > at all.
> > > > 
> > > > When you insert an Intel VT-d or AMD Vi
> chipset you have
> > > > the option
> > > > of passing in a native PCI device to the
> guest. If you
> > > > don't pass
> > > > in a PCI device then you are still using the
> old
> > > > mechanism.
> > > > 
> > > > 
> > > >
> _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.xensource.com
> > > > http://lists.xensource.com/xen-devel
> > > > 
> > > 
> > > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, XenServer Engineering
> Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>