How to intercept interrupts from guest domains

How to intercept interrupts from guest domains

[Mads Bergdal]

Hi!

I am writing my master thesis on virtualization with Xen. I am trying to 
    intercept the hypercalls coming from the guest domains. More 
specific I am trying to determine where in memory a guest domain is 
writing. Does anyone have a hint on where in the code I should try to do 
this?

Rgds
Mads

Re: How to intercept interrupts from guest domains

[Keir Fraser]

On 19/9/06 10:14, "Mads Bergdal" <mbergdal@gmail.com> wrote:

> I am writing my master thesis on virtualization with Xen. I am trying to
>     intercept the hypercalls coming from the guest domains. More
> specific I am trying to determine where in memory a guest domain is
> writing. Does anyone have a hint on where in the code I should try to do
> this?

If you want to know where a guest is writing you need to do more than
intercept hypercalls. You want to intercept memory accesses to, which would
liekly mean you need to run on shadow pagetables and manipulate access
permissions to trap on first access to a page.

 -- Keir

Re: How to intercept interrupts from guest domains

[Mads Bergdal]

Keir Fraser wrote:
> On 19/9/06 10:14, "Mads Bergdal" <mbergdal@gmail.com> wrote:
> 
>> I am writing my master thesis on virtualization with Xen. I am trying to
>>     intercept the hypercalls coming from the guest domains. More
>> specific I am trying to determine where in memory a guest domain is
>> writing. Does anyone have a hint on where in the code I should try to do
>> this?
> 
> If you want to know where a guest is writing you need to do more than
> intercept hypercalls. You want to intercept memory accesses to, which would
> liekly mean you need to run on shadow pagetables and manipulate access
> permissions to trap on first access to a page.
> 
>  -- Keir
Yes, that sounds reasonable. Do you know where in the code this could be 
achieved?

Mads

Re: Re: How to intercept interrupts from guest domains

[Keir Fraser]

On 19/9/06 10:52, "Mads Bergdal" <mbergdal@gmail.com> wrote:

>>> I am writing my master thesis on virtualization with Xen. I am trying to
>>>     intercept the hypercalls coming from the guest domains. More
>>> specific I am trying to determine where in memory a guest domain is
>>> writing. Does anyone have a hint on where in the code I should try to do
>>> this?
>> 
>> If you want to know where a guest is writing you need to do more than
>> intercept hypercalls. You want to intercept memory accesses to, which would
>> liekly mean you need to run on shadow pagetables and manipulate access
>> permissions to trap on first access to a page.
>> 
>>  -- Keir
> Yes, that sounds reasonable. Do you know where in the code this could be
> achieved?

What's the intended purpose? You could perhaps look at the log-dirty shadow
mode. This is used to track which pages have been modified by the guest -- a
page which the guest maps writeable is not made writable in the shadow page
tables until the time of the first write access (when that page is added to
a 'dirty log' for further processing).

Be warned that modifying the shadow code is rather more difficult than a
project that would simply involve adding a hook point to every hypercall!

 -- Keir

Re: How to intercept interrupts from guest domains

[Mads Bergdal]

Keir Fraser wrote:
> On 19/9/06 10:52, "Mads Bergdal" <mbergdal@gmail.com> wrote:
> 
>>>> I am writing my master thesis on virtualization with Xen. I am trying to
>>>>     intercept the hypercalls coming from the guest domains. More
>>>> specific I am trying to determine where in memory a guest domain is
>>>> writing. Does anyone have a hint on where in the code I should try to do
>>>> this?
>>> If you want to know where a guest is writing you need to do more than
>>> intercept hypercalls. You want to intercept memory accesses to, which would
>>> liekly mean you need to run on shadow pagetables and manipulate access
>>> permissions to trap on first access to a page.
>>>
>>>  -- Keir
>> Yes, that sounds reasonable. Do you know where in the code this could be
>> achieved?
> 
> What's the intended purpose? You could perhaps look at the log-dirty shadow
> mode. This is used to track which pages have been modified by the guest -- a
> page which the guest maps writeable is not made writable in the shadow page
> tables until the time of the first write access (when that page is added to
> a 'dirty log' for further processing).
> 
> Be warned that modifying the shadow code is rather more difficult than a
> project that would simply involve adding a hook point to every hypercall!
> 
>  -- Keir
Thanks for the hints. I really appreciate it.

My main purpose of this is to try to monitor when a guestdomain tries to 
write to a specific address in it's memory. (An address that it should 
not write to) And then get the Hypervisor to notify my userspace 
"surveillance" program about this.
I have spent quite some time now reading the code. I must admit I am a 
bit lost. I am not sure where in the code I should be looking to get 
started.

 From what you write above I take it that you think the easiest approach 
  is to hook into the hypercalls? I that case which hypercalls and where?

If not, where should I look to learn he internals of the shadow mode?

Hope I am not wasting your time...

Mads

RE: Re: How to intercept interrupts from guest domains

[Petersson, Mats]

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> Mads Bergdal
> Sent: 21 September 2006 12:46
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] Re: How to intercept interrupts from 
> guest domains
> 
> Keir Fraser wrote:
> > On 19/9/06 10:52, "Mads Bergdal" <mbergdal@gmail.com> wrote:
> > 
> >>>> I am writing my master thesis on virtualization with 
> Xen. I am trying to
> >>>>     intercept the hypercalls coming from the guest domains. More
> >>>> specific I am trying to determine where in memory a 
> guest domain is
> >>>> writing. Does anyone have a hint on where in the code I 
> should try to do
> >>>> this?
> >>> If you want to know where a guest is writing you need to 
> do more than
> >>> intercept hypercalls. You want to intercept memory 
> accesses to, which would
> >>> liekly mean you need to run on shadow pagetables and 
> manipulate access
> >>> permissions to trap on first access to a page.
> >>>
> >>>  -- Keir
> >> Yes, that sounds reasonable. Do you know where in the code 
> this could be
> >> achieved?
> > 
> > What's the intended purpose? You could perhaps look at the 
> log-dirty shadow
> > mode. This is used to track which pages have been modified 
> by the guest -- a
> > page which the guest maps writeable is not made writable in 
> the shadow page
> > tables until the time of the first write access (when that 
> page is added to
> > a 'dirty log' for further processing).
> > 
> > Be warned that modifying the shadow code is rather more 
> difficult than a
> > project that would simply involve adding a hook point to 
> every hypercall!
> > 
> >  -- Keir
> Thanks for the hints. I really appreciate it.
> 
> My main purpose of this is to try to monitor when a 
> guestdomain tries to 
> write to a specific address in it's memory. (An address that 
> it should 
> not write to) And then get the Hypervisor to notify my userspace 
> "surveillance" program about this.
> I have spent quite some time now reading the code. I must 
> admit I am a 
> bit lost. I am not sure where in the code I should be looking to get 
> started.
> 
>  From what you write above I take it that you think the 
> easiest approach 
>   is to hook into the hypercalls? I that case which 
> hypercalls and where?
> 
> If not, where should I look to learn he internals of the shadow mode?


To catch a kernel writing to a specific address, you can't rely on
hypercalls - there is no hypercall to indicate that a particular address
(or any address) has been written to. 

When the kernel writes to memory, the processor will read the page-table
entry according to CR3 and the relevant page-table entries after that
(depending on whether it's 32-bit, 32-bit+PAE or 64-bit and whether the
pages are "large" 2MB/4MB or "small" 4KB). If the page indicated by the
page-table is writable, then the processor will do nothing other than
write the requested data to the memory. If the page indicated by the
page-table is not writable, then the a page-fault is issued. In the case
of Xen, this is trapped by Xen directly, and handled in the
page-fault-handling routine. Depending on what the reason for page-fault
is, Xen either changes the page to writable, or issues the page-fault
back to the kernel (if Xen didn't think this page should be writable). 

Xen uses non-writable pages to keep track of various things, so it's
perfectly "normal" for Xen to take a page-fault for write to a
non-writable page, and Xen "knows" whether a page SHOULD be written to
or not. A typical case is the shadow page-table handling itself.
Page-tables in the guest are write-protected, Xen takes the page-fault
and figures out what the new value for the page-table-entry should be
(remember that the Kernel doesn't actually know that memory mnay not be
organized like the kernel expects) and the ACTUAL page-table (shadow
version) used by the processor is updated accordingly, as well as
writing the new value to the kernel's "visible" page-table [but this
page-table is never used by the processor to address memory]. 

To make a memory address "trap" in the hypervisor, you would therefor
have to make sure the page that contains that address is always
write-protected, and when the page gets a hit, compare with your
"magical" value, and if so, indicate it back to your "supervisor"
application. Depending on the granularity, you could potentially just
write-protect the page and never have to worry about
write-enable/disable again. 

[This gets even more tricky on SMP systems, I hope you're only planning
on doing this for a single processor!]

Is the target address you're after a physical or a virtual address?

This is just an overview of how paging (and shadow paging) works...
There's more to it than the above couple of paragraphs, as you can
imagine. 

--
Mats

> 
> Hope I am not wasting your time...
> 
> Mads
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 
> 

Re: How to intercept interrupts from guest domains

[Mads Bergdal]

Petersson, Mats wrote:
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com 
>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
>> Mads Bergdal
>> Sent: 21 September 2006 12:46
>> To: xen-devel@lists.xensource.com
>> Subject: [Xen-devel] Re: How to intercept interrupts from 
>> guest domains
>>
>> Keir Fraser wrote:
>>> On 19/9/06 10:52, "Mads Bergdal" <mbergdal@gmail.com> wrote:
>>>
>>>>>> I am writing my master thesis on virtualization with 
>> Xen. I am trying to
>>>>>>     intercept the hypercalls coming from the guest domains. More
>>>>>> specific I am trying to determine where in memory a 
>> guest domain is
>>>>>> writing. Does anyone have a hint on where in the code I 
>> should try to do
>>>>>> this?
>>>>> If you want to know where a guest is writing you need to 
>> do more than
>>>>> intercept hypercalls. You want to intercept memory 
>> accesses to, which would
>>>>> liekly mean you need to run on shadow pagetables and 
>> manipulate access
>>>>> permissions to trap on first access to a page.
>>>>>
>>>>>  -- Keir
>>>> Yes, that sounds reasonable. Do you know where in the code 
>> this could be
>>>> achieved?
>>> What's the intended purpose? You could perhaps look at the 
>> log-dirty shadow
>>> mode. This is used to track which pages have been modified 
>> by the guest -- a
>>> page which the guest maps writeable is not made writable in 
>> the shadow page
>>> tables until the time of the first write access (when that 
>> page is added to
>>> a 'dirty log' for further processing).
>>>
>>> Be warned that modifying the shadow code is rather more 
>> difficult than a
>>> project that would simply involve adding a hook point to 
>> every hypercall!
>>>  -- Keir
>> Thanks for the hints. I really appreciate it.
>>
>> My main purpose of this is to try to monitor when a 
>> guestdomain tries to 
>> write to a specific address in it's memory. (An address that 
>> it should 
>> not write to) And then get the Hypervisor to notify my userspace 
>> "surveillance" program about this.
>> I have spent quite some time now reading the code. I must 
>> admit I am a 
>> bit lost. I am not sure where in the code I should be looking to get 
>> started.
>>
>>  From what you write above I take it that you think the 
>> easiest approach 
>>   is to hook into the hypercalls? I that case which 
>> hypercalls and where?
>>
>> If not, where should I look to learn he internals of the shadow mode?
> 
> 
> To catch a kernel writing to a specific address, you can't rely on
> hypercalls - there is no hypercall to indicate that a particular address
> (or any address) has been written to. 
> 
> When the kernel writes to memory, the processor will read the page-table
> entry according to CR3 and the relevant page-table entries after that
> (depending on whether it's 32-bit, 32-bit+PAE or 64-bit and whether the
> pages are "large" 2MB/4MB or "small" 4KB). If the page indicated by the
> page-table is writable, then the processor will do nothing other than
> write the requested data to the memory. If the page indicated by the
> page-table is not writable, then the a page-fault is issued. In the case
> of Xen, this is trapped by Xen directly, and handled in the
> page-fault-handling routine. Depending on what the reason for page-fault
> is, Xen either changes the page to writable, or issues the page-fault
> back to the kernel (if Xen didn't think this page should be writable). 
> 
> Xen uses non-writable pages to keep track of various things, so it's
> perfectly "normal" for Xen to take a page-fault for write to a
> non-writable page, and Xen "knows" whether a page SHOULD be written to
> or not. A typical case is the shadow page-table handling itself.
> Page-tables in the guest are write-protected, Xen takes the page-fault
> and figures out what the new value for the page-table-entry should be
> (remember that the Kernel doesn't actually know that memory mnay not be
> organized like the kernel expects) and the ACTUAL page-table (shadow
> version) used by the processor is updated accordingly, as well as
> writing the new value to the kernel's "visible" page-table [but this
> page-table is never used by the processor to address memory]. 
> 
> To make a memory address "trap" in the hypervisor, you would therefor
> have to make sure the page that contains that address is always
> write-protected, and when the page gets a hit, compare with your
> "magical" value, and if so, indicate it back to your "supervisor"
> application. Depending on the granularity, you could potentially just
> write-protect the page and never have to worry about
> write-enable/disable again. 
> 
> [This gets even more tricky on SMP systems, I hope you're only planning
> on doing this for a single processor!]

> Is the target address you're after a physical or a virtual address?
> 
> This is just an overview of how paging (and shadow paging) works...
> There's more to it than the above couple of paragraphs, as you can
> imagine. 
> 
> --
> Mats
> 
>> Hope I am not wasting your time...
>>
>> Mads
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>>
>>
Thanks a lot!

Well clearly this is a bit more complicated than I first excepted....
I'm only doing this for single 32-bit processors....

I guess I'm primarily interested in the virtual addresses. (I think...). 
If my 'supervisor' program first finds out which page the guest kernels 
syscall table (for instance...) maps to, the I would only need the 
hypervisor to send an message if this page is being accessed and let the 
'supervisor' program take care of the rest?

The most promising thing to do then is to:
- find out which page my 'untouchable' address is on and mark this 
page 	as non-writable. (how?) Or are all pages marked as non-writable by 
Xen as default?
- This way  the guest will always get a page fault, and trap to Xen. 
(Where in Xen is this trap handled?)
-I maintain (in Xen) an array of pages that is not supposed to be 
written to, and compare this to the page the guest kernel wants to write 
to.
-If the page is in the array I send a message (how?) to my program 
saying that the guest is trying to access a marked page.
-My program then decides if the guest kernel is allowed to proceed, and 
responds to the Hypervisor (how?)

Does this sound doable?
If so, where in the xen code is the xen page-fault-handling routine? And 
how can I send messages to and from the Hypervisor?

Mads

RE: Re: How to intercept interrupts from guest domains

[Petersson, Mats]

> Thanks a lot!
> 
> Well clearly this is a bit more complicated than I first excepted....
> I'm only doing this for single 32-bit processors....
> 
> I guess I'm primarily interested in the virtual addresses. (I 
> think...). 
> If my 'supervisor' program first finds out which page the 
> guest kernels 
> syscall table (for instance...) maps to, the I would only need the 
> hypervisor to send an message if this page is being accessed 
> and let the 
> 'supervisor' program take care of the rest?

Ok, so you're trying to write something that detects writes to the
syscall page. Isn't that write-protected already [I haven't checked, but
I expect it to be]. In which case any kernel write to that page would be
trapped with a kernel-oops... 

But perhaps that's not really the page you're looking to see if it's
being written to?

> 
> The most promising thing to do then is to:
> - find out which page my 'untouchable' address is on and mark this 
> page 	as non-writable. (how?) Or are all pages marked as 
> non-writable by 
> Xen as default?

Yes. How? I don't know... It may be that you'd have to invent your own
way to "fiddle" with the page-table. 

> - This way  the guest will always get a page fault, and trap to Xen. 
> (Where in Xen is this trap handled?)

You'd end up in "do_page_fault" in xen/arch/x86/traps.c - note that
there is also x86_64 and x86_32/traps.c - these are additional
architecture specific trap handlers. The function do_page_fault is 32/64
independent [obviously only to the x86 architecture, this code DOES NOT
work on ia64 or ppc for example], so it lives at the upper level... 

do_page_fault calls "fixup_page_fault" which calls "shadow_fault", which
in the current shadow page-table code ends up in "sh_page_fault". 

Any of these functions should be able to detect your fault. It depends
on how you want to treat the handling, and what information you need to
have available, which function is best for you. 

> -I maintain (in Xen) an array of pages that is not supposed to be 
> written to, and compare this to the page the guest kernel 
> wants to write 
> to.

Seems reasonable. 

> -If the page is in the array I send a message (how?) to my program 
> saying that the guest is trying to access a marked page.

The usual method is by "event channel". See
xen/include/public/event_channel.h

> -My program then decides if the guest kernel is allowed to 
> proceed, and 
> responds to the Hypervisor (how?)

Event channel?

> 
> Does this sound doable?

Certainly possible. Non-trivial, but doable. 


> If so, where in the xen code is the xen page-fault-handling 
> routine? And 
> how can I send messages to and from the Hypervisor?
> 
> Mads
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 
>