Logging External Page Writes for a Given Domain

Logging External Page Writes for a Given Domain

[Scott Baker]

Hello, all:

I'm new to Xen developing, and am hoping that you experts can help me 
out here (or at least point me in the right direction).

My team's goal is to be able to log all writes that are made to any 
memory page of a certain domain, except those writes that the domain 
itself makes.  That is to say, if Domain 2 is the domain we want to log 
page writes for (where logging is capturing what was written and its 
location), then we want to be able to log all the writes made by any 
domain /except/ Domain 2 -- i.e., writes made to shared pages that 
belong to Domain 2.

Ideally, we would like to be able to have these writes for domain /x/ 
detected and trap to a process running on Domain 0, with minimal VMM 
modification.  Preferably, the method used would only cause significant 
overhead when a write is made from outside domain /x/ (rather than for 
every write /x/ and everyone else makes), but I'm not picky about 
efficiency at the moment.

The rough idea we have at this point is to make the monitor process on 
Domain 0 mark all the pages of Domain /x/ as read-only (while 
remembering which are actually read-only).  Then, when a write-fault 
occurred, the VMM would pass it on to the monitoring process, which 
would then let Domain /x/ finish the write, the monitor would record 
what was written, and let everything continue as normal.

Unfortunately, I'm not sure how that vague sketch fits into Xen.  For 
the page table read-only flag setting, would we use the 
update_va_mapping() hypercall?  And, how would the monitoring process 
let Domain /x/ finish the write and then get control back?  (Or, does 
the VMM know what is about to be written, so we could just pass that to 
the monitoring process?)  Finally, where in Xen's code would we have to 
go to modify the fault-handling behavior so the callback could be made?

Hope I'm clear, and hope you can help a newbie!

Thanks,
Scott

Re: Logging External Page Writes for a Given Domain

[Anthony Liguori]

Scott Baker wrote:
> Hello, all:
>
> My team's goal is to be able to log all writes that are made to any 
> memory page of a certain domain, except those writes that the domain 
> itself makes.  That is to say, if Domain 2 is the domain we want to 
> log page writes for (where logging is capturing what was written and 
> its location), then we want to be able to log all the writes made by 
> any domain /except/ Domain 2 -- i.e., writes made to shared pages that 
> belong to Domain 2.
The details here are going to make a big difference.  Do you want to 
know the content of every write?

You'll have to modify Xen.  You can probably reuse some of the shadow 
paging code to track the dirty mappings of foreign pages for a domain.  
However, this won't track the contents of the write.

If you want to do that, you're going to have to implement a large amount 
of emulation to track what data gets written to the page so you can 
emulate the writes completely.  If this is an important requirement, you 
may wish to try to user an emulator (Bochs or qemu).

Regards,

Anthony Liguori
>
> Ideally, we would like to be able to have these writes for domain /x/ 
> detected and trap to a process running on Domain 0, with minimal VMM 
> modification.  Preferably, the method used would only cause 
> significant overhead when a write is made from outside domain /x/ 
> (rather than for every write /x/ and everyone else makes), but I'm not 
> picky about efficiency at the moment.
>
> The rough idea we have at this point is to make the monitor process on 
> Domain 0 mark all the pages of Domain /x/ as read-only (while 
> remembering which are actually read-only).  Then, when a write-fault 
> occurred, the VMM would pass it on to the monitoring process, which 
> would then let Domain /x/ finish the write, the monitor would record 
> what was written, and let everything continue as normal.
>
> Unfortunately, I'm not sure how that vague sketch fits into Xen.  For 
> the page table read-only flag setting, would we use the 
> update_va_mapping() hypercall?  And, how would the monitoring process 
> let Domain /x/ finish the write and then get control back?  (Or, does 
> the VMM know what is about to be written, so we could just pass that 
> to the monitoring process?)  Finally, where in Xen's code would we 
> have to go to modify the fault-handling behavior so the callback could 
> be made?
>
> Hope I'm clear, and hope you can help a newbie!
>
> Thanks,
> Scott
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Logging External Page Writes for a Given Domain

[Scott Baker]

Anthony,

Thanks for your response!  I've been thinking more about exactly what 
the requirements are, and it amounts to the following.

The end goal is to know the net result of any foreign-domain page writes 
to all of a domain's (call it Domain P) pages.  I don't actually need to 
know the contents of every individual foreign write to Domain P's pages 
per se, just which the contents of the pages that were modified by 
foreign domains since Domain P was last run.

Let me give an example to illustrate.  Domain P is running.  It is then 
suspended to allow another domain, Domain X, to start execution.  Domain 
X happens to have a few pages of Domain P mapped into its address 
space.  (This may be due to a device driver's page mapping from Domain 
0, or just another Domain mapping -- I don't care why the pages are 
mapped).  Now, during its time slice, Domain X modifies (over a period 
of several writes) two of Domain P's pages: page 19 and 94.  Then, 
Domain X is suspended and Domain Y is run.  Domain Y also modifies 
Domain P's page 94.  Finally, Domain Y is suspended so Domain P can 
resume.  It is at this point, *just before Domain P resumes*, that I 
would like to be able to have a list of the pages (if any) that have 
been written by a foreign domain (94 and 19, in this example).  With 
that list, I would like to log the *entire contents* of those modified 
pages of Domain P for my later use -- I don't need to know who wrote 
what when, just which pages were modified so I can log their new contents.

So, in essence, *just before* every time that Domain P is allowed to 
resume, I want to have a list of pages that have been modified *since 
Domain P finished its previous time slice*, for the purpose of logging 
the contents of each of those changed pages.

Having said all that, what is the most straightforward way to implement 
this (it needn't be the most efficient, though gross inefficiency would 
be nice to avoid)?

I really appreciate the feedback I've received.
Thanks again,

Scott

Anthony Liguori wrote:
> Scott Baker wrote:
>> Hello, all:
>>
>> My team's goal is to be able to log all writes that are made to any 
>> memory page of a certain domain, except those writes that the domain 
>> itself makes.  That is to say, if Domain 2 is the domain we want to 
>> log page writes for (where logging is capturing what was written and 
>> its location), then we want to be able to log all the writes made by 
>> any domain /except/ Domain 2 -- i.e., writes made to shared pages 
>> that belong to Domain 2.
> The details here are going to make a big difference.  Do you want to 
> know the content of every write?
>
> You'll have to modify Xen.  You can probably reuse some of the shadow 
> paging code to track the dirty mappings of foreign pages for a 
> domain.  However, this won't track the contents of the write.
>
> If you want to do that, you're going to have to implement a large 
> amount of emulation to track what data gets written to the page so you 
> can emulate the writes completely.  If this is an important 
> requirement, you may wish to try to user an emulator (Bochs or qemu).
>
> Regards,
>
> Anthony Liguori
>>
>> Ideally, we would like to be able to have these writes for domain /x/ 
>> detected and trap to a process running on Domain 0, with minimal VMM 
>> modification.  Preferably, the method used would only cause 
>> significant overhead when a write is made from outside domain /x/ 
>> (rather than for every write /x/ and everyone else makes), but I'm 
>> not picky about efficiency at the moment.
>>
>> The rough idea we have at this point is to make the monitor process 
>> on Domain 0 mark all the pages of Domain /x/ as read-only (while 
>> remembering which are actually read-only).  Then, when a write-fault 
>> occurred, the VMM would pass it on to the monitoring process, which 
>> would then let Domain /x/ finish the write, the monitor would record 
>> what was written, and let everything continue as normal.
>>
>> Unfortunately, I'm not sure how that vague sketch fits into Xen.  For 
>> the page table read-only flag setting, would we use the 
>> update_va_mapping() hypercall?  And, how would the monitoring process 
>> let Domain /x/ finish the write and then get control back?  (Or, does 
>> the VMM know what is about to be written, so we could just pass that 
>> to the monitoring process?)  Finally, where in Xen's code would we 
>> have to go to modify the fault-handling behavior so the callback 
>> could be made?
>>
>> Hope I'm clear, and hope you can help a newbie!
>>
>> Thanks,
>> Scott
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
>
>

Re: Logging External Page Writes for a Given Domain

[Anthony Liguori]

Scott Baker wrote:
> Anthony,
>
> Thanks for your response!  I've been thinking more about exactly what 
> the requirements are, and it amounts to the following.
>
> The end goal is to know the net result of any foreign-domain page 
> writes to all of a domain's (call it Domain P) pages.  I don't 
> actually need to know the contents of every individual foreign write 
> to Domain P's pages per se, just which the contents of the pages that 
> were modified by foreign domains since Domain P was last run.
>
> Let me give an example to illustrate.  Domain P is running.  It is 
> then suspended to allow another domain, Domain X, to start execution.  
> Domain X happens to have a few pages of Domain P mapped into its 
> address space.  (This may be due to a device driver's page mapping 
> from Domain 0, or just another Domain mapping -- I don't care why the 
> pages are mapped).  Now, during its time slice, Domain X modifies 
> (over a period of several writes) two of Domain P's pages: page 19 and 
> 94.  Then, Domain X is suspended and Domain Y is run.  Domain Y also 
> modifies Domain P's page 94.  Finally, Domain Y is suspended so Domain 
> P can resume.  It is at this point, *just before Domain P resumes*, 
> that I would like to be able to have a list of the pages (if any) that 
> have been written by a foreign domain (94 and 19, in this example).  
> With that list, I would like to log the *entire contents* of those 
> modified pages of Domain P for my later use -- I don't need to know 
> who wrote what when, just which pages were modified so I can log their 
> new contents.
Are you ignoring the SMP case (where two domains may be executing 
simultaneously)?  If so, I think the most direct approach would be to 
enable the dirty page tracking in shadow paging mode, and then dump any 
share pages before world switching that are dirty.  It still requires 
hypervisor modification, but it's reasonably straight forward.

Regards,

Anthony Liguori
> So, in essence, *just before* every time that Domain P is allowed to 
> resume, I want to have a list of pages that have been modified *since 
> Domain P finished its previous time slice*, for the purpose of logging 
> the contents of each of those changed pages.
>
> Having said all that, what is the most straightforward way to 
> implement this (it needn't be the most efficient, though gross 
> inefficiency would be nice to avoid)?
>
> I really appreciate the feedback I've received.
> Thanks again,
>
> Scott
>
> Anthony Liguori wrote:
>> Scott Baker wrote:
>>> Hello, all:
>>>
>>> My team's goal is to be able to log all writes that are made to any 
>>> memory page of a certain domain, except those writes that the domain 
>>> itself makes.  That is to say, if Domain 2 is the domain we want to 
>>> log page writes for (where logging is capturing what was written and 
>>> its location), then we want to be able to log all the writes made by 
>>> any domain /except/ Domain 2 -- i.e., writes made to shared pages 
>>> that belong to Domain 2.
>> The details here are going to make a big difference.  Do you want to 
>> know the content of every write?
>>
>> You'll have to modify Xen.  You can probably reuse some of the shadow 
>> paging code to track the dirty mappings of foreign pages for a 
>> domain.  However, this won't track the contents of the write.
>>
>> If you want to do that, you're going to have to implement a large 
>> amount of emulation to track what data gets written to the page so 
>> you can emulate the writes completely.  If this is an important 
>> requirement, you may wish to try to user an emulator (Bochs or qemu).
>>
>> Regards,
>>
>> Anthony Liguori
>>>
>>> Ideally, we would like to be able to have these writes for domain 
>>> /x/ detected and trap to a process running on Domain 0, with minimal 
>>> VMM modification.  Preferably, the method used would only cause 
>>> significant overhead when a write is made from outside domain /x/ 
>>> (rather than for every write /x/ and everyone else makes), but I'm 
>>> not picky about efficiency at the moment.
>>>
>>> The rough idea we have at this point is to make the monitor process 
>>> on Domain 0 mark all the pages of Domain /x/ as read-only (while 
>>> remembering which are actually read-only).  Then, when a write-fault 
>>> occurred, the VMM would pass it on to the monitoring process, which 
>>> would then let Domain /x/ finish the write, the monitor would record 
>>> what was written, and let everything continue as normal.
>>>
>>> Unfortunately, I'm not sure how that vague sketch fits into Xen.  
>>> For the page table read-only flag setting, would we use the 
>>> update_va_mapping() hypercall?  And, how would the monitoring 
>>> process let Domain /x/ finish the write and then get control back?  
>>> (Or, does the VMM know what is about to be written, so we could just 
>>> pass that to the monitoring process?)  Finally, where in Xen's code 
>>> would we have to go to modify the fault-handling behavior so the 
>>> callback could be made?
>>>
>>> Hope I'm clear, and hope you can help a newbie!
>>>
>>> Thanks,
>>> Scott
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>
>>
>>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Logging External Page Writes for a Given Domain

[Scott Baker]

Anthony,

I am indeed ignoring multiprocessor scenarios.  Thank you for your very 
timely input -- this is good news; I appreciate it.  I imagine I'll be 
back later with more questions about this, but I'll start giving this a 
shot.

--Scott Baker

Anthony Liguori wrote:
> Scott Baker wrote:
>> Anthony,
>>
>> Thanks for your response!  I've been thinking more about exactly what 
>> the requirements are, and it amounts to the following.
>>
>> The end goal is to know the net result of any foreign-domain page 
>> writes to all of a domain's (call it Domain P) pages.  I don't 
>> actually need to know the contents of every individual foreign write 
>> to Domain P's pages per se, just which the contents of the pages that 
>> were modified by foreign domains since Domain P was last run.
>>
>> Let me give an example to illustrate.  Domain P is running.  It is 
>> then suspended to allow another domain, Domain X, to start 
>> execution.  Domain X happens to have a few pages of Domain P mapped 
>> into its address space.  (This may be due to a device driver's page 
>> mapping from Domain 0, or just another Domain mapping -- I don't care 
>> why the pages are mapped).  Now, during its time slice, Domain X 
>> modifies (over a period of several writes) two of Domain P's pages: 
>> page 19 and 94.  Then, Domain X is suspended and Domain Y is run.  
>> Domain Y also modifies Domain P's page 94.  Finally, Domain Y is 
>> suspended so Domain P can resume.  It is at this point, *just before 
>> Domain P resumes*, that I would like to be able to have a list of the 
>> pages (if any) that have been written by a foreign domain (94 and 19, 
>> in this example).  With that list, I would like to log the *entire 
>> contents* of those modified pages of Domain P for my later use -- I 
>> don't need to know who wrote what when, just which pages were 
>> modified so I can log their new contents.
> Are you ignoring the SMP case (where two domains may be executing 
> simultaneously)?  If so, I think the most direct approach would be to 
> enable the dirty page tracking in shadow paging mode, and then dump 
> any share pages before world switching that are dirty.  It still 
> requires hypervisor modification, but it's reasonably straight forward.
>
> Regards,
>
> Anthony Liguori
>> So, in essence, *just before* every time that Domain P is allowed to 
>> resume, I want to have a list of pages that have been modified *since 
>> Domain P finished its previous time slice*, for the purpose of 
>> logging the contents of each of those changed pages.
>>
>> Having said all that, what is the most straightforward way to 
>> implement this (it needn't be the most efficient, though gross 
>> inefficiency would be nice to avoid)?
>>
>> I really appreciate the feedback I've received.
>> Thanks again,
>>
>> Scott
>>
>> Anthony Liguori wrote:
>>> Scott Baker wrote:
>>>> Hello, all:
>>>>
>>>> My team's goal is to be able to log all writes that are made to any 
>>>> memory page of a certain domain, except those writes that the 
>>>> domain itself makes.  That is to say, if Domain 2 is the domain we 
>>>> want to log page writes for (where logging is capturing what was 
>>>> written and its location), then we want to be able to log all the 
>>>> writes made by any domain /except/ Domain 2 -- i.e., writes made to 
>>>> shared pages that belong to Domain 2.
>>> The details here are going to make a big difference.  Do you want to 
>>> know the content of every write?
>>>
>>> You'll have to modify Xen.  You can probably reuse some of the 
>>> shadow paging code to track the dirty mappings of foreign pages for 
>>> a domain.  However, this won't track the contents of the write.
>>>
>>> If you want to do that, you're going to have to implement a large 
>>> amount of emulation to track what data gets written to the page so 
>>> you can emulate the writes completely.  If this is an important 
>>> requirement, you may wish to try to user an emulator (Bochs or qemu).
>>>
>>> Regards,
>>>
>>> Anthony Liguori
>>>>
>>>> Ideally, we would like to be able to have these writes for domain 
>>>> /x/ detected and trap to a process running on Domain 0, with 
>>>> minimal VMM modification.  Preferably, the method used would only 
>>>> cause significant overhead when a write is made from outside domain 
>>>> /x/ (rather than for every write /x/ and everyone else makes), but 
>>>> I'm not picky about efficiency at the moment.
>>>>
>>>> The rough idea we have at this point is to make the monitor process 
>>>> on Domain 0 mark all the pages of Domain /x/ as read-only (while 
>>>> remembering which are actually read-only).  Then, when a 
>>>> write-fault occurred, the VMM would pass it on to the monitoring 
>>>> process, which would then let Domain /x/ finish the write, the 
>>>> monitor would record what was written, and let everything continue 
>>>> as normal.
>>>>
>>>> Unfortunately, I'm not sure how that vague sketch fits into Xen.  
>>>> For the page table read-only flag setting, would we use the 
>>>> update_va_mapping() hypercall?  And, how would the monitoring 
>>>> process let Domain /x/ finish the write and then get control back?  
>>>> (Or, does the VMM know what is about to be written, so we could 
>>>> just pass that to the monitoring process?)  Finally, where in Xen's 
>>>> code would we have to go to modify the fault-handling behavior so 
>>>> the callback could be made?
>>>>
>>>> Hope I'm clear, and hope you can help a newbie!
>>>>
>>>> Thanks,
>>>> Scott
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>>>
>>>
>>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
>
>

Re: Logging External Page Writes for a Given Domain

[Scott Baker]

Hello, again!

I'm back, with another question or two regarding logging external page 
writes.  I was hoping that someone could point me to the proper source 
files where I might make the following modifications:

1.   Enable dirty page tracking (in shadow paging mode), so I can know 
which domain pages were modified.
2.   Reset dirty pages /right after/ the domain being logged is stopped 
(before any other domain runs), so that I know which pages are dirtied 
after the domain stops running.
3.   Dump (record) the contents of all dirty pages /right before/ the 
domain being logged is about to start again.

I've been poking around in the code base, but it's a little 
overwhelming.  Another pointer in the right direction (e.g., appropriate 
source files/locations, etc.) would be *very* helpful.  Also, a pointer 
to any general documentation on how Xen's shadow paging and dirty page 
tracking work would be helpful as well.

Oh, one last thing -- could these changes be made successfully on 
versions of Xen >= 2.0; if not, what's the minimum/maximum version?

Thank you all,
Scott

P.S.  The below is included to ensure the context of the request is 
understood.

Anthony Liguori wrote:
> Scott Baker wrote:
>> Anthony,
>>
>> Thanks for your response!  I've been thinking more about exactly what 
>> the requirements are, and it amounts to the following.
>>
>> The end goal is to know the net result of any foreign-domain page 
>> writes to all of a domain's (call it Domain P) pages.  I don't 
>> actually need to know the contents of every individual foreign write 
>> to Domain P's pages per se, just which the contents of the pages that 
>> were modified by foreign domains since Domain P was last run.
>>
>> Let me give an example to illustrate.  Domain P is running.  It is 
>> then suspended to allow another domain, Domain X, to start 
>> execution.  Domain X happens to have a few pages of Domain P mapped 
>> into its address space.  (This may be due to a device driver's page 
>> mapping from Domain 0, or just another Domain mapping -- I don't care 
>> why the pages are mapped).  Now, during its time slice, Domain X 
>> modifies (over a period of several writes) two of Domain P's pages: 
>> page 19 and 94.  Then, Domain X is suspended and Domain Y is run.  
>> Domain Y also modifies Domain P's page 94.  Finally, Domain Y is 
>> suspended so Domain P can resume.  It is at this point, *just before 
>> Domain P resumes*, that I would like to be able to have a list of the 
>> pages (if any) that have been written by a foreign domain (94 and 19, 
>> in this example).  With that list, I would like to log the *entire 
>> contents* of those modified pages of Domain P for my later use -- I 
>> don't need to know who wrote what when, just which pages were 
>> modified so I can log their new contents.
> Are you ignoring the SMP case (where two domains may be executing 
> simultaneously)?  If so, I think the most direct approach would be to 
> enable the dirty page tracking in shadow paging mode, and then dump 
> any share pages before world switching that are dirty.  It still 
> requires hypervisor modification, but it's reasonably straight forward.
>
> Regards,
>
> Anthony Liguori
>> So, in essence, *just before* every time that Domain P is allowed to 
>> resume, I want to have a list of pages that have been modified *since 
>> Domain P finished its previous time slice*, for the purpose of 
>> logging the contents of each of those changed pages.
>>
>> Having said all that, what is the most straightforward way to 
>> implement this (it needn't be the most efficient, though gross 
>> inefficiency would be nice to avoid)?
>>
>> I really appreciate the feedback I've received.
>> Thanks again,
>>
>> Scott
>>
>> Anthony Liguori wrote:
>>> Scott Baker wrote:
>>>> Hello, all:
>>>>
>>>> My team's goal is to be able to log all writes that are made to any 
>>>> memory page of a certain domain, except those writes that the 
>>>> domain itself makes.  That is to say, if Domain 2 is the domain we 
>>>> want to log page writes for (where logging is capturing what was 
>>>> written and its location), then we want to be able to log all the 
>>>> writes made by any domain /except/ Domain 2 -- i.e., writes made to 
>>>> shared pages that belong to Domain 2.
>>> The details here are going to make a big difference.  Do you want to 
>>> know the content of every write?
>>>
>>> You'll have to modify Xen.  You can probably reuse some of the 
>>> shadow paging code to track the dirty mappings of foreign pages for 
>>> a domain.  However, this won't track the contents of the write.
>>>
>>> If you want to do that, you're going to have to implement a large 
>>> amount of emulation to track what data gets written to the page so 
>>> you can emulate the writes completely.  If this is an important 
>>> requirement, you may wish to try to user an emulator (Bochs or qemu).
>>>
>>> Regards,
>>>
>>> Anthony Liguori
>>>>
>>>> Ideally, we would like to be able to have these writes for domain 
>>>> /x/ detected and trap to a process running on Domain 0, with 
>>>> minimal VMM modification.  Preferably, the method used would only 
>>>> cause significant overhead when a write is made from outside domain 
>>>> /x/ (rather than for every write /x/ and everyone else makes), but 
>>>> I'm not picky about efficiency at the moment.
>>>>
>>>> The rough idea we have at this point is to make the monitor process 
>>>> on Domain 0 mark all the pages of Domain /x/ as read-only (while 
>>>> remembering which are actually read-only).  Then, when a 
>>>> write-fault occurred, the VMM would pass it on to the monitoring 
>>>> process, which would then let Domain /x/ finish the write, the 
>>>> monitor would record what was written, and let everything continue 
>>>> as normal.
>>>>
>>>> Unfortunately, I'm not sure how that vague sketch fits into Xen.  
>>>> For the page table read-only flag setting, would we use the 
>>>> update_va_mapping() hypercall?  And, how would the monitoring 
>>>> process let Domain /x/ finish the write and then get control back?  
>>>> (Or, does the VMM know what is about to be written, so we could 
>>>> just pass that to the monitoring process?)  Finally, where in Xen's 
>>>> code would we have to go to modify the fault-handling behavior so 
>>>> the callback could be made?
>>>>
>>>> Hope I'm clear, and hope you can help a newbie!
>>>>
>>>> Thanks,
>>>> Scott
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>>>
>>>
>>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
>
>

Re: Logging External Page Writes for a Given Domain

[Anthony Liguori]

Take a look at tools/libxc/xc_linux_save.c.  It does most of what you're 
looking for but in a different context (live migration of a domain).  
There is a paper, IIRC, that describes how migration in Xen works too 
that would be a good starter for understanding what that code's doing.

Regards,

Anthony Liguori

Scott Baker wrote:
> Hello, again!
>
> I'm back, with another question or two regarding logging external page 
> writes.  I was hoping that someone could point me to the proper source 
> files where I might make the following modifications:
>
> 1.   Enable dirty page tracking (in shadow paging mode), so I can know 
> which domain pages were modified.
> 2.   Reset dirty pages /right after/ the domain being logged is 
> stopped (before any other domain runs), so that I know which pages are 
> dirtied after the domain stops running.
> 3.   Dump (record) the contents of all dirty pages /right before/ the 
> domain being logged is about to start again.
>
> I've been poking around in the code base, but it's a little 
> overwhelming.  Another pointer in the right direction (e.g., 
> appropriate source files/locations, etc.) would be *very* helpful.  
> Also, a pointer to any general documentation on how Xen's shadow 
> paging and dirty page tracking work would be helpful as well.
>
> Oh, one last thing -- could these changes be made successfully on 
> versions of Xen >= 2.0; if not, what's the minimum/maximum version?
>
> Thank you all,
> Scott
>
> P.S.  The below is included to ensure the context of the request is 
> understood.
>
> Anthony Liguori wrote:
>> Scott Baker wrote:
>>> Anthony,
>>>
>>> Thanks for your response!  I've been thinking more about exactly 
>>> what the requirements are, and it amounts to the following.
>>>
>>> The end goal is to know the net result of any foreign-domain page 
>>> writes to all of a domain's (call it Domain P) pages.  I don't 
>>> actually need to know the contents of every individual foreign write 
>>> to Domain P's pages per se, just which the contents of the pages 
>>> that were modified by foreign domains since Domain P was last run.
>>>
>>> Let me give an example to illustrate.  Domain P is running.  It is 
>>> then suspended to allow another domain, Domain X, to start 
>>> execution.  Domain X happens to have a few pages of Domain P mapped 
>>> into its address space.  (This may be due to a device driver's page 
>>> mapping from Domain 0, or just another Domain mapping -- I don't 
>>> care why the pages are mapped).  Now, during its time slice, Domain 
>>> X modifies (over a period of several writes) two of Domain P's 
>>> pages: page 19 and 94.  Then, Domain X is suspended and Domain Y is 
>>> run.  Domain Y also modifies Domain P's page 94.  Finally, Domain Y 
>>> is suspended so Domain P can resume.  It is at this point, *just 
>>> before Domain P resumes*, that I would like to be able to have a 
>>> list of the pages (if any) that have been written by a foreign 
>>> domain (94 and 19, in this example).  With that list, I would like 
>>> to log the *entire contents* of those modified pages of Domain P for 
>>> my later use -- I don't need to know who wrote what when, just which 
>>> pages were modified so I can log their new contents.
>> Are you ignoring the SMP case (where two domains may be executing 
>> simultaneously)?  If so, I think the most direct approach would be to 
>> enable the dirty page tracking in shadow paging mode, and then dump 
>> any share pages before world switching that are dirty.  It still 
>> requires hypervisor modification, but it's reasonably straight forward.
>>
>> Regards,
>>
>> Anthony Liguori
>>> So, in essence, *just before* every time that Domain P is allowed to 
>>> resume, I want to have a list of pages that have been modified 
>>> *since Domain P finished its previous time slice*, for the purpose 
>>> of logging the contents of each of those changed pages.
>>>
>>> Having said all that, what is the most straightforward way to 
>>> implement this (it needn't be the most efficient, though gross 
>>> inefficiency would be nice to avoid)?
>>>
>>> I really appreciate the feedback I've received.
>>> Thanks again,
>>>
>>> Scott
>>>
>>> Anthony Liguori wrote:
>>>> Scott Baker wrote:
>>>>> Hello, all:
>>>>>
>>>>> My team's goal is to be able to log all writes that are made to 
>>>>> any memory page of a certain domain, except those writes that the 
>>>>> domain itself makes.  That is to say, if Domain 2 is the domain we 
>>>>> want to log page writes for (where logging is capturing what was 
>>>>> written and its location), then we want to be able to log all the 
>>>>> writes made by any domain /except/ Domain 2 -- i.e., writes made 
>>>>> to shared pages that belong to Domain 2.
>>>> The details here are going to make a big difference.  Do you want 
>>>> to know the content of every write?
>>>>
>>>> You'll have to modify Xen.  You can probably reuse some of the 
>>>> shadow paging code to track the dirty mappings of foreign pages for 
>>>> a domain.  However, this won't track the contents of the write.
>>>>
>>>> If you want to do that, you're going to have to implement a large 
>>>> amount of emulation to track what data gets written to the page so 
>>>> you can emulate the writes completely.  If this is an important 
>>>> requirement, you may wish to try to user an emulator (Bochs or qemu).
>>>>
>>>> Regards,
>>>>
>>>> Anthony Liguori
>>>>>
>>>>> Ideally, we would like to be able to have these writes for domain 
>>>>> /x/ detected and trap to a process running on Domain 0, with 
>>>>> minimal VMM modification.  Preferably, the method used would only 
>>>>> cause significant overhead when a write is made from outside 
>>>>> domain /x/ (rather than for every write /x/ and everyone else 
>>>>> makes), but I'm not picky about efficiency at the moment.
>>>>>
>>>>> The rough idea we have at this point is to make the monitor 
>>>>> process on Domain 0 mark all the pages of Domain /x/ as read-only 
>>>>> (while remembering which are actually read-only).  Then, when a 
>>>>> write-fault occurred, the VMM would pass it on to the monitoring 
>>>>> process, which would then let Domain /x/ finish the write, the 
>>>>> monitor would record what was written, and let everything continue 
>>>>> as normal.
>>>>>
>>>>> Unfortunately, I'm not sure how that vague sketch fits into Xen.  
>>>>> For the page table read-only flag setting, would we use the 
>>>>> update_va_mapping() hypercall?  And, how would the monitoring 
>>>>> process let Domain /x/ finish the write and then get control 
>>>>> back?  (Or, does the VMM know what is about to be written, so we 
>>>>> could just pass that to the monitoring process?)  Finally, where 
>>>>> in Xen's code would we have to go to modify the fault-handling 
>>>>> behavior so the callback could be made?
>>>>>
>>>>> Hope I'm clear, and hope you can help a newbie!
>>>>>
>>>>> Thanks,
>>>>> Scott
>>>>>
>>>>> _______________________________________________
>>>>> Xen-devel mailing list
>>>>> Xen-devel@lists.xensource.com
>>>>> http://lists.xensource.com/xen-devel
>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>
>>
>>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel