From mboxrd@z Thu Jan 1 00:00:00 1970 From: Anthony Liguori Subject: Re: Logging External Page Writes for a Given Domain Date: Mon, 27 Feb 2006 18:17:45 -0600 Message-ID: <440396A9.9090309@us.ibm.com> References: <43F81EEB.40108@Juno.com> <43F957B0.4090803@us.ibm.com> <43FB7475.5050208@Juno.com> <43FB753E.20802@us.ibm.com> <440394A9.2070004@Juno.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <440394A9.2070004@Juno.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Scott Baker Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org Take a look at tools/libxc/xc_linux_save.c. It does most of what you're looking for but in a different context (live migration of a domain). There is a paper, IIRC, that describes how migration in Xen works too that would be a good starter for understanding what that code's doing. Regards, Anthony Liguori Scott Baker wrote: > Hello, again! > > I'm back, with another question or two regarding logging external page > writes. I was hoping that someone could point me to the proper source > files where I might make the following modifications: > > 1. Enable dirty page tracking (in shadow paging mode), so I can know > which domain pages were modified. > 2. Reset dirty pages /right after/ the domain being logged is > stopped (before any other domain runs), so that I know which pages are > dirtied after the domain stops running. > 3. Dump (record) the contents of all dirty pages /right before/ the > domain being logged is about to start again. > > I've been poking around in the code base, but it's a little > overwhelming. Another pointer in the right direction (e.g., > appropriate source files/locations, etc.) would be *very* helpful. > Also, a pointer to any general documentation on how Xen's shadow > paging and dirty page tracking work would be helpful as well. > > Oh, one last thing -- could these changes be made successfully on > versions of Xen >= 2.0; if not, what's the minimum/maximum version? > > Thank you all, > Scott > > P.S. The below is included to ensure the context of the request is > understood. > > Anthony Liguori wrote: >> Scott Baker wrote: >>> Anthony, >>> >>> Thanks for your response! I've been thinking more about exactly >>> what the requirements are, and it amounts to the following. >>> >>> The end goal is to know the net result of any foreign-domain page >>> writes to all of a domain's (call it Domain P) pages. I don't >>> actually need to know the contents of every individual foreign write >>> to Domain P's pages per se, just which the contents of the pages >>> that were modified by foreign domains since Domain P was last run. >>> >>> Let me give an example to illustrate. Domain P is running. It is >>> then suspended to allow another domain, Domain X, to start >>> execution. Domain X happens to have a few pages of Domain P mapped >>> into its address space. (This may be due to a device driver's page >>> mapping from Domain 0, or just another Domain mapping -- I don't >>> care why the pages are mapped). Now, during its time slice, Domain >>> X modifies (over a period of several writes) two of Domain P's >>> pages: page 19 and 94. Then, Domain X is suspended and Domain Y is >>> run. Domain Y also modifies Domain P's page 94. Finally, Domain Y >>> is suspended so Domain P can resume. It is at this point, *just >>> before Domain P resumes*, that I would like to be able to have a >>> list of the pages (if any) that have been written by a foreign >>> domain (94 and 19, in this example). With that list, I would like >>> to log the *entire contents* of those modified pages of Domain P for >>> my later use -- I don't need to know who wrote what when, just which >>> pages were modified so I can log their new contents. >> Are you ignoring the SMP case (where two domains may be executing >> simultaneously)? If so, I think the most direct approach would be to >> enable the dirty page tracking in shadow paging mode, and then dump >> any share pages before world switching that are dirty. It still >> requires hypervisor modification, but it's reasonably straight forward. >> >> Regards, >> >> Anthony Liguori >>> So, in essence, *just before* every time that Domain P is allowed to >>> resume, I want to have a list of pages that have been modified >>> *since Domain P finished its previous time slice*, for the purpose >>> of logging the contents of each of those changed pages. >>> >>> Having said all that, what is the most straightforward way to >>> implement this (it needn't be the most efficient, though gross >>> inefficiency would be nice to avoid)? >>> >>> I really appreciate the feedback I've received. >>> Thanks again, >>> >>> Scott >>> >>> Anthony Liguori wrote: >>>> Scott Baker wrote: >>>>> Hello, all: >>>>> >>>>> My team's goal is to be able to log all writes that are made to >>>>> any memory page of a certain domain, except those writes that the >>>>> domain itself makes. That is to say, if Domain 2 is the domain we >>>>> want to log page writes for (where logging is capturing what was >>>>> written and its location), then we want to be able to log all the >>>>> writes made by any domain /except/ Domain 2 -- i.e., writes made >>>>> to shared pages that belong to Domain 2. >>>> The details here are going to make a big difference. Do you want >>>> to know the content of every write? >>>> >>>> You'll have to modify Xen. You can probably reuse some of the >>>> shadow paging code to track the dirty mappings of foreign pages for >>>> a domain. However, this won't track the contents of the write. >>>> >>>> If you want to do that, you're going to have to implement a large >>>> amount of emulation to track what data gets written to the page so >>>> you can emulate the writes completely. If this is an important >>>> requirement, you may wish to try to user an emulator (Bochs or qemu). >>>> >>>> Regards, >>>> >>>> Anthony Liguori >>>>> >>>>> Ideally, we would like to be able to have these writes for domain >>>>> /x/ detected and trap to a process running on Domain 0, with >>>>> minimal VMM modification. Preferably, the method used would only >>>>> cause significant overhead when a write is made from outside >>>>> domain /x/ (rather than for every write /x/ and everyone else >>>>> makes), but I'm not picky about efficiency at the moment. >>>>> >>>>> The rough idea we have at this point is to make the monitor >>>>> process on Domain 0 mark all the pages of Domain /x/ as read-only >>>>> (while remembering which are actually read-only). Then, when a >>>>> write-fault occurred, the VMM would pass it on to the monitoring >>>>> process, which would then let Domain /x/ finish the write, the >>>>> monitor would record what was written, and let everything continue >>>>> as normal. >>>>> >>>>> Unfortunately, I'm not sure how that vague sketch fits into Xen. >>>>> For the page table read-only flag setting, would we use the >>>>> update_va_mapping() hypercall? And, how would the monitoring >>>>> process let Domain /x/ finish the write and then get control >>>>> back? (Or, does the VMM know what is about to be written, so we >>>>> could just pass that to the monitoring process?) Finally, where >>>>> in Xen's code would we have to go to modify the fault-handling >>>>> behavior so the callback could be made? >>>>> >>>>> Hope I'm clear, and hope you can help a newbie! >>>>> >>>>> Thanks, >>>>> Scott >>>>> >>>>> _______________________________________________ >>>>> Xen-devel mailing list >>>>> Xen-devel@lists.xensource.com >>>>> http://lists.xensource.com/xen-devel >>>> >>>> >>>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >> >> >> > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel