From mboxrd@z Thu Jan 1 00:00:00 1970 From: Shriram Rajagopalan Subject: Re: Accessing Dom0 Physical memory from xen, via direct mappings (PML4:262-271) Date: Tue, 13 Mar 2012 09:32:28 -0700 Message-ID: References: <02FF9937-5F09-4CB1-93ED-A55FCC6F625C@cs.ubc.ca> <20120313160817.GA71026@ocelot.phlegethon.org> Reply-To: rshriram@cs.ubc.ca Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0531809382128773997==" Return-path: In-Reply-To: <20120313160817.GA71026@ocelot.phlegethon.org> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Tim Deegan Cc: "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org --===============0531809382128773997== Content-Type: multipart/alternative; boundary=00151747805cfdfcf104bb2269e4 --00151747805cfdfcf104bb2269e4 Content-Type: text/plain; charset=ISO-8859-1 On Tue, Mar 13, 2012 at 9:08 AM, Tim Deegan wrote: > At 08:45 -0700 on 13 Mar (1331628358), Shriram Rajagopalan wrote: > > In config.h (include/asm-x86/config.h) I found this: > > > > #if __x86_64__ > > ... > > * 0xffff830000000000 - 0xffff87ffffffffff [5TB, 5*2^40 bytes, > PML4:262-271] > > * 1:1 direct mapping of all physical memory. > > ... > > > > I was wondering if it's possible for dom0 to malloc a huge chunk of > memory > > and let xen know the starting address of this range. > > Inside xen, I can translate dom0 virt address to a virt address in the > above > > range and access the entire chunk via these virtual addresses. > > Eh, maybe? But it's harder than you'd think. Memory malloc()ed in dom0 > may not be contiguous in PFN-space, and dom0 PFNs may not be contiguous > in MFN-space, so you can't just translate the base of the buffer and use > it with offsets, you have to translate again for every 4k page. Also, you > need to make sure dom0 doesn't page out or relocate your user-space > buffer while Xen is accessing the MFNs. mlock() might do what you need > but AIUI there's no guarantee that it won't, e.g., move the buffer > around to defragment memory. > > Yep. I am aware of the above issues. As far as contiguity is concerned, I was hoping (*naively/lazily*) that if I allocate a huge chunk (1G or so) using posix_memalign, it would start at page boundary and also be contiguous *most* of the time. I need this setup only for some temporary analysis and not for a production quality system. And the machine has more than enough ram, with swap usage being 0 all the time. If you do handle all that, the correct way to get at the mappings in > this range is with map_domain_page(). But remember, this only works > like this on 64-bit Xen. On 32-bit, only a certain amount of memory can > be mapped at one time so if the buffer is really big, you'll need to map > an unmap parts of it on demand. > > 64-bit. The comments I pointed out were under the #if x86_64 region. But maybe back up a bit: why do you want to do this? What's the buffer > for? Is it something you could do more easily by having Xen allocate > the buffer and let dom0 map it? > > Well, the buffer acts as a huge log dirty "byte" map (a byte per word). I am skipping the reason for doing this huge byte map, for the sake of brevity. Can I have xen allocate this huge buffer ? (a byte per 8-byte word means about 128M for a 1G guest). And if I were to have this byte-map per-vcpu, it would mean 512M worth of RAM, for a 4-vcpu guest. Is there a way I could increase the xen heap size to be able to allocate this much memory? And how do I map the xen memory in dom0 ? I vaguely remember seeing similar code in xentrace, but if you could point me in the right direction, it would be great. > The catch here is that I want this virtual address range to be accessible > > across all vcpu contexts in xen (whether it's servicing a hypercall from > dom0 > > or a vmx fault caused by Guest). > > > > So far, I have only been able to achieve the former. In the latter case, > > where the "current" vcpu belongs to a guest (eg in a vmx fault handler), > > I can't access this address range inside xen. Do I have to add EPT > > mappings to guest's p2m to do this ? Or can I do something else ? > > If you really have got a pointer into the 1-1 mapping it should work > from any vcpu. > But again, that's not going to work on 32-bit Xen. > There, you have to use map_domain_page_global() to get a mapping that > persists across all vcpus, and that's even more limited in how much it > can map at once. > > Cheers, > > Tim. > > cheers shriram --00151747805cfdfcf104bb2269e4 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
On Tue, Mar 13, 2012 at 9:08 AM, Tim Deegan <tim@xen.= org> wrote:
At 08:45 -0700 on 13 Mar (1331628358), Shriram Rajagopalan wrote:
> In config.h (include/asm-x86/config.h) I found this:
>
> #if __x86_64__
> ...
> =A0* =A00xffff830000000000 - 0xffff87ffffffffff [5TB, 5*2^40 bytes, PM= L4:262-271]
> =A0* =A0 =A01:1 direct mapping of all physical memory.
> ...
>
> I was wondering if it's possible for dom0 to malloc a huge chunk o= f memory
> and let xen know the starting address of this range.
> Inside xen, I can translate dom0 virt address to a virt address in the= above
> range and access the entire chunk via these virtual addresses.

Eh, maybe? =A0But it's harder than you'd think. =A0Memory mal= loc()ed in dom0
may not be contiguous in PFN-space, and dom0 PFNs may not be contiguous
in MFN-space, so you can't just translate the base of the buffer and us= e
it with offsets, you have to translate again for every 4k page. =A0Also, yo= u
need to make sure dom0 doesn't page out or relocate your user-space
buffer while Xen is accessing the MFNs. =A0mlock() might do what you need but AIUI there's no guarantee that it won't, e.g., move the buffer<= br> around to defragment memory.


Yep. I am aware of the above issues. As far as co= ntiguity is concerned,
I was hoping (*naively/lazily*) that if I alloca= te a huge chunk (1G or so)
using posix_memalign, it would start at page = boundary and also be contiguous
*most* of the time.=A0 I need this setup only for some temporary analysis a= nd
not for a production quality system.
And the machine has more th= an enough ram, with swap usage being 0 all the time.
=A0

If you do handle all that, the correct way to get at the mappings in
this range is with map_domain_page(). =A0But remember, this only works
like this on 64-bit Xen. =A0On 32-bit, only a certain amount of memory can<= br> be mapped at one time so if the buffer is really big, you'll need to ma= p
an unmap parts of it on demand.


64-bit. The comments I pointed out were under the= #if x86_64 region.

But maybe back up a bit: why do you want to do this? =A0What's the buff= er
for? =A0Is it something you could do more easily by having Xen allocate
the buffer and let dom0 map it?


Well, the buffer acts as a huge log di= rty "byte" map (a byte per word).
I am skipping the reason for= doing this huge byte map, for the sake of brevity.

Can I have xen allocate this huge buffer ? (a byte per 8-byte word means ab= out
128M for a 1G guest). And if I were to have this byte-map per-vcpu, = it would mean
512M worth of RAM, for a 4-vcpu guest.

Is there a = way I could increase the xen heap size to be able to allocate this much mem= ory?
And how do I map the xen memory in dom0 ? I vaguely remember seeing similar= code in
xentrace, but if you could point me in the right direction, it = would be great.


> The catch here is that I want this virtual address range to be accessi= ble
> across all vcpu contexts in xen (whether it's servicing a hypercal= l from dom0
> or a vmx fault caused by Guest).
>
> So far, I have only been able to achieve the former. In the latter cas= e,
> where the "current" vcpu belongs to a guest (eg in a vmx fau= lt handler),
> I can't access this address range inside xen. Do I have to add EPT=
> mappings to guest's p2m to do this ? Or can I do something else ?<= br>
If you really have got a pointer into the 1-1 mapping it should work<= br> from any vcpu.=A0
=A0But again, that's not going to work on 32-bit Xen.
There, you have to use map_domain_page_global() to get a mapping that
persists across all vcpus, and that's even more limited in how much it<= br> can map at once.

Cheers,

Tim.


cheers
shriram
--00151747805cfdfcf104bb2269e4-- --===============0531809382128773997== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============0531809382128773997==--