Memory Sharing

Memory Sharing

[hanji unit]

Hello, does Xen support sharing memory pages between multiple domains
(such as as Dom0, DomU1, DomU2)? The Grant Table hypercalls seem
limited to:

IOCTL_GNTALLOC_ALLOC_GREF
IOCTL_GNTALLOC_DEALLOC_GREF
IOCTL_GNTALLOC_SET_UNMAP_NOTIFY

Is there any mechanism to not allocate, but only share existing pages?

Re: Memory Sharing

[Tamas K Lengyel]

[-- Attachment #1.1: Type: text/plain, Size: 892 bytes --]

On Thu, Feb 4, 2016 at 2:04 PM, hanji unit <hanjiunit@gmail.com> wrote:

> Hello, does Xen support sharing memory pages between multiple domains
> (such as as Dom0, DomU1, DomU2)? The Grant Table hypercalls seem
> limited to:
>
> IOCTL_GNTALLOC_ALLOC_GREF
> IOCTL_GNTALLOC_DEALLOC_GREF
> IOCTL_GNTALLOC_SET_UNMAP_NOTIFY
>
> Is there any mechanism to not allocate, but only share existing pages?
>

There is, it's call mem_share in Xen, but it's not very well (khm at all)
documented. You can take a look at
http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/tests/mem-sharing/memshrtool.c;h=437c7c9c472c0eff29b60e54f3de90e878de6d34;hb=HEAD
to see an outline of the tools-side functions that control it. I also have
a simple clone routine that deduplicates (clones) two identical VMs here
https://github.com/tklengyel/drakvuf/blob/master/src/xen_helper/xen_helper.c#L197
.

Cheers,
Tamas

[-- Attachment #1.2: Type: text/html, Size: 1563 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Memory Sharing

[David Vrabel]

On 04/02/16 21:04, hanji unit wrote:
> Hello, does Xen support sharing memory pages between multiple domains
> (such as as Dom0, DomU1, DomU2)? The Grant Table hypercalls seem
> limited to:
> 
> IOCTL_GNTALLOC_ALLOC_GREF
> IOCTL_GNTALLOC_DEALLOC_GREF
> IOCTL_GNTALLOC_SET_UNMAP_NOTIFY

These are the ioctls provided by the gntalloc driver.  The grant table
mechanism itself can share any page of guest memory.

If you need to share an already allocated page of userspace memory, the
gntalloc driver would need to be extended with a new ioctl.

David

memory sharing

[sepanta s]

[-- Attachment #1.1: Type: text/plain, Size: 85 bytes --]

hi,
what is the difference between memory sharing in Xen and memory
de-duplication ?

[-- Attachment #1.2: Type: text/html, Size: 125 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Memory Sharing

[Lakshitha Harshan]

[-- Attachment #1.1: Type: text/plain, Size: 577 bytes --]

Hi,

I'm doing a research project related to memory sharing. In my project I'm
trying to share the memory pages in kernel code segments. These kernels are
identical.
I have carried out some experiments to analyze these memory pages. Results
concluded that no modifications occurs on these memory pages at runtime.

But after sharing these pages using nominate and share functions, 5 memory
pages calls unshare() operation just after sharing the code pages.

So my question is, if no modifications occurs at runtime, how could be
possible to call "unshare()"?

Thanks,

Harshan

[-- Attachment #1.2: Type: text/html, Size: 731 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Memory Sharing

[Tim Deegan]

Hi, 

At 13:03 +0530 on 26 Jul (1311685428), Lakshitha Harshan wrote:
> I'm doing a research project related to memory sharing. In my project I'm
> trying to share the memory pages in kernel code segments. These kernels are
> identical.
> I have carried out some experiments to analyze these memory pages. Results
> concluded that no modifications occurs on these memory pages at runtime.
> 
> But after sharing these pages using nominate and share functions, 5 memory
> pages calls unshare() operation just after sharing the code pages.
> 
> So my question is, if no modifications occurs at runtime, how could be
> possible to call "unshare()"?

Have you tried tracing the callers of the function to find out what they
are?  The WARN() macro is handy for that - it prints a complete
backtrace of the hypervisor call chain. 

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Memory Sharing

[Lakshitha Harshan]

Hi,er

I am playing with memory sharing functions (nominate & share), for my
research project. In my project, i get the GFNs of starting & ending
pages of kernel code segment using xc_translate_foreign_address and
then using a loop i nominate the memory pages of kernel code segment &
share them.

Both nominate & share() functions returns 0, on most sharing requests
(almost 99%).
My problem is that after sharing the memory, all of the other domUs
except the domU which is used for getting the handles (shandles) for
sharing memory pages are frozen, if i start one VM after another and
try to share code pages after VMs have logged in. If i try to run the
same procedure on already started set of guest domains, the VMs will
be restarted (except the shandle domU).

So what could be the reason for this?
I'm using Xen -4.1.1-rc3-pre.

Thanks,
Harshan

Re: Memory Sharing

[Jui-Hao Chiang]

Hi, Harshan;

The share() in mem_sharing.c doesn't check if the content two pages
(handles) are the same. The function simply drop the 2nd one (client
handle), and that's why you got all VMs crash except the domU which is
used for getting the handles (shandles).
You have to use memcmp or some other way in share() function to make
sure they are actually the same.

Bests,
Jui-Hao

On Thu, Jun 16, 2011 at 10:15 AM, Lakshitha Harshan
<harshan.dll@gmail.com> wrote:
> Hi,er
>
> I am playing with memory sharing functions (nominate & share), for my
> research project. In my project, i get the GFNs of starting & ending
> pages of kernel code segment using xc_translate_foreign_address and
> then using a loop i nominate the memory pages of kernel code segment &
> share them.
>
> Both nominate & share() functions returns 0, on most sharing requests
> (almost 99%).
> My problem is that after sharing the memory, all of the other domUs
> except the domU which is used for getting the handles (shandles) for
> sharing memory pages are frozen, if i start one VM after another and
> try to share code pages after VMs have logged in. If i try to run the
> same procedure on already started set of guest domains, the VMs will
> be restarted (except the shandle domU).
>
> So what could be the reason for this?
> I'm using Xen -4.1.1-rc3-pre.
>
> Thanks,
> Harshan
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Memory Sharing

[Lakshitha Harshan]

[-- Attachment #1.1: Type: text/plain, Size: 95 bytes --]

Hi,

Is there a way to find out the amount of memory shared in-between domUs?

Thanks,
Harshan

[-- Attachment #1.2: Type: text/html, Size: 155 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Memory Sharing

[Tim Deegan]

At 10:34 +0530 on 14 Jun (1308047652), Lakshitha Harshan wrote:
> Hi,
> 
> Is there a way to find out the amount of memory shared in-between domUs?

XENMEM_get_sharing_freed_pages will get you the total saving from
sharing.  I don't think there's a per-VM equivalent. 

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Memory Sharing

[Lakshitha Harshan]

[-- Attachment #1.1: Type: text/plain, Size: 390 bytes --]

Hi,

This is regarding memory page sharing.
I traced the code for page_sharing_nominate. But couldn't find the exact
location
which requires HAP. So if somebody could tell me the location it would be a
great help for me.
My other questions are,
I don't understand the role HAP plays in page sharing?
Why can't we use the shadow page tables with page sharing?

Please help.

Thanks,
Harshan

[-- Attachment #1.2: Type: text/html, Size: 606 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Memory Sharing

[Tim Deegan]

At 09:15 +0530 on 01 Jun (1306919734), Lakshitha Harshan wrote:
> This is regarding memory page sharing.
> I traced the code for page_sharing_nominate. But couldn't find the exact
> location
> which requires HAP. So if somebody could tell me the location it would be a
> great help for me.

Grrr, it doesn't look like there is a test. :(  I'll add one. 

> My other questions are,
> I don't understand the role HAP plays in page sharing?
> Why can't we use the shadow page tables with page sharing?

Briefly: because nobody did the work to make it happen.  The shadow
pagetable code would need to be updated to deal with the p2m code
telling it that a page was shared (possibly just reporting a pagefault
as fixed would do).  I think it would also need a check that an in-use
top-level pagetable could never be shared or paged out, and I'm sure
there would be a few more bugs here and there.

It's possible that once Olaf and Keir have finished with the code to
suspend a vcpus in mid-action, the shadow pagetable code will just
work[tm] with memory sharing. 

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Memory Sharing

[Lakshitha Harshan]

[-- Attachment #1.1: Type: text/plain, Size: 1462 bytes --]

Hi,

I'm using Xen 4.1.1-rc1-pre on linux-xen-2.6.32.39-g057b171. I'm trying to
use memory sharing functions. In my code i call,

uint64_t shandle, chandle;
 xc_memshr_nominate_gfn(xc_handle,dom1,4200,&shandle); //dom1 & dom2 are
Debian Squeeze domUs
 xc_memshr_nominate_gfn(xc_handle,dom2,4200,&shandle); // gfn 4200 is a
memory page in kernel code segment
xc_memshr_share(xc_handle, shandle, chandle);

I use xl dmesg and following is the output.

(XEN) sh error: sh_remove_all_mappings(): can't find all mappings of mfn
42f01: c=8000000000000002 t=8400000000000001
(XEN) sh error: sh_remove_all_mappings(): can't find all mappings of mfn
42f01: c=8000000000000002 t=8400000000000001
(XEN) sh error: sh_remove_all_mappings(): can't find all mappings of mfn
9c502: c=8000000000000002 t=8400000000000001
(XEN) sh error: sh_remove_all_mappings(): can't find all mappings of mfn
42f02: c=8000000000000002 t=8400000000000001
(XEN) sh error: sh_remove_all_mappings(): can't find all mappings of mfn
42f02: c=8000000000000002 t=8400000000000001
(XEN) printk: 2695593 messages suppressed.
(XEN) mm.c:907:d1 Error getting mfn 9c625 (pfn fffffffffffffffe) from L1
entry 000000009c625021 for l1e_owner=1, pg_owner=1
(XEN) printk: 2768333 messages suppressed.

After this operations the cpu usage of domUs goes to 97-100% and I can't
shutdown them. So when i issue "xl destroy""the system reboots.

What could be the reason? Any help greatly appreciated.

Thanks,
Harshan

[-- Attachment #1.2: Type: text/html, Size: 1975 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Memory Sharing

[Tim Deegan]

At 08:17 +0100 on 24 May (1306225059), Lakshitha Harshan wrote:
> Hi,
> 
> I'm using Xen 4.1.1-rc1-pre on linux-xen-2.6.32.39-g057b171. I'm trying to use memory sharing functions. In my code i call,
> 
> uint64_t shandle, chandle;
>  xc_memshr_nominate_gfn(xc_handle,dom1,4200,&shandle); //dom1 & dom2 are Debian Squeeze domUs
>  xc_memshr_nominate_gfn(xc_handle,dom2,4200,&shandle); // gfn 4200 is a memory page in kernel code segment
> xc_memshr_share(xc_handle, shandle, chandle);
> 
> I use xl dmesg and following is the output.
> 
> (XEN) sh error: sh_remove_all_mappings(): can't find all mappings of mfn 42f01: c=8000000000000002 t=8400000000000001
> (XEN) sh error: sh_remove_all_mappings(): can't find all mappings of mfn 42f01: c=8000000000000002 t=8400000000000001
> (XEN) sh error: sh_remove_all_mappings(): can't find all mappings of mfn 9c502: c=8000000000000002 t=8400000000000001
> (XEN) sh error: sh_remove_all_mappings(): can't find all mappings of mfn 42f02: c=8000000000000002 t=8400000000000001
> (XEN) sh error: sh_remove_all_mappings(): can't find all mappings of mfn 42f02: c=8000000000000002 t=8400000000000001
> (XEN) printk: 2695593 messages suppressed.
> (XEN) mm.c:907:d1 Error getting mfn 9c625 (pfn fffffffffffffffe) from L1 entry 000000009c625021 for l1e_owner=1, pg_owner=1
> (XEN) printk: 2768333 messages suppressed.
> 
> After this operations the cpu usage of domUs goes to 97-100% and I can't shutdown them. So when i issue "xl destroy""the system reboots.
> 
> What could be the reason? Any help greatly 

Memory sharing is only supportes with HAP (that is, you must have a CPU
with NPT or EPT support, and not have "hap=0" in your VM configuration
file). 

The reboot is unfortunate; there should really be a more graceful
failure mode if memory sharing is turned on with shadow pagetables.

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Memory Sharing

[Lakshitha Harshan]

[-- Attachment #1.1: Type: text/plain, Size: 236 bytes --]

Hi all,

I'm doing some tests on memory sharing feature. Current version i'm having
is not working (Xen -4.0.2-rc3 on Debian Dom0 (linux 2.6.32.27)).

So can anybody please send me the details of working configuration?

Thanks,
Harshan

[-- Attachment #1.2: Type: text/html, Size: 344 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Memory Sharing

[Tim Deegan]

Hi, 

At 05:49 +0100 on 03 May (1304401799), Lakshitha Harshan wrote:
> I'm doing some tests on memory sharing feature. Current version i'm
> having is not working (Xen -4.0.2-rc3 on Debian Dom0 (linux
> 2.6.32.27)).

That's a very old version of Xen.  I don't use memory sharing, and maybe
someone who does can suggest a better starting point, but I would
suggest using at least the tip of 4.1-testing, and the Debian packaged
dom0 kernel.

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Memory Sharing

[Lakshitha Harshan]

[-- Attachment #1.1: Type: text/plain, Size: 627 bytes --]

Hi,

I'm trying to create a code which invokes xc_memshr_nominate_gfn() and
xc_memshr_share().

In my testing code I use,

uint64_t handle; int y;
y = xc_memshr_nominate_gfn(xc_handle,dom,1500,&handle);

It returns -1 even if I use another gfn number instead of 1500. So my
question is why does i returns -1 every time. My guest VMs are HVM (64-bit),
PAE & HAP enabled. I'm using Xen-4.0.2-rc3.

Is it  possible to use these memory sharing functions in this manner
 without any support from blktap2?

If someone could provide me some sample code on how to invoke this functions
it would be a great help to me.

Thanks,
Harshan

[-- Attachment #1.2: Type: text/html, Size: 1069 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Memory Sharing

[Lakshitha Harshan]

[-- Attachment #1.1: Type: text/plain, Size: 622 bytes --]

Hi guys,

I'm receiving following errors when trying to create a VM.

Error: Failed to create device.
    stdout:
    stderr: MEMSHR support not compiled in.
Check that target "home/images/ubuntuX.img" exists and that blktap2 driver
installed in dom0.

In configuration file this is how i specify my disks.

 disk = ['tap2:tapdisk:aio:home/images/ubuntuX.img,hda,w',
'phy:/dev/sr0,hdc:cdrom,r' ]

But if I use
 tapdisk2  -n aio:/home/images/ubuntuX.img  command it executes and returns
"/dev/xen/blktap-2/tapdev0".

What I want is to test the memory sharing feature. So can anybody tell me
how to do that?
Thanks,
Harshan

[-- Attachment #1.2: Type: text/html, Size: 897 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Memory Sharing

[Keir Fraser]

On 09/04/2011 07:37, "Lakshitha Harshan" <harshan.dll@gmail.com> wrote:

> Hi guys,
> 
> I'm receiving following errors when trying to create a VM.
> 
> Error: Failed to create device.
>     stdout: 
>     stderr: MEMSHR support not compiled in.
> Check that target "home/images/ubuntuX.img" exists and that blktap2 driver
> installed in dom0.
> 
> In configuration file this is how i specify my disks.
> 
>  disk = ['tap2:tapdisk:aio:home/images/ubuntuX.img,hda,w',
> 'phy:/dev/sr0,hdc:cdrom,r' ]
> 
> But if I use
>  tapdisk2  -n aio:/home/images/ubuntuX.img  command it executes and returns
> "/dev/xen/blktap-2/tapdev0". 
> 
> What I want is to test the memory sharing feature. So can anybody tell me how
> to do that?

You could try reverting changeset 20927:b882c558d4e4 and cross your fingers.
Looks like blktap2 devs disabled memshr extensions due to them not working
properly.

 -- Keir

> Thanks,
> Harshan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Memory sharing

[Lakshitha Harshan]

[-- Attachment #1.1: Type: text/plain, Size: 290 bytes --]

Hi,

Currently i'm going through the source codes of tools/blktap2. I need to
find-out where blktap2 calls
1. mem_share_nominate()
2. mem_sharing_share_page() functions to do the memory sharing on HVM
guests.

Can someone please provide me some guidance on blktap2 driver.

Thanks,
Harshan

[-- Attachment #1.2: Type: text/html, Size: 1211 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Memory sharing

[Keir Fraser]

On 03/04/2011 09:28, "Lakshitha Harshan" <harshan.dll@gmail.com> wrote:

> Hi,
> 
> Currently i'm going through the source codes of tools/blktap2. I need to
> find-out where blktap2 calls 
> 1. mem_share_nominate()
> 2. mem_sharing_share_page() functions to do the memory sharing on HVM guests.

Use recursive grep or similar to find uses of those functions in Xen. Find
out what DOMCTL command they are associated with, search for that in similar
way in tools/libxc directory, search for uses of the libxc interface
function elsewhere in tools/, and so on.

You'll find mem_sharing_share_page will take you from Xen to tools/libxc to
tools/memshr to tools/blktap2.

find . -name '*.c' | xargs grep -n <word_to_search_for>

 -- Keir

> Can someone please provide me some guidance on blktap2 driver. 
> 
> Thanks,
> Harshan 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel