save & restore failed when tmem enabled in Xen 4.1 & Xen 4.3

save & restore failed when tmem enabled in Xen 4.1 & Xen 4.3

[yunfang tai]

[-- Attachment #1.1: Type: text/plain, Size: 942 bytes --]

Hi all,
    Recently, I am testing the TMEM support on Xen. I discovered that when
enabled TMEM in ubuntu 14.10 as guest on Xen 4.1 & Xen 4.3, "xm save" & "xm
restore“ failed after there are more than 1000 pages put in persistent pool
of TMEM in Xen. My operations are list as follows:

In ubuntu guest (8 cores , 8GB):
sudo modprobe tmem
(than wait for the selfballoon to finish)
dd if=/dev/zero of=/tmp/test.img bs=10M count=1000
dd if=/tmp/test.img of=/dev/null bs=10M
dd if=/tmp/test.img of=/dev/null bs=10M
.........
(until more than 1000 pages put in persistent pool)

In Domain 0:
(add tmem in grub.cfg)
xm save ubuntu test.save
xm restore ubuntu test.save

When TMEM is not enabled, save & restore success after these operations.
But if TMEM is enabled, save & restore fail.

Does anyone test about save & restore when enabled TMEM in Xen?? Is there
anything I do wrong?
Thanks!!

Best Regards,
Yunfang

[-- Attachment #1.2: Type: text/html, Size: 1228 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: save & restore failed when tmem enabled in Xen 4.1 & Xen 4.3

[Konrad Rzeszutek Wilk]

On Thu, Jun 04, 2015 at 10:27:06PM +0800, yunfang tai wrote:
> Hi all,

Hey!
>     Recently, I am testing the TMEM support on Xen. I discovered that when
> enabled TMEM in ubuntu 14.10 as guest on Xen 4.1 & Xen 4.3, "xm save" & "xm
> restore“ failed after there are more than 1000 pages put in persistent pool
> of TMEM in Xen. My operations are list as follows:

Is it exactly 1000 or just about? I presume it does not matter how much but
that you discovered it by having 1000 of them?

> 
> In ubuntu guest (8 cores , 8GB):
> sudo modprobe tmem
> (than wait for the selfballoon to finish)
> dd if=/dev/zero of=/tmp/test.img bs=10M count=1000
> dd if=/tmp/test.img of=/dev/null bs=10M
> dd if=/tmp/test.img of=/dev/null bs=10M
> .........
> (until more than 1000 pages put in persistent pool)

> 
> In Domain 0:
> (add tmem in grub.cfg)
> xm save ubuntu test.save
> xm restore ubuntu test.save
> 
> When TMEM is not enabled, save & restore success after these operations.
> But if TMEM is enabled, save & restore fail.

Are there any errors from the logs? Anything?
> 
> Does anyone test about save & restore when enabled TMEM in Xen?? Is there
> anything I do wrong?

Well lets see what broke. But I think Andrew discovered that the
migration protocol when it came to 'tmem' was not up to snuff. CC-ing him
just to confirm.

(Andrew, for the persistent part of this - it conceptually should
get all of the tmem memory that pushed to the hypervisor back in the
image. When you were looking at migrationv2 did you just skim through
that or mostly ignored it?)

Thanks.
> Thanks!!
> 
> Best Regards,
> Yunfang

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: save & restore failed when tmem enabled in Xen 4.1 & Xen 4.3

[Andrew Cooper]

On 05/06/15 19:45, Konrad Rzeszutek Wilk wrote:
> On Thu, Jun 04, 2015 at 10:27:06PM +0800, yunfang tai wrote:
>> Hi all,
> Hey!
>>     Recently, I am testing the TMEM support on Xen. I discovered that when
>> enabled TMEM in ubuntu 14.10 as guest on Xen 4.1 & Xen 4.3, "xm save" & "xm
>> restore“ failed after there are more than 1000 pages put in persistent pool
>> of TMEM in Xen. My operations are list as follows:
> Is it exactly 1000 or just about? I presume it does not matter how much but
> that you discovered it by having 1000 of them?
>
>> In ubuntu guest (8 cores , 8GB):
>> sudo modprobe tmem
>> (than wait for the selfballoon to finish)
>> dd if=/dev/zero of=/tmp/test.img bs=10M count=1000
>> dd if=/tmp/test.img of=/dev/null bs=10M
>> dd if=/tmp/test.img of=/dev/null bs=10M
>> .........
>> (until more than 1000 pages put in persistent pool)
>> In Domain 0:
>> (add tmem in grub.cfg)
>> xm save ubuntu test.save
>> xm restore ubuntu test.save
>>
>> When TMEM is not enabled, save & restore success after these operations.
>> But if TMEM is enabled, save & restore fail.
> Are there any errors from the logs? Anything?
>> Does anyone test about save & restore when enabled TMEM in Xen?? Is there
>> anything I do wrong?
> Well lets see what broke. But I think Andrew discovered that the
> migration protocol when it came to 'tmem' was not up to snuff. CC-ing him
> just to confirm.
>
> (Andrew, for the persistent part of this - it conceptually should
> get all of the tmem memory that pushed to the hypervisor back in the
> image. When you were looking at migrationv2 did you just skim through
> that or mostly ignored it?)

Took a look at the code, attempted to figure out what was going on, then
decided to ignore it for the time being.

As a baseline, there is no error checking of hypercalls or their
returned data putting the data into the stream.

Migration v2 currently has no TMEM support, and I would suggest
re-implementing it from scratch over attempting to port what currently
exists for legacy.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: save & restore failed when tmem enabled in Xen 4.1 & Xen 4.3

[yunfang tai]

[-- Attachment #1.1: Type: text/plain, Size: 2463 bytes --]

Hi Andrew,
    Thank you for your reply!
    I do not know much about migration V2. Was it integrated to Xen? If
integrated, from which version?
    Thank you!!

Best Regards,
Yunfang

2015-06-06 3:00 GMT+08:00 Andrew Cooper <andrew.cooper3@citrix.com>:

> On 05/06/15 19:45, Konrad Rzeszutek Wilk wrote:
> > On Thu, Jun 04, 2015 at 10:27:06PM +0800, yunfang tai wrote:
> >> Hi all,
> > Hey!
> >>     Recently, I am testing the TMEM support on Xen. I discovered that
> when
> >> enabled TMEM in ubuntu 14.10 as guest on Xen 4.1 & Xen 4.3, "xm save" &
> "xm
> >> restore“ failed after there are more than 1000 pages put in persistent
> pool
> >> of TMEM in Xen. My operations are list as follows:
> > Is it exactly 1000 or just about? I presume it does not matter how much
> but
> > that you discovered it by having 1000 of them?
> >
> >> In ubuntu guest (8 cores , 8GB):
> >> sudo modprobe tmem
> >> (than wait for the selfballoon to finish)
> >> dd if=/dev/zero of=/tmp/test.img bs=10M count=1000
> >> dd if=/tmp/test.img of=/dev/null bs=10M
> >> dd if=/tmp/test.img of=/dev/null bs=10M
> >> .........
> >> (until more than 1000 pages put in persistent pool)
> >> In Domain 0:
> >> (add tmem in grub.cfg)
> >> xm save ubuntu test.save
> >> xm restore ubuntu test.save
> >>
> >> When TMEM is not enabled, save & restore success after these operations.
> >> But if TMEM is enabled, save & restore fail.
> > Are there any errors from the logs? Anything?
> >> Does anyone test about save & restore when enabled TMEM in Xen?? Is
> there
> >> anything I do wrong?
> > Well lets see what broke. But I think Andrew discovered that the
> > migration protocol when it came to 'tmem' was not up to snuff. CC-ing him
> > just to confirm.
> >
> > (Andrew, for the persistent part of this - it conceptually should
> > get all of the tmem memory that pushed to the hypervisor back in the
> > image. When you were looking at migrationv2 did you just skim through
> > that or mostly ignored it?)
>
> Took a look at the code, attempted to figure out what was going on, then
> decided to ignore it for the time being.
>
> As a baseline, there is no error checking of hypercalls or their
> returned data putting the data into the stream.
>
> Migration v2 currently has no TMEM support, and I would suggest
> re-implementing it from scratch over attempting to port what currently
> exists for legacy.
>
> ~Andrew
>

[-- Attachment #1.2: Type: text/html, Size: 3233 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: save & restore failed when tmem enabled in Xen 4.1 & Xen 4.3

[Yang Hongyang]

On 06/08/2015 02:30 PM, yunfang tai wrote:
> Hi Andrew,
>      Thank you for your reply!
>      I do not know much about migration V2. Was it integrated to Xen? If
> integrated, from which version?

It's intended to be integrated to Xen4.6. However, the libxc part has already
been merged into upstream, but the libxl part still work in progress.

>      Thank you!!
>
> Best Regards,
> Yunfang
>
> 2015-06-06 3:00 GMT+08:00 Andrew Cooper <andrew.cooper3@citrix.com
> <mailto:andrew.cooper3@citrix.com>>:
>
>     On 05/06/15 19:45, Konrad Rzeszutek Wilk wrote:
>      > On Thu, Jun 04, 2015 at 10:27:06PM +0800, yunfang tai wrote:
>      >> Hi all,
>      > Hey!
>      >>     Recently, I am testing the TMEM support on Xen. I discovered that when
>      >> enabled TMEM in ubuntu 14.10 as guest on Xen 4.1 & Xen 4.3, "xm save" & "xm
>      >> restore“ failed after there are more than 1000 pages put in persistent pool
>      >> of TMEM in Xen. My operations are list as follows:
>      > Is it exactly 1000 or just about? I presume it does not matter how much but
>      > that you discovered it by having 1000 of them?
>      >
>      >> In ubuntu guest (8 cores , 8GB):
>      >> sudo modprobe tmem
>      >> (than wait for the selfballoon to finish)
>      >> dd if=/dev/zero of=/tmp/test.img bs=10M count=1000
>      >> dd if=/tmp/test.img of=/dev/null bs=10M
>      >> dd if=/tmp/test.img of=/dev/null bs=10M
>      >> .........
>      >> (until more than 1000 pages put in persistent pool)
>      >> In Domain 0:
>      >> (add tmem in grub.cfg)
>      >> xm save ubuntu test.save
>      >> xm restore ubuntu test.save
>      >>
>      >> When TMEM is not enabled, save & restore success after these operations.
>      >> But if TMEM is enabled, save & restore fail.
>      > Are there any errors from the logs? Anything?
>      >> Does anyone test about save & restore when enabled TMEM in Xen?? Is there
>      >> anything I do wrong?
>      > Well lets see what broke. But I think Andrew discovered that the
>      > migration protocol when it came to 'tmem' was not up to snuff. CC-ing him
>      > just to confirm.
>      >
>      > (Andrew, for the persistent part of this - it conceptually should
>      > get all of the tmem memory that pushed to the hypervisor back in the
>      > image. When you were looking at migrationv2 did you just skim through
>      > that or mostly ignored it?)
>
>     Took a look at the code, attempted to figure out what was going on, then
>     decided to ignore it for the time being.
>
>     As a baseline, there is no error checking of hypercalls or their
>     returned data putting the data into the stream.
>
>     Migration v2 currently has no TMEM support, and I would suggest
>     re-implementing it from scratch over attempting to port what currently
>     exists for legacy.
>
>     ~Andrew
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>

-- 
Thanks,
Yang.

Re: save & restore failed when tmem enabled in Xen 4.1 & Xen 4.3

[yunfang tai]

[-- Attachment #1.1: Type: text/plain, Size: 2872 bytes --]

Hi Konrad,
    Thank you for your reply!
    It does not matter whether it is 1000 or not. Most of the save &
restore operations will be failed
when there are more than 1000 pages put in the persistent pool. Some of the
operations
will be success when there are not so much pages put in the persistent pool.
    Attached file is the screenshot of the error message (no log files),
and it seems to be panic in libnss-files.
    Migration operations have the same phenomenon as save & restore
operations, and some of them are success
when there are not so much pages put in the persistent pool.

    Also, my test environment is as follows:

Xen server: Xen 4.1 (Oracle VM server release 3.2.6), Xen 4.3 (Oracle VM
server release 3.3.1)
Guest:    Ubuntu 14.10 (kernel 3.16.0)
Params:  8cores, 8G ram， 100G disk (using "file" protocol).

Best Regards,
Yunfang

2015-06-06 2:45 GMT+08:00 Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>:

> On Thu, Jun 04, 2015 at 10:27:06PM +0800, yunfang tai wrote:
> > Hi all,
>
> Hey!
> >     Recently, I am testing the TMEM support on Xen. I discovered that
> when
> > enabled TMEM in ubuntu 14.10 as guest on Xen 4.1 & Xen 4.3, "xm save" &
> "xm
> > restore“ failed after there are more than 1000 pages put in persistent
> pool
> > of TMEM in Xen. My operations are list as follows:
>
> Is it exactly 1000 or just about? I presume it does not matter how much but
> that you discovered it by having 1000 of them?
>
> >
> > In ubuntu guest (8 cores , 8GB):
> > sudo modprobe tmem
> > (than wait for the selfballoon to finish)
> > dd if=/dev/zero of=/tmp/test.img bs=10M count=1000
> > dd if=/tmp/test.img of=/dev/null bs=10M
> > dd if=/tmp/test.img of=/dev/null bs=10M
> > .........
> > (until more than 1000 pages put in persistent pool)
>
> >
> > In Domain 0:
> > (add tmem in grub.cfg)
> > xm save ubuntu test.save
> > xm restore ubuntu test.save
> >
> > When TMEM is not enabled, save & restore success after these operations.
> > But if TMEM is enabled, save & restore fail.
>
> Are there any errors from the logs? Anything?
> >
> > Does anyone test about save & restore when enabled TMEM in Xen?? Is there
> > anything I do wrong?
>
> Well lets see what broke. But I think Andrew discovered that the
> migration protocol when it came to 'tmem' was not up to snuff. CC-ing him
> just to confirm.
>
> (Andrew, for the persistent part of this - it conceptually should
> get all of the tmem memory that pushed to the hypervisor back in the
> image. When you were looking at migrationv2 did you just skim through
> that or mostly ignored it?)
>
> Thanks.
> > Thanks!!
> >
> > Best Regards,
> > Yunfang
>
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
>
>

[-- Attachment #1.2: Type: text/html, Size: 3794 bytes --]

[-- Attachment #2: tmem.png --]
[-- Type: image/png, Size: 50060 bytes --]

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel