Xen-unstable save error

Xen-unstable save error

[Michal Novotny]

Hi,
I'm trying to run Xen-unstable but I have a problem for save 
functionality, apparently some symbol is missing there:

[2010-06-21 13:02:14 4333] DEBUG (XendCheckpoint:124) [xc_save]: 
/usr/lib64/xen/bin/xc_save 56 1 0 0 4
[2010-06-21 13:02:14 4333] INFO (XendCheckpoint:408) 
/usr/lib64/xen/bin/xc_save: symbol lookup error: 
/usr/lib64/xen/bin/xc_save: undefined symbol: xs_suspend_evtchn_port
[2010-06-21 13:02:14 4333] ERROR (XendCheckpoint:178) Save failed on 
domain rhel5-32fv-stubdom (1) - resuming.
Traceback (most recent call last):
   File "usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", 
line 146, in save
     forkHelper(cmd, fd, saveInputHandler, False)
   File "usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", 
line 394, in forkHelper
     raise XendError("%s failed: popen failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_save 56 1 0 0 4 failed: popen failed
[2010-06-21 13:02:14 4333] DEBUG (XendDomainInfo:3131) 
XendDomainInfo.resumeDomain(1)

Any ideas?

Michal

-- 
Michal Novotny<minovotn@redhat.com>, RHCE
Virtualization Team (xen userspace), Red Hat

Re: Xen-unstable save error

[Keir Fraser]

On 21/06/2010 12:15, "Michal Novotny" <minovotn@redhat.com> wrote:

> Hi,
> I'm trying to run Xen-unstable but I have a problem for save
> functionality, apparently some symbol is missing there:

I don't think it's a msising symbol. Looks like popen threw an error.
Perhaps xc_save binary couldn't be found, or something like that?

 -- Keir

> [2010-06-21 13:02:14 4333] DEBUG (XendCheckpoint:124) [xc_save]:
> /usr/lib64/xen/bin/xc_save 56 1 0 0 4
> [2010-06-21 13:02:14 4333] INFO (XendCheckpoint:408)
> /usr/lib64/xen/bin/xc_save: symbol lookup error:
> /usr/lib64/xen/bin/xc_save: undefined symbol: xs_suspend_evtchn_port
> [2010-06-21 13:02:14 4333] ERROR (XendCheckpoint:178) Save failed on
> domain rhel5-32fv-stubdom (1) - resuming.
> Traceback (most recent call last):
>    File "usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
> line 146, in save
>      forkHelper(cmd, fd, saveInputHandler, False)
>    File "usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
> line 394, in forkHelper
>      raise XendError("%s failed: popen failed" % string.join(cmd))
> XendError: /usr/lib64/xen/bin/xc_save 56 1 0 0 4 failed: popen failed
> [2010-06-21 13:02:14 4333] DEBUG (XendDomainInfo:3131)
> XendDomainInfo.resumeDomain(1)
> 
> Any ideas?
> 
> Michal

Re: Re: Xen-unstable save error

[Michal Novotny]

On 06/21/2010 02:05 PM, Keir Fraser wrote:
> On 21/06/2010 12:15, "Michal Novotny"<minovotn@redhat.com>  wrote:
>
>    
>> Hi,
>> I'm trying to run Xen-unstable but I have a problem for save
>> functionality, apparently some symbol is missing there:
>>      
> I don't think it's a msising symbol. Looks like popen threw an error.
> Perhaps xc_save binary couldn't be found, or something like that?
>
>   -- Keir
>
>    

Well, Keir, it does exist:

# ls -al /usr/lib64/xen/bin/xc_save
-rwxr-xr-x 1 root root 25245 Jun 18 21:37 /usr/lib64/xen/bin/xc_save
# /usr/lib64/xen/bin/xc_save
xc_save: usage: /usr/lib64/xen/bin/xc_save iofd domid maxit maxf flags

Also, when I try to run it manually with the domain running and testing 
the save it returns:

# /usr/lib64/xen/bin/xc_save 56 1 0 0 4
/usr/lib64/xen/bin/xc_save: symbol lookup error: undefined symbol: xs_suspend_evtchn_port


When I try it to domain that is not running (or the iofd is not opened 
yet) it returns segmentation fault:

#  /usr/lib64/xen/bin/xc_save 56 1 0 0 4
Segmentation fault
# gdb -arg /usr/lib64/xen/bin/xc_save 56 1 0 0 4
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
(gdb) run
Starting program: /usr/lib64/xen/bin/xc_save 56 1 0 0 4
[Thread debugging using libthread_db enabled]
[New Thread 0x7feb772b46e0 (LWP 6411)]

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00000000004014f5 in main (argc=<value optimized out>, 
argv=0x7fff6a1f9868) at xc_save.c:192
(gdb) disassemble
No function contains program counter for selected frame.
(gdb)

line 192 of xc_save is closing the xc_interface by 
"xc_interface_close(xc_fd);".

Any ideas?

Michal

>> [2010-06-21 13:02:14 4333] DEBUG (XendCheckpoint:124) [xc_save]:
>> /usr/lib64/xen/bin/xc_save 56 1 0 0 4
>> [2010-06-21 13:02:14 4333] INFO (XendCheckpoint:408)
>> /usr/lib64/xen/bin/xc_save: symbol lookup error:
>> /usr/lib64/xen/bin/xc_save: undefined symbol: xs_suspend_evtchn_port
>> [2010-06-21 13:02:14 4333] ERROR (XendCheckpoint:178) Save failed on
>> domain rhel5-32fv-stubdom (1) - resuming.
>> Traceback (most recent call last):
>>     File "usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
>> line 146, in save
>>       forkHelper(cmd, fd, saveInputHandler, False)
>>     File "usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
>> line 394, in forkHelper
>>       raise XendError("%s failed: popen failed" % string.join(cmd))
>> XendError: /usr/lib64/xen/bin/xc_save 56 1 0 0 4 failed: popen failed
>> [2010-06-21 13:02:14 4333] DEBUG (XendDomainInfo:3131)
>> XendDomainInfo.resumeDomain(1)
>>
>> Any ideas?
>>
>> Michal
>>      
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>    


-- 
Michal Novotny<minovotn@redhat.com>, RHCE
Virtualization Team (xen userspace), Red Hat

Re: Re: Xen-unstable save error

[Keir Fraser]

On 21/06/2010 13:38, "Michal Novotny" <minovotn@redhat.com> wrote:

> Well, Keir, it does exist:
> 
> # ls -al /usr/lib64/xen/bin/xc_save
> -rwxr-xr-x 1 root root 25245 Jun 18 21:37 /usr/lib64/xen/bin/xc_save
> # /usr/lib64/xen/bin/xc_save
> xc_save: usage: /usr/lib64/xen/bin/xc_save iofd domid maxit maxf flags
> 
> Also, when I try to run it manually with the domain running and testing
> the save it returns:
> 
> # /usr/lib64/xen/bin/xc_save 56 1 0 0 4
> /usr/lib64/xen/bin/xc_save: symbol lookup error: undefined symbol:
> xs_suspend_evtchn_port

Ah, the dynamic linker is picking up an old version of libxenstore.
xs_suspend_evctnn_port was introduced in Xen 3.4.0. I missed that in your
log-file snippet.

 -- Keir

Re: Re: Xen-unstable save error

[Michal Novotny]

On 06/21/2010 02:44 PM, Keir Fraser wrote:
> On 21/06/2010 13:38, "Michal Novotny"<minovotn@redhat.com>  wrote:
>
>    
>> Well, Keir, it does exist:
>>
>> # ls -al /usr/lib64/xen/bin/xc_save
>> -rwxr-xr-x 1 root root 25245 Jun 18 21:37 /usr/lib64/xen/bin/xc_save
>> # /usr/lib64/xen/bin/xc_save
>> xc_save: usage: /usr/lib64/xen/bin/xc_save iofd domid maxit maxf flags
>>
>> Also, when I try to run it manually with the domain running and testing
>> the save it returns:
>>
>> # /usr/lib64/xen/bin/xc_save 56 1 0 0 4
>> /usr/lib64/xen/bin/xc_save: symbol lookup error: undefined symbol:
>> xs_suspend_evtchn_port
>>      
> Ah, the dynamic linker is picking up an old version of libxenstore.
> xs_suspend_evctnn_port was introduced in Xen 3.4.0. I missed that in your
> log-file snippet.
>
>   -- Keir
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>    
Oh, ok, should it be in /usr/lib then or how to fix it ?

Thanks,
Michal

-- 
Michal Novotny<minovotn@redhat.com>, RHCE
Virtualization Team (xen userspace), Red Hat

Re: Re: Xen-unstable save error

[Keir Fraser]

On 21/06/2010 13:47, "Michal Novotny" <minovotn@redhat.com> wrote:

>> Ah, the dynamic linker is picking up an old version of libxenstore.
>> xs_suspend_evctnn_port was introduced in Xen 3.4.0. I missed that in your
>> log-file snippet.

> Oh, ok, should it be in /usr/lib then or how to fix it ?

It should be where your dynamic linker will find it. :-)

 -- Keir

Re: Re: Xen-unstable save error

[Michal Novotny]

On 06/21/2010 02:44 PM, Keir Fraser wrote:
> On 21/06/2010 13:38, "Michal Novotny"<minovotn@redhat.com>  wrote:
>
>    
>> Well, Keir, it does exist:
>>
>> # ls -al /usr/lib64/xen/bin/xc_save
>> -rwxr-xr-x 1 root root 25245 Jun 18 21:37 /usr/lib64/xen/bin/xc_save
>> # /usr/lib64/xen/bin/xc_save
>> xc_save: usage: /usr/lib64/xen/bin/xc_save iofd domid maxit maxf flags
>>
>> Also, when I try to run it manually with the domain running and testing
>> the save it returns:
>>
>> # /usr/lib64/xen/bin/xc_save 56 1 0 0 4
>> /usr/lib64/xen/bin/xc_save: symbol lookup error: undefined symbol:
>> xs_suspend_evtchn_port
>>      
> Ah, the dynamic linker is picking up an old version of libxenstore.
> xs_suspend_evctnn_port was introduced in Xen 3.4.0. I missed that in your
> log-file snippet.
>
>   -- Keir
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>    
Well, those are objdumps:

$ objdump -x /xen-unstable.hg/tools/xcutils/xc_save | grep xs_sus
0000000000000000       F *UND*  00000000000000ad              
xs_suspend_evtchn_port
$ objdump -x /usr/lib64/xen/bin/xc_save | grep xs_suspend
0000000000000000       F *UND*  00000000000000ad              
xs_suspend_evtchn_port
$ ls -al /usr/lib64/libxenctrl.so
lrwxrwxrwx 1 root root 17 Jun 21 13:52 /usr/lib64/libxenctrl.so -> 
libxenctrl.so.4.0

I was having some libxenctrl* files at /lib64 so I removed them in order 
to make linked link those from /usr/lib64 so it did. However, the error 
now is:

[2010-06-21 14:52:22 6151] DEBUG (XendCheckpoint:126) [xc_save]: 
/usr/lib64/xen/bin/xc_save 57 1 0 0 4
[2010-06-21 14:52:22 6151] ERROR (XendCheckpoint:180) Save failed on 
domain rhel5-32fv-stubdom (1) - resuming.
Traceback (most recent call last):
   File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", 
line 148, in save
     forkHelper(cmd, fd, saveInputHandler, False)
   File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", 
line 398, in forkHelper
     raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_save 57 1 0 0 4 failed
[2010-06-21 14:52:22 6151] DEBUG (XendDomainInfo:3131) 
XendDomainInfo.resumeDomain(1)

and when I run xc_save manually it's returning segmentation fault:

  gdb -arg /usr/lib64/xen/bin/xc_save 57 1 0 0 4
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
(gdb) run
Starting program: /usr/lib64/xen/bin/xc_save 57 1 0 0 4
[Thread debugging using libthread_db enabled]
[New Thread 0x7f6490b9d6e0 (LWP 6785)]

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00000000004014f5 in main (argc=<value optimized out>, 
argv=0x7fffb5942ac8) at xc_save.c:192
(gdb) up
#1  0x00000000004014f5 in main (argc=<value optimized out>, 
argv=0x7fffb5942ac8) at xc_save.c:192
192             port = xs_suspend_evtchn_port(si.domid);

So I'm sorry, I was wrong about the line. This is from some other file. 
According to strace the "/usr/lib64/libxenctrl.so.4.0" (which has been 
built today) is loaded there but "objdump -x 
/usr/lib64/libxenctrl.so.4.0 | grep xs_" doesn't return anything.

Any ideas now?

Michal

-- 
Michal Novotny<minovotn@redhat.com>, RHCE
Virtualization Team (xen userspace), Red Hat

Re: Re: Xen-unstable save error

[Keir Fraser]

On 21/06/2010 13:57, "Michal Novotny" <minovotn@redhat.com> wrote:

> Well, those are objdumps:
> 
> $ objdump -x /xen-unstable.hg/tools/xcutils/xc_save | grep xs_sus
> 0000000000000000       F *UND*  00000000000000ad
> xs_suspend_evtchn_port
> $ objdump -x /usr/lib64/xen/bin/xc_save | grep xs_suspend
> 0000000000000000       F *UND*  00000000000000ad
> xs_suspend_evtchn_port
> $ ls -al /usr/lib64/libxenctrl.so
> lrwxrwxrwx 1 root root 17 Jun 21 13:52 /usr/lib64/libxenctrl.so ->
> libxenctrl.so.4.0
> 
> I was having some libxenctrl* files at /lib64 so I removed them in order
> to make linked link those from /usr/lib64 so it did. However, the error
> now is:

Well what was the above supposed to achieve? xs_suspend_evtchn_port is
provided by libxenstore. It's that library which is getting mis-linked.
Moving the xc_save binary itself, and/or libxenctrl, isn't going to change
that. Go use ldd to find what libxenstore is being linked against; work out
why it's the wrong one; put the right one in its place.

 -- Keir

Re: Re: Xen-unstable save error

[Michal Novotny]

On 06/21/2010 02:56 PM, Keir Fraser wrote:
> On 21/06/2010 13:47, "Michal Novotny"<minovotn@redhat.com>  wrote:
>
>    
>>> Ah, the dynamic linker is picking up an old version of libxenstore.
>>> xs_suspend_evctnn_port was introduced in Xen 3.4.0. I missed that in your
>>> log-file snippet.
>>>        
>    
>> Oh, ok, should it be in /usr/lib then or how to fix it ?
>>      
> It should be where your dynamic linker will find it. :-)
>
>   -- Keir
>
>
>    
One more thing, should the xenstore daemon be running or not? Isn't it 
in kernel and xenfs now ? I saw there's a file:

/usr/src/redhat/BUILD/kernel-2.6.32.15/drivers/xen/xenfs/xenstored.c

in the kernel source codes. But I'm still having /usr/sbin/xenstored. 
Should this be running or not?

Also, when I kill it there's an error:
[2010-06-21 15:08:59 6953] ERROR (SrvDaemon:349) Exception starting xend 
((111, 'Connection refused'))
Traceback (most recent call last):
   File 
"usr/lib64/python2.4/site-packages/xen/xend/server/SrvDaemon.py", line 
341, in run
     servers = SrvServer.create()
   File 
"usr/lib64/python2.4/site-packages/xen/xend/server/SrvServer.py", line 
258, in create
     root.putChild('xend', SrvRoot())
   File "usr/lib64/python2.4/site-packages/xen/xend/server/SrvRoot.py", 
line 40, in __init__
     self.get(name)
   File "usr/lib64/python2.4/site-packages/xen/web/SrvDir.py", line 84, 
in get
     val = val.getobj()
   File "usr/lib64/python2.4/site-packages/xen/web/SrvDir.py", line 52, 
in getobj
     self.obj = klassobj()
   File "usr/lib64/python2.4/site-packages/xen/xend/server/SrvNode.py", 
line 30, in __init__
     self.xn = XendNode.instance()
   File "usr/lib64/python2.4/site-packages/xen/xend/XendNode.py", line 
1176, in instance
     inst = XendNode()
   File "usr/lib64/python2.4/site-packages/xen/xend/XendNode.py", line 
163, in __init__
     self._init_cpu_pools()
   File "usr/lib64/python2.4/site-packages/xen/xend/XendNode.py", line 
377, in _init_cpu_pools
     XendCPUPool.recreate_active_pools()
   File "usr/lib64/python2.4/site-packages/xen/xend/XendCPUPool.py", 
line 754, in recreate_active_pools
     uuid = xstransact.Read(path, 'uuid')
   File 
"usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", 
line 307, in Read
     return complete(path, lambda t: t.read(*args))
   File 
"usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", 
line 361, in complete
     t = xstransact(path)
   File 
"usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", 
line 29, in __init__
     self.transaction = xshandle().transaction_start()
   File "usr/lib64/python2.4/site-packages/xen/xend/xenstore/xsutil.py", 
line 18, in xshandle
     xs_handle = xen.lowlevel.xs.xs()
Error: (111, 'Connection refused')

Did xenstore location change or something?

# ls -al /usr/sbin/xenstored
-rwxr-xr-x 1 root root 257580 Jun 18 21:37 /usr/sbin/xenstored

Michal

-- 
Michal Novotny<minovotn@redhat.com>, RHCE
Virtualization Team (xen userspace), Red Hat

Re: Re: Xen-unstable save error

[Keir Fraser]

On 21/06/2010 14:10, "Michal Novotny" <minovotn@redhat.com> wrote:

> One more thing, should the xenstore daemon be running or not? Isn't it
> in kernel and xenfs now ?
>
> I saw there's a file:
> /usr/src/redhat/BUILD/kernel-2.6.32.15/drivers/xen/xenfs/xenstored.c

That will eb the kernel's own interface to the xenstore daemon. Or the
interface it exposes to userspace to allow it to communicate to the xenstore
daemon via the kernel's xenstore connection.
 
> in the kernel source codes. But I'm still having /usr/sbin/xenstored.
> Should this be running or not?

It should still run. The xenstore daemon remains a user-space daemon.

 -- Keir

Re: Re: Xen-unstable save error

[Michal Novotny]

On 06/21/2010 03:08 PM, Keir Fraser wrote:
> On 21/06/2010 13:57, "Michal Novotny"<minovotn@redhat.com>  wrote:
>
>    
>> Well, those are objdumps:
>>
>> $ objdump -x /xen-unstable.hg/tools/xcutils/xc_save | grep xs_sus
>> 0000000000000000       F *UND*  00000000000000ad
>> xs_suspend_evtchn_port
>> $ objdump -x /usr/lib64/xen/bin/xc_save | grep xs_suspend
>> 0000000000000000       F *UND*  00000000000000ad
>> xs_suspend_evtchn_port
>> $ ls -al /usr/lib64/libxenctrl.so
>> lrwxrwxrwx 1 root root 17 Jun 21 13:52 /usr/lib64/libxenctrl.so ->
>> libxenctrl.so.4.0
>>
>> I was having some libxenctrl* files at /lib64 so I removed them in order
>> to make linked link those from /usr/lib64 so it did. However, the error
>> now is:
>>      
> Well what was the above supposed to achieve? xs_suspend_evtchn_port is
> provided by libxenstore. It's that library which is getting mis-linked.
> Moving the xc_save binary itself, and/or libxenctrl, isn't going to change
> that. Go use ldd to find what libxenstore is being linked against; work out
> why it's the wrong one; put the right one in its place.
>
>   -- Keir
>
>
>    
Well, I was trying to show that the version that's being linked it the 
4.0 version and not 3.4 or older one.

Also,
# ldd /usr/lib64/libxenctrl.so
         linux-vdso.so.1 =>  (0x00007fff275ff000)
         libpthread.so.0 => /lib64/libpthread.so.0 (0x000000391ec00000)
         libc.so.6 => /lib64/libc.so.6 (0x000000391e000000)
         /lib64/ld-linux-x86-64.so.2 (0x000000391dc00000)
# objdump -x /usr/lib64/libxenctrl.so | grep xs_
#

So there's no xs_suspend_evtchn_port (or anything xs_*) function being 
exported by /usr/lib64/libxenctrl.so (which is the symlink to 
/usr/lib64/libxenctrl.so.4.0.0), therefore:

#objdump -x /usr/lib64/libxenctrl.so.4.0.0 | grep xs_
#objdump -x /xen-unstable.hg/tools/libxc/libxenctrl.so.4.0.0  | grep xs_
#

So the problem here is the missing xs_*.

Note: /xen-unstable.hg/* is the path with Xen-4.1-unstable source codes 
downloaded from Mercurial and compiled.

Michal

-- 
Michal Novotny<minovotn@redhat.com>, RHCE
Virtualization Team (xen userspace), Red Hat

Re: Re: Xen-unstable save error

[Keir Fraser]

On 21/06/2010 14:18, "Michal Novotny" <minovotn@redhat.com> wrote:

> So there's no xs_suspend_evtchn_port (or anything xs_*) function being
> exported by /usr/lib64/libxenctrl.so (which is the symlink to
> /usr/lib64/libxenctrl.so.4.0.0), therefore:
> 
> #objdump -x /usr/lib64/libxenctrl.so.4.0.0 | grep xs_
> #objdump -x /xen-unstable.hg/tools/libxc/libxenctrl.so.4.0.0  | grep xs_

Libxenstore, libxenstore, libxen**STORE**.

#nm /my/path/to/libxenstore.so.3.0.0 | grep xs_sus
000000000000346a T xs_suspend_evtchn_port

 -- Keir

Re: Re: Xen-unstable save error

[Michal Novotny]

On 06/21/2010 03:24 PM, Keir Fraser wrote:
> On 21/06/2010 14:18, "Michal Novotny"<minovotn@redhat.com>  wrote:
>
>    
>> So there's no xs_suspend_evtchn_port (or anything xs_*) function being
>> exported by /usr/lib64/libxenctrl.so (which is the symlink to
>> /usr/lib64/libxenctrl.so.4.0.0), therefore:
>>
>> #objdump -x /usr/lib64/libxenctrl.so.4.0.0 | grep xs_
>> #objdump -x /xen-unstable.hg/tools/libxc/libxenctrl.so.4.0.0  | grep xs_
>>      
> Libxenstore, libxenstore, libxen**STORE**.
>
> #nm /my/path/to/libxenstore.so.3.0.0 | grep xs_sus
> 000000000000346a T xs_suspend_evtchn_port
>
>   -- Keir
>
>
>    
Oh, sorry for that and thanks for noticing my mistake. I overlooked this 
one, nevertheless it's still not working:

[2010-06-21 17:27:39 4305] DEBUG (XendCheckpoint:126) [xc_save]: 
/usr/lib64/xen/bin/xc_save 56 1 0 0 4
[2010-06-21 17:27:39 4305] INFO (XendCheckpoint:410) xc_save: failed to 
get the suspend evtchn port
[2010-06-21 17:27:39 4305] INFO (XendCheckpoint:410)
[2010-06-21 17:27:39 4305] DEBUG (XendCheckpoint:381) suspend
[2010-06-21 17:27:39 4305] DEBUG (XendCheckpoint:129) In 
saveInputHandler suspend
[2010-06-21 17:27:39 4305] DEBUG (XendCheckpoint:131) Suspending 1 ...
[2010-06-21 17:27:39 4305] DEBUG (XendDomainInfo:521) 
XendDomainInfo.shutdown(suspend)
[2010-06-21 17:27:39 4305] DEBUG (XendDomainInfo:1877) 
XendDomainInfo.handleShutdownWatch
[2010-06-21 17:27:39 4305] INFO (XendDomainInfo:538) HVM save:remote 
shutdown dom 1!
[2010-06-21 17:27:39 4305] INFO (XendCheckpoint:137) Domain 1 suspended.
[2010-06-21 17:27:39 4305] INFO (XendDomainInfo:2074) Domain has 
shutdown: name=migrating-rhel5-32fv-stubdom id=1 reason=suspend.
[2010-06-21 17:27:40 4305] INFO (image:538) signalDeviceModel:restore dm 
state to running
[2010-06-21 17:27:40 4305] DEBUG (XendCheckpoint:146) Written done
[2010-06-21 17:27:46 4305] DEBUG (XendDomainInfo:3067) 
XendDomainInfo.destroy: domid=1
[2010-06-21 17:27:46 4305] DEBUG (XendDomainInfo:2397) Destroying device 
model
[2010-06-21 17:27:47 4305] INFO (image:615) migrating-rhel5-32fv-stubdom 
device model terminated


# ls -al rhel5-32fv.sav
-rwxr-xr-x 1 root root 54657427 Jun 21 17:27 rhel5-32fv.sav

# xm restore rhel5-32fv.sav
Error: /usr/lib64/xen/bin/xc_restore 4 2 2 3 1 1 1 0 failed
Usage: xm restore <CheckpointFile> [-p]

Restore a domain from a saved state.
   -p, --paused                   Do not unpause domain after restoring it

# tail /var/log/xen/xend.log
[2010-06-21 17:29:21 4305] INFO (image:822) Need to create platform 
device.[domid:2]
[2010-06-21 17:29:21 4305] DEBUG (XendCheckpoint:273) 
restore:shadow=0x9, _static_max=0x40000000, _static_min=0x0,
[2010-06-21 17:29:21 4305] DEBUG (XendCheckpoint:292) [xc_restore]: 
/usr/lib64/xen/bin/xc_restore 4 2 2 3 1 1 1 0
[2010-06-21 17:29:22 4305] INFO (XendCheckpoint:410) xc: error: Error 
when reading batch size (0 = Success): Internal error
[2010-06-21 17:29:22 4305] INFO (XendCheckpoint:410) xc: error: error 
when buffering batch, finishing (0 = Success): Internal error
[2010-06-21 17:29:22 4305] INFO (XendCheckpoint:410) xc: error: error 
zeroing magic pages (22 = Invalid argument): Internal error
[2010-06-21 17:29:22 4305] DEBUG (XendDomainInfo:3067) 
XendDomainInfo.destroy: domid=2
[2010-06-21 17:29:22 4305] ERROR (XendDomainInfo:3081) 
XendDomainInfo.destroy: domain destruction failed.
Traceback (most recent call last):
   File "usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", 
line 3074, in destroy
     xc.domain_pause(self.domid)
Error: (3, 'No such process')
[2010-06-21 17:29:22 4305] DEBUG (XendDomainInfo:2402) No device model
[2010-06-21 17:29:22 4305] DEBUG (XendDomainInfo:2404) Releasing devices
[2010-06-21 17:29:22 4305] DEBUG (XendDomainInfo:2410) Removing vif/0
[2010-06-21 17:29:22 4305] DEBUG (XendDomainInfo:1272) 
XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2010-06-21 17:29:22 4305] DEBUG (XendDomainInfo:2410) Removing vbd/768
[2010-06-21 17:29:22 4305] DEBUG (XendDomainInfo:1272) 
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/768
[2010-06-21 17:29:22 4305] DEBUG (XendDomainInfo:2410) Removing vbd/2048
[2010-06-21 17:29:22 4305] DEBUG (XendDomainInfo:1272) 
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/2048
[2010-06-21 17:29:22 4305] DEBUG (XendDomainInfo:2410) Removing vfb/0
[2010-06-21 17:29:22 4305] DEBUG (XendDomainInfo:1272) 
XendDomainInfo.destroyDevice: deviceClass = vfb, device = vfb/0
[2010-06-21 17:29:22 4305] DEBUG (XendDomainInfo:2410) Removing console/0
[2010-06-21 17:29:22 4305] DEBUG (XendDomainInfo:1272) 
XendDomainInfo.destroyDevice: deviceClass = console, device = console/0
[2010-06-21 17:29:22 4305] ERROR (XendCheckpoint:344) 
/usr/lib64/xen/bin/xc_restore 4 2 2 3 1 1 1 0 failed
Traceback (most recent call last):
   File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", 
line 296, in restore
     forkHelper(cmd, fd, handler.handler, True)
   File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", 
line 398, in forkHelper
     raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_restore 4 2 2 3 1 1 1 0 failed
[2010-06-21 17:29:22 4305] ERROR (XendDomain:1182) Restore failed
Traceback (most recent call last):
   File "usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 
1166, in domain_restore_fd
     dominfo = XendCheckpoint.restore(self, fd, paused=paused, 
relocating=relocating)
   File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", 
line 345, in restore
     raise exn
XendError: /usr/lib64/xen/bin/xc_restore 4 2 2 3 1 1 1 0 failed

So now it seems to be linked with the correct library but it can't get 
the suspend port by now:
 >> xc_save: failed to get the suspend evtchn port

This is being called from xc_save.c (snippet from line 192):
...
        port = xs_suspend_evtchn_port(si.domid);

         if (port < 0)
             warnx("failed to get the suspend evtchn port\n");
         else {
             ... suspend...
         }

I had a look at the code for xenstore/xs.c I saw it's reading the value at:

/local/domain/%d/device/suspend/event-channel

but when I try to get it using:

xenstore-ls /local/domain/3/device/suspendupstream

where 3 is my domid I saw nothing, I saw just:

# xenstore-ls /local/domain/3/device
vfb = ""
  0 = ""
   state = "1"
   backend-id = "0"
   backend = "/local/domain/0/backend/vfb/3/0"
vbd = ""
  768 = ""
   backend-id = "0"
   virtual-device = "768"
   device-type = "disk"
   state = "1"
   backend = "/local/domain/0/backend/vbd/3/768"
  2048 = ""
   backend-id = "0"
   virtual-device = "2048"
   device-type = "disk"
   state = "1"
   backend = "/local/domain/0/backend/vbd/3/2048"
vif = ""
  0 = ""
   state = "1"
   backend-id = "0"
   backend = "/local/domain/0/backend/vif/3/0"
console = ""
  0 = ""
   state = "1"
   backend-id = "0"
   backend = "/local/domain/0/backend/console/3/0"
#

My guest is RHEL-5 i386 guest but this seems that the suspend port is 
missing. AFAIK, you started using the SUSPEND_CANCEL some time ago which 
requires the modified kernel.

Isn't it possible that's the issue or how is it with the SUSPEND_CANCEL 
functionality?

Thanks,
Michal

-- 
Michal Novotny<minovotn@redhat.com>, RHCE
Virtualization Team (xen userspace), Red Hat

Re: Re: Xen-unstable save error

[Keir Fraser]

On 21/06/2010 14:37, "Michal Novotny" <minovotn@redhat.com> wrote:

> My guest is RHEL-5 i386 guest but this seems that the suspend port is
> missing. AFAIK, you started using the SUSPEND_CANCEL some time ago which
> requires the modified kernel.
> 
> Isn't it possible that's the issue or how is it with the SUSPEND_CANCEL
> functionality?

SUSPEND_CANCEL is a different thing. The suspend port is simply a quicker
way for suspend notifications to be passed back and forth between the guest
and the dom0 toolstack. We fall back okay if the guest kernel does not
support the new faster method.

I'm not sure why the domain restore operation fails. Unfortunately some
error messages are now expected in the logs, since Remus functionality went
into the tree. So it's hard to work out what the first error is.

 -- Keir

Re: Re: Xen-unstable save error

[Michal Novotny]

On 06/21/2010 03:45 PM, Keir Fraser wrote:
> On 21/06/2010 14:37, "Michal Novotny"<minovotn@redhat.com>  wrote:
>
>    
>> My guest is RHEL-5 i386 guest but this seems that the suspend port is
>> missing. AFAIK, you started using the SUSPEND_CANCEL some time ago which
>> requires the modified kernel.
>>
>> Isn't it possible that's the issue or how is it with the SUSPEND_CANCEL
>> functionality?
>>      
> SUSPEND_CANCEL is a different thing. The suspend port is simply a quicker
> way for suspend notifications to be passed back and forth between the guest
> and the dom0 toolstack. We fall back okay if the guest kernel does not
> support the new faster method.
>
> I'm not sure why the domain restore operation fails. Unfortunately some
> error messages are now expected in the logs, since Remus functionality went
> into the tree. So it's hard to work out what the first error is.
>
>   -- Keir
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>    
Ok Keir, but what I don't understand is why there's nothing in 
`/local/domain/%d/device/suspend/event-channel`. So this is OK?

For the restore functionality:

# ls -ahl rhel5-32fv.sav
-rwxr-xr-x 1 root root 53M Jun 21  2010 rhel5-32fv.sav

As you can see the save file is 53M big but the guest was having 1G of 
memory and I think this is why it's failing.
You can see it should be having 1G of memory here too:
...
[2010-06-21 17:29:20 4305] DEBUG (XendDomainInfo:237) 
XendDomainInfo.restore(['domain', ['domid', '1'], ['cpu_weight', '256'], 
['cpu_cap', '0'], ['on_crash', 'restart'], ['uuid', 
'c91ec802-2015-cb49-80e5-810c808bf725'], ['bootloader_args'], 
['pool_name', 'Pool-0'], ['vcpus', '1'], ['name', 'rhel5-32fv-stubdom'], 
['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['cpus', [[]]], 
['description'], ['bootloader'], ['maxmem', '1024'],* ['memory', 
'1024'],* ['shadow_memory', '9'], ['vcpu_avail', '1'], ['features'], 
['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['start_time', 
'1277134046.11'], ['cpu_time', '1.550284835'], ['online_vcpus', '1'], 
['image', ['hvm', ['kernel'], ['superpages', '0'], ['tsc_mode', '0'], 
['videoram', '4'], ['hpet', '0'], ['boot', 'c'], ['loader', 
'/usr/lib/xen/boot/hvmloader'], ['serial', 'pty'], ['vpt_align', '1'], 
['xen_platform_pci', '1'], ['opengl', '1'], ['vncunused', '1'], 
['rtc_timeoffset', '0'], ['pci', []], ['pae', '1'], ['stdvga', '0'], 
['hap', '1'], ['viridian', '0'], ['acpi', '1'], ['localtime', '0'], 
['timer_mode', '1'], ['vnc', '1'], ['nographic', '0'], ['guest_os_type', 
'default'], ['vncdisplay', '1'], ['pci_msitranslate', '1'], ['oos', 
'1'], ['apic', '1'], ['sdl', '0'], ['nomigrate', '0'], ['device_model', 
'/usr/lib/xen/bin/qemu-dm'], ['pci_power_mgmt', '0'], ['usb', '0'], 
['xauthority', '/root/.Xauthority'], ['isa', '0'], ['display', 
'localhost:10.0'], ['notes', ['SUSPEND_CANCEL', '1']]]], ['status', 
'2'], ['state', 'r-----'], ['store_mfn', '1044476'], ['device', ['vif', 
['bridge', 'virbr0'], ['uuid', 'dcd99a20-2e8f-2692-8e56-dc4051579923'], 
['script', '/etc/xen/scripts/vif-bridge'], ['mac', '00:16:3e:5b:bd:9c'], 
['type', 'ioemu'], ['backend', '0']]], ['device', ['vbd', ['uuid', 
'e7e07da9-c104-800d-ee3f-5fe9757167fd'], ['bootable', '1'], ['dev', 
'hda:disk'], ['uname', 
'file:/var/lib/xen/images/colossus/rhel5-32fv.img'], ['mode', 'w'], 
['backend', '0'], ['VDI']]], ['device', ['vbd', ['uuid', 
'0180089b-8394-cbfa-0da4-b8c1fc688617'], ['bootable', '0'], ['dev', 
'sda:disk'], ['uname', 'file:/home2/test.img'], ['mode', 'w'], 
['backend', '0'], ['VDI']]], ['device', ['vfb', ['vncunused', '1'], 
['location', '127.0.0.1:5901'], ['vnc', '1'], ['vncdisplay', '1'], 
['uuid', '7fa1bcc0-797d-66ac-eb88-6ef15f1209f0']]], ['device', 
['console', ['protocol', 'vt100'], ['location', '3'], ['uuid', 
'd77b182b-4152-a4d2-f577-8b610b5cd6ff']]]])

The first error (Error when reading batch size (0 = Success): Internal 
error) is coming from libxc/xc_domain_restore.c in pagebuf_get_one() 
function where it is there:
...
     if ( RDEXACT(fd, &count, sizeof(count)) )
     {
         PERROR("Error when reading batch size");
         return -1;
     }
...
so I guess the data are not well-written for this guest (since the file 
is smaller than the original guest memory) and that's why the error 
occurs. As you can see there's nothing in xend.log except "failed to get 
the suspend evtchn port" message:

[2010-06-21 15:59:55 4305] DEBUG (XendCheckpoint:126) [xc_save]: 
/usr/lib64/xen/bin/xc_save 56 5 0 0 4
[2010-06-21 15:59:55 4305] INFO (XendCheckpoint:410) xc_save: failed to 
get the suspend evtchn port
[2010-06-21 15:59:55 4305] INFO (XendCheckpoint:410)
[2010-06-21 15:59:55 4305] DEBUG (XendCheckpoint:381) suspend
[2010-06-21 15:59:55 4305] DEBUG (XendCheckpoint:129) In 
saveInputHandler suspend
[2010-06-21 15:59:55 4305] DEBUG (XendCheckpoint:131) Suspending 5 ...
[2010-06-21 15:59:55 4305] DEBUG (XendDomainInfo:521) 
XendDomainInfo.shutdown(suspend)
[2010-06-21 15:59:55 4305] DEBUG (XendDomainInfo:1877) 
XendDomainInfo.handleShutdownWatch
[2010-06-21 15:59:55 4305] INFO (XendDomainInfo:538) HVM save:remote 
shutdown dom 5!
[2010-06-21 15:59:55 4305] INFO (XendDomainInfo:2074) Domain has 
shutdown: name=migrating-rhel5-32fv-stubdom id=5 reason=suspend.
[2010-06-21 15:59:55 4305] INFO (XendCheckpoint:137) Domain 5 suspended.
[2010-06-21 15:59:56 4305] INFO (image:538) signalDeviceModel:restore dm 
state to running
[2010-06-21 15:59:56 4305] DEBUG (XendCheckpoint:146) Written done
[2010-06-21 16:00:02 4305] DEBUG (XendDomainInfo:3067) 
XendDomainInfo.destroy: domid=5
[2010-06-21 16:00:03 4305] DEBUG (XendDomainInfo:2397) Destroying device 
model
[2010-06-21 16:00:03 4305] INFO (image:615) migrating-rhel5-32fv-stubdom 
device model terminated
[2010-06-21 16:00:03 4305] DEBUG (XendDomainInfo:2404) Releasing devices
[2010-06-21 16:00:03 4305] DEBUG (XendDomainInfo:2410) Removing vif/0
[2010-06-21 16:00:03 4305] DEBUG (XendDomainInfo:1272) 
XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2010-06-21 16:00:03 4305] DEBUG (XendDomainInfo:2410) Removing vbd/768
[2010-06-21 16:00:03 4305] DEBUG (XendDomainInfo:1272) 
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/768
[2010-06-21 16:00:03 4305] DEBUG (XendDomainInfo:2410) Removing vbd/2048
[2010-06-21 16:00:03 4305] DEBUG (XendDomainInfo:1272) 
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/2048
[2010-06-21 16:00:03 4305] DEBUG (XendDomainInfo:2410) Removing vfb/0
[2010-06-21 16:00:03 4305] DEBUG (XendDomainInfo:1272) 
XendDomainInfo.destroyDevice: deviceClass = vfb, device = vfb/0
[2010-06-21 16:00:03 4305] DEBUG (XendDomainInfo:2410) Removing console/0
[2010-06-21 16:00:03 4305] DEBUG (XendDomainInfo:1272) 
XendDomainInfo.destroyDevice: deviceClass = console, device = console/0

Any ideas why the save file is that small (it should be 1024M at least, 
right? ) ?

Thanks,
Michal

-- 
Michal Novotny<minovotn@redhat.com>, RHCE
Virtualization Team (xen userspace), Red Hat

Re: Re: Xen-unstable save error

[Keir Fraser]

On 21/06/2010 15:02, "Michal Novotny" <minovotn@redhat.com> wrote:

> Ok Keir, but what I don't understand is why there's nothing in
> `/local/domain/%d/device/suspend/event-channel`. So this is OK?

Yes that's okay. The guest writes that location only If it supports the new
event-channel notification method for suspend.

> For the restore functionality:
> 
> # ls -ahl rhel5-32fv.sav
> -rwxr-xr-x 1 root root 53M Jun 21  2010 rhel5-32fv.sav

Okay, yeah, if the guest really has 1G of memory allocated to it at the time
of the save, then something has gone wrong on the save side. But no errors
were apparent on the save side, apart from that event-channel port warning
which is benign? That's weird.

 -- Keir

Re: Re: Xen-unstable save error

[Michal Novotny]

On 06/21/2010 04:06 PM, Keir Fraser wrote:
> On 21/06/2010 15:02, "Michal Novotny"<minovotn@redhat.com>  wrote:
>
>    
>> Ok Keir, but what I don't understand is why there's nothing in
>> `/local/domain/%d/device/suspend/event-channel`. So this is OK?
>>      
> Yes that's okay. The guest writes that location only If it supports the new
> event-channel notification method for suspend.
>
>    
>> For the restore functionality:
>>
>> # ls -ahl rhel5-32fv.sav
>> -rwxr-xr-x 1 root root 53M Jun 21  2010 rhel5-32fv.sav
>>      
> Okay, yeah, if the guest really has 1G of memory allocated to it at the time
> of the save, then something has gone wrong on the save side. But no errors
> were apparent on the save side, apart from that event-channel port warning
> which is benign? That's weird.
>
>   -- Keir
>
>
>    
Exactly Keir. Nothing wrong there except the event-channel port missing 
warning. So this is something strange I guess. Well, I'll try to 
investigate this further then.

Michal

-- 
Michal Novotny<minovotn@redhat.com>, RHCE
Virtualization Team (xen userspace), Red Hat

Re: Re: Xen-unstable save error

[Michal Novotny]

On 06/21/2010 04:06 PM, Keir Fraser wrote:
> On 21/06/2010 15:02, "Michal Novotny"<minovotn@redhat.com>  wrote:
>
>    
>> Ok Keir, but what I don't understand is why there's nothing in
>> `/local/domain/%d/device/suspend/event-channel`. So this is OK?
>>      
> Yes that's okay. The guest writes that location only If it supports the new
> event-channel notification method for suspend.
>
>    
>> For the restore functionality:
>>
>> # ls -ahl rhel5-32fv.sav
>> -rwxr-xr-x 1 root root 53M Jun 21  2010 rhel5-32fv.sav
>>      
> Okay, yeah, if the guest really has 1G of memory allocated to it at the time
> of the save, then something has gone wrong on the save side. But no errors
> were apparent on the save side, apart from that event-channel port warning
> which is benign? That's weird.
>
>   -- Keir
>
>
>    
Well, I see what's going on now but I had to turn on the debug flag now. 
According to the code, this is what's being hit per almost each page and 
MFN:

...
                 if ( pfn_type[j] == XEN_DOMCTL_PFINFO_XTAB )
                 {
                     printf("type fail: page %i mfn %08lx\n", j, gmfn);
                     continue;
                 }
...

Xend.log is having many occurences of type failures:

[2010-06-21 16:48:53 10573] DEBUG (XendCheckpoint:381) type fail: page 
414 mfn 0000659e
[2010-06-21 16:48:53 10573] DEBUG (XendCheckpoint:129) In 
saveInputHandler type fail: page 414 mfn 0000659e
[2010-06-21 16:48:53 10573] DEBUG (XendCheckpoint:381) type fail: page 
415 mfn 0000659f
[2010-06-21 16:48:53 10573] DEBUG (XendCheckpoint:129) In 
saveInputHandler type fail: page 415 mfn 0000659f
[2010-06-21 16:48:53 10573] DEBUG (XendCheckpoint:381) type fail: page 
416 mfn 000065a0
[2010-06-21 16:48:53 10573] DEBUG (XendCheckpoint:129) In 
saveInputHandler type fail: page 416 mfn 000065a0

Well, did somebody hit those save issues or similar?

Thanks,
Michal

-- 
Michal Novotny<minovotn@redhat.com>, RHCE
Virtualization Team (xen userspace), Red Hat

Re: Re: Xen-unstable save error

[Keir Fraser]

On 21/06/2010 15:50, "Michal Novotny" <minovotn@redhat.com> wrote:

> Well, I see what's going on now but I had to turn on the debug flag now.
> According to the code, this is what's being hit per almost each page and
> MFN:
> 
> ...
>                  if ( pfn_type[j] == XEN_DOMCTL_PFINFO_XTAB )
>                  {
>                      printf("type fail: page %i mfn %08lx\n", j, gmfn);
>                      continue;
>                  }
> ...
> 
> Xend.log is having many occurences of type failures:

Hm, well I just saved/restored a 750MB PV guest and a 750MB HVM guest okay
with xen-unstable tip. However I think the above type-checking code may only
have been introduced for HVM guests as of Jan Beulich's recent changeset
21615. If your guest is an HVM guest, perhaps this changeset is to blame? I
would suggest reverting it and re-testing. I've cc'ed Jan in case he can
comment on the info you've given.

 -- Keir

Re: Re: Xen-unstable save error

[Michal Novotny]

On 06/21/2010 07:57 PM, Keir Fraser wrote:
> On 21/06/2010 15:50, "Michal Novotny"<minovotn@redhat.com>  wrote:
>
>    
>> Well, I see what's going on now but I had to turn on the debug flag now.
>> According to the code, this is what's being hit per almost each page and
>> MFN:
>>
>> ...
>>                   if ( pfn_type[j] == XEN_DOMCTL_PFINFO_XTAB )
>>                   {
>>                       printf("type fail: page %i mfn %08lx\n", j, gmfn);
>>                       continue;
>>                   }
>> ...
>>
>> Xend.log is having many occurences of type failures:
>>      
> Hm, well I just saved/restored a 750MB PV guest and a 750MB HVM guest okay
> with xen-unstable tip. However I think the above type-checking code may only
> have been introduced for HVM guests as of Jan Beulich's recent changeset
> 21615. If your guest is an HVM guest, perhaps this changeset is to blame? I
> would suggest reverting it and re-testing. I've cc'ed Jan in case he can
> comment on the info you've given.
>    

Keir,
you were right about the c/s 21615. I did try reverting it and it was 
working fine to both save and restore when I was having this patch reverted.

Michal

-- 
Michal Novotny<minovotn@redhat.com>, RHCE
Virtualization Team (xen userspace), Red Hat

Re: Re: Xen-unstable save error

[Keir Fraser]

On 22/06/2010 06:38, "Michal Novotny" <minovotn@redhat.com> wrote:

>> Hm, well I just saved/restored a 750MB PV guest and a 750MB HVM guest okay
>> with xen-unstable tip. However I think the above type-checking code may only
>> have been introduced for HVM guests as of Jan Beulich's recent changeset
>> 21615. If your guest is an HVM guest, perhaps this changeset is to blame? I
>> would suggest reverting it and re-testing. I've cc'ed Jan in case he can
>> comment on the info you've given.
> 
> you were right about the c/s 21615. I did try reverting it and it was
> working fine to both save and restore when I was having this patch reverted.

Okay, Jan should be ebale to help then. We don't really want to revert that
changeset as it is itself a bugfix.

 -- Keir

Re: Re: Xen-unstable save error

[Michal Novotny]

On 06/22/2010 07:58 AM, Keir Fraser wrote:
> On 22/06/2010 06:38, "Michal Novotny"<minovotn@redhat.com>  wrote:
>
>    
>>> Hm, well I just saved/restored a 750MB PV guest and a 750MB HVM guest okay
>>> with xen-unstable tip. However I think the above type-checking code may only
>>> have been introduced for HVM guests as of Jan Beulich's recent changeset
>>> 21615. If your guest is an HVM guest, perhaps this changeset is to blame? I
>>> would suggest reverting it and re-testing. I've cc'ed Jan in case he can
>>> comment on the info you've given.
>>>        
>> you were right about the c/s 21615. I did try reverting it and it was
>> working fine to both save and restore when I was having this patch reverted.
>>      
> Okay, Jan should be ebale to help then. We don't really want to revert that
> changeset as it is itself a bugfix.
>
>   -- Keir
>
>
>    
Jan, as we were solving this with Keir on the list I was unable to save 
the HVM guest properly and reverting your c/s 21615 helped and with your 
patch reverted it was working fine. Any idea why there were so type 
failures for all the pages of HVM guest?

Thanks,
Michal

-- 
Michal Novotny<minovotn@redhat.com>, RHCE
Virtualization Team (xen userspace), Red Hat

Re: Re: Xen-unstable save error

[Jan Beulich]

>>> On 22.06.10 at 08:23, Michal Novotny <minovotn@redhat.com> wrote:
> Jan, as we were solving this with Keir on the list I was unable to save 
> the HVM guest properly and reverting your c/s 21615 helped and with your 
> patch reverted it was working fine. Any idea why there were so type 
> failures for all the pages of HVM guest?

Not really, and I think you will need to find out by instrumenting the
hypercall implementation (XEN_DOMCTL_getpageframeinfo{2,3}) -
really, type failures should only happen for grant frames and I/O
memory pages (as you can see in the code, for the function to
report XTAB either the MFN must be invalid, must be a Xen frame,
or it must be impossible to obtain a reference to the page - all of
which shouldn't hold for the majority of a HVM guest's pages).

Btw., Keir, is it really correct for the type to be returned as zero
(normal page) when xsm_getpageframeinfo() returns non-zero?

Is there anything special about the guest you're trying to save? That
patch was tested quite extensively here before submission...

Jan

Re: Re: Xen-unstable save error

[Keir Fraser]

On 22/06/2010 07:48, "Jan Beulich" <JBeulich@novell.com> wrote:

> Btw., Keir, is it really correct for the type to be returned as zero
> (normal page) when xsm_getpageframeinfo() returns non-zero?

Well, maybe it should return XTAB... Really, who knows. I'm sure if you did
use XSM and hit that path, no good would come of it for the caller whatever!

 -- Keir

Re: Re: Xen-unstable save error

[Jan Beulich]

>>> On 22.06.10 at 08:51, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> On 22/06/2010 07:48, "Jan Beulich" <JBeulich@novell.com> wrote:
> 
>> Btw., Keir, is it really correct for the type to be returned as zero
>> (normal page) when xsm_getpageframeinfo() returns non-zero?
> 
> Well, maybe it should return XTAB... Really, who knows. I'm sure if you did
> use XSM and hit that path, no good would come of it for the caller whatever!

If you don't know, really who does? Or is the XSM stuff dead code?

Jan

Re: Re: Xen-unstable save error

[Keir Fraser]

On 22/06/2010 08:12, "Jan Beulich" <JBeulich@novell.com> wrote:

>>> Btw., Keir, is it really correct for the type to be returned as zero
>>> (normal page) when xsm_getpageframeinfo() returns non-zero?
>> 
>> Well, maybe it should return XTAB... Really, who knows. I'm sure if you did
>> use XSM and hit that path, no good would come of it for the caller whatever!
> 
> If you don't know, really who does? Or is the XSM stuff dead code?

The NSA guys (George Coker et al) maintain it.

 -- Keir

Re: Re: Xen-unstable save error

[Michal Novotny]

On 06/22/2010 08:48 AM, Jan Beulich wrote:
>>>> On 22.06.10 at 08:23, Michal Novotny<minovotn@redhat.com>  wrote:
>>>>          
>> Jan, as we were solving this with Keir on the list I was unable to save
>> the HVM guest properly and reverting your c/s 21615 helped and with your
>> patch reverted it was working fine. Any idea why there were so type
>> failures for all the pages of HVM guest?
>>      
> Not really, and I think you will need to find out by instrumenting the
> hypercall implementation (XEN_DOMCTL_getpageframeinfo{2,3}) -
> really, type failures should only happen for grant frames and I/O
> memory pages (as you can see in the code, for the function to
> report XTAB either the MFN must be invalid, must be a Xen frame,
> or it must be impossible to obtain a reference to the page - all of
> which shouldn't hold for the majority of a HVM guest's pages).
>
> Btw., Keir, is it really correct for the type to be returned as zero
> (normal page) when xsm_getpageframeinfo() returns non-zero?
>
> Is there anything special about the guest you're trying to save? That
> patch was tested quite extensively here before submission...
>
> Jan
>    

Nothing special, the guest is RHEL-5 i386 HVM guest with the serial 
console enabled in /boot/grub/grub.conf to be able to access it using 
the `xm console` command. It returns no error printed to the xend.log 
(except when I added the debug output) but the total size of the 'ls 
-alh /path/to/save/file' (after the save) is 54M instead of > 1G (the 
guest memory size).

Michal

-- 
Michal Novotny<minovotn@redhat.com>, RHCE
Virtualization Team (xen userspace), Red Hat

Re: Re: Xen-unstable save error

[Keir Fraser]

On 22/06/2010 07:48, "Jan Beulich" <JBeulich@novell.com> wrote:

>>>> On 22.06.10 at 08:23, Michal Novotny <minovotn@redhat.com> wrote:
>> Jan, as we were solving this with Keir on the list I was unable to save
>> the HVM guest properly and reverting your c/s 21615 helped and with your
>> patch reverted it was working fine. Any idea why there were so type
>> failures for all the pages of HVM guest?
> 
> Not really, and I think you will need to find out by instrumenting the
> hypercall implementation (XEN_DOMCTL_getpageframeinfo{2,3}) -
> really, type failures should only happen for grant frames and I/O
> memory pages (as you can see in the code, for the function to
> report XTAB either the MFN must be invalid, must be a Xen frame,
> or it must be impossible to obtain a reference to the page - all of
> which shouldn't hold for the majority of a HVM guest's pages).

Michal, is it possible you are running new tools against a slightly older
hypervisor? The tools changes had a corresponding, non-version-checked,
modification to a hypercall. Without running the modified hypervisor, HVM
save would indeed behave very oddly!

 -- Keir

Re: Re: Xen-unstable save error

[Michal Novotny]

On 06/22/2010 12:01 PM, Keir Fraser wrote:
> On 22/06/2010 07:48, "Jan Beulich"<JBeulich@novell.com>  wrote:
>
>    
>>>>> On 22.06.10 at 08:23, Michal Novotny<minovotn@redhat.com>  wrote:
>>>>>            
>>> Jan, as we were solving this with Keir on the list I was unable to save
>>> the HVM guest properly and reverting your c/s 21615 helped and with your
>>> patch reverted it was working fine. Any idea why there were so type
>>> failures for all the pages of HVM guest?
>>>        
>> Not really, and I think you will need to find out by instrumenting the
>> hypercall implementation (XEN_DOMCTL_getpageframeinfo{2,3}) -
>> really, type failures should only happen for grant frames and I/O
>> memory pages (as you can see in the code, for the function to
>> report XTAB either the MFN must be invalid, must be a Xen frame,
>> or it must be impossible to obtain a reference to the page - all of
>> which shouldn't hold for the majority of a HVM guest's pages).
>>      
> Michal, is it possible you are running new tools against a slightly older
> hypervisor? The tools changes had a corresponding, non-version-checked,
> modification to a hypercall. Without running the modified hypervisor, HVM
> save would indeed behave very oddly!
>
>   -- Keir
>
>
>    
Keir, I don't think so since the hypervisor datetime in /boot is the 
same like for kernel and tools.

Michal

-- 
Michal Novotny<minovotn@redhat.com>, RHCE
Virtualization Team (xen userspace), Red Hat

Re: Re: Xen-unstable save error

[Keir Fraser]

On 22/06/2010 11:05, "Michal Novotny" <minovotn@redhat.com> wrote:

>> Michal, is it possible you are running new tools against a slightly older
>> hypervisor? The tools changes had a corresponding, non-version-checked,
>> modification to a hypercall. Without running the modified hypervisor, HVM
>> save would indeed behave very oddly!
>>    
> Keir, I don't think so since the hypervisor datetime in /boot is the
> same like for kernel and tools.

Well, it remains something to doublecheck at least. There's no good reason
that HVM save should work okay for everyone else and only fail so miserably
for you.

 -- Keir

Re: Re: Xen-unstable save error

[Michal Novotny]

On 06/22/2010 12:23 PM, Keir Fraser wrote:
> On 22/06/2010 11:05, "Michal Novotny"<minovotn@redhat.com>  wrote:
>
>    
>>> Michal, is it possible you are running new tools against a slightly older
>>> hypervisor? The tools changes had a corresponding, non-version-checked,
>>> modification to a hypercall. Without running the modified hypervisor, HVM
>>> save would indeed behave very oddly!
>>>
>>>        
>> Keir, I don't think so since the hypervisor datetime in /boot is the
>> same like for kernel and tools.
>>      
> Well, it remains something to doublecheck at least. There's no good reason
> that HVM save should work okay for everyone else and only fail so miserably
> for you.
>
>   -- Keir
>
>
>    
I don't know but I'll investigate it further but it's working with the 
patch 21615 patch reverted. Maybe something's wrong with my test setup. 
Some further investigation is necessary.

Michal

-- 
Michal Novotny<minovotn@redhat.com>, RHCE
Virtualization Team (xen userspace), Red Hat