Issues with latest staging tree starting PV guest.

Issues with latest staging tree starting PV guest.

[Mukesh Rathor]

Hi,

I reinstalled my system with fedora 19, then installed xen. Got the
latest xen tree, cs 79de2d3. Then I built copied xen, tools. But running
into minor issues, which are new in 4.5 for me:

  1. xl cr -c vm.cfg.pv : it doesn't throw the pygrub menu on terminal,
     and the guest remains paused. However, if I contrl C, xl unp, then
     xl console, I see guest booting.

  2. When I do halt in it, it remains suspended but doesn't get
  destroyed.

  3. I setup xenbr, added vif line as always:
        vif = ['type=netfront,bridge=xenbr0']

    But now get:
     libxl: error: libxl_create.c:1226:domcreate_attach_vtpms: unable
    to add nic devices


xenbr seems ok:

bridge name     bridge id               STP enabled     interfaces
xenbr0          8000.003048cef05e       no              em1


Wondering if anyone's seen/dealt with them.

thanks a lot,
Mukesh

Re: Issues with latest staging tree starting PV guest.

[Jan Beulich]

>>> On 04.03.14 at 03:52, Mukesh Rathor <mukesh.rathor@oracle.com> wrote:
> I reinstalled my system with fedora 19, then installed xen. Got the
> latest xen tree, cs 79de2d3. Then I built copied xen, tools. But running
> into minor issues, which are new in 4.5 for me:
> 
>   1. xl cr -c vm.cfg.pv : it doesn't throw the pygrub menu on terminal,
>      and the guest remains paused. However, if I contrl C, xl unp, then
>      xl console, I see guest booting.

Sounds relatively similar to what d6ac84ca was supposed to
address; odd.

>   2. When I do halt in it, it remains suspended but doesn't get
>   destroyed.
> 
>   3. I setup xenbr, added vif line as always:
>         vif = ['type=netfront,bridge=xenbr0']
> 
>     But now get:
>      libxl: error: libxl_create.c:1226:domcreate_attach_vtpms: unable
>     to add nic devices

I've been seeing this relatively regularly for quite some time, but
only ever on the first try to start a guest after host boot. The
second try always succeeded for me so far, so I attributed it to
some race that I didn't find time to investigate so far.

Jan

Re: Issues with latest staging tree starting PV guest.

[Mukesh Rathor]

On Mon, 3 Mar 2014 18:52:39 -0800
Mukesh Rathor <mukesh.rathor@oracle.com> wrote:

> Hi,
... 
>   3. I setup xenbr, added vif line as always:
>         vif = ['type=netfront,bridge=xenbr0']
> 
>     But now get:
>      libxl: error: libxl_create.c:1226:domcreate_attach_vtpms: unable
>     to add nic devices

This is how far I got debugging this:

Looks like the issue is not getting domid, because stepping thru I see
in devstate_watch_callback() it goes thru:

    } else {
        int got = atoi(sstate);
        if (got == ds->wanted) {
            LIBXL__LOG(CTX, LIBXL__LOG_DEBUG, "backend %s wanted state %d ok",
                       watch_path, ds->wanted);
            rc = 0;
logging:
libxl: debug: libxl_event.c:653:devstate_watch_callback: backend /local/domain/0/backend/vif/5/0/state wanted state 2 ok

then it calls:
...
ds->callback(egc, ds, rc); --> device_backend_callback --> device_hotplug
which does:
...
    rc = libxl__get_domid(gc, &domid);  ---> libxl__xs_read_checked
     --> libxl__xs_read (, , path="domid")...
      This returns result == 0 and errno == ENOENT.

logging:
libxl: debug: libxl_dm.c:1473:kill_device_model: Device Model signaled
libxl: error: libxl_device.c:780:libxl__initiate_device_remove: unable to get my domid

However, xenstore-ls, I see domid being there:
  bash> xenstore-ls |grep domid
     domid = "11"

Zooming in libxl__xs_read, I got as far as :

libxl__xs_read -> xs_read((gc=0x636880, t=0x0, path=0x7ffff7bc13ba "domid")
 --> .. --> xs_talkv() where:

        ret = read_reply(h, &msg.type, len);
        if (!ret)
                goto fail;

        mutex_unlock(&h->request_mutex);

        sigaction(SIGPIPE, &oldact, NULL);
        if (msg.type == XS_ERROR) {
                saved_errno = get_error(ret); <=========

Ian(s), any thoughts/ideas?

thanks
mukesh

Re: Issues with latest staging tree starting PV guest.

[Roger Pau Monné]

On 05/03/14 02:49, Mukesh Rathor wrote:
> On Mon, 3 Mar 2014 18:52:39 -0800
> Mukesh Rathor <mukesh.rathor@oracle.com> wrote:
> 
>> Hi,
> ... 
>>   3. I setup xenbr, added vif line as always:
>>         vif = ['type=netfront,bridge=xenbr0']
>>
>>     But now get:
>>      libxl: error: libxl_create.c:1226:domcreate_attach_vtpms: unable
>>     to add nic devices
> 
> This is how far I got debugging this:
> 
> Looks like the issue is not getting domid, because stepping thru I see
> in devstate_watch_callback() it goes thru:
> 
>     } else {
>         int got = atoi(sstate);
>         if (got == ds->wanted) {
>             LIBXL__LOG(CTX, LIBXL__LOG_DEBUG, "backend %s wanted state %d ok",
>                        watch_path, ds->wanted);
>             rc = 0;
> logging:
> libxl: debug: libxl_event.c:653:devstate_watch_callback: backend /local/domain/0/backend/vif/5/0/state wanted state 2 ok
> 
> then it calls:
> ...
> ds->callback(egc, ds, rc); --> device_backend_callback --> device_hotplug
> which does:
> ...
>     rc = libxl__get_domid(gc, &domid);  ---> libxl__xs_read_checked
>      --> libxl__xs_read (, , path="domid")...
>       This returns result == 0 and errno == ENOENT.
> 
> logging:
> libxl: debug: libxl_dm.c:1473:kill_device_model: Device Model signaled
> libxl: error: libxl_device.c:780:libxl__initiate_device_remove: unable to get my domid
> 
> However, xenstore-ls, I see domid being there:
>   bash> xenstore-ls |grep domid
>      domid = "11"

What's missing is the domid from Dom0, not the domid of the guest. You
are missing an entry like:

/local/domain/0/domid = "0"

> 
> Zooming in libxl__xs_read, I got as far as :
> 
> libxl__xs_read -> xs_read((gc=0x636880, t=0x0, path=0x7ffff7bc13ba "domid")
>  --> .. --> xs_talkv() where:
> 
>         ret = read_reply(h, &msg.type, len);
>         if (!ret)
>                 goto fail;
> 
>         mutex_unlock(&h->request_mutex);
> 
>         sigaction(SIGPIPE, &oldact, NULL);
>         if (msg.type == XS_ERROR) {
>                 saved_errno = get_error(ret); <=========
> 
> Ian(s), any thoughts/ideas?

You need to update your init scripts, since commit
02ebea7768fe723deaa56c2cfcb860874544b47d Dom0 domid is written to
xenstore by xencommons init script, and libxl will fail to work properly
if the domid of Dom0 is not found.

Roger.

Re: Issues with latest staging tree starting PV guest.

[Mukesh Rathor]

On Wed, 5 Mar 2014 10:11:35 +0100
Roger Pau Monné <roger.pau@citrix.com> wrote:

> On 05/03/14 02:49, Mukesh Rathor wrote:
> > On Mon, 3 Mar 2014 18:52:39 -0800
> > Mukesh Rathor <mukesh.rathor@oracle.com> wrote:
> > 
....
> > then it calls:
> > ...
> > ds->callback(egc, ds, rc); --> device_backend_callback -->
> > device_hotplug which does:
> > ...
> >     rc = libxl__get_domid(gc, &domid);  ---> libxl__xs_read_checked
> >      --> libxl__xs_read (, , path="domid")...
> >       This returns result == 0 and errno == ENOENT.
> > 
> > logging:
> > libxl: debug: libxl_dm.c:1473:kill_device_model: Device Model
> > signaled libxl: error:
> > libxl_device.c:780:libxl__initiate_device_remove: unable to get my
> > domid
> > 
> > However, xenstore-ls, I see domid being there:
> >   bash> xenstore-ls |grep domid
> >      domid = "11"
> 
> What's missing is the domid from Dom0, not the domid of the guest. You
> are missing an entry like:
> 
> /local/domain/0/domid = "0"

Right...

> > 
> > Zooming in libxl__xs_read, I got as far as :
> > 
> > libxl__xs_read -> xs_read((gc=0x636880, t=0x0, path=0x7ffff7bc13ba
> > "domid") --> .. --> xs_talkv() where:
> > 
> >         ret = read_reply(h, &msg.type, len);
> >         if (!ret)
> >                 goto fail;
> > 
> >         mutex_unlock(&h->request_mutex);
> > 
> >         sigaction(SIGPIPE, &oldact, NULL);
> >         if (msg.type == XS_ERROR) {
> >                 saved_errno = get_error(ret); <=========
> > 
> > Ian(s), any thoughts/ideas?
> 
> You need to update your init scripts, since commit
> 02ebea7768fe723deaa56c2cfcb860874544b47d Dom0 domid is written to
> xenstore by xencommons init script, and libxl will fail to work
> properly if the domid of Dom0 is not found.

Ah, I already had seen your post googling for error and already made
sure I had latest init script. The problem was xencommons was not running
automatically. Once I added the sevice via systemctl, it works now. Thanks 
for the tip.

Mukesh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel