After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[erin.balid]

Hi All.

I posted this over on the OpenSUSE Virtual mailing list, since I use
that distro.  But I found on the Xen wiki site,

  http://wiki.xen.org/wiki/MigrationGuideToXen4.1%2B#Toolstack_upgrade_notes

so I guess we should post xm->xl migration configuration problems here
too, or instead.


I inherited an OpenSUSE Version 11.4 Xen host machine here at work.

Without too much trouble I finished upgrading it to Version 12.1.

The old setup was running using the Xen "xm" toolstack and works like
you'd expect.  The new setup works ok if I use the "xm" stack as well.

I learned that we should move to the "xl" toolstack.  I'm trying to do
that now and having some problems with networking, but only with the xl
approach.

I've been running up to date Xen for awhile.

rpm -qa | grep -i xen
 xen-tools-4.1.2_11-164.5.x86_64
 xen-doc-html-4.1.2_11-164.5.x86_64
 xen-libs-4.1.2_11-164.5.x86_64
 xen-4.1.2_11-164.5.x86_64
 patterns-openSUSE-xen_server-12.1-25.21.1.x86_64
 xen-devel-4.1.2_11-164.5.x86_64
 xen-doc-pdf-4.1.2_11-164.5.x86_64
 kernel-xen-devel-3.1.0-1.2.1.x86_64
 kernel-xen-3.1.0-1.2.1.x86_64

I already setup the networking on the Host to use OpenSUSE
configuration.

brctl show
 bridge name     bridge id               STP enabled     interfaces
 br0             8000.0052351d5337       no              eth0

cat /etc/sysconfig/network/ifcfg-br0  
 STARTMODE='auto'
 BOOTPROTO='static'
 BRIDGE='yes'
 BRIDGE_PORTS='eth0'
 BRIDGE_FORWARDDELAY='0'
 BRIDGE_STP='off'

 NAME='Motherboard'
 IPADDR='192.168.1.32/24'
 BROADCAST=''
 REMOTE_IPADDR=''
 NETWORK=''
 MTU=''
 ETHTOOL_OPTIONS=''
 USERCONTROL='no'
 NM_CONTROLLED='no'

cat /etc/sysconfig/network/ifcfg-eth0
 STARTMODE='manual'
 BOOTPROTO='none'
 BRIDGE='no'

The Guest cfg file I'm using for testing is:

cat test.cfg
 name='test'
 builder='linux'
 bootloader='/usr/bin/pygrub'
 disk=['phy:/dev/XenVols/test,xvda,w']
 vif=[mac=00:16:3e:12:34:01, bridge=br0  , vifname=vifBR']
 vfb=[ 'type=vnc, vncdisplay=1, vnclisten=127.0.0.1']
 extra='textmode=1 xencons=xvc0 noirqdebug'
 on_shutdown='destroy'
 on_reboot='restart'
 on_crash='destroy'
 vcpus=2
 maxmem=2048
 memory=2048

With a clean-booted Host and no launched Guests,

xl list
 Name                                        ID   Mem VCPUs      State  
 Time(s)
 Domain-0                                     0  1010     4     r-----  
    86.3

xl create -c test.cfg
	    pyGRUB  version 0.6
	 ┌────────────────────────────────────────────────────────────────────────┐
	 │ Xen4 12.1 DomU                                               
	          │
	 │                                                              
	          │
	 │                                                              
	          │
	 │                                                              
	          │
	 │                                                              
	          │
	 │                                                              
	          │
	 │                                                              
	          │
	 │                                                              
	          │
	 └────────────────────────────────────────────────────────────────────────┘
	     Use the ^ and v keys to select which entry is highlighted.
	     Press enter to boot the selected OS, 'e' to edit the
	     commands before booting, 'a' to modify the kernel arguments
	     before booting, or 'c' for a command line.

	     Will boot selected entry in  1 seconds


	Daemon running with PID 6779

I get no other output here (why not?) on the console for about 30
seconds. Then just,

 Welcome to openSUSE 12.1 "Asparagus" - Kernel 3.1.0-1.2-xen (xvc0).

 testserver login:


The output of 'tail -f /var/log/messages /var/log/xen/*log' is just

	==> /var/log/messages <==
	Jan 21 10:40:39 testserver logger: /etc/xen/scripts/block: add
	XENBUS_PATH=backend/vbd/5/51712
	Jan 21 10:40:39 testserver logger: /etc/xen/scripts/block: add
	XENBUS_PATH=backend/vbd/5/51728
	Jan 21 10:40:39 testserver logger: /etc/xen/scripts/block: add
	XENBUS_PATH=backend/vbd/5/51744
	Jan 21 10:40:40 testserver logger: /etc/xen/scripts/vif-bridge:
	online type_if=vif XENBUS_PATH=backend/vif/5/0
	Jan 21 10:40:40 testserver logger: /etc/xen/scripts/vif-bridge:
	No device details in backend/vif/5/0, exiting.
	Jan 21 10:40:40 testserver logger: /etc/xen/scripts/vif-bridge:
	online XENBUS_PATH=backend/vif/5/0
	Jan 21 10:40:40 testserver logger: /etc/xen/scripts/vif-bridge:
	No device details in backend/vif/5/0, exiting.
	Jan 21 10:40:40 testserver kernel: [ 2711.951822] blkback:
	ring-ref 8, event-channel 12, protocol 1 (x86_64-abi)
	Jan 21 10:40:40 testserver kernel: [ 2712.015731] blkback:
	ring-ref 9, event-channel 13, protocol 1 (x86_64-abi)
	Jan 21 10:40:40 testserver kernel: [ 2712.066467] blkback:
	ring-ref 10, event-channel 14, protocol 1 (x86_64-abi)


I can login to the guest at the console and the Guest's networking looks
like it's recognized OK.

dmesg | grep eth
[    0.896597] netfront: Initialising virtual ethernet driver.

hwinfo --network
05: None 00.0: 10700 Loopback
  [Created at net.124]
  Unique ID: ZsBS.GQNx7L4uPNA
  SysFS ID: /class/net/lo
  Hardware Class: network interface
  Model: "Loopback network interface"
  Device File: lo
  Link detected: yes
  Config Status: cfg=new, avail=yes, need=no, active=unknown

06: None 00.0: 10701 Ethernet
  [Created at net.124]
  Unique ID: usDW.ndpeucax6V1
  Parent ID: +jsg.Sd+ykfyvlK4
  SysFS ID: /class/net/eth0
  SysFS Device Link: /devices/xen/vif-0
  Hardware Class: network interface
  Model: "Ethernet network interface"
  Driver: "vif"
  Driver Modules: "xennet"
  Device File: eth0
  HW Address: 00:16:3e:12:34:01
  Link detected: yes
  Config Status: cfg=new, avail=yes, need=no, active=unknown
  Attached to: #4 (Ethernet controller)

The Guest's network routes look OK.

route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use
Iface
default         192.168.1.1     0.0.0.0         UG    0      0        0
eth0
loopback        *               255.0.0.0       U     0      0        0
lo
link-local      *               255.255.0.0     U     0      0        0
eth0
192.168.1.0     *               255.255.255.0   U     0      0        0
eth0

But when I try to ping anywhere off the Guest I get HostUnreachable
errors.

ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
From 192.168.1.32 icmp_seq=1 Destination Host Unreachable
From 192.168.1.32 icmp_seq=2 Destination Host Unreachable
From 192.168.1.32 icmp_seq=3 Destination Host Unreachable
etc etc

It feels like I'm pretty close to getting this working.  Am I still
missing some part of the configuration changes needed for using the xl
toolstack?

Regards,

Erin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Roger Pau Monné]

2012/1/21  <erin.balid@inoutbox.com>:
> Hi All.
>
> I posted this over on the OpenSUSE Virtual mailing list, since I use
> that distro.  But I found on the Xen wiki site,
>
>  http://wiki.xen.org/wiki/MigrationGuideToXen4.1%2B#Toolstack_upgrade_notes
>
> so I guess we should post xm->xl migration configuration problems here
> too, or instead.
>
>
> I inherited an OpenSUSE Version 11.4 Xen host machine here at work.
>
> Without too much trouble I finished upgrading it to Version 12.1.
>
> The old setup was running using the Xen "xm" toolstack and works like
> you'd expect.  The new setup works ok if I use the "xm" stack as well.
>
> I learned that we should move to the "xl" toolstack.  I'm trying to do
> that now and having some problems with networking, but only with the xl
> approach.
>
> I've been running up to date Xen for awhile.
>
> rpm -qa | grep -i xen
>  xen-tools-4.1.2_11-164.5.x86_64
>  xen-doc-html-4.1.2_11-164.5.x86_64
>  xen-libs-4.1.2_11-164.5.x86_64
>  xen-4.1.2_11-164.5.x86_64
>  patterns-openSUSE-xen_server-12.1-25.21.1.x86_64
>  xen-devel-4.1.2_11-164.5.x86_64
>  xen-doc-pdf-4.1.2_11-164.5.x86_64
>  kernel-xen-devel-3.1.0-1.2.1.x86_64
>  kernel-xen-3.1.0-1.2.1.x86_64
>
> I already setup the networking on the Host to use OpenSUSE
> configuration.
>
> brctl show
>  bridge name     bridge id               STP enabled     interfaces
>  br0             8000.0052351d5337       no              eth0
>
> cat /etc/sysconfig/network/ifcfg-br0
>  STARTMODE='auto'
>  BOOTPROTO='static'
>  BRIDGE='yes'
>  BRIDGE_PORTS='eth0'
>  BRIDGE_FORWARDDELAY='0'
>  BRIDGE_STP='off'
>
>  NAME='Motherboard'
>  IPADDR='192.168.1.32/24'
>  BROADCAST=''
>  REMOTE_IPADDR=''
>  NETWORK=''
>  MTU=''
>  ETHTOOL_OPTIONS=''
>  USERCONTROL='no'
>  NM_CONTROLLED='no'
>
> cat /etc/sysconfig/network/ifcfg-eth0
>  STARTMODE='manual'
>  BOOTPROTO='none'
>  BRIDGE='no'
>
> The Guest cfg file I'm using for testing is:
>
> cat test.cfg
>  name='test'
>  builder='linux'
>  bootloader='/usr/bin/pygrub'
>  disk=['phy:/dev/XenVols/test,xvda,w']
>  vif=[mac=00:16:3e:12:34:01, bridge=br0  , vifname=vifBR']

The xl toolstack present in version 4.1.2 doesn't support vifname[0]
(newer versions do), so I think if you remove that it should be ok.

[0] http://lists.xen.org/archives/html/xen-users/2011-12/msg00225.html

>  vfb=[ 'type=vnc, vncdisplay=1, vnclisten=127.0.0.1']
>  extra='textmode=1 xencons=xvc0 noirqdebug'
>  on_shutdown='destroy'
>  on_reboot='restart'
>  on_crash='destroy'
>  vcpus=2
>  maxmem=2048
>  memory=2048
>
> With a clean-booted Host and no launched Guests,
>
> xl list
>  Name                                        ID   Mem VCPUs      State
>  Time(s)
>  Domain-0                                     0  1010     4     r-----
>    86.3
>
> xl create -c test.cfg
>            pyGRUB  version 0.6
>         ┌────────────────────────────────────────────────────────────────────────┐
>         │ Xen4 12.1 DomU
>                  │
>         │
>                  │
>         │
>                  │
>         │
>                  │
>         │
>                  │
>         │
>                  │
>         │
>                  │
>         │
>                  │
>         └────────────────────────────────────────────────────────────────────────┘
>             Use the ^ and v keys to select which entry is highlighted.
>             Press enter to boot the selected OS, 'e' to edit the
>             commands before booting, 'a' to modify the kernel arguments
>             before booting, or 'c' for a command line.
>
>             Will boot selected entry in  1 seconds
>
>
>        Daemon running with PID 6779
>
> I get no other output here (why not?) on the console for about 30
> seconds. Then just,
>
>  Welcome to openSUSE 12.1 "Asparagus" - Kernel 3.1.0-1.2-xen (xvc0).
>
>  testserver login:
>
>
> The output of 'tail -f /var/log/messages /var/log/xen/*log' is just
>
>        ==> /var/log/messages <==
>        Jan 21 10:40:39 testserver logger: /etc/xen/scripts/block: add
>        XENBUS_PATH=backend/vbd/5/51712
>        Jan 21 10:40:39 testserver logger: /etc/xen/scripts/block: add
>        XENBUS_PATH=backend/vbd/5/51728
>        Jan 21 10:40:39 testserver logger: /etc/xen/scripts/block: add
>        XENBUS_PATH=backend/vbd/5/51744
>        Jan 21 10:40:40 testserver logger: /etc/xen/scripts/vif-bridge:
>        online type_if=vif XENBUS_PATH=backend/vif/5/0
>        Jan 21 10:40:40 testserver logger: /etc/xen/scripts/vif-bridge:
>        No device details in backend/vif/5/0, exiting.
>        Jan 21 10:40:40 testserver logger: /etc/xen/scripts/vif-bridge:
>        online XENBUS_PATH=backend/vif/5/0
>        Jan 21 10:40:40 testserver logger: /etc/xen/scripts/vif-bridge:
>        No device details in backend/vif/5/0, exiting.
>        Jan 21 10:40:40 testserver kernel: [ 2711.951822] blkback:
>        ring-ref 8, event-channel 12, protocol 1 (x86_64-abi)
>        Jan 21 10:40:40 testserver kernel: [ 2712.015731] blkback:
>        ring-ref 9, event-channel 13, protocol 1 (x86_64-abi)
>        Jan 21 10:40:40 testserver kernel: [ 2712.066467] blkback:
>        ring-ref 10, event-channel 14, protocol 1 (x86_64-abi)
>
>
> I can login to the guest at the console and the Guest's networking looks
> like it's recognized OK.
>
> dmesg | grep eth
> [    0.896597] netfront: Initialising virtual ethernet driver.
>
> hwinfo --network
> 05: None 00.0: 10700 Loopback
>  [Created at net.124]
>  Unique ID: ZsBS.GQNx7L4uPNA
>  SysFS ID: /class/net/lo
>  Hardware Class: network interface
>  Model: "Loopback network interface"
>  Device File: lo
>  Link detected: yes
>  Config Status: cfg=new, avail=yes, need=no, active=unknown
>
> 06: None 00.0: 10701 Ethernet
>  [Created at net.124]
>  Unique ID: usDW.ndpeucax6V1
>  Parent ID: +jsg.Sd+ykfyvlK4
>  SysFS ID: /class/net/eth0
>  SysFS Device Link: /devices/xen/vif-0
>  Hardware Class: network interface
>  Model: "Ethernet network interface"
>  Driver: "vif"
>  Driver Modules: "xennet"
>  Device File: eth0
>  HW Address: 00:16:3e:12:34:01
>  Link detected: yes
>  Config Status: cfg=new, avail=yes, need=no, active=unknown
>  Attached to: #4 (Ethernet controller)
>
> The Guest's network routes look OK.
>
> route
> Kernel IP routing table
> Destination     Gateway         Genmask         Flags Metric Ref    Use
> Iface
> default         192.168.1.1     0.0.0.0         UG    0      0        0
> eth0
> loopback        *               255.0.0.0       U     0      0        0
> lo
> link-local      *               255.255.0.0     U     0      0        0
> eth0
> 192.168.1.0     *               255.255.255.0   U     0      0        0
> eth0
>
> But when I try to ping anywhere off the Guest I get HostUnreachable
> errors.
>
> ping 192.168.1.1
> PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
> From 192.168.1.32 icmp_seq=1 Destination Host Unreachable
> From 192.168.1.32 icmp_seq=2 Destination Host Unreachable
> From 192.168.1.32 icmp_seq=3 Destination Host Unreachable
> etc etc
>
> It feels like I'm pretty close to getting this working.  Am I still
> missing some part of the configuration changes needed for using the xl
> toolstack?
>
> Regards,
>
> Erin
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[erin.balid]

Hi Roger,

On Sat, Jan 21, 2012, at 11:57 PM, Roger Pau Monné wrote:
> 2012/1/21  <erin.balid@inoutbox.com>:
> >  vif=[mac=00:16:3e:12:34:01, bridge=br0  , vifname=vifBR']
> 
> The xl toolstack present in version 4.1.2 doesn't support vifname[0]
> (newer versions do), so I think if you remove that it should be ok.
> 
> [0] http://lists.xen.org/archives/html/xen-users/2011-12/msg00225.html

That looked hopeful.

I changed

 - vif=[mac=00:16:3e:12:34:01, bridge=br0  , vifname=vifBR']
 + vif=[mac=00:16:3e:12:34:01, bridge=br0']

and launched again.  I still ended up with no attached VIF interface.

brctl show
 bridge name     bridge id               STP enabled     interfaces
 br0             8000.0052351d5337       yes             eth0

xl list-vm template
 UUID                                  ID    name
 7d46ca9a-9d86-476e-948d-3eddf5d070e4  4    test

xl network-list template
 Idx BE Mac Addr.         handle state evt-ch   tx-/rx-ring-ref BE-path
 0   0  00:16:3e:12:34:01      0     4     15   768/769        
 /local/domain/0/backend/vif/4/0

And, I still can't ping just like my orginal post.

:-(

Regards,

Erin

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[erin.balid]

Hi All.

I wanted to do an apples-to-apples comparison of "xm" and "xl".

Using a config file VIF of

	vif=['mac=00:16:3E:12:34:01,bridge=br0']

If I turn on  xend and start the Guest with 'xm create', networking at
the Guest is OK.
If I turn off xend and start the Guest with 'xl create', networking at
the Guest fails, I think because the Guest's VIF doesn't get assigned to
the Host bridge.


service xend start
	Starting xend                                                   
	                                         done

brctl show
	bridge name     bridge id               STP enabled    
	interfaces
	br0             8000.0052351d5337       yes             eth0

xm list
	Name                                        ID   Mem VCPUs     
	State   Time(s)
	Domain-0                                     0  1010     1    
	r-----    169.1

xm create /etc/xen/vm/tmpl_run.cfg

xm list
	Name                                        ID   Mem VCPUs     
	State   Time(s)
	Domain-0                                     0  1010     1    
	r-----    175.2
	test                                         6  1024     2    
	-b----      8.7

brctl show
	bridge name     bridge id               STP enabled    
	interfaces
	br0             8000.0052351d5337       yes             eth0
	                                                        vif6.0

xm console test    
	Welcome to openSUSE 12.1 "Asparagus" - Kernel 3.1.0-1.2-xen
	(xvc0).
	testserver login: test
	Password:
	Last login: Sat Jan 21 14:57:14 PST 2012 on xvc0
	You have new mail.
	Have a lot of fun...

ping -c 2 192.168.1.1
	PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
	64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.500 ms
	64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.388 ms
	--- 192.168.1.1 ping statistics ---
	2 packets transmitted, 2 received, 0% packet loss, time 999ms
	rtt min/avg/max/mdev = 0.388/0.444/0.500/0.056 ms

shutdown -h now
	The system is going down for system halt NOW! (Sat Jan 21
	16:34:48 2012):
	...
	[  147.792590] System halted.

service xend stop

brctl show
	bridge name     bridge id               STP enabled    
	interfaces
	br0             8000.0052351d5337       yes             eth0

xl list
	Name                                        ID   Mem VCPUs     
	State   Time(s)
	Domain-0                                     0  1010     1    
	r-----     182.8

xl create /etc/xen/vm/tmpl_run.cfg

xl list
	Name                                        ID   Mem VCPUs     
	State   Time(s)
	Domain-0                                     0  1010     1    
	r-----     186.6
	test                                         7  1024     2    
	-b----       6.9

brctl show
	bridge name     bridge id               STP enabled    
	interfaces
	br0             8000.0052351d5337       yes             eth0

xl console test    
	Welcome to openSUSE 12.1 "Asparagus" - Kernel 3.1.0-1.2-xen
	(xvc0).
	testserver login: test
	Password:
	Last login: Sat Jan 21 16:34:08 PST 2012 on xvc0
	You have new mail.
	Have a lot of fun...

ping -c 2 192.168.1.1
	PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
	From 192.168.1.32 icmp_seq=1 Destination Host Unreachable
	From 192.168.1.32 icmp_seq=2 Destination Host Unreachable
	--- 192.168.1.1 ping statistics ---
	2 packets transmitted, 0 received, +2 errors, 100% packet loss,
	time 1001ms
	pipe 2

Regards,

Erin

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Roger Pau Monné]

2012/1/22  <erin.balid@inoutbox.com>:
> Hi Roger,
>
> On Sat, Jan 21, 2012, at 11:57 PM, Roger Pau Monné wrote:
>> 2012/1/21  <erin.balid@inoutbox.com>:
>> >  vif=[mac=00:16:3e:12:34:01, bridge=br0  , vifname=vifBR']
>>
>> The xl toolstack present in version 4.1.2 doesn't support vifname[0]
>> (newer versions do), so I think if you remove that it should be ok.
>>
>> [0] http://lists.xen.org/archives/html/xen-users/2011-12/msg00225.html
>
> That looked hopeful.
>
> I changed
>
>  - vif=[mac=00:16:3e:12:34:01, bridge=br0  , vifname=vifBR']
>  + vif=[mac=00:16:3e:12:34:01, bridge=br0']
>
> and launched again.  I still ended up with no attached VIF interface.
>
> brctl show
>  bridge name     bridge id               STP enabled     interfaces
>  br0             8000.0052351d5337       yes             eth0
>
> xl list-vm template
>  UUID                                  ID    name
>  7d46ca9a-9d86-476e-948d-3eddf5d070e4  4    test
>
> xl network-list template
>  Idx BE Mac Addr.         handle state evt-ch   tx-/rx-ring-ref BE-path
>  0   0  00:16:3e:12:34:01      0     4     15   768/769
>  /local/domain/0/backend/vif/4/0
>
> And, I still can't ping just like my orginal post.

Can you post "ifconfig -a" output from the DomU when launched with xl.
Also paste the output from "xenstore-ls -fp" on the Dom0 when the xl
DomU is up and running. From what I saw above in /var/log/messages the
problem seem to be related to hotplug scripts.

> :-(
>
> Regards,
>
> Erin
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[erin.balid]

Hi Roger.

Here is "ifconfig -a" output from the DomU when launched with xm.

xm create test.cfg
xm list
 Name                                        ID   Mem VCPUs      State  
 Time(s)
 Domain-0                                     0  1010     1     r-----  
  913.4
 template                                     1  2048     2     -b----  
   86.0
xm console template
 ...
ifconfig -a
 eth0      Link encap:Ethernet  HWaddr 00:16:3E:12:34:01
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:38658 errors:0 dropped:24935 overruns:0 frame:0
           TX packets:11009 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:2621879 (2.5 Mb)  TX bytes:10432153 (9.9 Mb)
 
 eth0:1    Link encap:Ethernet  HWaddr 00:16:3E:12:34:01
           inet addr:192.168.1.32   Bcast:192.168.1.255 
           Mask:255.255.255.0
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
 
 lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:7 errors:0 dropped:0 overruns:0 frame:0
           TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:563 (563.0 b)  TX bytes:563 (563.0 b)


And, "ifconfig -a" output from the DomU when launched with xl.

xl create test.cfg
xl list
 Name                                        ID   Mem VCPUs      State  
 Time(s)
 Domain-0                                     0  1010     1     r-----  
    66.8
 template                                     1  2048     2     -b----  
     6.8

ifconfig -a
 eth0      Link encap:Ethernet  HWaddr 00:16:3E:12:34:01
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:114 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:0 (0.0 b)  TX bytes:5241 (5.1 Kb)
 
 eth0:1    Link encap:Ethernet  HWaddr 00:16:3E:12:34:01
           inet addr:192.168.1.32   Bcast:192.168.1.255 
           Mask:255.255.255.0
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
 
 lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:52 errors:0 dropped:0 overruns:0 frame:0
           TX packets:52 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:4431 (4.3 Kb)  TX bytes:4431 (4.3 Kb)

> Also paste the output from "xenstore-ls -fp" on the Dom0 when the xl
> DomU is up and running. From what I saw above in /var/log/messages the
> problem seem to be related to hotplug scripts.

Here is the paste of that output. ===> http://pastebin.com/s0e03VnW


On Sun, Jan 22, 2012, at 10:49 AM, djmagee@mageenet.net wrote:
> You're missing the opening single quote on your vif line.

That's just a copy/paste typo in the email.

Regards,

Erin

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Roger Pau Monné]

2012/1/22  <erin.balid@inoutbox.com>:
> Hi Roger.
>
> Here is "ifconfig -a" output from the DomU when launched with xm.
>
> xm create test.cfg
> xm list
>  Name                                        ID   Mem VCPUs      State
>  Time(s)
>  Domain-0                                     0  1010     1     r-----
>  913.4
>  template                                     1  2048     2     -b----
>   86.0
> xm console template
>  ...
> ifconfig -a
>  eth0      Link encap:Ethernet  HWaddr 00:16:3E:12:34:01
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:38658 errors:0 dropped:24935 overruns:0 frame:0
>           TX packets:11009 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:2621879 (2.5 Mb)  TX bytes:10432153 (9.9 Mb)
>
>  eth0:1    Link encap:Ethernet  HWaddr 00:16:3E:12:34:01
>           inet addr:192.168.1.32   Bcast:192.168.1.255
>           Mask:255.255.255.0
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>
>  lo        Link encap:Local Loopback
>           inet addr:127.0.0.1  Mask:255.0.0.0
>           UP LOOPBACK RUNNING  MTU:16436  Metric:1
>           RX packets:7 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:563 (563.0 b)  TX bytes:563 (563.0 b)
>
>
> And, "ifconfig -a" output from the DomU when launched with xl.
>
> xl create test.cfg
> xl list
>  Name                                        ID   Mem VCPUs      State
>  Time(s)
>  Domain-0                                     0  1010     1     r-----
>    66.8
>  template                                     1  2048     2     -b----
>     6.8
>
> ifconfig -a
>  eth0      Link encap:Ethernet  HWaddr 00:16:3E:12:34:01
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:114 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:0 (0.0 b)  TX bytes:5241 (5.1 Kb)
>
>  eth0:1    Link encap:Ethernet  HWaddr 00:16:3E:12:34:01
>           inet addr:192.168.1.32   Bcast:192.168.1.255
>           Mask:255.255.255.0
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>
>  lo        Link encap:Local Loopback
>           inet addr:127.0.0.1  Mask:255.0.0.0
>           UP LOOPBACK RUNNING  MTU:16436  Metric:1
>           RX packets:52 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:52 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:4431 (4.3 Kb)  TX bytes:4431 (4.3 Kb)
>
>> Also paste the output from "xenstore-ls -fp" on the Dom0 when the xl
>> DomU is up and running. From what I saw above in /var/log/messages the
>> problem seem to be related to hotplug scripts.
>
> Here is the paste of that output. ===> http://pastebin.com/s0e03VnW


Apart from the fact that you have quite a lot of junk in xenstore I
don't see anything strange, devices seem to be connected ok. I've
realized that you have installed
patterns-openSUSE-xen_server-12.1-25.21.1.x86_64, what's this? I've
never used Xen with OpenSUSE, but this seems related to XenServer
comercial version, not open source Xen. Maybe someone familiar with
Xen on OpenSUSE can provide more helpful information.

The only strange thing I see are the hotplug error messages in
/var/log/messages, but I don't know what could cause them. Maybe old
udev rules or hotplug scripts that didn't get cleaned when updating?

> On Sun, Jan 22, 2012, at 10:49 AM, djmagee@mageenet.net wrote:
>> You're missing the opening single quote on your vif line.
>
> That's just a copy/paste typo in the email.
>
> Regards,
>
> Erin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[erin.balid]

Hi Roger.

On Sun, Jan 22, 2012, at 06:28 PM, Roger Pau Monné wrote:
> Apart from the fact that you have quite a lot of junk in xenstore I
> don't see anything strange, devices seem to be connected ok. I've
> realized that you have installed
> patterns-openSUSE-xen_server-12.1-25.21.1.x86_64, what's this? I've
> never used Xen with OpenSUSE, but this seems related to XenServer
> comercial version, not open source Xen. Maybe someone familiar with
> Xen on OpenSUSE can provide more helpful information.

That package is a meta-package for the xen_server package. 'patterns' on
OpenSuse are just functional collections of packages.

zypper info patterns-openSUSE-xen_server
	Information for package patterns-openSUSE-xen_server:

	Repository: OS12-oss
	Name: patterns-openSUSE-xen_server
	Version: 12.1-25.21.1
	Arch: x86_64
	Vendor: openSUSE
	Installed: Yes
	Status: up-to-date
	Installed Size: 1.0 KiB
	Summary: Meta package for pattern xen_server
	Description:
	This package is installed if a pattern is selected to have a
	working update path

zypper info -t pattern xen_server
	Information for pattern xen_server:

	Repository: OS12-oss
	Name: xen_server
	Version: 12.1-25.21.1
	Arch: x86_64
	Vendor: openSUSE
	Installed: Yes
	Summary: Xen Virtual Machine Host Server
	Description:
	Software to set up a server for configuring, managing, and
	monitoring virtual machines on a single physical machine.
	Contents:

	S | Name                         | Type    | Dependency
	--+------------------------------+---------+-----------
	i | kernel-xen                   | package |
	i | xen-tools                    | package |
	  | vm-install                   | package |
	i | yast2-vm                     | package |
	  | virt-manager                 | package |
	i | patterns-openSUSE-xen_server | package |
	i | bridge-utils                 | package |
	i | install-initrd               | package |
	  | virt-viewer                  | package |
	i | xterm                        | package |
	i | xen                          | package |
	i | xen-doc-html                 | package |
	i | xen-doc-pdf                  | package |
	i | xen-libs                     | package |

It has nothing to do with commercial Xen.

> The only strange thing I see are the hotplug error messages in
> /var/log/messages, but I don't know what could cause them. Maybe old
> udev rules or hotplug scripts that didn't get cleaned when updating?

I know this machine was originally installed with at least opensuse
11.4. I upgraded it from 11.4 ==> 12.1.  I don't know if it was
originally installed on 11.4, or sommthing even earlier and then
upgraded.

Short of reinstalling the machine from scratch, I don't know to clean
things up.  It sounds like it's more that just deleting the xenstore
database somehow.

I found this post from a year ago by an OpenSuse Xen developer -

http://lists.opensuse.org/opensuse-features/2011-03/msg00212.html

"Xen 4.1.0 (due February 2011) will be the first release in which thenew
xl/libxl (aka libxenlight [1]) toolstack will be the default,preferred
tool set. Although still included in the release, the legacyxm/xend
toolstack will be disabled and in process of deprecation."

That was for 4.1.0. The current version is 4.1.2.  The post reads then
like we already should be using xl in OpenSuse.  Looks like the author
posts on this list too.  I'll CC him directly on just this email. Maybe
he can participate here a bit and has some ideas.

Regards,

Erin

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Jim Fehlig]

erin.balid@inoutbox.com wrote:
> I found this post from a year ago by an OpenSuse Xen developer -
>
> http://lists.opensuse.org/opensuse-features/2011-03/msg00212.html
>   

A mailing list to discuss openSUSE features, not an authoritative
resource on functionality in any given openSUSE release...

> "Xen 4.1.0 (due February 2011) will be the first release in which thenew
> xl/libxl (aka libxenlight [1]) toolstack will be the default,preferred
> tool set. Although still included in the release, the legacyxm/xend
> toolstack will be disabled and in process of deprecation."
>
> That was for 4.1.0. The current version is 4.1.2.  The post reads then
> like we already should be using xl in OpenSuse.

That talks about upstream xen, not xen packaging in openSUSE.  xl/libxl
is the default, preferred toolstack in upstream Xen 4.1.x.

>   Looks like the author
> posts on this list too.  I'll CC him directly on just this email. Maybe
> he can participate here a bit and has some ideas.
>   

Well, our plan was to switch to xl/libxl in openSUSE12.1, but we
switched back to the legacy toolstack before 12.1 released.  In the end,
we didn't have time to ensure xl/libxl was baked enough serve as a
replacement for xm/xend, let alone ensuring the tools consuming libxl
(e.g. libvirt) were feature parity with the legacy toolstack
counterparts.  Hopefully we will have time to do this during
openSUSE12.2 development cycle.

BTW, I can reproduce your problem and will take a look as time permits.

Regards,
Jim

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[erin.balid]

Hi Jim.

Thanks for chiming in.

On Tue, Jan 24, 2012, at 04:13 PM, Jim Fehlig wrote:
> > http://lists.opensuse.org/opensuse-features/2011-03/msg00212.html
> 
> A mailing list to discuss openSUSE features, not an authoritative
> resource on functionality in any given openSUSE release...

It's been a big challenge finding anything that IS "an authoritative
resource on functionality" for Xen in Opensuse.

Where is the Opensuse-specific and Up-to-Date Xen documentation?  I
guess we cant rely on xen.org docs for Opensuse's Xen?  Is it not really
closely tied to the xen-devel project then?

> Well, our plan was to switch to xl/libxl in openSUSE12.1, but we
> switched back to the legacy toolstack before 12.1 released.  In the end,
> we didn't have time to ensure xl/libxl was baked enough serve as a
> replacement for xm/xend, let alone ensuring the tools consuming libxl
> (e.g. libvirt) were feature parity with the legacy toolstack
> counterparts.  Hopefully we will have time to do this during
> openSUSE12.2 development cycle.
> 
> BTW, I can reproduce your problem and will take a look as time permits.

Okay, so that looks like the answer here.  At the moment, it doesn't
work on Opensuse.  Thanks for verifying you can reproduce it and looking
into it more.

That leaves me back at my earlier question -- What are good docs for Xen
on Opensuse?  I've wasted a lot of time - mine and other people's -
doing what I was told (RTFM!) just to find out that 'real' Xen manual is
the wrong one.

Regards,

Erin

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Jim Fehlig]

erin.balid@inoutbox.com wrote:
> Hi Jim.
>
> Thanks for chiming in.
>
> On Tue, Jan 24, 2012, at 04:13 PM, Jim Fehlig wrote:
>   
>>> http://lists.opensuse.org/opensuse-features/2011-03/msg00212.html
>>>       
>> A mailing list to discuss openSUSE features, not an authoritative
>> resource on functionality in any given openSUSE release...
>>     
>
> It's been a big challenge finding anything that IS "an authoritative
> resource on functionality" for Xen in Opensuse.
>
> Where is the Opensuse-specific and Up-to-Date Xen documentation?  I
> guess we cant rely on xen.org docs for Opensuse's Xen?  Is it not really
> closely tied to the xen-devel project then?
>   

Generally speaking, xen.org documentation applies to the xen packages in
openSUSE.  In the case of toolstacks, openSUSE still enables the legacy
toolstack by default, and all upstream documentation wrt the legacy
toolstack applies.  You can use the new toolstack as well, in which case
the upstream xl/libxl documentation applies - including recommendation
that the toolstacks should not be used in parallel.

What you have described in this thread is a bug in openSUSE12.1.  You
followed the documentation and it didn't work :-).

> That leaves me back at my earlier question -- What are good docs for Xen
> on Opensuse?  I've wasted a lot of time - mine and other people's -
> doing what I was told (RTFM!) just to find out that 'real' Xen manual is
> the wrong one.
>   

I'm not sure what the problem is with the manual.  You've hit a bug I
introduced in openSUSE.

Regards,
Jim

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[erin.balid]

Hi Jim.

On Wed, Jan 25, 2012, at 08:55 AM, Jim Fehlig wrote:
> Generally speaking, xen.org documentation applies to the xen packages in openSUSE.

Okay then.  That simplifies life.  Thanks.

What will be the best place to track fixes to this issue in
Opensuse-land?  Xen-users list?  Opensuse-virtual list? I'd like to get
off _this_ list if possible.  I don't really fit here.  Unless it really
is the right forum to follow.

I got your comment that you'd look for fixes in the Opensuse 12.2
time-frame.  As I read the openSUSE Roadmap, the scheduled release date
for openSUSE 12.2 is July 2012.

We won't switch to a new OS release until after it's in production.  We
_are_ willing to use a few non-production repositories for key packages.
 Xen's one of them.

Should we plan to see these Xen fixes in a shorter timeframe in a repo
built for use with 12.1 release/kernel prior to that 12.2 release date? 
Or to meet our requirements, would you - or others here - suggest we get
onto a different OS+Xen stack?

Regards,

Erin

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Jim Fehlig]

erin.balid@inoutbox.com wrote:
> Hi Jim.
>
> On Wed, Jan 25, 2012, at 08:55 AM, Jim Fehlig wrote:
>   
>> Generally speaking, xen.org documentation applies to the xen packages in openSUSE.
>>     
>
> Okay then.  That simplifies life.  Thanks.
>
> What will be the best place to track fixes to this issue in
> Opensuse-land?  Xen-users list?  Opensuse-virtual list?

bugzilla.novell.com

>  I'd like to get
> off _this_ list if possible.  I don't really fit here.  Unless it really
> is the right forum to follow.
>   

Yes, let's stop this thread here :-).  This is a downstream issue.

> I got your comment that you'd look for fixes in the Opensuse 12.2
> time-frame.  As I read the openSUSE Roadmap, the scheduled release date
> for openSUSE 12.2 is July 2012.
>
> We won't switch to a new OS release until after it's in production.  We
> _are_ willing to use a few non-production repositories for key packages.
>  Xen's one of them.
>
> Should we plan to see these Xen fixes in a shorter timeframe in a repo
> built for use with 12.1 release/kernel prior to that 12.2 release date?

We do maintenance on virtualization packages in openSUSE, so a fix for
this could be delivered to the openSUSE12.1 update repository.  But the
process starts with a bugzilla entry...

Regards,
Jim

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[djmagee]

> cat test.cfg
>  name='test'
>  builder='linux'
>  bootloader='/usr/bin/pygrub'
>  disk=['phy:/dev/XenVols/test,xvda,w']
>  vif=[mac=00:16:3e:12:34:01, bridge=br0  , vifname=vifBR']

You're missing the opening single quote on your vif line.  I'm not sure how the guest started at all, when I try it this way I get a parse error on xl create.

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Ian Campbell]

On Sat, 2012-01-21 at 20:06 +0000, erin.balid@inoutbox.com wrote:
> The Guest cfg file I'm using for testing is:
> 
> cat test.cfg
>  name='test'
>  builder='linux'
>  bootloader='/usr/bin/pygrub'
>  disk=['phy:/dev/XenVols/test,xvda,w']
>  vif=[mac=00:16:3e:12:34:01, bridge=br0  , vifname=vifBR']

ISTRT xl had a bug at one point where it would fail to strip whitespace
in the network KVP lists. i.e. the above ended up expecting to find a
bridge named "br0  ". Can you try without the spaces please?

(hmm, I wonder if that was ever fixed. I appear to have the following
tucked away in my queue. I suspect I got sidetracked with never getting
round to fixing it properly instead of pushing the bandaid)

8<--------------------------------------------------

xl: strip whitespace from vif key/values when parsing

Otherwise we can end up with bridge = "br0   " etc which is usually not
what is desired.

This whole parser could do with a rewrite (perhaps a generic KVP parser
in flex would be useful) but this quick fix appears to suffice for now.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>


diff -r 3c6eaf62996d -r 31eee4e06a20 tools/libxl/xl_cmdimpl.c
--- a/tools/libxl/xl_cmdimpl.c	Thu Nov 10 07:56:40 2011 +0000
+++ b/tools/libxl/xl_cmdimpl.c	Thu Nov 10 08:04:16 2011 +0000
@@ -515,6 +515,20 @@ static void parse_disk_config(XLU_Config
     parse_disk_config_multistring(config, 1, &spec, disk);
 }
 
+/*
+ * Duplicate s stripping leading and trailing whitespace.
+ * Destroys contents of s.
+ */
+static char *strdup_ws(char *s)
+{
+    char *e = s + strlen(s) - 1;
+    while (*s == ' ')
+        s++;
+    while (*e == ' ')
+        *(e--) = '\0';
+    return strndup(s, e-s+1);
+}
+
 static void parse_config_data(const char *configfile_filename_report,
                               const char *configfile_data,
                               int configfile_len,
@@ -758,7 +772,7 @@ static void parse_config_data(const char
         while ((buf = xlu_cfg_get_listitem (nics, d_config->num_vifs)) != NULL) {
             libxl_device_nic *nic;
             char *buf2 = strdup(buf);
-            char *p, *p2;
+            char *p, *p2, *p3;
 
             d_config->vifs = (libxl_device_nic *) realloc(d_config->vifs, sizeof (libxl_device_nic) * (d_config->num_vifs+1));
             nic = d_config->vifs + d_config->num_vifs;
@@ -779,6 +793,9 @@ static void parse_config_data(const char
                 if ((p2 = strchr(p, '=')) == NULL)
                     break;
                 *p2 = '\0';
+                p3 = p2 - 1;
+                while (*p3 == ' ' && p3 > p)
+                    *(p3--) = '\0';
                 if (!strcmp(p, "model")) {
                     free(nic->model);
                     nic->model = strdup(p2 + 1);
@@ -802,8 +820,8 @@ static void parse_config_data(const char
                     *(p3 + 2) = '\0';
                     nic->mac[5] = strtol(p3, NULL, 16);
                 } else if (!strcmp(p, "bridge")) {
                     free(nic->bridge);
-                    nic->bridge = strdup(p2 + 1);
+                    nic->bridge = strdup_ws(p2 + 1);
                 } else if (!strcmp(p, "type")) {
                     if (!strcmp(p2 + 1, "ioemu"))
                         nic->nictype = LIBXL_NIC_TYPE_IOEMU;

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[erin.balid]

Hi Ian.

On Mon, Jan 23, 2012, at 11:27 AM, Ian Campbell wrote:
> ISTRT xl had a bug at one point where it would fail to strip whitespace
> in the network KVP lists. i.e. the above ended up expecting to find a
> bridge named "br0  ". Can you try without the spaces please?

Edit the configuration file.

 - vif          = [ 'mac=00:16:3E:12:34:01, bridge=br0' ]
 + vif=['mac=00:16:3E:12:34:01,bridge=br0']             

xl create test.cfg
xl list
Name                            ID   Mem VCPUs      State   Time(s)
 Domain-0                        0  1010     1     r-----    1072.3
 test                            6  2048     2     -b----       2.2

Looks like it didn't make any difference :-(

brctl show
 bridge name     bridge id               STP enabled     interfaces
 br0             8000.0052351d5337       yes             eth0

xl console test
login
ping -c 2 192.168.1.1
 PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
 From 192.168.1.32 icmp_seq=1 Destination Host Unreachable
 From 192.168.1.32 icmp_seq=2 Destination Host Unreachable
 
 --- 192.168.1.1 ping statistics ---
 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time
 1000ms
 pipe 2

Regards,

Erin

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Ian Campbell]

On Mon, 2012-01-23 at 15:28 +0000, erin.balid@inoutbox.com wrote:
> Hi Ian.
> 
> On Mon, Jan 23, 2012, at 11:27 AM, Ian Campbell wrote:
> > ISTRT xl had a bug at one point where it would fail to strip whitespace
> > in the network KVP lists. i.e. the above ended up expecting to find a
> > bridge named "br0  ". Can you try without the spaces please?
> 
> Edit the configuration file.
> 
>  - vif          = [ 'mac=00:16:3E:12:34:01, bridge=br0' ]
>  + vif=['mac=00:16:3E:12:34:01,bridge=br0']
>
[...]
> brctl show
>  bridge name     bridge id               STP enabled     interfaces
>  br0             8000.0052351d5337       yes             eth0

Hrm. The device simply isn't on the bridge. Does "ifconfig -a" in dom0
show a vifX.Y device for the domain? You previous xenstore log suggests
it should be there but lets double check.

Do you get anything in the logs under /var/log/xen?

Are your hotplug scripts running at all? One trick I usually use is to
add to the top of vif-bridge something like:
	exec 1>>/tmp/hotplog.log
	exec 2>&1
	echo "`date`: Running $0 $*"
Then any hotplug script which gets run will dump its output
to /tmp/hotplug.log.

http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen lists a bunch of
configuration files and logs which might be of interest, at least to
skim looking for anything odd.

Ian.

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[erin.balid]

Hi Ian.

> Does "ifconfig -a" in dom0 show a vifX.Y device for the domain? You previous xenstore log
> suggests it should be there but lets double check.

edit /etc/xen/scripts/vif-bridge
	#!/bin/bash
+       exec 1>>/tmp/hotplog.log
+       exec 2>&1
+       echo "`date`: Running $0 $*"

xl create test.cfg


ifconfig -a
brINT     Link encap:Ethernet  HWaddr 00:52:35:11:26:3B
          inet addr:192.168.1.10  Bcast:192.168.1.255 
          Mask:255.255.252.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:444732 errors:0 dropped:0 overruns:0 frame:0
          TX packets:287814 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1096020765 (1045.2 Mb)  TX bytes:95900515 (91.4 Mb)

eth0      Link encap:Ethernet  HWaddr 00:52:35:11:26:3B
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:840591 errors:0 dropped:0 overruns:0 frame:0
          TX packets:376159 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1132351110 (1079.8 Mb)  TX bytes:100533253 (95.8 Mb)
          Interrupt:48

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:3922 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3922 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:252238 (246.3 Kb)  TX bytes:252238 (246.3 Kb)

vif7.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

xl network-list test
 Idx BE Mac Addr.         handle state evt-ch   tx-/rx-ring-ref BE-path
 0   0  00:16:3e:12:34:01      0     4     15   768/769        
 /local/domain/0/backend/vif/7/0


> Do you get anything in the logs under /var/log/xen?

Just this after the Guest launches.  It looks a bit suspect maybe?

cat /var/log/xen/*log
 domid: 7
 Warning: vlan 0 is not connected to host network
 -videoram option does not work with cirrus vga device model. Videoram
 set to 4M.
 /home/abuild/rpmbuild/BUILD/xen-4.1.2-testing/tools/ioemu-dir/hw/xen_blktap.c:704:
 Init blktap pipes
 Could not open /var/run/tap/qemu-read-7
 char device redirected to /dev/pts/1
 xs_read(): target get error. /local/domain/7/target.
 xs_read(): vncpasswd get error.
 /vm/22eef8a2-7b68-4bb6-a69f-632d20db0d84/vncpasswd.
 Waiting for domain test (domid 7) to die [pid 28807]

> Are your hotplug scripts running at all? ...

cat /tmp/hotplog.log
 Tue Jan 24 07:23:49 PST 2012: Running /etc/xen/scripts/vif-bridge
 online type_if=vif
 Tue Jan 24 07:23:49 PST 2012: Running /etc/xen/scripts/vif-bridge
 online

> http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen lists a bunch of
> configuration files and logs which might be of interest, at least to
> skim looking for anything odd.

cat /var/log/xen/xen-hotplug.log
cat /var/log/xen/qemu-dm-*.log
 above ...

cat /var/log/xeb/console/*
 cat: /var/log/xeb/console/*: No such file or directory

I'm not completely sure what to look for that I'd know to be "odd".  I
think/hope the rest has already been reported in this thread.  If you
need anything else, please ID it and I can post it.

Regards,

Erin

p.s. Over on the opensuse-virtual list, at this thread,
http://lists.opensuse.org/opensuse-virtual/2012-01/msg00015.html

I just received a new post (It hasn't propagated to the list archive
just yet) says:

	"We also have what appears to be this problem, on Fedora 16 with
	Xen 4.1.2. We can create guests with xm that connect to network
	but the same guest will not connect, when created by xl. In both
	cases we can see the appropriate vif has been created and
	attached (xl network-list shows it) but no connectivity from
	within the guest. (If I convert the same guest config file to
	virsh format, then virsh also creates and connects it to the
	network successfully.)

	I post this to the list as evidence that this is not just a
	problem with SuSE."

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Ian Campbell]

On Tue, 2012-01-24 at 15:56 +0000, erin.balid@inoutbox.com wrote:
> Hi Ian.
> 
> > Does "ifconfig -a" in dom0 show a vifX.Y device for the domain? You previous xenstore log
> > suggests it should be there but lets double check.
> 
> edit /etc/xen/scripts/vif-bridge
> 	#!/bin/bash
> +       exec 1>>/tmp/hotplog.log
> +       exec 2>&1
> +       echo "`date`: Running $0 $*"
> 
> xl create test.cfg
> 
> 
> ifconfig -a
> brINT     Link encap:Ethernet  HWaddr 00:52:35:11:26:3B

brINT is not the name you've been quoting before (which was br0). Are
you running these tests on a variety of different systems with different
configurations?

It would be really helpful for those of us trying to help you if you
could use the same system with the same setup (modulo requested changes)
for the entirety of this conversation. Otherwise it becomes rather hard
for us to correlate the facts.

In this case I now have to ask if you are sure that you used the
appropriate "bridge=brINT" in your guest configuration instead of
"bridge=br0" which you had before.

[...]
> vif7.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF

Good that this exists.

> > Do you get anything in the logs under /var/log/xen?
> 
> Just this after the Guest launches.  It looks a bit suspect maybe?
> 
> cat /var/log/xen/*log
>  domid: 7
>  Warning: vlan 0 is not connected to host network
>  -videoram option does not work with cirrus vga device model. Videoram
>  set to 4M.
>  /home/abuild/rpmbuild/BUILD/xen-4.1.2-testing/tools/ioemu-dir/hw/xen_blktap.c:704:
>  Init blktap pipes
>  Could not open /var/run/tap/qemu-read-7
>  char device redirected to /dev/pts/1
>  xs_read(): target get error. /local/domain/7/target.
>  xs_read(): vncpasswd get error.
>  /vm/22eef8a2-7b68-4bb6-a69f-632d20db0d84/vncpasswd.
>  Waiting for domain test (domid 7) to die [pid 28807]

These are the logs for the qemu process which is serving as your vfb
backend. These all look reasonably normal (in the sense that all this
pointless noise is normal).

> > Are your hotplug scripts running at all? ...
> 
> cat /tmp/hotplog.log
>  Tue Jan 24 07:23:49 PST 2012: Running /etc/xen/scripts/vif-bridge
>  online type_if=vif
>  Tue Jan 24 07:23:49 PST 2012: Running /etc/xen/scripts/vif-bridge
>  online

So the hotplug script is running without any apparent error, at least
not one that was logged.

At this point I'm afraid my only suggesting is to drop "echo made it to
XXX" breadcrumbs throughout the script and try to narrow down to the
line which exits.

You could also perhaps run "udevadm monitor" in another window while
starting the guest, so wee can see what events you are actually seeing.

[...]
> cat /var/log/xeb/console/*
>  cat: /var/log/xeb/console/*: No such file or directory

Obviously a typo, but there's probably not too much of interest in the
guest console logs.

Ian.

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[erin.balid]

Ian.

On Tue, Jan 24, 2012, at 04:41 PM, Ian Campbell wrote:
> brINT is not the name you've been quoting before (which was br0). Are
> you running these tests on a variety of different systems with different
> configurations?
>
> It would be really helpful for those of us trying to help you if you
> could use the same system with the same setup (modulo requested changes)
> for the entirety of this conversation. Otherwise it becomes rather hard
> for us to correlate the facts.
> 
> In this case I now have to ask if you are sure that you used the
> appropriate "bridge=brINT" in your guest configuration instead of
> "bridge=br0" which you had before.

No, the same system.  Just a different bridge config file because I've
read some speculation re: bridge naming requirements, and have been
testing it -- in response to others that are trying to help me.

The email to this list was just a copy and paste problem from the notes
I'm trying to keep.

Returning to just =br0, the problem still exists exactly as I reported
above.  I can repost everything if you'd like to see that. Just to
remind, there are now at least 3 persons reporting similar problems with
no network access with xl-created Guests.

> At this point I'm afraid my only suggesting is to drop "echo made it to
> XXX" breadcrumbs throughout the script and try to narrow down to the
> line which exits.

I'll give that a try and report.

> You could also perhaps run "udevadm monitor" in another window while
> starting the guest, so wee can see what events you are actually seeing.

xl create /etc/xen/vm/test.cfg
 Parsing config file /etc/xen/vm/test.cfg
 Daemon running with PID 29685

xl list
 Name                                        ID   Mem VCPUs      State  
 Time(s)
 Domain-0                                     0  1010     1     r-----  
  2524.2
 test                                         8  2048     2     -b----  
     7.0

xl console test
 login ...

@ the Guest
dmesg | egrep -i "bus|eth|dri|pci"
 [    0.157505] PCI: Fatal: No config space access function found
 [    0.157512] PCI: setting up Xen PCI frontend stub
 [    0.157878] xen_mem: Initialising balloon driver.
 [    0.160157] PCI: System does not support PCI
 [    0.160157] PCI: System does not support PCI
 [    0.164020] PCI: max bus depth: 0 pci_try_num: 1
 [    0.166097] PCI: CLS 64 bytes
 [    0.227389] Block layer SCSI generic (bsg) driver version 0.4 loaded
 (major 253)
 [    0.227651] Non-volatile memory driver v1.3
 [    0.311840] Fixed MDIO Bus: probed
 [    0.412125] XENBUS: Device with no driver: device/vbd/51712
 [    0.412133] XENBUS: Device with no driver: device/vbd/51728
 [    0.412139] XENBUS: Device with no driver: device/vbd/51744
 [    0.412144] XENBUS: Device with no driver: device/vif/0
 [    0.412153]
 /home/abuild/rpmbuild/BUILD/kernel-xen-3.1.0/linux-3.1/drivers/rtc/hctosys.c:
 unable to open rtc device (rtc0)
 [    0.868480] netfront: Initialising virtual ethernet driver.

@ the Host
udevadm monitor
 monitor will print the received events for:
 UDEV - the event which udev sends out after rule processing
 KERNEL - the kernel uevent

 KERNEL[172855.382354] add      /devices/xen-backend/vbd-8-51712
 (xen-backend)
 KERNEL[172855.471281] add      /devices/xen-backend/vbd-8-51728
 (xen-backend)
 UDEV  [172855.491539] add      /devices/xen-backend/vbd-8-51712
 (xen-backend)
 KERNEL[172855.561121] add      /devices/xen-backend/vbd-8-51744
 (xen-backend)
 UDEV  [172855.579734] add      /devices/xen-backend/vbd-8-51728
 (xen-backend)
 KERNEL[172855.610413] add      /devices/xen-backend/vif-8-0
 (xen-backend)
 KERNEL[172855.635818] add      /devices/xen-backend/vif-8-0/net/vif8.0
 (net)
 KERNEL[172855.635839] add     
 /devices/xen-backend/vif-8-0/net/vif8.0/queues/rx-0 (queues)
 KERNEL[172855.635851] add     
 /devices/xen-backend/vif-8-0/net/vif8.0/queues/tx-0 (queues)
 KERNEL[172855.636154] online   /devices/xen-backend/vif-8-0
 (xen-backend)
 UDEV  [172855.655709] add      /devices/xen-backend/vif-8-0
 (xen-backend)
 UDEV  [172855.673354] add      /devices/xen-backend/vbd-8-51744
 (xen-backend)
 KERNEL[172855.700762] add      /devices/xen-backend/vfb-8-0
 (xen-backend)
 UDEV  [172855.706786] add      /devices/xen-backend/vfb-8-0
 (xen-backend)
 UDEV  [172855.720759] add      /devices/xen-backend/vif-8-0/net/vif8.0
 (net)
 UDEV  [172855.721130] add     
 /devices/xen-backend/vif-8-0/net/vif8.0/queues/rx-0 (queues)
 UDEV  [172855.721457] add     
 /devices/xen-backend/vif-8-0/net/vif8.0/queues/tx-0 (queues)
 KERNEL[172855.725625] add      /devices/xen-backend/vkbd-8-0
 (xen-backend)
 UDEV  [172855.732509] add      /devices/xen-backend/vkbd-8-0
 (xen-backend)
 KERNEL[172855.758430] add      /devices/xen-backend/console-8-0
 (xen-backend)
 UDEV  [172855.768396] add      /devices/xen-backend/console-8-0
 (xen-backend)
 UDEV  [172855.909030] online   /devices/xen-backend/vif-8-0
 (xen-backend)

> [...]
> > cat /var/log/xeb/console/*
> >  cat: /var/log/xeb/console/*: No such file or directory
> 
> Obviously a typo, but there's probably not too much of interest in the
> guest console logs.

Yes a typo.

 cat /var/log/xen/console/*
  cat: /var/log/xen/console/*: No such file or directory

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[erin.balid]

Hi Ian.

On Tue, Jan 24, 2012, at 09:10 AM, erin.balid@inoutbox.com wrote:
> > At this point I'm afraid my only suggesting is to drop "echo made it to
> > XXX" breadcrumbs throughout the script and try to narrow down to the
> > line which exits.
> 
> I'll give that a try and report.

I can't figure out how to get xl to output to a xend-debug.log (xm does,
as expected).

Using the same log redirection you mentioned before

edit /etc/xen/scripts/vif-bridge
	#!/bin/bash
+       exec 1>>/tmp/vif-bridge.log
+       exec 2>&1
+       echo "`date`: Running $0 $*"
...
+       ## TEST ##
+       echo "made it to 001"

	dir=$(dirname "$0")
	. "$dir/vif-common.sh"

+       ## TEST ##
+       echo "made it to 002"


	domu=$(xenstore_read_default "$XENBUS_PATH/domain" "")
	if [ -z "$domu" ]
	then
	    log debug "No device details in $XENBUS_PATH, exiting."
	    exit 0
	fi

+       ## TEST ##
+       echo "made it to 003"
...


Then

xl destroy test
xl create /etc/xen/vm/test.cfg

tail -f /tmp/vif-bridge.log
 Tue Jan 24 09:59:29 PST 2012: Running /etc/xen/scripts/vif-bridge
 online type_if=vif
  made it to 001
  made it to 002
  Tue Jan 24 09:59:29 PST 2012: Running /etc/xen/scripts/vif-bridge
  online
  made it to 001
  made it to 002

Never makes it to "003".

Erin

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[erin.balid]

Hi Ian.

On Tue, Jan 24, 2012, at 10:05 AM, erin.balid@inoutbox.com wrote:
> I can't figure out how to get xl to output to a xend-debug.log (xm does,
> as expected).


Reading http://www.gossamer-threads.com/lists/xen/devel/198734, and
checking "-vvv" console output

xl destroy test
xl -vvv create /etc/xen/vm/test.cfg

XEND_DEBUG = 1
Parsing config file /etc/xen/vm/test.cfg
libxl: debug: libxl.c:1208:libxl_device_disk_local_attach attaching PHY
disk /dev/XenVols/test_boot to domain 0
domainbuilder: detail: xc_dom_allocate: cmdline="root=/dev/xvdc1
resume=/dev/xvdb1 kbdtype=us headless text quiet nofb selinux=0
apparmor-0 edd=off splash=silent noshell showopts root=/dev/xvdc1
textmode=1 xencons=xvc0 noirqdebug", features="(null)"
domainbuilder: detail: xc_dom_kernel_mem: called
domainbuilder: detail: xc_dom_malloc            : 11850 kB
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x425f68 -> 0xb92a70
domainbuilder: detail: xc_dom_ramdisk_mem: called
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.1, caps
xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
hvm-3.0-x86_64
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying ELF-generic loader ...
domainbuilder: detail: loader probe OK
xc: detail: elf_parse_binary: phdr: paddr=0x2000 memsz=0x8a5000
xc: detail: elf_parse_binary: phdr: paddr=0x8a7000 memsz=0x7a0e8
xc: detail: elf_parse_binary: phdr: paddr=0x922000 memsz=0xaac0
xc: detail: elf_parse_binary: phdr: paddr=0x92d000 memsz=0x158000
xc: detail: elf_parse_binary: memory: 0x2000 -> 0xa85000
xc: detail: elf_xen_parse_note: GUEST_OS = "linux"
xc: detail: elf_xen_parse_note: GUEST_VERSION = "2.6"
xc: detail: elf_xen_parse_note: XEN_VERSION = "xen-3.0"
xc: detail: elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
xc: detail: elf_xen_parse_note: PADDR_OFFSET = 0x0
xc: detail: elf_xen_parse_note: ENTRY = 0xffffffff80002000
xc: detail: elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff80003000
xc: detail: elf_xen_parse_note: unknown xen elf note (0xd)
xc: detail: elf_xen_parse_note: MOD_START_PFN = 0x1
xc: detail: elf_xen_parse_note: INIT_P2M = 0xffffea0000000000
xc: detail: elf_xen_parse_note: FEATURES =
"writable_page_tables|writable_descriptor_tables|auto_translated_physmap|supervisor_mode_kernel"
xc: detail: elf_xen_parse_note: SUPPORTED_FEATURES = 0x80f
xc: detail: elf_xen_parse_note: LOADER = "generic"
xc: detail: elf_xen_parse_note: SUSPEND_CANCEL = 0x1
xc: detail: elf_xen_addr_calc_check: addresses:
xc: detail:     virt_base        = 0xffffffff80000000
xc: detail:     elf_paddr_offset = 0x0
xc: detail:     virt_offset      = 0xffffffff80000000
xc: detail:     virt_kstart      = 0xffffffff80002000
xc: detail:     virt_kend        = 0xffffffff80a85000
xc: detail:     virt_entry       = 0xffffffff80002000
xc: detail:     p2m_base         = 0xffffea0000000000
domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64:
0xffffffff80002000 -> 0xffffffff80a85000
domainbuilder: detail: xc_dom_mem_init: mem 2048 MB, pages 0x80000
pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x80000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64
domainbuilder: detail: xc_dom_malloc            : 4096 kB
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_alloc_segment:   kernel       :
0xffffffff80002000 -> 0xffffffff80a85000  (pfn 0x2 + 0xa83 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x2+0xa83 at
0x7fcd83941000
xc: detail: elf_load_binary: phdr 0 at 0x0x7fcd83941000 ->
0x0x7fcd841e6000
xc: detail: elf_load_binary: phdr 1 at 0x0x7fcd841e6000 ->
0x0x7fcd842600e8
xc: detail: elf_load_binary: phdr 2 at 0x0x7fcd84261000 ->
0x0x7fcd8426bac0
xc: detail: elf_load_binary: phdr 3 at 0x0x7fcd8426c000 ->
0x0x7fcd842d1000
domainbuilder: detail: xc_dom_alloc_segment:   ramdisk      :
0xffffffff80a85000 -> 0xffffffff82078000  (pfn 0xa85 + 0x15f3 pages)
domainbuilder: detail: xc_dom_malloc            : 131 kB
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0xa85+0x15f3
at 0x7fcd8234e000
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x94895f -> 0x15f2e10
domainbuilder: detail: xc_dom_alloc_segment:   phys2mach    :
0xffffffff82078000 -> 0xffffffff82478000  (pfn 0x2078 + 0x400 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x2078+0x400
at 0x7fcd81f4e000
domainbuilder: detail: xc_dom_alloc_page   :   start info   :
0xffffffff82478000 (pfn 0x2478)
domainbuilder: detail: xc_dom_alloc_page   :   xenstore     :
0xffffffff82479000 (pfn 0x2479)
domainbuilder: detail: xc_dom_alloc_page   :   console      :
0xffffffff8247a000 (pfn 0x247a)
domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48:
0xffff000000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x0000007fffffffff/39:
0xffffff8000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30:
0xffffffff80000000 -> 0xffffffffbfffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21:
0xffffffff80000000 -> 0xffffffff827fffff, 20 table(s)
domainbuilder: detail: xc_dom_alloc_segment:   page tables  :
0xffffffff8247b000 -> 0xffffffff82492000  (pfn 0x247b + 0x17 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x247b+0x17
at 0x7fcd81f37000
domainbuilder: detail: xc_dom_alloc_page   :   boot stack   :
0xffffffff82492000 (pfn 0x2492)
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end :
0xffffffff82493000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end :
0xffffffff82800000
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: arch_setup_bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type:
xen-3.0-x86_64 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type:
xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type:
hvm-3.0-x86_32
domainbuilder: detail: xc_dom_compat_check: supported guest type:
hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type:
hvm-3.0-x86_64
domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x80000
domainbuilder: detail: clear_page: pfn 0x247a, mfn 0x11fe01
domainbuilder: detail: clear_page: pfn 0x2479, mfn 0x11fe02
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x2478+0x1
at 0x7fcd889b0000
domainbuilder: detail: start_info_x86_64: called
domainbuilder: detail: setup_hypercall_page: vaddr=0xffffffff80003000
pfn=0x3
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 16168 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 0 bytes
domainbuilder: detail:       domU mmap          : 36 MB
domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn
0xcfc5c
domainbuilder: detail: shared_info_x86_64: called
domainbuilder: detail: vcpu_x86_64: called
domainbuilder: detail: vcpu_x86_64: cr3: pfn 0x247b mfn 0x11fe00
domainbuilder: detail: launch_vm: called, ctxt=0x7fffa5f0a510
domainbuilder: detail: xc_dom_release: called
Daemon running with PID 2864
xc: debug: hypercall buffer: total allocations:99 total releases:99
xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:94 misses:2 toobig:3

Does that help at all?

Erin

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Ian Campbell]

On Tue, 2012-01-24 at 18:19 +0000, erin.balid@inoutbox.com wrote:
> Hi Ian.
> 
> On Tue, Jan 24, 2012, at 10:05 AM, erin.balid@inoutbox.com wrote:
> > I can't figure out how to get xl to output to a xend-debug.log (xm does,
> > as expected).
> 
> 
> Reading http://www.gossamer-threads.com/lists/xen/devel/198734, and
> checking "-vvv" console output
> 
> xl destroy test
> xl -vvv create /etc/xen/vm/test.cfg
[...]
> Does that help at all?

I'm afraid not -- the failure is not at this level.

Ian.

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Ian Campbell]

On Tue, 2012-01-24 at 18:05 +0000, erin.balid@inoutbox.com wrote:
> Hi Ian.
> 
> On Tue, Jan 24, 2012, at 09:10 AM, erin.balid@inoutbox.com wrote:
> > > At this point I'm afraid my only suggesting is to drop "echo made it to
> > > XXX" breadcrumbs throughout the script and try to narrow down to the
> > > line which exits.
> > 
> > I'll give that a try and report.
> 
> I can't figure out how to get xl to output to a xend-debug.log (xm does,
> as expected).

They should be going to /var/log/xen/xen-hotplug.

> Using the same log redirection you mentioned before
> 
> edit /etc/xen/scripts/vif-bridge
> 	#!/bin/bash
> +       exec 1>>/tmp/vif-bridge.log
> +       exec 2>&1
> +       echo "`date`: Running $0 $*"
> ...
> +       ## TEST ##
> +       echo "made it to 001"
> 
> 	dir=$(dirname "$0")
> 	. "$dir/vif-common.sh"
> 
> +       ## TEST ##
> +       echo "made it to 002"
> 
> 
> 	domu=$(xenstore_read_default "$XENBUS_PATH/domain" "")
> 	if [ -z "$domu" ]
> 	then
> 	    log debug "No device details in $XENBUS_PATH, exiting."

So, presumably either the xenstore_read_default is failing or domu is
zero length and the script is explicitly bailing here.

However I do not see anything like this anywhere in the hotplug script
shipped with Xen. Either this is a local modification or something done
by your distribution, in which case you should contact them.

What does this node contain under xend?

What does your script go on to do with this domu value?

Perhaps given that xend writes this domain node so perhaps xl should
too. In a few cases (vkb, vfb, console) it does seem to add it but in
others (disk, nic, etc) etc does not. To be honest I don't really see
what purpose it serves to put the domain's name in each device's backend
subdirectory.

Ian.

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[erin.balid]

Hi Ian.

On Tue, Jan 24, 2012, at 09:20 PM, Ian Campbell wrote:
> > I can't figure out how to get xl to output to a xend-debug.log (xm does,
> > as expected).
> 
> They should be going to /var/log/xen/xen-hotplug.

Not a peep in there.  Not even a there, there.

 xl create test.cfg

 ls -al /var/log/xen/xen-hotplug
  ls: cannot access /var/log/xen/xen-hotplug: No such file or directory

> So, presumably either the xenstore_read_default is failing or domu is
> zero length and the script is explicitly bailing here.
> 
> However I do not see anything like this anywhere in the hotplug script
> shipped with Xen. Either this is a local modification

I haven't made any local modifications.  At least not intentionally.

Here's the vif-bridge script ==> http://pastebin.com/9nHESsqS

> or something done by your distribution

It's opensuse.

> in which case you should contact them.

I already have.  Well, I tried.  Both on their mailing list
(opensuse-virtual), and cc'ing this thread to one of their developers
seemingly active on this mailing list.

So far the only reponses have been - there - from someone on Fedora16
seeing the same problem, and - here - from non-suse types, such as
yourself, trying to help.  Other than asking for their help, I don't
know how to convince them to help.  I can only work with the assistance
of folks that'll talk to me about the problem.

> What does this node contain under xend?

I don't understand the question.  Is there some specific detail I can
give you ?  Is it the value of 'domu' you're after?

> What does your script go on to do with this domu value?

Does the script pastebin above answer that for you?

> Perhaps given that xend writes this domain node so perhaps xl should
> too. In a few cases (vkb, vfb, console) it does seem to add it but in
> others (disk, nic, etc) etc does not. To be honest I don't really see
> what purpose it serves to put the domain's name in each device's backend
> subdirectory.

Regards,

Erin

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[erin.balid]

Hi Ian.  Et al.

Thanks for the help in sorting this out.  We at least found out that
it's a problem at the distro.  Jim Fehlig says he can reproduce what I
see and will look into it - time permitting.   Which is great.

I don't know how this addresses the Fedora16 appearance of the problem,
but that's another discussion.

Regards.

Erin

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Ian Campbell]

On Tue, 2012-01-24 at 21:44 +0000, erin.balid@inoutbox.com wrote:
> Hi Ian.
> 
> On Tue, Jan 24, 2012, at 09:20 PM, Ian Campbell wrote:
> > > I can't figure out how to get xl to output to a xend-debug.log (xm does,
> > > as expected).
> > 
> > They should be going to /var/log/xen/xen-hotplug.
> 
> Not a peep in there.  Not even a there, there.
> 
>  xl create test.cfg
> 
>  ls -al /var/log/xen/xen-hotplug
>   ls: cannot access /var/log/xen/xen-hotplug: No such file or directory

My mistake, it is /var/log/xen/xen-hotplug.log.

> > So, presumably either the xenstore_read_default is failing or domu is
> > zero length and the script is explicitly bailing here.
> > 
> > However I do not see anything like this anywhere in the hotplug script
> > shipped with Xen. Either this is a local modification
> 
> I haven't made any local modifications.  At least not intentionally.
> 
> Here's the vif-bridge script ==> http://pastebin.com/9nHESsqS

Seems to read domu and abort if it is not there but subsequently never
makes any use of the variable.

Anyway it seems that SuSE are on the case so I will leave this thread
here.

Ian.

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Jim Fehlig]

Ian Campbell wrote:
> On Tue, 2012-01-24 at 18:05 +0000, erin.balid@inoutbox.com wrote:
>   
>> Using the same log redirection you mentioned before
>>
>> edit /etc/xen/scripts/vif-bridge
>> 	#!/bin/bash
>> +       exec 1>>/tmp/vif-bridge.log
>> +       exec 2>&1
>> +       echo "`date`: Running $0 $*"
>> ...
>> +       ## TEST ##
>> +       echo "made it to 001"
>>
>> 	dir=$(dirname "$0")
>> 	. "$dir/vif-common.sh"
>>
>> +       ## TEST ##
>> +       echo "made it to 002"
>>
>>
>> 	domu=$(xenstore_read_default "$XENBUS_PATH/domain" "")
>> 	if [ -z "$domu" ]
>> 	then
>> 	    log debug "No device details in $XENBUS_PATH, exiting."
>>     
>
> So, presumably either the xenstore_read_default is failing or domu is
> zero length and the script is explicitly bailing here.
>
> However I do not see anything like this anywhere in the hotplug script
> shipped with Xen. Either this is a local modification or something done
> by your distribution, in which case you should contact them.
>   

Grrr, that is my patch here

http://old-list-archives.xen.org/archives/html/xen-devel/2011-10/msg01790.html

which thankfully was never applied upstream but did make its way into
openSUSE12.1.  In that thread, Ian suggested having the toolstack write
the literal tap device name to xenstore and then read it in the vif
script.  I have this on my todo list with very low priority, but seems I
should bump it now :-).

> What does this node contain under xend?
>   

The xend toolstack writes the domain name under this node but xl/libxl
doesn't, hence the failure Erin is seeing.

Erin, you can revert the patch mentioned above to fix your problem. 
I'll try to work on the proper fix soon.

Regards,
Jim

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Ian Campbell]

On Wed, 2012-01-25 at 16:19 +0000, Jim Fehlig wrote:
> Ian Campbell wrote:
> > On Tue, 2012-01-24 at 18:05 +0000, erin.balid@inoutbox.com wrote:
> >   
> >> Using the same log redirection you mentioned before
> >>
> >> edit /etc/xen/scripts/vif-bridge
> >> 	#!/bin/bash
> >> +       exec 1>>/tmp/vif-bridge.log
> >> +       exec 2>&1
> >> +       echo "`date`: Running $0 $*"
> >> ...
> >> +       ## TEST ##
> >> +       echo "made it to 001"
> >>
> >> 	dir=$(dirname "$0")
> >> 	. "$dir/vif-common.sh"
> >>
> >> +       ## TEST ##
> >> +       echo "made it to 002"
> >>
> >>
> >> 	domu=$(xenstore_read_default "$XENBUS_PATH/domain" "")
> >> 	if [ -z "$domu" ]
> >> 	then
> >> 	    log debug "No device details in $XENBUS_PATH, exiting."
> >>     
> >
> > So, presumably either the xenstore_read_default is failing or domu is
> > zero length and the script is explicitly bailing here.
> >
> > However I do not see anything like this anywhere in the hotplug script
> > shipped with Xen. Either this is a local modification or something done
> > by your distribution, in which case you should contact them.
> >   
> 
> Grrr, that is my patch here
> 
> http://old-list-archives.xen.org/archives/html/xen-devel/2011-10/msg01790.html
> 
> which thankfully was never applied upstream but did make its way into
> openSUSE12.1.  In that thread, Ian suggested having the toolstack write
> the literal tap device name to xenstore and then read it in the vif
> script.  I have this on my todo list with very low priority, but seems I
> should bump it now :-).

There's been a fair bit of discussion recently on changing the way these
scripts are executed, this case is one which we need to remember to
handle too.

In particular Roger Pau Monet has posted a few series recently. You
should be able to find these and the discussion in the archives.

Ian.

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[John McDermott CIV]

This problem also happens in Fedora 16. Should we be making bugzilla entries for problems with Xen?

On Jan 25, 2012, at 11:47 AM, xen-devel-request@lists.xensource.com wrote:

> Yes, let's stop this thread here :-).  This is a downstream issue.
> 
>> I got your comment that you'd look for fixes in the Opensuse 12.2
>> time-frame.  As I read the openSUSE Roadmap, the scheduled release date
>> for openSUSE 12.2 is July 2012.
>> 
>> We won't switch to a new OS release until after it's in production.  We
>> _are_ willing to use a few non-production repositories for key packages.
>> Xen's one of them.
>> 
>> Should we plan to see these Xen fixes in a shorter timeframe in a repo
>> built for use with 12.1 release/kernel prior to that 12.2 release date?
> 
> We do maintenance on virtualization packages in openSUSE, so a fix for
> this could be delivered to the openSUSE12.1 update repository.  But the
> process starts with a bugzilla entry...

----
What is the formal meaning of the one-line program
#include "/dev/tty"

J.P. McDermott			building 12
Code 5542			john.mcdermott@nrl.navy.mil
Naval Research Laboratory	voice: +1 202.404.8301
Washington, DC 20375, US	fax:   +1 202.404.7942

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Ian Campbell]

Please don't top post.

On Wed, 2012-01-25 at 17:06 +0000, John McDermott CIV wrote:
> This problem also happens in Fedora 16. Should we be making bugzilla
> entries for problems with Xen?

There is more than one reason why networking may not work for you so
there isn't necessarily any reason (yet) to think you are seeing the
same issue.

I think xen-users or an equivalent fedora list would be an appropriate
forum for your bug report in the first instance.

Ian.

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[John McDermott CIV]

Ian,

I did not intend to top post. I have already posted this once on Xen-devel and got no response.

-----
	Subject: 	Re: [opensuse-virtual] After switching from "xm" to "xl" toolstack, can't get Guest networking to work.
	From: 	John McDermott CIV <john.mcdermott@nrl.navy.mil>
	Date: 	January 24, 2012 7:40:39 AM EST
	To: 	erin.balid@inoutbox.com
	Cc: 	xen-devel@lists.xensource.com


Erin,

We also have what appears to be this problem, on Fedora 16 with Xen 4.1.2. We can create guests with xm that connect to network but the same guest will not connect, when created by xl. In both cases we can see the appropriate vif has been created and attached (xl network-list shows it) but no connectivity from within the guest. (If I convert the same guest config file to virsh format, then virsh also creates and connects it to the network successfully.)

I post this to the list as evidence that this is not just a problem with SuSE.

Sincerely,

John
----

On Jan 23, 2012, at 6:39 PM, erin.balid@inoutbox.com wrote:

> Hi All.
> 
> This problem's got a little - not a lot! - of traction over at the
> xen-devel list.
> 
> http://lists.xen.org/archives/html/xen-devel/2012-01/msg01754.html. 
> 
> Rather than leave this open & uncommented here, I'm going to concentrate
> on that thread instead for now.
> 
> Erin
> -- 
> To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org
> To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

----
What is the formal meaning of the one-line program
#include "/dev/tty"

J.P. McDermott			building 12
Code 5542			john.mcdermott@nrl.navy.mil
Naval Research Laboratory	voice: +1 202.404.8301
Washington, DC 20375, US	fax:   +1 202.404.7942












On Jan 25, 2012, at 12:16 PM, Ian Campbell wrote:

> Please don't top post.
> 
> On Wed, 2012-01-25 at 17:06 +0000, John McDermott CIV wrote:
>> This problem also happens in Fedora 16. Should we be making bugzilla
>> entries for problems with Xen?
> 
> There is more than one reason why networking may not work for you so
> there isn't necessarily any reason (yet) to think you are seeing the
> same issue.
> 
> I think xen-users or an equivalent fedora list would be an appropriate
> forum for your bug report in the first instance.
> 
> Ian.

----
What is the formal meaning of the one-line program
#include "/dev/tty"

J.P. McDermott			building 12
Code 5542			john.mcdermott@nrl.navy.mil
Naval Research Laboratory	voice: +1 202.404.8301
Washington, DC 20375, US	fax:   +1 202.404.7942

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Jim Fehlig]

Ian Campbell wrote:
> Please don't top post.
>
> On Wed, 2012-01-25 at 17:06 +0000, John McDermott CIV wrote:
>   
>> This problem also happens in Fedora 16. Should we be making bugzilla
>> entries for problems with Xen?
>>     
>
> There is more than one reason why networking may not work for you so
> there isn't necessarily any reason (yet) to think you are seeing the
> same issue.
>   

Yes, different issue.  I'd be really surprised if Fedora had this patch

http://old-list-archives.xen.org/archives/html/xen-devel/2011-10/msg01790.html

Regards,
Jim

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[John McDermott CIV]

So I should try this patch and report back if it works?

On Jan 25, 2012, at 12:33 PM, Jim Fehlig wrote:

> Ian Campbell wrote:
>> Please don't top post.
>> 
>> On Wed, 2012-01-25 at 17:06 +0000, John McDermott CIV wrote:
>> 
>>> This problem also happens in Fedora 16. Should we be making bugzilla
>>> entries for problems with Xen?
>>> 
>> 
>> There is more than one reason why networking may not work for you so
>> there isn't necessarily any reason (yet) to think you are seeing the
>> same issue.
>> 
> 
> Yes, different issue.  I'd be really surprised if Fedora had this patch
> 
> http://old-list-archives.xen.org/archives/html/xen-devel/2011-10/msg01790.html
> 
> Regards,
> Jim

----
What is the formal meaning of the one-line program
#include "/dev/tty"

J.P. McDermott			building 12
Code 5542			john.mcdermott@nrl.navy.mil
Naval Research Laboratory	voice: +1 202.404.8301
Washington, DC 20375, US	fax:   +1 202.404.7942

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Jim Fehlig]

John McDermott CIV wrote:
> So I should try this patch and report back if it works?
>   

No, that patch causes the issue reported by Erin :-).  In Erin's case,
the vif was not connected to the bridge.  In the problem you described,
the vif was connected to the bridge but didn't work for other reasons.

Regards,
Jim

> On Jan 25, 2012, at 12:33 PM, Jim Fehlig wrote:
>
>   
>> Ian Campbell wrote:
>>     
>>> Please don't top post.
>>>
>>> On Wed, 2012-01-25 at 17:06 +0000, John McDermott CIV wrote:
>>>
>>>       
>>>> This problem also happens in Fedora 16. Should we be making bugzilla
>>>> entries for problems with Xen?
>>>>
>>>>         
>>> There is more than one reason why networking may not work for you so
>>> there isn't necessarily any reason (yet) to think you are seeing the
>>> same issue.
>>>
>>>       
>> Yes, different issue.  I'd be really surprised if Fedora had this patch
>>
>> http://old-list-archives.xen.org/archives/html/xen-devel/2011-10/msg01790.html
>>
>> Regards,
>> Jim
>>     
>
> ----
> What is the formal meaning of the one-line program
> #include "/dev/tty"
>
> J.P. McDermott			building 12
> Code 5542			john.mcdermott@nrl.navy.mil
> Naval Research Laboratory	voice: +1 202.404.8301
> Washington, DC 20375, US	fax:   +1 202.404.7942
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>   

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[John McDermott CIV]

Thanks, that's what I though from looking at the patch, but I am completely ignorant of the tools part of codebase.

Should I report this as a separate problem? We (NRL) really like xl and want to use it. What sort of debug dumps should I make?

Sincerely,

John
----


On Jan 25, 2012, at 1:02 PM, Jim Fehlig wrote:

> John McDermott CIV wrote:
>> So I should try this patch and report back if it works?
>> 
> 
> No, that patch causes the issue reported by Erin :-).  In Erin's case,
> the vif was not connected to the bridge.  In the problem you described,
> the vif was connected to the bridge but didn't work for other reasons.
> 
> Regards,
> Jim
> 
>> On Jan 25, 2012, at 12:33 PM, Jim Fehlig wrote:
>> 
>> 
>>> Ian Campbell wrote:
>>> 
>>>> Please don't top post.
>>>> 
>>>> On Wed, 2012-01-25 at 17:06 +0000, John McDermott CIV wrote:
>>>> 
>>>> 
>>>>> This problem also happens in Fedora 16. Should we be making bugzilla
>>>>> entries for problems with Xen?
>>>>> 
>>>>> 
>>>> There is more than one reason why networking may not work for you so
>>>> there isn't necessarily any reason (yet) to think you are seeing the
>>>> same issue.
>>>> 
>>>> 
>>> Yes, different issue.  I'd be really surprised if Fedora had this patch
>>> 
>>> http://old-list-archives.xen.org/archives/html/xen-devel/2011-10/msg01790.html
>>> 
>>> Regards,
>>> Jim
>>> 
>> 
>> ----
>> What is the formal meaning of the one-line program
>> #include "/dev/tty"
>> 
>> J.P. McDermott			building 12
>> Code 5542			john.mcdermott@nrl.navy.mil
>> Naval Research Laboratory	voice: +1 202.404.8301
>> Washington, DC 20375, US	fax:   +1 202.404.7942
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>> 
>> 

----
What is the formal meaning of the one-line program
#include "/dev/tty"

J.P. McDermott			building 12
Code 5542			john.mcdermott@nrl.navy.mil
Naval Research Laboratory	voice: +1 202.404.8301
Washington, DC 20375, US	fax:   +1 202.404.7942

Re: After switching from "xm" to "xl" toolstack, can't get Guest networking to work.

[Jim Fehlig]

John McDermott CIV wrote:
> Thanks, that's what I though from looking at the patch, but I am completely ignorant of the tools part of codebase.
>
> Should I report this as a separate problem? We (NRL) really like xl and want to use it. What sort of debug dumps should I make?
>   

As Ian suggested, I'd report this to Redhat bugzilla or the appropriate
fedora list.  I don't see the issue in openSUSE12.1.

Regards,
Jim

> Sincerely,
>
> John
> ----
>
>
> On Jan 25, 2012, at 1:02 PM, Jim Fehlig wrote:
>
>   
>> John McDermott CIV wrote:
>>     
>>> So I should try this patch and report back if it works?
>>>
>>>       
>> No, that patch causes the issue reported by Erin :-).  In Erin's case,
>> the vif was not connected to the bridge.  In the problem you described,
>> the vif was connected to the bridge but didn't work for other reasons.
>>
>> Regards,
>> Jim
>>
>>     
>>> On Jan 25, 2012, at 12:33 PM, Jim Fehlig wrote:
>>>
>>>
>>>       
>>>> Ian Campbell wrote:
>>>>
>>>>         
>>>>> Please don't top post.
>>>>>
>>>>> On Wed, 2012-01-25 at 17:06 +0000, John McDermott CIV wrote:
>>>>>
>>>>>
>>>>>           
>>>>>> This problem also happens in Fedora 16. Should we be making bugzilla
>>>>>> entries for problems with Xen?
>>>>>>
>>>>>>
>>>>>>             
>>>>> There is more than one reason why networking may not work for you so
>>>>> there isn't necessarily any reason (yet) to think you are seeing the
>>>>> same issue.
>>>>>
>>>>>
>>>>>           
>>>> Yes, different issue.  I'd be really surprised if Fedora had this patch
>>>>
>>>> http://old-list-archives.xen.org/archives/html/xen-devel/2011-10/msg01790.html
>>>>
>>>> Regards,
>>>> Jim
>>>>
>>>>         
>>> ----
>>> What is the formal meaning of the one-line program
>>> #include "/dev/tty"
>>>
>>> J.P. McDermott			building 12
>>> Code 5542			john.mcdermott@nrl.navy.mil
>>> Naval Research Laboratory	voice: +1 202.404.8301
>>> Washington, DC 20375, US	fax:   +1 202.404.7942
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>
>>>
>>>       
>
> ----
> What is the formal meaning of the one-line program
> #include "/dev/tty"
>
> J.P. McDermott			building 12
> Code 5542			john.mcdermott@nrl.navy.mil
> Naval Research Laboratory	voice: +1 202.404.8301
> Washington, DC 20375, US	fax:   +1 202.404.7942
>
>
>
>
>
>
>
>
>
>
>
>