unstable network trouble

unstable network trouble

[Gerd Knorr]

   Hi,

With current unstable (hg7500) I have the problem that networking for 
domU's doesn't work reliable.  Seems to be more or less random, havn't 
seen any pattern yet, sometimes it works normally and sometimes it doesn't.

In case it does _not_ work it looks like this (domain 1 runs fine, 
domain 3 is broken):

The xen-backend device is present in sysfs:

   master-xen root ~# ls /sys/devices/xen-backend
   .  ..  vbd-1-768  vbd-1-832  vbd-3-51712  vif-1-0  vif-3-0

The vif network interface is missing:

   master-xen root ~# ip link ls | grep vif
   3: vif0.0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
   6: vif1.0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue

/etc/xen/scripts/vif-bridge is not called according to /var/log/messages:

   master-xen root /var/log# grep vif-bridge /var/log/messages
   Oct 27 15:22:50 master-xen logger: /etc/xen/scripts/vif-bridge: up 
XENBUS_PATH=backend/vif/1/0

The backend entries in xenstore look just fine:

master-xen root ~# xenstore-dump backend/vif
1/
     0/
         bridge           xenbr0
         mac              aa:00:57:4f:25:a7
         handle           0
         script           /etc/xen/scripts/vif-bridge
         frontend-id      1
         domain           debian
         frontend         /local/domain/1/device/vif/0
3/
     0/
         bridge           xenbr0
         mac              aa:00:ca:d7:83:e1
         handle           0
         script           /etc/xen/scripts/vif-bridge
         frontend-id      3
         domain           gentoo
         frontend         /local/domain/3/device/vif/0

ideas anyone?

cheers,

   Gerd

Re: unstable network trouble

[Ewan Mellor]

On Thu, Oct 27, 2005 at 03:57:50PM +0200, Gerd Knorr wrote:

> With current unstable (hg7500) I have the problem that networking for 
> domU's doesn't work reliable.  Seems to be more or less random, havn't 
> seen any pattern yet, sometimes it works normally and sometimes it doesn't.

There does seem to be a problem with the hotplug scripts either not running at
all or bailing out early.  Please could you put some tracing into the relevant
scripts -- /etc/hotplug/xen-backend.agent,
/etc/hotplug.d/default/default.hotplug, and / or
/etc/udev/rules.d/xen-backend.rules, depending on your system, and see if you
can figure out whether the hotplug event is firing at all, and if it is, where
the script bails out.

To have got the device in /sys/devices/xen-backend tells us that it's managed
almost everything before the hotplug scripts get started, so this shouldn't be
difficult to track down.

Thanks,

Ewan.

Re: unstable network trouble

[Gerd Knorr]

> There does seem to be a problem with the hotplug scripts either not running at
> all or bailing out early.

Hmm, no, seems to be a missing hotplug event.

I've added a debug line to xen-backend.rules:

SUBSYSTEM=="xen-backend", RUN+="/bin/logger -p daemon.debug -- 
xen-backend: $env{ACTION} %k"

A successfull start looks like this:

   Oct 27 17:34:50 master-xen logger: xen-backend: add vbd-19-768
   Oct 27 17:34:50 master-xen logger: xen-backend: add vbd-19-832
   Oct 27 17:34:50 master-xen logger: xen-backend: add vif-19-0
   Oct 27 17:34:50 master-xen logger: xen-backend: online vif-19-0

With broken network the "online" event is missing:

   Oct 27 17:35:31 master-xen logger: xen-backend: add vbd-20-832
   Oct 27 17:35:31 master-xen logger: xen-backend: add vbd-20-768
   Oct 27 17:35:31 master-xen logger: xen-backend: add vif-20-0

Oh, and btw, reboot doesn't work all the time either.  Sometimes the 
domain comes up again as it should, and sometimes it doesn't.

cheers,

   Gerd

Re: unstable network trouble

[Ewan Mellor]

On Thu, Oct 27, 2005 at 05:43:38PM +0200, Gerd Knorr wrote:

> Oh, and btw, reboot doesn't work all the time either.  Sometimes the 
> domain comes up again as it should, and sometimes it doesn't.

That one really is odd.  Could you post your /var/log/xend.log when that
happens?

Ewan.

Re: unstable network trouble

[Gerd Knorr]

[-- Attachment #1: Type: text/plain, Size: 621 bytes --]

Ewan Mellor wrote:
> On Thu, Oct 27, 2005 at 05:43:38PM +0200, Gerd Knorr wrote:
> 
> That one really is odd.  Could you post your /var/log/xend.log when that
> happens?

Attached.  The "error: (12, 'Cannot allocate memory')" line looks 
strange.  The domain has 128 MB assigned.  Now (after the failed reboot) 
it looks like this:

    master-xen root ~# xm info | grep memory
    memory                 : 983
    free_memory            : 132

Should be enougth memory.  I also have "(dom0-min-mem 256)" configured 
for xend, so it should happily take away memory from dom0 in case it 
needs more ...

cheers,

   Gerd


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: xend.log --]
[-- Type: text/x-log; name="xend.log", Size: 11176 bytes --]

[2005-10-28 10:54:16 xend] INFO (SrvDaemon:269) Xend Daemon started
[2005-10-28 10:54:16 xend] INFO (SrvDaemon:273) Xend changeset: Thu Oct 27 11:53:02 2005 +0100 7502:e70ea9465b31.
[2005-10-28 10:54:16 xend.XendDomainInfo] DEBUG (XendDomainInfo:168) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 8738451167L, 'ssidref': 0, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'shutdown_reason': 0, 'dying': 0, 'dom': 0, 'mem_kb': 972800, 'maxmem_kb': -4, 'max_vcpu_id': 1, 'crashed': 0, 'running': 1, 'shutdown': 0, 'online_vcpus': 2, 'blocked': 0})
[2005-10-28 10:54:16 xend.XendDomainInfo] INFO (XendDomainInfo:180) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000.
[2005-10-28 10:54:16 xend.XendDomainInfo] WARNING (XendDomainInfo:202) No vm path in store for existing domain 0
[2005-10-28 10:54:16 xend.XendDomainInfo] DEBUG (XendDomainInfo:589) Storing VM details: {'ssidref': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart', 'on_poweroff': 'destroy', 'vcpus': '1', 'on_crash': 'restart', 'memory/target': '972800', 'vcpu_avail': '1', 'name': 'Domain-0'}
[2005-10-28 10:54:16 xend.XendDomainInfo] DEBUG (XendDomainInfo:614) Storing domain details: {'cpu/0/availability': 'online', 'memory/target': '972800', 'name': 'Domain-0', 'console/limit': '1048576', 'vm': '/vm/00000000-0000-0000-0000-000000000000', 'domid': '0'}
[2005-10-28 10:54:16 xend] DEBUG (XendDomain:140) number of vcpus to use is 0
[2005-10-28 10:54:16 xend] INFO (SrvServer:112) unix path=/var/lib/xend/xend-socket
[2005-10-28 10:54:32 xend.XendDomainInfo] DEBUG (XendDomainInfo:147) XendDomainInfo.create(['vm', ['name', 'debian'], ['memory', '128'], ['ssidref', '0'], ['image', ['linux', ['kernel', '/vm/debian/vmlinuz3'], ['ip', ':1.2.3.4::::eth0:dhcp'], ['root', '/dev/hda1 ro'], ['vcpus', '1'], ['vcpus', '1'], ['boot', 'c']]], ['device', ['vbd', ['uname', 'phy:xen/debian'], ['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'file:/vm/debian/disk.img'], ['dev', 'hdb'], ['mode', 'w']]], ['device', ['vif', ['mac', 'aa:00:57:4f:25:a7']]]])
[2005-10-28 10:54:32 xend.XendDomainInfo] DEBUG (XendDomainInfo:250) parseConfig: config is ['vm', ['name', 'debian'], ['memory', '128'], ['ssidref', '0'], ['image', ['linux', ['kernel', '/vm/debian/vmlinuz3'], ['ip', ':1.2.3.4::::eth0:dhcp'], ['root', '/dev/hda1 ro'], ['vcpus', '1'], ['vcpus', '1'], ['boot', 'c']]], ['device', ['vbd', ['uname', 'phy:xen/debian'], ['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'file:/vm/debian/disk.img'], ['dev', 'hdb'], ['mode', 'w']]], ['device', ['vif', ['mac', 'aa:00:57:4f:25:a7']]]]
[2005-10-28 10:54:32 xend.XendDomainInfo] DEBUG (XendDomainInfo:308) parseConfig: result is {'ssidref': 0, 'name': 'debian', 'on_reboot': None, 'maxmem': None, 'on_poweroff': None, 'mem_kb': None, 'maxmem_kb': None, 'on_crash': None, 'backend': [], 'vcpus': 1, 'cpu_weight': None, 'vcpu_avail': None, 'memory': 128, 'device': [('vbd', ['vbd', ['uname', 'phy:xen/debian'], ['dev', 'hda'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'file:/vm/debian/disk.img'], ['dev', 'hdb'], ['mode', 'w']]), ('vif', ['vif', ['mac', 'aa:00:57:4f:25:a7']])], 'bootloader': None, 'image': ['linux', ['kernel', '/vm/debian/vmlinuz3'], ['ip', ':1.2.3.4::::eth0:dhcp'], ['root', '/dev/hda1 ro'], ['vcpus', '1'], ['vcpus', '1'], ['boot', 'c']], 'cpu': None, 'uuid': None}
[2005-10-28 10:54:32 xend.XendDomainInfo] DEBUG (XendDomainInfo:1047) XendDomainInfo.construct: None 0
[2005-10-28 10:54:32 xend.XendDomainInfo] DEBUG (XendDomainInfo:1077) XendDomainInfo.initDomain: 1 131072 1.0
[2005-10-28 10:54:32 xend] INFO (image:131) buildDomain os=linux dom=1 vcpus=1
[2005-10-28 10:54:32 xend] DEBUG (image:169) dom            = 1
[2005-10-28 10:54:32 xend] DEBUG (image:170) image          = /vm/debian/vmlinuz3
[2005-10-28 10:54:32 xend] DEBUG (image:171) store_evtchn   = 1
[2005-10-28 10:54:32 xend] DEBUG (image:172) console_evtchn = 2
[2005-10-28 10:54:32 xend] DEBUG (image:173) cmdline        =  ip=:1.2.3.4::::eth0:dhcp root=/dev/hda1 ro
[2005-10-28 10:54:32 xend] DEBUG (image:174) ramdisk        = 
[2005-10-28 10:54:32 xend] DEBUG (image:175) vcpus          = 1
[2005-10-28 10:54:32 xend] DEBUG (DevController:230) DevController: writing {'virtual-device': '768', 'backend-id': '0', 'backend': '/local/domain/0/backend/vbd/1/768'} to /local/domain/1/device/vbd/768.
[2005-10-28 10:54:32 xend] DEBUG (DevController:232) DevController: writing {'domain': 'debian', 'params': 'xen/debian', 'frontend-id': '1', 'type': 'phy', 'dev': 'hda', 'frontend': '/local/domain/1/device/vbd/768'} to /local/domain/0/backend/vbd/1/768.
[2005-10-28 10:54:32 xend] DEBUG (blkif:24) exception looking up device number for hdb: [Errno 2] No such file or directory: '/dev/hdb'
[2005-10-28 10:54:32 xend] DEBUG (DevController:230) DevController: writing {'virtual-device': '832', 'backend-id': '0', 'backend': '/local/domain/0/backend/vbd/1/832'} to /local/domain/1/device/vbd/832.
[2005-10-28 10:54:32 xend] DEBUG (DevController:232) DevController: writing {'domain': 'debian', 'params': '/vm/debian/disk.img', 'frontend-id': '1', 'type': 'file', 'dev': 'hdb', 'frontend': '/local/domain/1/device/vbd/832'} to /local/domain/0/backend/vbd/1/832.
[2005-10-28 10:54:32 xend] DEBUG (DevController:230) DevController: writing {'backend-id': '0', 'mac': 'aa:00:57:4f:25:a7', 'handle': '0', 'backend': '/local/domain/0/backend/vif/1/0'} to /local/domain/1/device/vif/0.
[2005-10-28 10:54:32 xend] DEBUG (DevController:232) DevController: writing {'bridge': 'xenbr0', 'mac': 'aa:00:57:4f:25:a7', 'handle': '0', 'script': '/etc/xen/scripts/vif-bridge', 'frontend-id': '1', 'domain': 'debian', 'frontend': '/local/domain/1/device/vif/0'} to /local/domain/0/backend/vif/1/0.
[2005-10-28 10:54:32 xend.XendDomainInfo] DEBUG (XendDomainInfo:589) Storing VM details: {'ssidref': '0', 'uuid': '36e0c3fb-a7c3-b33d-3006-4cba42f26e79', 'on_reboot': 'restart', 'image': "(linux (kernel /vm/debian/vmlinuz3) (ip :1.2.3.4::::eth0:dhcp) (root '/dev/hda1 ro') (vcpus 1) (vcpus 1) (boot c))", 'on_poweroff': 'destroy', 'vcpus': '1', 'on_crash': 'restart', 'memory/target': '131072', 'vcpu_avail': '1', 'name': 'debian'}
[2005-10-28 10:54:32 xend.XendDomainInfo] DEBUG (XendDomainInfo:614) Storing domain details: {'console/ring-ref': '145391', 'console/port': '2', 'name': 'debian', 'console/limit': '1048576', 'vm': '/vm/36e0c3fb-a7c3-b33d-3006-4cba42f26e79', 'domid': '1', 'cpu/0/availability': 'online', 'memory/target': '131072', 'store/ring-ref': '145392', 'store/port': '1'}
[2005-10-28 10:54:33 xend] INFO (XendDomain:349) Domain debian (1) unpaused.
[2005-10-28 10:55:49 xend.XendDomainInfo] INFO (XendDomainInfo:741) Domain has shutdown: name=debian id=1 reason=reboot.
[2005-10-28 10:55:49 xend.XendDomainInfo] ERROR (XendDomainInfo:1278) None
[2005-10-28 10:55:49 xend.XendDomainInfo] DEBUG (XendDomainInfo:1164) XendDomainInfo.destroyDomain(1)
[2005-10-28 10:55:49 xend.XendDomainInfo] DEBUG (XendDomainInfo:147) XendDomainInfo.create(['domain', ['domid', 1], ['memory', 127], ['name', 'debian'], ['uuid', '36e0c3fb-a7c3-b33d-3006-4cba42f26e79'], ['ssidref', 0], ['vcpus', 1], ['vcpu_avail', 1], ['cpu_weight', 1.0], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['maxmem', 127], ['image', ['linux', ['kernel', '/vm/debian/vmlinuz3'], ['ip', ':1.2.3.4::::eth0:dhcp'], ['root', '/dev/hda1 ro'], ['vcpus', '1'], ['vcpus', '1'], ['boot', 'c']]], ['device', ['vbd', ['uname', 'phy:xen/debian'], ['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'file:/vm/debian/disk.img'], ['dev', 'hdb'], ['mode', 'w']]], ['device', ['vif', ['mac', 'aa:00:57:4f:25:a7']]], ['state', '---s--'], ['shutdown_reason', 'reboot'], ['cpu_time', 7.3716442630000003], ['vcpus', 1], ['up_time', '76.4017469883'], ['start_time', '1130489672.99'], ['store_mfn', 145392], ['console_mfn', 145391]])
[2005-10-28 10:55:49 xend.XendDomainInfo] DEBUG (XendDomainInfo:250) parseConfig: config is ['domain', ['domid', 1], ['memory', 127], ['name', 'debian'], ['uuid', '36e0c3fb-a7c3-b33d-3006-4cba42f26e79'], ['ssidref', 0], ['vcpus', 1], ['vcpu_avail', 1], ['cpu_weight', 1.0], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['maxmem', 127], ['image', ['linux', ['kernel', '/vm/debian/vmlinuz3'], ['ip', ':1.2.3.4::::eth0:dhcp'], ['root', '/dev/hda1 ro'], ['vcpus', '1'], ['vcpus', '1'], ['boot', 'c']]], ['device', ['vbd', ['uname', 'phy:xen/debian'], ['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'file:/vm/debian/disk.img'], ['dev', 'hdb'], ['mode', 'w']]], ['device', ['vif', ['mac', 'aa:00:57:4f:25:a7']]], ['state', '---s--'], ['shutdown_reason', 'reboot'], ['cpu_time', 7.3716442630000003], ['vcpus', 1], ['up_time', '76.4017469883'], ['start_time', '1130489672.99'], ['store_mfn', 145392], ['console_mfn', 145391]]
[2005-10-28 10:55:49 xend.XendDomainInfo] DEBUG (XendDomainInfo:308) parseConfig: result is {'ssidref': 0, 'name': 'debian', 'on_reboot': 'restart', 'maxmem': 127, 'on_poweroff': 'destroy', 'mem_kb': None, 'maxmem_kb': None, 'on_crash': 'restart', 'backend': [], 'vcpus': 1, 'cpu_weight': 1.0, 'vcpu_avail': 1, 'memory': 127, 'device': [('vbd', ['vbd', ['uname', 'phy:xen/debian'], ['dev', 'hda'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'file:/vm/debian/disk.img'], ['dev', 'hdb'], ['mode', 'w']]), ('vif', ['vif', ['mac', 'aa:00:57:4f:25:a7']])], 'bootloader': None, 'image': ['linux', ['kernel', '/vm/debian/vmlinuz3'], ['ip', ':1.2.3.4::::eth0:dhcp'], ['root', '/dev/hda1 ro'], ['vcpus', '1'], ['vcpus', '1'], ['boot', 'c']], 'cpu': None, 'uuid': '36e0c3fb-a7c3-b33d-3006-4cba42f26e79'}
[2005-10-28 10:55:49 xend.XendDomainInfo] DEBUG (XendDomainInfo:1047) XendDomainInfo.construct: None 0
[2005-10-28 10:55:49 xend.XendDomainInfo] DEBUG (XendDomainInfo:1077) XendDomainInfo.initDomain: 2 130048 1.0
[2005-10-28 10:55:49 xend.XendDomainInfo] ERROR (XendDomainInfo:158) Domain construction failed
Traceback (most recent call last):
  File "//usr/lib/python/xen/xend/XendDomainInfo.py", line 152, in create
    vm.initDomain()
  File "//usr/lib/python/xen/xend/XendDomainInfo.py", line 1093, in initDomain
    xc.domain_memory_increase_reservation(self.domid, m, 0, 0)
error: (12, 'Cannot allocate memory')
[2005-10-28 10:55:49 xend.XendDomainInfo] DEBUG (XendDomainInfo:1156) XendDomainInfo.destroy: domid=2
[2005-10-28 10:55:49 xend.XendDomainInfo] DEBUG (XendDomainInfo:1164) XendDomainInfo.destroyDomain(2)
[2005-10-28 10:55:49 xend.XendDomainInfo] ERROR (XendDomainInfo:1307) Failed to restart domain 1.
Traceback (most recent call last):
  File "//usr/lib/python/xen/xend/XendDomainInfo.py", line 1300, in restart
    new_dom = xd.domain_create(config)
  File "//usr/lib/python/xen/xend/XendDomain.py", line 215, in domain_create
    dominfo = XendDomainInfo.create(config)
  File "//usr/lib/python/xen/xend/XendDomainInfo.py", line 152, in create
    vm.initDomain()
  File "//usr/lib/python/xen/xend/XendDomainInfo.py", line 1093, in initDomain
    xc.domain_memory_increase_reservation(self.domid, m, 0, 0)
error: (12, 'Cannot allocate memory')

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: unstable network trouble

[Gerd Knorr]

   Hi,

>> That one really is odd.  Could you post your /var/log/xend.log when that
>> happens?
> 
> Attached.   The "error: (12, 'Cannot allocate memory')" line looks
> strange.  [ ... ]

Hmm, there seems to be more than one bug.  I've manually decreased the 
amount of memory for domain 0 so there is more free memory, which seems 
to sucessfully workaround the memory allocation issue.  Now I got this one:

   [2005-10-28 11:55:39 xend.XendDomainInfo] INFO (XendDomainInfo:741) 
Domain has shutdown: name=debian id=9 reason=reboot.
   [2005-10-28 11:55:39 xend.XendDomainInfo] ERROR (XendDomainInfo:1278) 
1130493287.56
   [2005-10-28 11:55:39 xend.XendDomainInfo] DEBUG (XendDomainInfo:1156) 
XendDomainInfo.destroy: domid=9
   [2005-10-28 11:55:39 xend.XendDomainInfo] DEBUG (XendDomainInfo:1164) 
XendDomainInfo.destroyDomain(9)

cheers,

   Gerd

Re: unstable network trouble

[Gerd Knorr]

> [2005-10-28 10:55:49 xend.XendDomainInfo] ERROR (XendDomainInfo:1307) Failed to restart domain 1.
> Traceback (most recent call last):
>   File "//usr/lib/python/xen/xend/XendDomainInfo.py", line 1300, in restart
>     new_dom = xd.domain_create(config)
>   File "//usr/lib/python/xen/xend/XendDomain.py", line 215, in domain_create
>     dominfo = XendDomainInfo.create(config)
>   File "//usr/lib/python/xen/xend/XendDomainInfo.py", line 152, in create
>     vm.initDomain()
>   File "//usr/lib/python/xen/xend/XendDomainInfo.py", line 1093, in initDomain
>     xc.domain_memory_increase_reservation(self.domid, m, 0, 0)
> error: (12, 'Cannot allocate memory')

Oh, xend-debug.log has something as well:

Failed allocation for dom 2: 32768 pages order 0 addr_bits 0

cheers,

   Gerd

Re: unstable network trouble

[Ewan Mellor]

On Fri, Oct 28, 2005 at 11:59:47AM +0200, Gerd Knorr wrote:

>   Hi,
> 
> >>That one really is odd.  Could you post your /var/log/xend.log when that
> >>happens?
> >
> >Attached.   The "error: (12, 'Cannot allocate memory')" line looks
> >strange.  [ ... ]
> 
> Hmm, there seems to be more than one bug.  I've manually decreased the 
> amount of memory for domain 0 so there is more free memory, which seems 
> to sucessfully workaround the memory allocation issue.  Now I got this one:
> 
>   [2005-10-28 11:55:39 xend.XendDomainInfo] INFO (XendDomainInfo:741) 
> Domain has shutdown: name=debian id=9 reason=reboot.
>   [2005-10-28 11:55:39 xend.XendDomainInfo] ERROR (XendDomainInfo:1278) 
> 1130493287.56
>   [2005-10-28 11:55:39 xend.XendDomainInfo] DEBUG (XendDomainInfo:1156) 
> XendDomainInfo.destroy: domid=9
>   [2005-10-28 11:55:39 xend.XendDomainInfo] DEBUG (XendDomainInfo:1164) 
> XendDomainInfo.destroyDomain(9)

I'm not sure about your Cannot allocate memory error -- I'll look into that.
The rebooting code in Xend, however, is just plain wrong.  I'll fix that
today.  Thanks for your help, Gerd.

Ewan.

Re: unstable network trouble

[Gerd Knorr]

> Hmm, no, seems to be a missing hotplug event.

Sticked some printk's into the net backend, looks like a xenbus issue:

domU boot with broken network:

   netback_probe: add backend watch
   netback_probe: add frontend watch
   netback_probe: done
   frontend_changed: called
   frontend_changed: called
   frontend_changed: called
   frontend_changed: called

domU boot with working network:

   netback_probe: add backend watch
   netback_probe: add frontend watch
   netback_probe: done
   backend_changed: called
   backend_changed: hotplug online
   netback_hotplug: called
   frontend_changed: called
   frontend_changed: called
   device vif2.0 entered promiscuous mode
   xenbr0: port 3(vif2.0) entering learning state
   xenbr0: topology change detected, propagating
   xenbr0: port 3(vif2.0) entering forwarding state
   frontend_changed: called
   frontend_changed: called
   frontend_changed: called

Note that backend_changed is never called in the first case.

cheers,

   Gerd