Re: PV networking issue in 2.6.38.6 and 2.6.39-rc7 pvops dom0 kernels

[Mario <mario@slackverse.org>]

On 05/16/2011 11:03 AM, Mario wrote:
> On 05/16/2011 10:59 AM, Ian Campbell wrote:
>> On Mon, 2011-05-16 at 09:56 +0100, Mario wrote:
>>> On 05/16/2011 09:58 AM, Ian Campbell wrote:
>>>> On Mon, 2011-05-16 at 07:51 +0100, Pasi Kärkkäinen wrote:
>>>>> Hello,
>>>>>
>>>>> Ian: Any idea what could cause this error message (more info below):
>>>>>
>>>>> "XENBUS: Timeout connecting to device: device/vif/0 (state 1)"
>>>>
>>>> Is this from the domU or the dom0?
>>>
>>> This is from the domU.
>>>
>>>>
>>>> Do other devices work ok?
>>>>
>>>
>>> Yes, everything else seems to be working fine.
>>>
>>>
>>>>> User says replacing 2.6.38/2.6.39 pvops dom0 kernel with xenlinux one
>>>>> fixes the problem..
>>>>
>>>> 2.6.38 didn't have netback, only 2.6.39-rc1 did. Mario, please can you
>>>> supply the .config you used with the 2.6.39 kernel and the exact commit
>>>> id you are using. It would also be useful to see a complete log from
>>>> both dom0 and domU.
>>>>
>>>


Pvops kernel configs are here:

http://dev.slackverse.org/tmp/config-2.6.39-rc7-xen
http://dev.slackverse.org/tmp/config-2.6.37.6-xenU

They are both based on default configs shipped with Slackware, with 
minimal xen-only options enabled.


>>> Well, when i saw that 2.6.39-rc7 did not work, and i did not expect
>>> anyone to care, i simply wiped all of the pvops stuff that i was
>>> testing, and replaced it with a gentoo patched kernel wich works.
>>> If it will not be in vain, i will compile it again and give you more
>>> details.
>>
>> Yes please, we care very much about pvops to working well for everyone.
>> (I'm not sure why you would think otherwise, this has been the focus of
>> our development efforts for several years now).
>>
>> Ian.
>>
>
> Fair enough, it is compiling as we speak. Report to follow shortly.
>
> m.
>
>
>>>
>>>> Do the vif entries appear in XenStore? Using:
>>>> xenstore-ls /local/domain/$(xl domid<name>)/devices/vif
>>>> you should see some entries and those entries should reference a
>>>> backend
>>>> path which should also contains entries, please post both.
>>>>
>>>


##################
This is from dom0:
##################

# xenstore-ls /local/domain/$(xl domid slackware)/devices/vif
xenstore-ls: xs_directory (/local/domain/1/devices/vif): No such file or 
directory


However it works when i do this:

# xenstore-ls /local/domain/0/backend/vif/1/0
frontend = "/local/domain/1/device/vif/0"
frontend-id = "1"
online = "1"
state = "1"
script = "/etc/xen/scripts/vif-bridge"
mac = "00:16:3e:00:00:01"
bridge = "xenbr0"
handle = "0"


# xl block-list 1
Vdev  BE  handle state evt-ch ring-ref BE-path
51713 0   1      4     8      8        /local/domain/0/backend/qdisk/1/51713
268440065 0   1      4     9      9 
/local/domain/0/backend/qdisk/1/268440065


# xl network-list 1
Idx BE Mac Addr.         handle state evt-ch   tx-/rx-ring-ref BE-path 

0   0  00:16:3e:00:00:01      0     1     -1    -1/-1 
/local/domain/0/backend/vif/1/0


# brctl show
bridge name	bridge id		STP enabled	interfaces
xenbr0		8000.00219bfbf76e	no		eth0


# ifconfig -a | grep -v '^ '
eth0      Link encap:Ethernet  HWaddr 00:21:9b:fb:f7:6e
eth1      Link encap:Ethernet  HWaddr 00:21:9b:fb:f7:6f
lo        Link encap:Local Loopback
xenbr0    Link encap:Ethernet  HWaddr 00:21:9b:fb:f7:6e


# cat /var/log/xen/qemu-dm-slackware.log
domid: 1
Warning: vlan 0 is not connected to host network
-videoram option does not work with cirrus vga device model. Videoram 
set to 4M.
/tmp/SBo/xen-4.1.0/tools/ioemu-qemu-xen/hw/xen_blktap.c:628: Init blktap 
pipes
Could not open /var/run/tap/qemu-read-1
char device redirected to /dev/pts/1
xs_read(): target get error. /local/domain/1/target.
(qemu) (qemu)


- /var/log/xen/xend.log - empty
- /var/log/xen/xen-hotplug.log - also empty


# dmesg | grep -i xen
[    1.354427] xen/balloon: Initialising balloon driver.
[    1.368850] xen-balloon: Initialising balloon driver.
[    1.370157] Switching to clocksource xen
[    1.371868] xen: registering gsi 13 triggering 1 polarity 0
[    1.371872] xen_map_pirq_gsi: returning irq 13 for gsi 13
[    1.371876] xen: --> pirq=13 -> irq=13
[    1.372379] xen: registering gsi 8 triggering 1 polarity 0
[    1.372382] xen_map_pirq_gsi: returning irq 8 for gsi 8
[    1.372386] xen: --> pirq=8 -> irq=8
[    1.372912] xen: registering gsi 4 triggering 1 polarity 0
[    1.372915] xen_map_pirq_gsi: returning irq 4 for gsi 4
[    1.372919] xen: --> pirq=4 -> irq=4
[    1.386083] xen: registering gsi 16 triggering 0 polarity 1
[    1.386091] xen: --> pirq=16 -> irq=16
[    1.386119] xen: registering gsi 16 triggering 0 polarity 1
[    1.386122] xen_map_pirq_gsi: returning irq 16 for gsi 16
[    1.386126] xen: --> pirq=16 -> irq=16
[    1.386151] xen: registering gsi 16 triggering 0 polarity 1
[    1.386153] xen_map_pirq_gsi: returning irq 16 for gsi 16
[    1.386157] xen: --> pirq=16 -> irq=16
[    1.386182] xen: registering gsi 17 triggering 0 polarity 1
[    1.386186] xen: --> pirq=17 -> irq=17
[    5.175257] xen: registering gsi 23 triggering 0 polarity 1
[    5.175265] xen: --> pirq=23 -> irq=23
[   10.135114] xen: registering gsi 21 triggering 0 polarity 1
[   10.135125] xen: --> pirq=21 -> irq=21
[   10.159141] xen: registering gsi 16 triggering 0 polarity 1
[   10.159146] xen_map_pirq_gsi: returning irq 16 for gsi 16
[   10.159150] xen: --> pirq=16 -> irq=16
[   10.191968] usb usb1: Manufacturer: Linux 2.6.39-rc7-xen ehci_hcd
[   10.195303] xen: registering gsi 17 triggering 0 polarity 1
[   10.195308] xen_map_pirq_gsi: returning irq 17 for gsi 17
[   10.195312] xen: --> pirq=17 -> irq=17
[   10.486610] xen: registering gsi 21 triggering 0 polarity 1
[   10.486615] xen_map_pirq_gsi: returning irq 21 for gsi 21
[   10.486619] xen: --> pirq=21 -> irq=21
[   10.500379] usb usb2: Manufacturer: Linux 2.6.39-rc7-xen uhci_hcd
[   10.500601] xen: registering gsi 20 triggering 0 polarity 1
[   10.500608] xen: --> pirq=20 -> irq=20
[   10.540318] usb usb3: Manufacturer: Linux 2.6.39-rc7-xen uhci_hcd
[   10.540557] xen: registering gsi 21 triggering 0 polarity 1
[   10.540563] xen_map_pirq_gsi: returning irq 21 for gsi 21
[   10.540567] xen: --> pirq=21 -> irq=21
[   10.570445] usb usb4: Manufacturer: Linux 2.6.39-rc7-xen uhci_hcd
[   16.992548] ADDRCONF(NETDEV_UP): xenbr0: link is not ready
[   19.463299] xenbr0: port 1(eth0) entering forwarding state
[   19.463303] xenbr0: port 1(eth0) entering forwarding state
[   19.463363] ADDRCONF(NETDEV_CHANGE): xenbr0: link becomes ready
[   29.870099] xenbr0: no IPv6 routers present
[   34.470061] xenbr0: port 1(eth0) entering forwarding state
[   57.355646] XENBUS: Unable to read cpu state
[   57.356049] XENBUS: Unable to read cpu state

# xl dmesg
  __  __            _  _    _   ___
  \ \/ /___ _ __   | || |  / | / _ \
   \  // _ \ '_ \  | || |_ | || | | |
   /  \  __/ | | | |__   _|| || |_| |
  /_/\_\___|_| |_|    |_|(_)_(_)___/

(XEN) Xen version 4.1.0 (root@example.net) (gcc version 4.5.2 (GCC) ) 
Fri May 13 20:30:34 CEST 2011
(XEN) Latest ChangeSet: unavailable
(XEN) Bootloader: mbootpack v0.6 (alpha)
(XEN) Command line: auto BOOT_IMAGE=Linux.XEN ro root=900 dom0_mem=512M
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 2 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 2 MBR signatures
(XEN)  Found 2 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 00000000000a0000 (usable)
(XEN)  0000000000100000 - 000000007feb2000 (usable)
(XEN)  000000007feb2000 - 000000007fec8000 (reserved)
(XEN)  000000007fec8000 - 000000007fee7c00 (ACPI data)
(XEN)  000000007fee7c00 - 0000000080000000 (reserved)
(XEN)  00000000e0000000 - 00000000f0000000 (reserved)
(XEN)  00000000fe000000 - 0000000100000000 (reserved)
(XEN) System RAM: 2046MB (2095432kB)
(XEN) ACPI: RSDP 000F2980, 0024 (r2 DELL  )
(XEN) ACPI: XSDT 000F2A08, 0094 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: FACP 000F2B20, 00F4 (r3 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: DSDT 7FEC8000, 2545 (r1 DELL   PE_SC3          1 INTL 20050624)
(XEN) ACPI: FACS 7FEE7C00, 0040
(XEN) ACPI: APIC 000F2C14, 0072 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: SPCR 000F2C93, 0050 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: HPET 000F2CE3, 0038 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: MCFG 000F2D1B, 003C (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: WD__ 000F2D57, 0134 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: SLIC 000F2E8B, 0024 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: ERST 7FECB114, 0210 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: HEST 7FECB324, 027C (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: BERT 7FECAF94, 0030 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: EINJ 7FECAFC4, 0150 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: SSDT 7FECA545, 032C (r1 DELL   PE_SC3         11 INTL 20050624)
(XEN) ACPI: SSDT 7FECA9BB, 032C (r1 DELL   PE_SC3         11 INTL 20050624)
(XEN) ACPI: SSDT 7FECAE31, 0162 (r1 DELL   PE_SC3         10 INTL 20050624)
(XEN) Domain heap initialised
(XEN) Processor #0 6:15 APIC version 20
(XEN) Processor #1 6:15 APIC version 20
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) ERST table is invalid
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2400.141 MHz processor.
(XEN) Initing memory sharing.
(XEN) I/O virtualisation disabled
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 16 KiB.
(XEN) Brought up 2 CPUs
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1b72000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   000000007a000000->000000007c000000 (121577 pages 
to be allocated)
(XEN)  Init. ramdisk: 000000007f4e9000->000000007f9ff200
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff81b72000
(XEN)  Init. ramdisk: ffffffff81b72000->ffffffff82088200
(XEN)  Phys-Mach map: ffffffff82089000->ffffffff82189000
(XEN)  Start info:    ffffffff82189000->ffffffff821894b4
(XEN)  Page tables:   ffffffff8218a000->ffffffff8219f000
(XEN)  Boot stack:    ffffffff8219f000->ffffffff821a0000
(XEN)  TOTAL:         ffffffff80000000->ffffffff82400000
(XEN)  ENTRY ADDRESS: ffffffff818dd200
(XEN) Dom0 has maximum 2 VCPUs
(XEN) Scrubbing Free RAM: ...............done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch 
input to Xen)
(XEN) Freed 220kB init memory.


And domU config:

# cat slackware
name = "slackware"
memory = 256
vcpus = 1
vif = ["bridge=xenbr0,mac=00:16:3e:00:00:01"]
disk = 
["file:/var/test/xvda1.img,xvda1,w","file:/var/test/xvds1.img,xvds1,w"]
bootloader = "/usr/bin/pygrub"
extra = "fastboot"
root = "/dev/xvda1"




#####################
Some stuff from domU:
#####################

# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:00:00:00:00:00
           BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)


# dmesg | grep -i xen
[    0.000000] Linux version 2.6.37.6-xenU (root@orion) (gcc version 
4.5.2 (GCC) ) #1 SMP Sun Apr 24 14:37:22 CEST 2011
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000000010800000 (usable)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.1.0 (preserve-AD)
[    0.000000] Xen: using vcpuop timer interface
[    0.000000] installing Xen timer for CPU 0
[    0.060643] PCI: setting up Xen PCI frontend stub
[    1.250219] xen_balloon: Initialising balloon driver.
[    1.261020] Switching to clocksource xen
[    1.308267] Initialising Xen virtual ethernet driver.
[    6.410098] XENBUS: Waiting for devices to initialise: 
295s...290s...285s...280s...275s...270s...265s...260s...255s...250s...245s...240s...235s...230s...225s...220s...215s...210s...205s...200s...195s...190s...185s...180s...175s...170s...165s...160s...155s...150s...145s...140s...135s...130s...125s...120s...115s...110s...105s...100s...95s...90s...85s...80s...75s...70s...65s...60s...55s...50s...45s...40s...35s...30s...25s...20s...15s...10s...5s...0s...
[  301.410245] XENBUS: Timeout connecting to device: device/vif/0 (local 
state 1, remote state 1)


Hope it helps, i might not be able to power this box on any more, but if 
you need something else i will keep the drives and boot them up in 
another server on your request..


Best regards,
mario




>>> Well, from what i remember, only vhd was being shown, but i will try and
>>> get you exact info today.
>>>
>>>
>>>> Ian.
>>>>
>>>>>
>>>>> -- Pasi
>>>>>
>>>>> On Sun, May 15, 2011 at 07:24:54PM +0200, Mario wrote:
>>>>>>
>>>>>>
>>>>>> On 05/15/2011 07:20 PM, Pasi Kärkkäinen wrote:
>>>>>>> On Sun, May 15, 2011 at 08:03:45PM +0300, Pasi Kärkkäinen wrote:
>>>>>>>> On Sun, May 15, 2011 at 06:41:25PM +0200, Mario wrote:
>>>>>>>>> On 05/15/2011 06:34 PM, Pasi Kärkkäinen wrote:
>>>>>>>>>> On Sun, May 15, 2011 at 06:24:11PM +0200, Mario wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 05/15/2011 06:12 PM, Pasi Kärkkäinen wrote:
>>>>>>>>>>>> On Sun, May 15, 2011 at 06:02:02PM +0200, Mario wrote:
>>>>>>>>>>>>> On 05/15/2011 05:53 PM, Pasi Kärkkäinen wrote:
>>>>>>>>>>>>>> On Sun, May 15, 2011 at 02:36:11PM +0200, Mario wrote:
>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have been trying to make Xen-4.1.0 work on kernels
>>>>>>>>>>>>>>> mentioned in the
>>>>>>>>>>>>>>> subject, without much sucess. HVM guests are not
>>>>>>>>>>>>>>> affected, only PV.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Below you mention gentoo xenlinux patches..
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So please be more specific about which kernel was vanilla
>>>>>>>>>>>>>> (pvops),
>>>>>>>>>>>>>> and which kernels had some external patches applied..
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -- Pasi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sorry, both kernels in subject were vanilla from
>>>>>>>>>>>>> kernel.org, with xen
>>>>>>>>>>>>> options selected for dom0 usage.
>>>>>>>>>>>>> I only wanted to point out that networking works if i use
>>>>>>>>>>>>> kernel patched
>>>>>>>>>>>>> with rebased gentoo patches.
>>>>>>>>>>>>> I am talking only about dom0 kernel here, domU kernel does
>>>>>>>>>>>>> not seem to
>>>>>>>>>>>>> be the source of the problem.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> OK. What kernel is the PV domU running? What Xen version are
>>>>>>>>>>>> you using? 4.1 ?
>>>>>>>>>>>>
>>>>>>>>>>>> -- Pasi
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Well, i have tried following kernels in domU:
>>>>>>>>>>>
>>>>>>>>>>> 2.6.31.6 (i got this one from stacklet)
>>>>>>>>>>> 2.6.37.6-domU
>>>>>>>>>>> 2.6.33.4-domU
>>>>>>>>>>
>>>>>>>>>> Are these pvops/vanilla or with some patches?
>>>>>>>>>
>>>>>>>>> Those are pvops/vanilla, no patches.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> 2.6.38.6-xen (dom0 kernel)
>>>>>>>>>>> 2.6.34.7-xen (dom0 kernel)
>>>>>>>>>>> 2.6.39-rc7-xen (dom0 kernel)
>>>>>>>>>>>
>>>>>>>>>>> None of them made any difference, they were all complaining
>>>>>>>>>>> about vif.
>>>>>>>>>>> Yeah, I am running Xen 4.1
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Are you using xm/xend or xl/libxl toolstack?
>>>>>>>>>
>>>>>>>>> I am using xl/libxl with Xen 4.1
>>>>>>>>>
>>>>>>>>
>>>>>>>> Ok. Have you tried Jeremy's xen.git xen/stable-2.6.32.x as a
>>>>>>>> dom0 kernel?
>>>>>>>> It would be interesting to know if that works for you or not.
>>>>>>>>
>>>>>>>
>>>>>>> Btw. Did you realize upstream 2.6.38 / 2.6.39 does not yet have
>>>>>>> xen-blkback driver included? also blktap2 driver is missing.
>>>>>>>
>>>>>>> So you you can only use the phy: or file: backends, assuming Xen
>>>>>>> 4.1,
>>>>>>> which has the (slow) fallback qemu-based userspace blkback
>>>>>>> implementation.
>>>>>>>
>>>>>>> -- Pasi
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Yes i have, but seeing that HVM networking works, i wanted to
>>>>>> report the
>>>>>> PV one that doesn't.
>>>>>>
>>>>>>
>>>>>> m.
>>>>>>
>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Error that i see in dmesg on the domU:
>>>>>>>>>>>>>>> XENBUS: Timeout connecting to device: device/vif/0 (state 1)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ifconfig -a on domU shows MAC address: 00:00:00:00:00:00
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> dom0 doesn't have any vif/tap interfaces assigned to domU
>>>>>>>>>>>>>>> when i do
>>>>>>>>>>>>>>> brctl show or ifconfig -a
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I don't see any errors worth mentioning in /var/log/xen,
>>>>>>>>>>>>>>> except mybe
>>>>>>>>>>>>>>> these two:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> /tmp/SBo/xen-4.1.0/tools/ioemu-qemu-xen/hw/xen_blktap.c:628:
>>>>>>>>>>>>>>> Init blktap
>>>>>>>>>>>>>>> pipes
>>>>>>>>>>>>>>> Could not open /var/run/tap/qemu-read-2
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Problem goes away when i boot 2.6.34.7 (it is the last
>>>>>>>>>>>>>>> kernel that i
>>>>>>>>>>>>>>> have built with gentoo rebased patches).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If there is something i can do to help debug this please
>>>>>>>>>>>>>>> do let me know.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>> mario
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> Xen-users mailing list
>>>>>>>>>>>>>>> Xen-users@lists.xensource.com
>>>>>>>>>>>>>>> http://lists.xensource.com/xen-users
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Xen-users mailing list
>>>>>>>>>>>>>> Xen-users@lists.xensource.com
>>>>>>>>>>>>>> http://lists.xensource.com/xen-users
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Xen-users mailing list
>>>>>>>>>>>> Xen-users@lists.xensource.com
>>>>>>>>>>>> http://lists.xensource.com/xen-users
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Xen-users mailing list
>>>>>>>>>> Xen-users@lists.xensource.com
>>>>>>>>>> http://lists.xensource.com/xen-users
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Xen-users mailing list
>>>>>>>> Xen-users@lists.xensource.com
>>>>>>>> http://lists.xensource.com/xen-users
>>>>
>>>>
>>
>>