Re: [Xen-users] wiki page on 10G SRIOV

Re: [Xen-users] wiki page on 10G SRIOV

[kishore kumar]

[-- Attachment #1.1: Type: text/plain, Size: 3197 bytes --]

Hello Everyone,

I am facing a strange issue with my 10G 82599 PCIe device.

The Device Id has to be 0x10FB for SRIOV to function properly, But all of
sudden the Device ID started showing me as 0x10D8, and because of this the
SRIOV feature is not working and I cannot assign the Virtual Function to  my
guest.

When I did lspci command it is showing me as Intel Corporation Unknown
device 10d8 (rev 01).  Previously it was showing me Intel Corporation
82599EB Gigabit Networking with 10FB, and I was able to assign the Virtual
Function to my guest.

I googled and found out that the device ID 0x10D8 is a 82599 Unprogrammed
device.

Can any one please let me know what is happening here and how to recover
back from this?
I am totally blocked for 2 days because of this issue.

Thank You.
On Fri, Jul 16, 2010 at 1:05 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:

> On Fri, Jul 16, 2010 at 02:17:30AM +0000, wangalalei wrote:
> >    hi all:
> >
> >       i read intel 82576 card datasheet,and finding it saying that the
> 82576
> >    can set the vf's bandwidth,
> >    but when i test this card,i can not find any parameter to it
> >       any one else notice this problem? or anyone knows whether the 82576
> >    card can control its vf's bandwidth?
> >
>
> I think bandwidth management is a work-in-progress by Intel..
>
> -- Pasi
>
> >    alaleiwang
> >
> >
>  --------------------------------------------------------------------------
> >
> >    Date: Thu, 15 Jul 2010 18:35:57 -0700
> >    Subject: Re: [Xen-users] wiki page on 10G SRIOV
> >    From: bodkekumar@gmail.com
> >    To: pasik@iki.fi
> >    CC: xen-users@lists.xensource.com
> >
> >    Yeah, I configured 64 Virtual Functions in DOM-0  and passthru 1 VF to
> >    assign it to the guest.  It worked for me.
> >
> >    I assign static IP to my guest and tried to ping the remote machine.
> It
> >    worked for me.
> >
> >    But I am wondering that ping to DOM-0 from guest makes any sense here?
> >    Can you let me know on this?  ping worked though from guest to DOM-0
> as
> >    well.
> >
> >    On Thu, Jul 15, 2010 at 1:11 PM, Pasi KÀrkkÀinen <[1]pasik@iki.fi>
> >    wrote:
> >
> >      On Thu, Jun 24, 2010 at 10:56:28AM -0700, kishore kumar wrote:
> >      >    Hi All,
> >      >
> >      >    Can any one please let me know if there is a WIKI page which
> talks
> >      about
> >      >    how to configure SRIOV in Xen and use Physical Functions  and
> >      Virtual
> >      >    Functions ?
> >      >
> >      >    I searched xen wiki, but could not find.
> >      >
> >
> >      Each virtual function should be a separate PCI device, so you can
> >      passthru
> >      them like any other normal PCI device.
> >
> >      Did you get it to work?
> >      -- Pasi
> >
> >
>  --------------------------------------------------------------------------
> >
> >    è**倩+æ**玢+é*®ç®± æ*³èŠ*蜻æ*Ÿå*ºæžž,æ**æ*ºMSNåž®äœ æ**å®*!
> >    [2]ç«*å*»äž*蜜ïŒ*
> >
> > References
> >
> >    Visible links
> >    1. mailto:pasik@iki.fi
> >    2. http://3g.msn.cn/
>

[-- Attachment #1.2: Type: text/html, Size: 4284 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Re: [Xen-users] wiki page on 10G SRIOV

[Byrne, John (HP Labs)]

[-- Attachment #1.1: Type: text/plain, Size: 5743 bytes --]

I’m a bit confused with your device IDs. The 10fb device is the PCI id for the physical NIC; the 10d8 device is the SRIOV Virtual Function device created by the driver.  With the latest xen-unstable, I have successfully used this NIC to create and assign the VF devices to HVM guests. The ixgbevf driver, which binds to the VF devices, expects device id 0x10ed. I simply assumed that the driver was not quite up to date and tweaked things to work.  So, if you are just a bit confused, the following might help.

I added the following in my modprobe.d directory to create four VF devices (ixgbe.conf, attached):

options ixgbe max_vfs=4

So, on my machine, I can see the physical NIC and the 4 VFs created.

lspci –n | grep 0a:
0a:00.0 0200: 8086:10fb (rev 01)
0a:10.0 0200: 8086:10d8 (rev 01)
0a:10.2 0200: 8086:10d8 (rev 01)
0a:10.4 0200: 8086:10d8 (rev 01)
0a:10.6 0200: 8086:10d8 (rev 01)

I added the ixgbevf driver to both my dom0 and domUs. While it is a simple tweak to add the device id to the driver, I decided to play with modprobe instead and added the following to my modprobe.d directory in both dom0 and domU (ixgbvevf.conf, attached):

install ixgbevf /sbin/modprobe --ignore-install ixgbevf; echo "0x8086 0x10d8" >/
sys/bus/pci/drivers/ixgbevf/new_id

If you are working with HVM guests, there has to be a patch to  qemu as well. (Attached)  You also have to add “(pci-passthrough-strict-check no)” to your xend-config.sxp. (I don’t think you need the xend-config change for PV guests.)

Then it is simply a matter of adding the VF PCI devices to your domU configurations and configuring the ixgbevf devices in the guests.

Both the ixgbe and ixgbevf drivers will generate random MAC addresses for the devices. What  I wanted to do was to set the MAC addresses for the VF devices in dom0 and then my guests’ networking configuration was much easier. The physical NIC must be up. It doesn’t have to have an IP address, but if it isn’t up the MAC address set in dom0 will be lost. I’ve attached my dom0 rc.local where I set the MAC addresses and assign the devices to pciback.

I can’t swear that this is the best way to do this, but this did work.

John Byrne


From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of kishore kumar
Sent: Monday, July 19, 2010 10:16 AM
To: Pasi Kärkkäinen; xen-devel@lists.xensource.com
Cc: wangalalei; xen-users@lists.xensource.com
Subject: [Xen-devel] Re: [Xen-users] wiki page on 10G SRIOV


Hello Everyone,

I am facing a strange issue with my 10G 82599 PCIe device.

The Device Id has to be 0x10FB for SRIOV to function properly, But all of sudden the Device ID started showing me as 0x10D8, and because of this the SRIOV feature is not working and I cannot assign the Virtual Function to  my guest.

When I did lspci command it is showing me as Intel Corporation Unknown device 10d8 (rev 01).  Previously it was showing me Intel Corporation 82599EB Gigabit Networking with 10FB, and I was able to assign the Virtual Function to my guest.

I googled and found out that the device ID 0x10D8 is a 82599 Unprogrammed device.

Can any one please let me know what is happening here and how to recover back from this?
I am totally blocked for 2 days because of this issue.

Thank You.
On Fri, Jul 16, 2010 at 1:05 AM, Pasi Kärkkäinen <pasik@iki.fi<mailto:pasik@iki.fi>> wrote:
On Fri, Jul 16, 2010 at 02:17:30AM +0000, wangalalei wrote:
>    hi all:
>
>       i read intel 82576 card datasheet,and finding it saying that the 82576
>    can set the vf's bandwidth,
>    but when i test this card,i can not find any parameter to it
>       any one else notice this problem? or anyone knows whether the 82576
>    card can control its vf's bandwidth?
>
I think bandwidth management is a work-in-progress by Intel..

-- Pasi

>    alaleiwang
>
>    --------------------------------------------------------------------------
>
>    Date: Thu, 15 Jul 2010 18:35:57 -0700
>    Subject: Re: [Xen-users] wiki page on 10G SRIOV
>    From: bodkekumar@gmail.com<mailto:bodkekumar@gmail.com>
>    To: pasik@iki.fi<mailto:pasik@iki.fi>
>    CC: xen-users@lists.xensource.com<mailto:xen-users@lists.xensource.com>
>
>    Yeah, I configured 64 Virtual Functions in DOM-0  and passthru 1 VF to
>    assign it to the guest.  It worked for me.
>
>    I assign static IP to my guest and tried to ping the remote machine. It
>    worked for me.
>
>    But I am wondering that ping to DOM-0 from guest makes any sense here?
>    Can you let me know on this?  ping worked though from guest to DOM-0 as
>    well.
>
>    On Thu, Jul 15, 2010 at 1:11 PM, Pasi KÀrkkÀinen <[1]pasik@iki.fi<mailto:pasik@iki.fi>>
>    wrote:
>
>      On Thu, Jun 24, 2010 at 10:56:28AM -0700, kishore kumar wrote:
>      >    Hi All,
>      >
>      >    Can any one please let me know if there is a WIKI page which talks
>      about
>      >    how to configure SRIOV in Xen and use Physical Functions  and
>      Virtual
>      >    Functions ?
>      >
>      >    I searched xen wiki, but could not find.
>      >
>
>      Each virtual function should be a separate PCI device, so you can
>      passthru
>      them like any other normal PCI device.
>
>      Did you get it to work?
>      -- Pasi
>
>    --------------------------------------------------------------------------
>
>    è**倩+æ**玢+é*®ç®± æ*³èŠ*蜻æ*Ÿå*ºæžž,æ**æ*ºMSNåž®äœ æ**å®*!
>    [2]ç«*å*»äž*蜜ïŒ*
>
> References
>
>    Visible links
>    1. mailto:pasik@iki.fi<mailto:pasik@iki.fi>
>    2. http://3g.msn.cn/


[-- Attachment #1.2: Type: text/html, Size: 14389 bytes --]

[-- Attachment #2: qemu-10d8.patch --]
[-- Type: application/octet-stream, Size: 1031 bytes --]

diff --git a/hw/pass-through.c b/hw/pass-through.c
index 5a76e8d..9e406b8 100644
--- a/hw/pass-through.c
+++ b/hw/pass-through.c
@@ -2606,7 +2606,8 @@ static int pt_hide_dev_cap(const struct pci_dev *dev, uint8_t grp_id)
          * Register is 0. We should not try to expose it to guest.
          */
         if (dev->vendor_id == PCI_VENDOR_ID_INTEL &&
-                dev->device_id == PCI_DEVICE_ID_INTEL_82599_VF)
+	    (dev->device_id == PCI_DEVICE_ID_INTEL_82599_VF ||
+	     dev->device_id == PCI_DEVICE_ID_INTEL_82599EB_VF))
             return 1;
         break;
     }
diff --git a/hw/pci.h b/hw/pci.h
index de5a4e1..769e7d6 100644
--- a/hw/pci.h
+++ b/hw/pci.h
@@ -113,6 +113,7 @@ extern target_phys_addr_t pci_mem_base;
 #define PCI_DEVICE_ID_INTEL_82371AB_2    0x7112
 #define PCI_DEVICE_ID_INTEL_82371AB_3    0x7113
 #define PCI_DEVICE_ID_INTEL_82599_VF     0x10ed
+#define PCI_DEVICE_ID_INTEL_82599EB_VF   0x10d8
 
 #define PCI_VENDOR_ID_FSL                0x1957
 #define PCI_DEVICE_ID_FSL_E500           0x0030

[-- Attachment #3: rc.local --]
[-- Type: application/octet-stream, Size: 950 bytes --]

#!/bin/sh

#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

[ -e /proc/xen ] || exit 0

modprobe ixgbevf
modprobe xen-pciback

rebind() {
    echo $1
    [ ! -e /sys/bus/pci/devices/$1/driver/unbind ] || \
	echo -n $1 >/sys/bus/pci/devices/$1/driver/unbind
    echo -n $1 >/sys/bus/pci/drivers/pciback/new_slot
    echo -n $1 >/sys/bus/pci/drivers/pciback/bind
}

#	 0000:0a:11.0 0000:0a:11.2 0000:0a:11.4 0000:0a:11.6 ; do

setmac() {
    eth=`(cd /sys/class/net ; ls -l  ) | grep /$1/ | awk '{ print $8 }'`
    if [[ -n "$eth" ]]; then
	echo $1 $eth $2;
	ifconfig $eth hw ether $2;
    fi
}

while read pci mac; do setmac $pci $mac; rebind $pci; done <<EOF
0000:0a:10.0 00:16:3e:31:88:91
0000:0a:10.2 00:16:3e:31:88:92
0000:0a:10.4 00:16:3e:31:88:93
0000:0a:10.6 00:16:3e:31:88:94
EOF


[-- Attachment #4: ixgbevf.conf --]
[-- Type: application/octet-stream, Size: 115 bytes --]

install ixgbevf /sbin/modprobe --ignore-install ixgbevf; echo "0x8086 0x10d8" >/sys/bus/pci/drivers/ixgbevf/new_id

[-- Attachment #5: ixgbe.conf --]
[-- Type: application/octet-stream, Size: 25 bytes --]

options ixgbe max_vfs=4


[-- Attachment #6: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Re: [Xen-users] wiki page on 10G SRIOV

[Rose, Gregory V]

[-- Attachment #1.1: Type: text/plain, Size: 6365 bytes --]

If you have an 82599EB that is using 10D8h for the Virtual Function Device ID then the adapter is a very old, preproduction version or else the correct EEPROM image has been over-written with an invalid image.  The correct Device ID value for an 82599EB virtual function is 10EDh.

- Greg

From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Byrne, John (HP Labs)
Sent: Monday, July 19, 2010 2:22 PM
To: kishore kumar
Cc: xen-devel@lists.xensource.com; wangalalei; xen-users@lists.xensource.com
Subject: RE: [Xen-devel] Re: [Xen-users] wiki page on 10G SRIOV

I’m a bit confused with your device IDs. The 10fb device is the PCI id for the physical NIC; the 10d8 device is the SRIOV Virtual Function device created by the driver.  With the latest xen-unstable, I have successfully used this NIC to create and assign the VF devices to HVM guests. The ixgbevf driver, which binds to the VF devices, expects device id 0x10ed. I simply assumed that the driver was not quite up to date and tweaked things to work.  So, if you are just a bit confused, the following might help.

I added the following in my modprobe.d directory to create four VF devices (ixgbe.conf, attached):

options ixgbe max_vfs=4

So, on my machine, I can see the physical NIC and the 4 VFs created.

lspci –n | grep 0a:
0a:00.0 0200: 8086:10fb (rev 01)
0a:10.0 0200: 8086:10d8 (rev 01)
0a:10.2 0200: 8086:10d8 (rev 01)
0a:10.4 0200: 8086:10d8 (rev 01)
0a:10.6 0200: 8086:10d8 (rev 01)

I added the ixgbevf driver to both my dom0 and domUs. While it is a simple tweak to add the device id to the driver, I decided to play with modprobe instead and added the following to my modprobe.d directory in both dom0 and domU (ixgbvevf.conf, attached):

install ixgbevf /sbin/modprobe --ignore-install ixgbevf; echo "0x8086 0x10d8" >/
sys/bus/pci/drivers/ixgbevf/new_id

If you are working with HVM guests, there has to be a patch to  qemu as well. (Attached)  You also have to add “(pci-passthrough-strict-check no)” to your xend-config.sxp. (I don’t think you need the xend-config change for PV guests.)

Then it is simply a matter of adding the VF PCI devices to your domU configurations and configuring the ixgbevf devices in the guests.

Both the ixgbe and ixgbevf drivers will generate random MAC addresses for the devices. What  I wanted to do was to set the MAC addresses for the VF devices in dom0 and then my guests’ networking configuration was much easier. The physical NIC must be up. It doesn’t have to have an IP address, but if it isn’t up the MAC address set in dom0 will be lost. I’ve attached my dom0 rc.local where I set the MAC addresses and assign the devices to pciback.

I can’t swear that this is the best way to do this, but this did work.

John Byrne


From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of kishore kumar
Sent: Monday, July 19, 2010 10:16 AM
To: Pasi Kärkkäinen; xen-devel@lists.xensource.com
Cc: wangalalei; xen-users@lists.xensource.com
Subject: [Xen-devel] Re: [Xen-users] wiki page on 10G SRIOV


Hello Everyone,

I am facing a strange issue with my 10G 82599 PCIe device.

The Device Id has to be 0x10FB for SRIOV to function properly, But all of sudden the Device ID started showing me as 0x10D8, and because of this the SRIOV feature is not working and I cannot assign the Virtual Function to  my guest.

When I did lspci command it is showing me as Intel Corporation Unknown device 10d8 (rev 01).  Previously it was showing me Intel Corporation 82599EB Gigabit Networking with 10FB, and I was able to assign the Virtual Function to my guest.

I googled and found out that the device ID 0x10D8 is a 82599 Unprogrammed device.

Can any one please let me know what is happening here and how to recover back from this?
I am totally blocked for 2 days because of this issue.

Thank You.
On Fri, Jul 16, 2010 at 1:05 AM, Pasi Kärkkäinen <pasik@iki.fi<mailto:pasik@iki.fi>> wrote:
On Fri, Jul 16, 2010 at 02:17:30AM +0000, wangalalei wrote:
>    hi all:
>
>       i read intel 82576 card datasheet,and finding it saying that the 82576
>    can set the vf's bandwidth,
>    but when i test this card,i can not find any parameter to it
>       any one else notice this problem? or anyone knows whether the 82576
>    card can control its vf's bandwidth?
>
I think bandwidth management is a work-in-progress by Intel..

-- Pasi

>    alaleiwang
>
>    --------------------------------------------------------------------------
>
>    Date: Thu, 15 Jul 2010 18:35:57 -0700
>    Subject: Re: [Xen-users] wiki page on 10G SRIOV
>    From: bodkekumar@gmail.com<mailto:bodkekumar@gmail.com>
>    To: pasik@iki.fi<mailto:pasik@iki.fi>
>    CC: xen-users@lists.xensource.com<mailto:xen-users@lists.xensource.com>
>
>    Yeah, I configured 64 Virtual Functions in DOM-0  and passthru 1 VF to
>    assign it to the guest.  It worked for me.
>
>    I assign static IP to my guest and tried to ping the remote machine. It
>    worked for me.
>
>    But I am wondering that ping to DOM-0 from guest makes any sense here?
>    Can you let me know on this?  ping worked though from guest to DOM-0 as
>    well.
>
>    On Thu, Jul 15, 2010 at 1:11 PM, Pasi KÀrkkÀinen <[1]pasik@iki.fi<mailto:pasik@iki.fi>>
>    wrote:
>
>      On Thu, Jun 24, 2010 at 10:56:28AM -0700, kishore kumar wrote:
>      >    Hi All,
>      >
>      >    Can any one please let me know if there is a WIKI page which talks
>      about
>      >    how to configure SRIOV in Xen and use Physical Functions  and
>      Virtual
>      >    Functions ?
>      >
>      >    I searched xen wiki, but could not find.
>      >
>
>      Each virtual function should be a separate PCI device, so you can
>      passthru
>      them like any other normal PCI device.
>
>      Did you get it to work?
>      -- Pasi
>
>    --------------------------------------------------------------------------
>
>    è**倩+æ**玢+é*®ç®± æ*³èŠ*蜻æ*Ÿå*ºæžž,æ**æ*ºMSNåž®äœ æ**å®*!
>    [2]ç«*å*»äž*蜜ïŒ*
>
> References
>
>    Visible links
>    1. mailto:pasik@iki.fi<mailto:pasik@iki.fi>
>    2. http://3g.msn.cn/


[-- Attachment #1.2: Type: text/html, Size: 18709 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [Xen-users] wiki page on 10G SRIOV

[kishore kumar]

[-- Attachment #1.1: Type: text/plain, Size: 7151 bytes --]

I am using Ethernet Server Adapater X520 - DA2 (E10G42BTDA) Card.

For the Device Id 0x10FB, I was able to see all the 64 Virtual Functions,
when I run lspci command.
But when the Device ID started showing 0x10D8, I cannot see any Virtual
Functions, when I run lspci command.
On Mon, Jul 19, 2010 at 2:35 PM, Rose, Gregory V
<gregory.v.rose@intel.com>wrote:

>  If you have an 82599EB that is using 10D8h for the Virtual Function
> Device ID then the adapter is a very old, preproduction version or else the
> correct EEPROM image has been over-written with an invalid image.  The
> correct Device ID value for an 82599EB virtual function is 10EDh.
>
>
>
> - Greg
>
>
>
> *From:* xen-devel-bounces@lists.xensource.com [mailto:
> xen-devel-bounces@lists.xensource.com] *On Behalf Of *Byrne, John (HP
> Labs)
> *Sent:* Monday, July 19, 2010 2:22 PM
> *To:* kishore kumar
> *Cc:* xen-devel@lists.xensource.com; wangalalei;
> xen-users@lists.xensource.com
> *Subject:* RE: [Xen-devel] Re: [Xen-users] wiki page on 10G SRIOV
>
>
>
> I’m a bit confused with your device IDs. The 10fb device is the PCI id for
> the physical NIC; the 10d8 device is the SRIOV Virtual Function device
> created by the driver.  With the latest xen-unstable, I have successfully
> used this NIC to create and assign the VF devices to HVM guests. The ixgbevf
> driver, which binds to the VF devices, expects device id 0x10ed. I simply
> assumed that the driver was not quite up to date and tweaked things to work.
>  So, if you are just a bit confused, the following might help.
>
>
>
> I added the following in my modprobe.d directory to create four VF devices
> (ixgbe.conf, attached):
>
>
>
> options ixgbe max_vfs=4
>
>
>
> So, on my machine, I can see the physical NIC and the 4 VFs created.
>
>
>
> lspci –n | grep 0a:
>
> 0a:00.0 0200: 8086:10fb (rev 01)
>
> 0a:10.0 0200: 8086:10d8 (rev 01)
>
> 0a:10.2 0200: 8086:10d8 (rev 01)
>
> 0a:10.4 0200: 8086:10d8 (rev 01)
>
> 0a:10.6 0200: 8086:10d8 (rev 01)
>
>
>
> I added the ixgbevf driver to both my dom0 and domUs. While it is a simple
> tweak to add the device id to the driver, I decided to play with modprobe
> instead and added the following to my modprobe.d directory in both dom0 and
> domU (ixgbvevf.conf, attached):
>
>
>
> install ixgbevf /sbin/modprobe --ignore-install ixgbevf; echo "0x8086
> 0x10d8" >/
>
> sys/bus/pci/drivers/ixgbevf/new_id
>
>
>
> If you are working with HVM guests, there has to be a patch to  qemu as
> well. (Attached)  You also have to add “(pci-passthrough-strict-check no)”
> to your xend-config.sxp. (I don’t think you need the xend-config change for
> PV guests.)
>
>
>
> Then it is simply a matter of adding the VF PCI devices to your domU
> configurations and configuring the ixgbevf devices in the guests.
>
>
>
> Both the ixgbe and ixgbevf drivers will generate random MAC addresses for
> the devices. What  I wanted to do was to set the MAC addresses for the VF
> devices in dom0 and then my guests’ networking configuration was much
> easier. The physical NIC must be up. It doesn’t have to have an IP address,
> but if it isn’t up the MAC address set in dom0 will be lost. I’ve attached
> my dom0 rc.local where I set the MAC addresses and assign the devices to
> pciback.
>
>
>
> I can’t swear that this is the best way to do this, but this did work.
>
>
>
> John Byrne
>
>
>
>
>
> *From:* xen-devel-bounces@lists.xensource.com [mailto:
> xen-devel-bounces@lists.xensource.com] *On Behalf Of *kishore kumar
> *Sent:* Monday, July 19, 2010 10:16 AM
> *To:* Pasi Kärkkäinen; xen-devel@lists.xensource.com
> *Cc:* wangalalei; xen-users@lists.xensource.com
> *Subject:* [Xen-devel] Re: [Xen-users] wiki page on 10G SRIOV
>
>
>
>
> Hello Everyone,
>
>
>
> I am facing a strange issue with my 10G 82599 PCIe device.
>
>
>
> The Device Id has to be 0x10FB for SRIOV to function properly, But all of
> sudden the Device ID started showing me as 0x10D8, and because of this the
> SRIOV feature is not working and I cannot assign the Virtual Function to  my
> guest.
>
>
>
> When I did lspci command it is showing me as Intel Corporation Unknown
> device 10d8 (rev 01).  Previously it was showing me Intel Corporation
> 82599EB Gigabit Networking with 10FB, and I was able to assign the Virtual
> Function to my guest.
>
>
>
> I googled and found out that the device ID 0x10D8 is a 82599 Unprogrammed
> device.
>
>
>
> Can any one please let me know what is happening here and how to recover
> back from this?
>
> I am totally blocked for 2 days because of this issue.
>
>
>
> Thank You.
>
> On Fri, Jul 16, 2010 at 1:05 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
>
> On Fri, Jul 16, 2010 at 02:17:30AM +0000, wangalalei wrote:
> >    hi all:
> >
> >       i read intel 82576 card datasheet,and finding it saying that the
> 82576
> >    can set the vf's bandwidth,
> >    but when i test this card,i can not find any parameter to it
> >       any one else notice this problem? or anyone knows whether the 82576
> >    card can control its vf's bandwidth?
> >
>
> I think bandwidth management is a work-in-progress by Intel..
>
> -- Pasi
>
>
> >    alaleiwang
> >
> >
>  --------------------------------------------------------------------------
> >
> >    Date: Thu, 15 Jul 2010 18:35:57 -0700
> >    Subject: Re: [Xen-users] wiki page on 10G SRIOV
> >    From: bodkekumar@gmail.com
> >    To: pasik@iki.fi
> >    CC: xen-users@lists.xensource.com
> >
> >    Yeah, I configured 64 Virtual Functions in DOM-0  and passthru 1 VF to
> >    assign it to the guest.  It worked for me.
> >
> >    I assign static IP to my guest and tried to ping the remote machine.
> It
> >    worked for me.
> >
> >    But I am wondering that ping to DOM-0 from guest makes any sense here?
> >    Can you let me know on this?  ping worked though from guest to DOM-0
> as
> >    well.
> >
>
> >    On Thu, Jul 15, 2010 at 1:11 PM, Pasi KÀrkkÀinen <[1]pasik@iki.fi>
>
> >    wrote:
> >
> >      On Thu, Jun 24, 2010 at 10:56:28AM -0700, kishore kumar wrote:
> >      >    Hi All,
> >      >
> >      >    Can any one please let me know if there is a WIKI page which
> talks
> >      about
> >      >    how to configure SRIOV in Xen and use Physical Functions  and
> >      Virtual
> >      >    Functions ?
> >      >
> >      >    I searched xen wiki, but could not find.
> >      >
> >
> >      Each virtual function should be a separate PCI device, so you can
> >      passthru
> >      them like any other normal PCI device.
> >
> >      Did you get it to work?
> >      -- Pasi
> >
>
> >
>  --------------------------------------------------------------------------
> >
> >    è**倩+æ**玢+é*®ç®± æ*³èŠ*蜻æ*Ÿå*ºæžž,æ**æ*ºMSNåž®äœ æ**å®*!
> >    [2]ç«*å*»äž*蜜ïŒ*
>
> >
> > References
> >
> >    Visible links
> >    1. mailto:pasik@iki.fi
>
> >    2. http://3g.msn.cn/
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 13309 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [Xen-users] wiki page on 10G SRIOV

[Pasi Kärkkäinen]

On Mon, Jul 19, 2010 at 02:59:52PM -0700, kishore kumar wrote:
>    I am using Ethernet Server Adapater X520 - DA2 (E10G42BTDA) Card.
>    For the Device Id 0x10FB, I was able to see all the 64 Virtual Functions,
>    when I run lspci command.
>    But when the Device ID started showing 0x10D8, I cannot see any Virtual
>    Functions, when I run lspci command.
>

So the question is what happened? How/when did it change device ID? 
Did you flash/upgrade the firmware?

-- Pasi

>    On Mon, Jul 19, 2010 at 2:35 PM, Rose, Gregory V
>    <[1]gregory.v.rose@intel.com> wrote:
> 
>      If you have an 82599EB that is using 10D8h for the Virtual Function
>      Device ID then the adapter is a very old, preproduction version or else
>      the correct EEPROM image has been over-written with an invalid image.Â
>      The correct Device ID value for an 82599EB virtual function is 10EDh.
> 
>      Â
> 
>      - Greg
> 
>      Â
> 
>      From: [2]xen-devel-bounces@lists.xensource.com
>      [mailto:[3]xen-devel-bounces@lists.xensource.com] On Behalf Of Byrne,
>      John (HP Labs)
>      Sent: Monday, July 19, 2010 2:22 PM
>      To: kishore kumar
>      Cc: [4]xen-devel@lists.xensource.com; wangalalei;
>      [5]xen-users@lists.xensource.com
>      Subject: RE: [Xen-devel] Re: [Xen-users] wiki page on 10G SRIOV
> 
>      Â
> 
>      Iâ**m a bit confused with your device IDs. The 10fb device is the PCI id
>      for the physical NIC; the 10d8 device is the SRIOV Virtual Function
>      device created by the driver. Â With the latest xen-unstable, I have
>      successfully used this NIC to create and assign the VF devices to HVM
>      guests. The ixgbevf driver, which binds to the VF devices, expects
>      device id 0x10ed. I simply assumed that the driver was not quite up to
>      date and tweaked things to work. Â So, if you are just a bit confused,
>      the following might help.
> 
>      Â
> 
>      I added the following in my modprobe.d directory to create four VF
>      devices (ixgbe.conf, attached):
> 
>      Â
> 
>      options ixgbe max_vfs=4
> 
>      Â
> 
>      So, on my machine, I can see the physical NIC and the 4 VFs created.
> 
>      Â
> 
>      lspci â**n | grep 0a:
> 
>      0a:00.0 0200: 8086:10fb (rev 01)
> 
>      0a:10.0 0200: 8086:10d8 (rev 01)
> 
>      0a:10.2 0200: 8086:10d8 (rev 01)
> 
>      0a:10.4 0200: 8086:10d8 (rev 01)
> 
>      0a:10.6 0200: 8086:10d8 (rev 01)
> 
>      Â
> 
>      I added the ixgbevf driver to both my dom0 and domUs. While it is a
>      simple tweak to add the device id to the driver, I decided to play with
>      modprobe instead and added the following to my modprobe.d directory in
>      both dom0 and domU (ixgbvevf.conf, attached):
> 
>      Â
> 
>      install ixgbevf /sbin/modprobe --ignore-install ixgbevf; echo "0x8086
>      0x10d8" >/
> 
>      sys/bus/pci/drivers/ixgbevf/new_id
> 
>      Â
> 
>      If you are working with HVM guests, there has to be a patch to  qemu as
>      well. (Attached) Â You also have to add â**(pci-passthrough-strict-check
>      no)â** to your xend-config.sxp. (I donâ**t think you need the
>      xend-config change for PV guests.)
> 
>      Â
> 
>      Then it is simply a matter of adding the VF PCI devices to your domU
>      configurations and configuring the ixgbevf devices in the guests.
> 
>      Â
> 
>      Both the ixgbe and ixgbevf drivers will generate random MAC addresses
>      for the devices. What  I wanted to do was to set the MAC addresses for
>      the VF devices in dom0 and then my guestsâ** networking configuration
>      was much easier. The physical NIC must be up. It doesnâ**t have to have
>      an IP address, but if it isnâ**t up the MAC address set in dom0 will be
>      lost. Iâ**ve attached my dom0 rc.local where I set the MAC addresses and
>      assign the devices to pciback.
> 
>      Â
> 
>      I canâ**t swear that this is the best way to do this, but this did work.
> 
>      Â
> 
>      John Byrne
> 
>      Â
> 
>      Â
> 
>      From: [6]xen-devel-bounces@lists.xensource.com
>      [mailto:[7]xen-devel-bounces@lists.xensource.com] On Behalf Of kishore
>      kumar
>      Sent: Monday, July 19, 2010 10:16 AM
>      To: Pasi KÀrkkÀinen; [8]xen-devel@lists.xensource.com
>      Cc: wangalalei; [9]xen-users@lists.xensource.com
>      Subject: [Xen-devel] Re: [Xen-users] wiki page on 10G SRIOV
> 
>      Â
> 
>      Hello Everyone,
> 
>      Â
> 
>      I am facing a strange issue with my 10G 82599 PCIe device.
> 
>      Â
> 
>      The Device Id has to be 0x10FB for SRIOV to function properly, But all
>      of sudden the Device IDÂ started showing me as 0x10D8, and because of
>      this the SRIOV feature is not working and I cannot assign the Virtual
>      Function to  my guest.
> 
>      Â
> 
>      When I did lspci command it is showing me as Intel Corporation Unknown
>      device 10d8 (rev 01).  Previously it was showing me Intel Corporation
>      82599EB Gigabit Networking with 10FB, and I was able to assign the
>      Virtual Function to my guest.
> 
>      Â
> 
>      I googled and found out that the device ID 0x10D8 is a 82599
>      Unprogrammed device.
> 
>      Â
> 
>      Can any one please let me know what is happening here and how to recover
>      back from this?
> 
>      I am totally blocked for 2 days because of this issue.
> 
>      Â
> 
>      Thank You.
> 
>      On Fri, Jul 16, 2010 at 1:05 AM, Pasi KÀrkkÀinen <[10]pasik@iki.fi>
>      wrote:
> 
>      On Fri, Jul 16, 2010 at 02:17:30AM +0000, wangalalei wrote:
>      > Â  Â hi all:
>      >
>      > Â  Â  Â  i read intel 82576 card datasheet,and finding it saying that
>      the 82576
>      > Â  Â can set the vf's bandwidth,
>      > Â  Â but when i test this card,i can not find any parameter to it
>      > Â  Â  Â  any one else notice this problem? or anyone knows whether the
>      82576
>      > Â  Â card can control its vf's bandwidth?
>      >
> 
>      I think bandwidth management is a work-in-progress by Intel..
> 
>      -- Pasi
> 
>      > Â  Â alaleiwang
>      >
>      > Â
>      Â --------------------------------------------------------------------------
>      >
>      > Â  Â Date: Thu, 15 Jul 2010 18:35:57 -0700
>      > Â  Â Subject: Re: [Xen-users] wiki page on 10G SRIOV
>      > Â  Â From: [11]bodkekumar@gmail.com
>      > Â  Â To: [12]pasik@iki.fi
>      > Â  Â CC: [13]xen-users@lists.xensource.com
>      >
>      > Â  Â Yeah, I configured 64 Virtual Functions in DOM-0 Â and passthru 1
>      VF to
>      > Â  Â assign it to the guest. Â It worked for me.
>      >
>      > Â  Â I assign static IP to my guest and tried to ping the remote
>      machine. It
>      > Â  Â worked for me.
>      >
>      > Â  Â But I am wondering that ping to DOM-0 from guest makes any sense
>      here?
>      > Â  Â Can you let me know on this? Â ping worked though from guest to
>      DOM-0 as
>      > Â  Â well.
>      >
> 
>      > Â  Â On Thu, Jul 15, 2010 at 1:11 PM, Pasi KÃ*â*¬rkkÃ*â*¬inen
>      <[1][14]pasik@iki.fi>
> 
>      > Â  Â wrote:
>      >
>      > Â  Â  Â On Thu, Jun 24, 2010 at 10:56:28AM -0700, kishore kumar wrote:
>      > Â  Â  Â > Â  Â Hi All,
>      > Â  Â  Â >
>      > Â  Â  Â > Â  Â Can any one please let me know if there is a WIKI page
>      which talks
>      > Â  Â  Â about
>      > Â  Â  Â > Â  Â how to configure SRIOV in Xen and use Physical
>      Functions  and
>      > Â  Â  Â Virtual
>      > Â  Â  Â > Â  Â Functions ?
>      > Â  Â  Â >
>      > Â  Â  Â > Â  Â I searched xen wiki, but could not find.
>      > Â  Â  Â >
>      >
>      > Â  Â  Â Each virtual function should be a separate PCI device, so you
>      can
>      > Â  Â  Â passthru
>      > Â  Â  Â them like any other normal PCI device.
>      >
>      > Â  Â  Â Did you get it to work?
>      > Â  Â  Â -- Pasi
>      >
> 
>      > Â
>      Â --------------------------------------------------------------------------
>      >
>      >    Ú**Ã¥â*¬Â©+Ê**çŜ¢+é*®ç®±
>      Ê*³ÚŠ*ÚÅ*»Ê*Şå*ºÊşş,Ê**Ê*ºMSNåş®ÀÅ* Ê**å®*!
>      >    [2]ç«*Ã¥*»Àş*ÚÅ*Å*ïÅ**
> 
>      >
>      > References
>      >
>      > Â  Â Visible links
>      > Â  Â 1. mailto:[15]pasik@iki.fi
> 
>      > Â  Â 2. [16]http://3g.msn.cn/
> 
>      Â
> 
> References
> 
>    Visible links
>    1. mailto:gregory.v.rose@intel.com
>    2. mailto:xen-devel-bounces@lists.xensource.com
>    3. mailto:xen-devel-bounces@lists.xensource.com
>    4. mailto:xen-devel@lists.xensource.com
>    5. mailto:xen-users@lists.xensource.com
>    6. mailto:xen-devel-bounces@lists.xensource.com
>    7. mailto:xen-devel-bounces@lists.xensource.com
>    8. mailto:xen-devel@lists.xensource.com
>    9. mailto:xen-users@lists.xensource.com
>   10. mailto:pasik@iki.fi
>   11. mailto:bodkekumar@gmail.com
>   12. mailto:pasik@iki.fi
>   13. mailto:xen-users@lists.xensource.com
>   14. mailto:pasik@iki.fi
>   15. mailto:pasik@iki.fi
>   16. http://3g.msn.cn/

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Re: [Xen-users] wiki page on 10G SRIOV

[kishore kumar]

[-- Attachment #1.1: Type: text/plain, Size: 10485 bytes --]

The EEPROM was corrupted .  I Programmed the EEPROM.

Now it is working fine.

Thank You.

On Tue, Jul 20, 2010 at 2:00 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:

> On Mon, Jul 19, 2010 at 02:59:52PM -0700, kishore kumar wrote:
> >    I am using Ethernet Server Adapater X520 - DA2 (E10G42BTDA) Card.
> >    For the Device Id 0x10FB, I was able to see all the 64 Virtual
> Functions,
> >    when I run lspci command.
> >    But when the Device ID started showing 0x10D8, I cannot see any
> Virtual
> >    Functions, when I run lspci command.
> >
>
> So the question is what happened? How/when did it change device ID?
> Did you flash/upgrade the firmware?
>
> -- Pasi
>
> >    On Mon, Jul 19, 2010 at 2:35 PM, Rose, Gregory V
> >    <[1]gregory.v.rose@intel.com> wrote:
> >
> >      If you have an 82599EB that is using 10D8h for the Virtual Function
> >      Device ID then the adapter is a very old, preproduction version or
> else
> >      the correct EEPROM image has been over-written with an invalid
> image.Â
> >      The correct Device ID value for an 82599EB virtual function is
> 10EDh.
> >
> >      Â
> >
> >      - Greg
> >
> >      Â
> >
> >      From: [2]xen-devel-bounces@lists.xensource.com
> >      [mailto:[3]xen-devel-bounces@lists.xensource.com] On Behalf Of
> Byrne,
> >      John (HP Labs)
> >      Sent: Monday, July 19, 2010 2:22 PM
> >      To: kishore kumar
> >      Cc: [4]xen-devel@lists.xensource.com; wangalalei;
> >      [5]xen-users@lists.xensource.com
> >      Subject: RE: [Xen-devel] Re: [Xen-users] wiki page on 10G SRIOV
> >
> >      Â
> >
> >      Iâ**m a bit confused with your device IDs. The 10fb device is the
> PCI id
> >      for the physical NIC; the 10d8 device is the SRIOV Virtual Function
> >      device created by the driver. Â With the latest xen-unstable, I have
> >      successfully used this NIC to create and assign the VF devices to
> HVM
> >      guests. The ixgbevf driver, which binds to the VF devices, expects
> >      device id 0x10ed. I simply assumed that the driver was not quite up
> to
> >      date and tweaked things to work. Â So, if you are just a bit
> confused,
> >      the following might help.
> >
> >      Â
> >
> >      I added the following in my modprobe.d directory to create four VF
> >      devices (ixgbe.conf, attached):
> >
> >      Â
> >
> >      options ixgbe max_vfs=4
> >
> >      Â
> >
> >      So, on my machine, I can see the physical NIC and the 4 VFs created.
> >
> >      Â
> >
> >      lspci â**n | grep 0a:
> >
> >      0a:00.0 0200: 8086:10fb (rev 01)
> >
> >      0a:10.0 0200: 8086:10d8 (rev 01)
> >
> >      0a:10.2 0200: 8086:10d8 (rev 01)
> >
> >      0a:10.4 0200: 8086:10d8 (rev 01)
> >
> >      0a:10.6 0200: 8086:10d8 (rev 01)
> >
> >      Â
> >
> >      I added the ixgbevf driver to both my dom0 and domUs. While it is a
> >      simple tweak to add the device id to the driver, I decided to play
> with
> >      modprobe instead and added the following to my modprobe.d directory
> in
> >      both dom0 and domU (ixgbvevf.conf, attached):
> >
> >      Â
> >
> >      install ixgbevf /sbin/modprobe --ignore-install ixgbevf; echo
> "0x8086
> >      0x10d8" >/
> >
> >      sys/bus/pci/drivers/ixgbevf/new_id
> >
> >      Â
> >
> >      If you are working with HVM guests, there has to be a patch to Â
> qemu as
> >      well. (Attached) Â You also have to add
> â**(pci-passthrough-strict-check
> >      no)â** to your xend-config.sxp. (I donâ**t think you need the
> >      xend-config change for PV guests.)
> >
> >      Â
> >
> >      Then it is simply a matter of adding the VF PCI devices to your domU
> >      configurations and configuring the ixgbevf devices in the guests.
> >
> >      Â
> >
> >      Both the ixgbe and ixgbevf drivers will generate random MAC
> addresses
> >      for the devices. What  I wanted to do was to set the MAC addresses
> for
> >      the VF devices in dom0 and then my guestsâ** networking
> configuration
> >      was much easier. The physical NIC must be up. It doesnâ**t have to
> have
> >      an IP address, but if it isnâ**t up the MAC address set in dom0 will
> be
> >      lost. Iâ**ve attached my dom0 rc.local where I set the MAC addresses
> and
> >      assign the devices to pciback.
> >
> >      Â
> >
> >      I canâ**t swear that this is the best way to do this, but this did
> work.
> >
> >      Â
> >
> >      John Byrne
> >
> >      Â
> >
> >      Â
> >
> >      From: [6]xen-devel-bounces@lists.xensource.com
> >      [mailto:[7]xen-devel-bounces@lists.xensource.com] On Behalf Of
> kishore
> >      kumar
> >      Sent: Monday, July 19, 2010 10:16 AM
> >      To: Pasi KÀrkkÀinen; [8]xen-devel@lists.xensource.com
> >      Cc: wangalalei; [9]xen-users@lists.xensource.com
> >      Subject: [Xen-devel] Re: [Xen-users] wiki page on 10G SRIOV
> >
> >      Â
> >
> >      Hello Everyone,
> >
> >      Â
> >
> >      I am facing a strange issue with my 10G 82599 PCIe device.
> >
> >      Â
> >
> >      The Device Id has to be 0x10FB for SRIOV to function properly, But
> all
> >      of sudden the Device IDÂ started showing me as 0x10D8, and because
> of
> >      this the SRIOV feature is not working and I cannot assign the
> Virtual
> >      Function to  my guest.
> >
> >      Â
> >
> >      When I did lspci command it is showing me as Intel Corporation
> Unknown
> >      device 10d8 (rev 01).  Previously it was showing me Intel
> Corporation
> >      82599EB Gigabit Networking with 10FB, and I was able to assign the
> >      Virtual Function to my guest.
> >
> >      Â
> >
> >      I googled and found out that the device ID 0x10D8 is a 82599
> >      Unprogrammed device.
> >
> >      Â
> >
> >      Can any one please let me know what is happening here and how to
> recover
> >      back from this?
> >
> >      I am totally blocked for 2 days because of this issue.
> >
> >      Â
> >
> >      Thank You.
> >
> >      On Fri, Jul 16, 2010 at 1:05 AM, Pasi KÀrkkÀinen <[10]
> pasik@iki.fi>
> >      wrote:
> >
> >      On Fri, Jul 16, 2010 at 02:17:30AM +0000, wangalalei wrote:
> >      > Â  Â hi all:
> >      >
> >      > Â  Â  Â  i read intel 82576 card datasheet,and finding it saying
> that
> >      the 82576
> >      > Â  Â can set the vf's bandwidth,
> >      > Â  Â but when i test this card,i can not find any parameter to it
> >      > Â  Â  Â  any one else notice this problem? or anyone knows whether
> the
> >      82576
> >      > Â  Â card can control its vf's bandwidth?
> >      >
> >
> >      I think bandwidth management is a work-in-progress by Intel..
> >
> >      -- Pasi
> >
> >      > Â  Â alaleiwang
> >      >
> >      > Â
> >      Â
> --------------------------------------------------------------------------
> >      >
> >      > Â  Â Date: Thu, 15 Jul 2010 18:35:57 -0700
> >      > Â  Â Subject: Re: [Xen-users] wiki page on 10G SRIOV
> >      > Â  Â From: [11]bodkekumar@gmail.com
> >      > Â  Â To: [12]pasik@iki.fi
> >      > Â  Â CC: [13]xen-users@lists.xensource.com
> >      >
> >      > Â  Â Yeah, I configured 64 Virtual Functions in DOM-0 Â and
> passthru 1
> >      VF to
> >      > Â  Â assign it to the guest. Â It worked for me.
> >      >
> >      > Â  Â I assign static IP to my guest and tried to ping the remote
> >      machine. It
> >      > Â  Â worked for me.
> >      >
> >      > Â  Â But I am wondering that ping to DOM-0 from guest makes any
> sense
> >      here?
> >      > Â  Â Can you let me know on this? Â ping worked though from guest
> to
> >      DOM-0 as
> >      > Â  Â well.
> >      >
> >
> >      > Â  Â On Thu, Jul 15, 2010 at 1:11 PM, Pasi KÃ*â*¬rkkÃ*â*¬inen
> >      <[1][14]pasik@iki.fi>
> >
> >      > Â  Â wrote:
> >      >
> >      > Â  Â  Â On Thu, Jun 24, 2010 at 10:56:28AM -0700, kishore kumar
> wrote:
> >      > Â  Â  Â > Â  Â Hi All,
> >      > Â  Â  Â >
> >      > Â  Â  Â > Â  Â Can any one please let me know if there is a WIKI
> page
> >      which talks
> >      > Â  Â  Â about
> >      > Â  Â  Â > Â  Â how to configure SRIOV in Xen and use Physical
> >      Functions  and
> >      > Â  Â  Â Virtual
> >      > Â  Â  Â > Â  Â Functions ?
> >      > Â  Â  Â >
> >      > Â  Â  Â > Â  Â I searched xen wiki, but could not find.
> >      > Â  Â  Â >
> >      >
> >      > Â  Â  Â Each virtual function should be a separate PCI device, so
> you
> >      can
> >      > Â  Â  Â passthru
> >      > Â  Â  Â them like any other normal PCI device.
> >      >
> >      > Â  Â  Â Did you get it to work?
> >      > Â  Â  Â -- Pasi
> >      >
> >
> >      > Â
> >      Â
> --------------------------------------------------------------------------
> >      >
> >      >    Ú**Ã¥â*¬Â©+Ê**çŜ¢+é*®ç®±
> >      Ê*³ÚŠ*ÚÅ*»Ê*Şå*ºÊşş,Ê**Ê*ºMSNåş®ÀÅ* Ê**å®*!
> >      >    [2]ç«*Ã¥*»Àş*ÚÅ*Å*ïÅ**
> >
> >      >
> >      > References
> >      >
> >      > Â  Â Visible links
> >      > Â  Â 1. mailto:[15]pasik@iki.fi
> >
> >      > Â  Â 2. [16]http://3g.msn.cn/
> >
> >      Â
> >
> > References
> >
> >    Visible links
> >    1. mailto:gregory.v.rose@intel.com
> >    2. mailto:xen-devel-bounces@lists.xensource.com
> >    3. mailto:xen-devel-bounces@lists.xensource.com
> >    4. mailto:xen-devel@lists.xensource.com
> >    5. mailto:xen-users@lists.xensource.com
> >    6. mailto:xen-devel-bounces@lists.xensource.com
> >    7. mailto:xen-devel-bounces@lists.xensource.com
> >    8. mailto:xen-devel@lists.xensource.com
> >    9. mailto:xen-users@lists.xensource.com
> >   10. mailto:pasik@iki.fi
> >   11. mailto:bodkekumar@gmail.com
> >   12. mailto:pasik@iki.fi
> >   13. mailto:xen-users@lists.xensource.com
> >   14. mailto:pasik@iki.fi
> >   15. mailto:pasik@iki.fi
> >   16. http://3g.msn.cn/
>
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>
>

[-- Attachment #1.2: Type: text/html, Size: 14581 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel