From mboxrd@z Thu Jan 1 00:00:00 1970 From: kishore kumar Subject: Re: Re: [Xen-users] wiki page on 10G SRIOV Date: Mon, 19 Jul 2010 14:59:52 -0700 Message-ID: References: <20100716080509.GA17817@reaktio.net> <469F2699A483D44BA6D2B311B1089D3A7F89281271@GVW1096EXB.americas.hpqcorp.net> <43F901BD926A4E43B106BF17856F0755F1510021@orsmsx508.amr.corp.intel.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1918192192==" Return-path: In-Reply-To: <43F901BD926A4E43B106BF17856F0755F1510021@orsmsx508.amr.corp.intel.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Rose, Gregory V" Cc: "xen-users@lists.xensource.com" , "xen-devel@lists.xensource.com" , wangalalei , "Byrne, John (HP Labs)" List-Id: xen-devel@lists.xenproject.org --===============1918192192== Content-Type: multipart/alternative; boundary=000e0cd3281633f5a2048bc4b0aa --000e0cd3281633f5a2048bc4b0aa Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I am using Ethernet Server Adapater X520 - DA2 (E10G42BTDA) Card. For the Device Id 0x10FB, I was able to see all the 64 Virtual Functions, when I run lspci command. But when the Device ID started showing 0x10D8, I cannot see any Virtual Functions, when I run lspci command. On Mon, Jul 19, 2010 at 2:35 PM, Rose, Gregory V wrote: > If you have an 82599EB that is using 10D8h for the Virtual Function > Device ID then the adapter is a very old, preproduction version or else t= he > correct EEPROM image has been over-written with an invalid image. The > correct Device ID value for an 82599EB virtual function is 10EDh. > > > > - Greg > > > > *From:* xen-devel-bounces@lists.xensource.com [mailto: > xen-devel-bounces@lists.xensource.com] *On Behalf Of *Byrne, John (HP > Labs) > *Sent:* Monday, July 19, 2010 2:22 PM > *To:* kishore kumar > *Cc:* xen-devel@lists.xensource.com; wangalalei; > xen-users@lists.xensource.com > *Subject:* RE: [Xen-devel] Re: [Xen-users] wiki page on 10G SRIOV > > > > I=E2=80=99m a bit confused with your device IDs. The 10fb device is the P= CI id for > the physical NIC; the 10d8 device is the SRIOV Virtual Function device > created by the driver. With the latest xen-unstable, I have successfully > used this NIC to create and assign the VF devices to HVM guests. The ixgb= evf > driver, which binds to the VF devices, expects device id 0x10ed. I simply > assumed that the driver was not quite up to date and tweaked things to wo= rk. > So, if you are just a bit confused, the following might help. > > > > I added the following in my modprobe.d directory to create four VF device= s > (ixgbe.conf, attached): > > > > options ixgbe max_vfs=3D4 > > > > So, on my machine, I can see the physical NIC and the 4 VFs created. > > > > lspci =E2=80=93n | grep 0a: > > 0a:00.0 0200: 8086:10fb (rev 01) > > 0a:10.0 0200: 8086:10d8 (rev 01) > > 0a:10.2 0200: 8086:10d8 (rev 01) > > 0a:10.4 0200: 8086:10d8 (rev 01) > > 0a:10.6 0200: 8086:10d8 (rev 01) > > > > I added the ixgbevf driver to both my dom0 and domUs. While it is a simpl= e > tweak to add the device id to the driver, I decided to play with modprobe > instead and added the following to my modprobe.d directory in both dom0 a= nd > domU (ixgbvevf.conf, attached): > > > > install ixgbevf /sbin/modprobe --ignore-install ixgbevf; echo "0x8086 > 0x10d8" >/ > > sys/bus/pci/drivers/ixgbevf/new_id > > > > If you are working with HVM guests, there has to be a patch to qemu as > well. (Attached) You also have to add =E2=80=9C(pci-passthrough-strict-c= heck no)=E2=80=9D > to your xend-config.sxp. (I don=E2=80=99t think you need the xend-config = change for > PV guests.) > > > > Then it is simply a matter of adding the VF PCI devices to your domU > configurations and configuring the ixgbevf devices in the guests. > > > > Both the ixgbe and ixgbevf drivers will generate random MAC addresses for > the devices. What I wanted to do was to set the MAC addresses for the VF > devices in dom0 and then my guests=E2=80=99 networking configuration was = much > easier. The physical NIC must be up. It doesn=E2=80=99t have to have an I= P address, > but if it isn=E2=80=99t up the MAC address set in dom0 will be lost. I=E2= =80=99ve attached > my dom0 rc.local where I set the MAC addresses and assign the devices to > pciback. > > > > I can=E2=80=99t swear that this is the best way to do this, but this did = work. > > > > John Byrne > > > > > > *From:* xen-devel-bounces@lists.xensource.com [mailto: > xen-devel-bounces@lists.xensource.com] *On Behalf Of *kishore kumar > *Sent:* Monday, July 19, 2010 10:16 AM > *To:* Pasi K=C3=A4rkk=C3=A4inen; xen-devel@lists.xensource.com > *Cc:* wangalalei; xen-users@lists.xensource.com > *Subject:* [Xen-devel] Re: [Xen-users] wiki page on 10G SRIOV > > > > > Hello Everyone, > > > > I am facing a strange issue with my 10G 82599 PCIe device. > > > > The Device Id has to be 0x10FB for SRIOV to function properly, But all of > sudden the Device ID started showing me as 0x10D8, and because of this th= e > SRIOV feature is not working and I cannot assign the Virtual Function to = my > guest. > > > > When I did lspci command it is showing me as Intel Corporation Unknown > device 10d8 (rev 01). Previously it was showing me Intel Corporation > 82599EB Gigabit Networking with 10FB, and I was able to assign the Virtua= l > Function to my guest. > > > > I googled and found out that the device ID 0x10D8 is a 82599 Unprogrammed > device. > > > > Can any one please let me know what is happening here and how to recover > back from this? > > I am totally blocked for 2 days because of this issue. > > > > Thank You. > > On Fri, Jul 16, 2010 at 1:05 AM, Pasi K=C3=A4rkk=C3=A4inen = wrote: > > On Fri, Jul 16, 2010 at 02:17:30AM +0000, wangalalei wrote: > > hi all: > > > > i read intel 82576 card datasheet,and finding it saying that the > 82576 > > can set the vf's bandwidth, > > but when i test this card,i can not find any parameter to it > > any one else notice this problem? or anyone knows whether the 825= 76 > > card can control its vf's bandwidth? > > > > I think bandwidth management is a work-in-progress by Intel.. > > -- Pasi > > > > alaleiwang > > > > > ------------------------------------------------------------------------= -- > > > > Date: Thu, 15 Jul 2010 18:35:57 -0700 > > Subject: Re: [Xen-users] wiki page on 10G SRIOV > > From: bodkekumar@gmail.com > > To: pasik@iki.fi > > CC: xen-users@lists.xensource.com > > > > Yeah, I configured 64 Virtual Functions in DOM-0 and passthru 1 VF = to > > assign it to the guest. It worked for me. > > > > I assign static IP to my guest and tried to ping the remote machine. > It > > worked for me. > > > > But I am wondering that ping to DOM-0 from guest makes any sense her= e? > > Can you let me know on this? ping worked though from guest to DOM-0 > as > > well. > > > > > On Thu, Jul 15, 2010 at 1:11 PM, Pasi K=C3=83=E2=82=ACrkk=C3=83=E2= =82=ACinen <[1]pasik@iki.fi> > > > wrote: > > > > On Thu, Jun 24, 2010 at 10:56:28AM -0700, kishore kumar wrote: > > > Hi All, > > > > > > Can any one please let me know if there is a WIKI page which > talks > > about > > > how to configure SRIOV in Xen and use Physical Functions and > > Virtual > > > Functions ? > > > > > > I searched xen wiki, but could not find. > > > > > > > Each virtual function should be a separate PCI device, so you can > > passthru > > them like any other normal PCI device. > > > > Did you get it to work? > > -- Pasi > > > > > > ------------------------------------------------------------------------= -- > > > > =C3=A8**=C3=A5=E2=82=AC=C2=A9+=C3=A6**=C3=A7=C5=BD=C2=A2+=C3=A9*=C2= =AE=C3=A7=C2=AE=C2=B1 =C3=A6*=C2=B3=C3=A8=C5=A0*=C3=A8=C5=93=C2=BB=C3=A6*= =C5=B8=C3=A5*=C2=BA=C3=A6=C5=BE=C5=BE,=C3=A6**=C3=A6*=C2=BAMSN=C3=A5=C5=BE= =C2=AE=C3=A4=C5=93 =C3=A6**=C3=A5=C2=AE*! > > [2]=C3=A7=C2=AB*=C3=A5*=C2=BB=C3=A4=C5=BE*=C3=A8=C5=93=C5=93=C3=AF= =C5=92* > > > > > References > > > > Visible links > > 1. mailto:pasik@iki.fi > > > 2. http://3g.msn.cn/ > > > --000e0cd3281633f5a2048bc4b0aa Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I am using Ethernet Server Adapater X520 - DA2 (E10G42BTDA) Card.
=C2=A0
For the Device Id 0x10FB, I was able to see all the 64 Virtual Functio= ns, when I run lspci command.
But when the Device ID started showing 0x1= 0D8, I cannot see any Virtual Functions, when I run lspci command.
On Mon, Jul 19, 2010 at 2:35 PM, Rose, Gregory V= <gregory.= v.rose@intel.com> wrote:

If y= ou have an 82599EB that is using 10D8h for the Virtual Function Device ID t= hen the adapter is a very old, preproduction version or else the correct EE= PROM image has been over-written with an invalid image.=C2=A0 The correct D= evice ID value for an 82599EB virtual function is 10EDh.

=C2= =A0

- Gr= eg

=C2= =A0

From:<= span style=3D"FONT-SIZE: 10pt"> xen-devel-bounces@lists.xensource.com [= mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Byrne= , John (HP Labs)
Sent: Monday, July 19, 2010 2:22 PM
To: kishore kumar
<= b>Cc: xen-devel@lists.xensource.com; wangalalei; xen-users@lists.xensource.com<= br> Subject: RE: [Xen-devel] Re: [Xen-users] wiki page on 10G SRIOV

=C2=A0

I=E2= =80=99m a bit confused with your device IDs. The 10fb device is the PCI id = for the physical NIC; the 10d8 device is the SRIOV Virtual Function device = created by the driver. =C2=A0With the latest xen-unstable, I have successfu= lly used this NIC to create and assign the VF devices to HVM guests. The ix= gbevf driver, which binds to the VF devices, expects device id 0x10ed. I si= mply assumed that the driver was not quite up to date and tweaked things to= work. =C2=A0So, if you are just a bit confused, the following might help.<= /span>

=C2= =A0

I ad= ded the following in my modprobe.d directory to create four VF devices (ixg= be.conf, attached):

=C2= =A0

opti= ons ixgbe max_vfs=3D4

=C2= =A0

So, = on my machine, I can see the physical NIC and the 4 VFs created.

=C2= =A0

lspc= i =E2=80=93n | grep 0a:

0a:0= 0.0 0200: 8086:10fb (rev 01)

0a:1= 0.0 0200: 8086:10d8 (rev 01)

0a:1= 0.2 0200: 8086:10d8 (rev 01)

0a:1= 0.4 0200: 8086:10d8 (rev 01)

0a:1= 0.6 0200: 8086:10d8 (rev 01)

=C2= =A0

I ad= ded the ixgbevf driver to both my dom0 and domUs. While it is a simple twea= k to add the device id to the driver, I decided to play with modprobe inste= ad and added the following to my modprobe.d directory in both dom0 and domU= (ixgbvevf.conf, attached):

=C2= =A0

inst= all ixgbevf /sbin/modprobe --ignore-install ixgbevf; echo "0x8086 0x10= d8" >/

sys/= bus/pci/drivers/ixgbevf/new_id

=C2= =A0

If y= ou are working with HVM guests, there has to be a patch to =C2=A0qemu as we= ll. (Attached) =C2=A0You also have to add =E2=80=9C(pci-passthrough-strict-= check no)=E2=80=9D to your xend-config.sxp. (I don=E2=80=99t think you need= the xend-config change for PV guests.)

=C2= =A0

Then= it is simply a matter of adding the VF PCI devices to your domU configurat= ions and configuring the ixgbevf devices in the guests.

=C2= =A0

Both= the ixgbe and ixgbevf drivers will generate random MAC addresses for the d= evices. What=C2=A0 I wanted to do was to set the MAC addresses for the VF d= evices in dom0 and then my guests=E2=80=99 networking configuration was muc= h easier. The physical NIC must be up. It doesn=E2=80=99t have to have an I= P address, but if it isn=E2=80=99t up the MAC address set in dom0 will be l= ost. I=E2=80=99ve attached my dom0 rc.local where I set the MAC addresses a= nd assign the devices to pciback.

=C2= =A0

I ca= n=E2=80=99t swear that this is the best way to do this, but this did work.<= /span>

=C2= =A0

John= Byrne

=C2= =A0

=C2= =A0

From:<= span style=3D"FONT-SIZE: 10pt"> xen-devel-bounces@lists.xensource.com [= mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of kisho= re kumar
Sent: Monday, July 19, 2010 10:16 AM
To: Pasi K=C3=A4rkk= =C3=A4inen; xen-devel@lists.xensource.com
Cc: wangalalei; xen-users@lists.xen= source.com
Subject: [Xen-devel] Re: [Xen-users] wiki page on 10G SRIOV

=C2=A0


Hello Everyone,

=C2=A0

I am facing a strange issue with my 10G 82599 PCIe d= evice.

=C2=A0

The Device Id has to be 0x10FB for SRIOV to function= properly, But all of sudden the Device ID=C2=A0started showing me as 0x10D= 8, and because of this the SRIOV feature is not working and I cannot assign= the Virtual Function to=C2=A0 my guest.

=C2=A0

When I did lspci command it is showing me as Intel C= orporation Unknown device 10d8 (rev 01).=C2=A0 Previously it was showing me= Intel Corporation 82599EB Gigabit Networking with 10FB, and I was able to = assign the Virtual Function to my guest.

=C2=A0

I googled and found out that the device ID 0x10D8 is= a 82599 Unprogrammed device.

=C2=A0

Can any one please let me know what is happening her= e and how to recover back from this?

I am totally blocked for 2 days because of this issu= e.

=C2=A0

Thank You.

On Fri, Jul 16, 2010 at 1:05 AM, Pasi K=C3=A4rkk=C3= =A4inen <pasik@iki.fi<= /a>> wrote:

On Fri, Jul 16, 2010 a= t 02:17:30AM +0000, wangalalei wrote:
> =C2=A0 =C2=A0hi all:
><= br>> =C2=A0 =C2=A0 =C2=A0 i read intel 82576 card datasheet,and finding = it saying that the 82576
> =C2=A0 =C2=A0can set the vf's bandwidth,
> =C2=A0 =C2=A0but = when i test this card,i can not find any parameter to it
> =C2=A0 =C2= =A0 =C2=A0 any one else notice this problem? or anyone knows whether the 82= 576
> =C2=A0 =C2=A0card can control its vf's bandwidth?
>

I think bandwidth management is a work-in-progress b= y Intel..

-- Pasi


> =C2=A0 =C2=A0alaleiwang
>
> =C2= =A0 =C2=A0-----------------------------------------------------------------= ---------
>
> =C2=A0 =C2=A0Date: Thu, 15 Jul 2010 18:35:57 -070= 0
> =C2=A0 =C2=A0Subject: Re: [Xen-users] wiki page on 10G SRIOV
> =C2=A0 =C2=A0From: bodkekumar@gmail.com
> =C2=A0 =C2=A0To: pasik@iki.fi
> =C2=A0 =C2=A0CC: <= a href=3D"mailto:xen-users@lists.xensource.com" target=3D"_blank">xen-users= @lists.xensource.com
>
> =C2=A0 =C2=A0Yeah, I configured 64 Virtual Functions in DOM-0 = =C2=A0and passthru 1 VF to
> =C2=A0 =C2=A0assign it to the guest. =C2= =A0It worked for me.
>
> =C2=A0 =C2=A0I assign static IP to my = guest and tried to ping the remote machine. It
> =C2=A0 =C2=A0worked for me.
>
> =C2=A0 =C2=A0But I am wond= ering that ping to DOM-0 from guest makes any sense here?
> =C2=A0 = =C2=A0Can you let me know on this? =C2=A0ping worked though from guest to D= OM-0 as
> =C2=A0 =C2=A0well.
>

> =C2=A0 =C2=A0On Thu, Jul 15, 2010 at 1:11 PM, P= asi K=C3=83=E2=82=ACrkk=C3=83=E2=82=ACinen <[1]pasik@iki.fi>

> =C2=A0 =C2=A0wrote:
>
> =C2=A0 =C2= =A0 =C2=A0On Thu, Jun 24, 2010 at 10:56:28AM -0700, kishore kumar wrote:> =C2=A0 =C2=A0 =C2=A0> =C2=A0 =C2=A0Hi All,
> =C2=A0 =C2=A0 = =C2=A0>
> =C2=A0 =C2=A0 =C2=A0> =C2=A0 =C2=A0Can any one please= let me know if there is a WIKI page which talks
> =C2=A0 =C2=A0 =C2=A0about
> =C2=A0 =C2=A0 =C2=A0> =C2=A0 =C2= =A0how to configure SRIOV in Xen and use Physical Functions =C2=A0and
&g= t; =C2=A0 =C2=A0 =C2=A0Virtual
> =C2=A0 =C2=A0 =C2=A0> =C2=A0 =C2= =A0Functions ?
> =C2=A0 =C2=A0 =C2=A0>
> =C2=A0 =C2=A0 =C2= =A0> =C2=A0 =C2=A0I searched xen wiki, but could not find.
> =C2=A0 =C2=A0 =C2=A0>
>
> =C2=A0 =C2=A0 =C2=A0Each virt= ual function should be a separate PCI device, so you can
> =C2=A0 =C2= =A0 =C2=A0passthru
> =C2=A0 =C2=A0 =C2=A0them like any other normal P= CI device.
>
> =C2=A0 =C2=A0 =C2=A0Did you get it to work?
> =C2=A0 =C2=A0 =C2=A0-- Pasi
>

> =C2=A0 =C2=A0----------------------------------= ----------------------------------------
>
> =C2=A0 =C2=A0=C3= =A8**=C3=A5=E2=82=AC=C2=A9+=C3=A6**=C3=A7=C5=BD=C2=A2+=C3=A9*=C2=AE=C3=A7= =C2=AE=C2=B1 =C3=A6*=C2=B3=C3=A8=C5=A0*=C3=A8=C5=93=C2=BB=C3=A6*=C5=B8=C3= =A5*=C2=BA=C3=A6=C5=BE=C5=BE,=C3=A6**=C3=A6*=C2=BAMSN=C3=A5=C5=BE=C2=AE=C3= =A4=C5=93 =C3=A6**=C3=A5=C2=AE*!
> =C2=A0 =C2=A0[2]=C3=A7=C2=AB*=C3= =A5*=C2=BB=C3=A4=C5=BE*=C3=A8=C5=93=C5=93=C3=AF=C5=92*

>
> References
>
> =C2=A0 =C2= =A0Visible links
> =C2=A0 =C2=A01. mailto:pasik@iki.fi

> =C2=A0 =C2=A02. http://3g.msn.cn/

=C2=A0


--000e0cd3281633f5a2048bc4b0aa-- --===============1918192192== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============1918192192==--