igb bandwidth allocation configuration

igb bandwidth allocation configuration

[Simon Horman]

Hi,

I have been looking into adding support the 82586's per-PF/VF
bandwidth allocation to the igb driver. It seems that the trickiest
part is working out how to expose things to user-space.

I was thinking along the lines of an ethtool option as follows:

	ethtool --bandwidth ethN LIMIT...

	where:
		* There is one LIMIT per PF/VF.
		  The 82576 can have up to 7 VFs per PF,
		  so there would be up to 8 LIMITS
		* A keyword (none?) can be used to denote that
		  bandwidth allocation should be disabled for the
		  corresponding VM
		* Otherwise LIMITS are in Megabits/s

This may get a bit combersome if there are a lot of VFs per PF,
perhaps a better syntax would be:

	ethtool --bandwidth ethN M=LIMIT...

	where:
		* LIMIT is as above
		* M is some key to denote which VF/PF is
		  having its limit set.

Internally it seems that actually the limits are applied to HW Tx queues
rather than directly VMs. There are 16 such queues. Accordingly it might
be useful to design an interface to set limits per-queue using ethtool.
But this would seem to also require exposing which queues are associated
with which PF/VF.


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july

Re: igb bandwidth allocation configuration

[Patrick McHardy]

Simon Horman wrote:
> Hi,
> 
> I have been looking into adding support the 82586's per-PF/VF
> bandwidth allocation to the igb driver. It seems that the trickiest
> part is working out how to expose things to user-space.
> 
> I was thinking along the lines of an ethtool option as follows:
> 
> 	ethtool --bandwidth ethN LIMIT...
> 
> 	where:
> 		* There is one LIMIT per PF/VF.
> 		  The 82576 can have up to 7 VFs per PF,
> 		  so there would be up to 8 LIMITS
> 		* A keyword (none?) can be used to denote that
> 		  bandwidth allocation should be disabled for the
> 		  corresponding VM
> 		* Otherwise LIMITS are in Megabits/s
> 
> This may get a bit combersome if there are a lot of VFs per PF,
> perhaps a better syntax would be:
> 
> 	ethtool --bandwidth ethN M=LIMIT...
> 
> 	where:
> 		* LIMIT is as above
> 		* M is some key to denote which VF/PF is
> 		  having its limit set.
> 
> Internally it seems that actually the limits are applied to HW Tx queues
> rather than directly VMs. There are 16 such queues. Accordingly it might
> be useful to design an interface to set limits per-queue using ethtool.
> But this would seem to also require exposing which queues are associated
> with which PF/VF.

Just an idea since I don't know much about this stuff:

Since we now have the mq packet scheduler, which exposes the device
queues as qdisc classes, how about adding driver-specific configuration
attributes that are passed to the driver by the mq scheduler? This
would allow to configure per-queue bandwidth limits using regular TC
commands and also use those limits without VFs for any kind of traffic.
Drivers not supporting this would refuse unsupported options.

Re: igb bandwidth allocation configuration

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 1123 bytes --]

Patrick McHardy wrote:
> Simon Horman wrote:
>>
>> I have been looking into adding support the 82586's per-PF/VF
>> bandwidth allocation to the igb driver. It seems that the trickiest
>> part is working out how to expose things to user-space.
>>
>> ...
>> Internally it seems that actually the limits are applied to HW Tx queues
>> rather than directly VMs. There are 16 such queues. Accordingly it might
>> be useful to design an interface to set limits per-queue using ethtool.
>> But this would seem to also require exposing which queues are associated
>> with which PF/VF.
> 
> Just an idea since I don't know much about this stuff:
> 
> Since we now have the mq packet scheduler, which exposes the device
> queues as qdisc classes, how about adding driver-specific configuration
> attributes that are passed to the driver by the mq scheduler? This
> would allow to configure per-queue bandwidth limits using regular TC
> commands and also use those limits without VFs for any kind of traffic.
> Drivers not supporting this would refuse unsupported options.

Attached patch demonstrates the idea. Compile-tested only.


[-- Attachment #2: x --]
[-- Type: text/plain, Size: 3012 bytes --]

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index a44118b..388841c 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -178,6 +178,7 @@ enum {
 struct neighbour;
 struct neigh_parms;
 struct sk_buff;
+struct nlattr;
 
 struct netif_rx_stats
 {
@@ -636,6 +637,12 @@ struct net_device_ops {
 	int			(*ndo_fcoe_ddp_done)(struct net_device *dev,
 						     u16 xid);
 #endif
+	int			(*ndo_queue_config)(struct net_device *dev,
+						    unsigned int qnum,
+						    const struct nlattr *nla[]);
+	int			(*ndo_get_queue_config)(struct net_device *dev,
+							struct sk_buff *skb,
+							unsigned int qnum);
 };
 
 /*
diff --git a/include/linux/pkt_sched.h b/include/linux/pkt_sched.h
index d51a2b3..742db43 100644
--- a/include/linux/pkt_sched.h
+++ b/include/linux/pkt_sched.h
@@ -518,4 +518,14 @@ struct tc_drr_stats
 	__u32	deficit;
 };
 
+/* MQ */
+
+enum
+{
+	TCA_MQ_UNSPEC,
+	__TCA_MQ_MAX
+};
+
+#define TCA_MQ_MAX	(__TCA_MQ_MAX - 1)
+
 #endif
diff --git a/net/sched/sch_mq.c b/net/sched/sch_mq.c
index dd5ee02..13132b9 100644
--- a/net/sched/sch_mq.c
+++ b/net/sched/sch_mq.c
@@ -171,15 +171,61 @@ static void mq_put(struct Qdisc *sch, unsigned long cl)
 	return;
 }
 
+static const struct nla_policy mq_policy[TCA_MQ_MAX + 1] = {
+	/* nothing so far */
+};
+
+static int mq_change_class(struct Qdisc *sch, u32 classid, u32 parentid,
+			   struct nlattr **tca, unsigned long *arg)
+{
+	struct net_device *dev = qdisc_dev(sch);
+	struct nlattr *tb[TCA_MQ_MAX + 1];
+	unsigned long ntx;
+	int err;
+
+	if (*arg == 0)
+		return -EOPNOTSUPP;
+	if (mq_queue_get(sch, *arg))
+		return -ENOENT;
+	ntx = *arg - 1;
+
+	if (tca == NULL)
+		return -EINVAL;
+
+	err = nla_parse_nested(tb, TCA_MQ_MAX, tca[TCA_OPTIONS], mq_policy);
+	if (err < 0)
+		return err;
+
+	if (dev->netdev_ops->ndo_queue_config == NULL)
+		return -EOPNOTSUPP;
+	return dev->netdev_ops->ndo_queue_config(dev, ntx, (void *)tb);
+}
+
 static int mq_dump_class(struct Qdisc *sch, unsigned long cl,
 			 struct sk_buff *skb, struct tcmsg *tcm)
 {
 	struct netdev_queue *dev_queue = mq_queue_get(sch, cl);
+	struct net_device *dev = qdisc_dev(sch);
+	struct nlattr *nest;
 
 	tcm->tcm_parent = TC_H_ROOT;
 	tcm->tcm_handle |= TC_H_MIN(cl);
 	tcm->tcm_info = dev_queue->qdisc_sleeping->handle;
-	return 0;
+
+	if (dev->netdev_ops->ndo_get_queue_config) {
+		nest = nla_nest_start(skb, TCA_OPTIONS);
+		if (nest == NULL)
+			goto nla_put_failure;
+		if (dev->netdev_ops->ndo_get_queue_config(dev, skb, cl - 1) < 0)
+			goto nla_put_failure;
+		nla_nest_end(skb, nest);
+	}
+
+	return skb->len;
+
+nla_put_failure:
+	nla_nest_cancel(skb, nest);
+	return -EMSGSIZE;
 }
 
 static int mq_dump_class_stats(struct Qdisc *sch, unsigned long cl,
@@ -214,6 +260,7 @@ static void mq_walk(struct Qdisc *sch, struct qdisc_walker *arg)
 
 static const struct Qdisc_class_ops mq_class_ops = {
 	.select_queue	= mq_select_queue,
+	.change		= mq_change_class,
 	.graft		= mq_graft,
 	.leaf		= mq_leaf,
 	.get		= mq_get,

Re: igb bandwidth allocation configuration

[Simon Horman]

On Thu, Sep 10, 2009 at 01:55:37PM +0200, Patrick McHardy wrote:
> Patrick McHardy wrote:
> > Simon Horman wrote:
> >>
> >> I have been looking into adding support the 82586's per-PF/VF
> >> bandwidth allocation to the igb driver. It seems that the trickiest
> >> part is working out how to expose things to user-space.
> >>
> >> ...
> >> Internally it seems that actually the limits are applied to HW Tx queues
> >> rather than directly VMs. There are 16 such queues. Accordingly it might
> >> be useful to design an interface to set limits per-queue using ethtool.
> >> But this would seem to also require exposing which queues are associated
> >> with which PF/VF.
> > 
> > Just an idea since I don't know much about this stuff:
> > 
> > Since we now have the mq packet scheduler, which exposes the device
> > queues as qdisc classes, how about adding driver-specific configuration
> > attributes that are passed to the driver by the mq scheduler? This
> > would allow to configure per-queue bandwidth limits using regular TC
> > commands and also use those limits without VFs for any kind of traffic.
> > Drivers not supporting this would refuse unsupported options.
> 
> Attached patch demonstrates the idea. Compile-tested only.
> 

Thanks, that seems like a pretty good idea to me.
I'll see if I can make it work.

Re: igb bandwidth allocation configuration

[Simon Horman]

On Fri, Sep 11, 2009 at 10:38:38AM +1000, Simon Horman wrote:
> On Thu, Sep 10, 2009 at 01:55:37PM +0200, Patrick McHardy wrote:
> > Patrick McHardy wrote:
> > > Simon Horman wrote:
> > >>
> > >> I have been looking into adding support the 82586's per-PF/VF
> > >> bandwidth allocation to the igb driver. It seems that the trickiest
> > >> part is working out how to expose things to user-space.
> > >>
> > >> ...
> > >> Internally it seems that actually the limits are applied to HW Tx queues
> > >> rather than directly VMs. There are 16 such queues. Accordingly it might
> > >> be useful to design an interface to set limits per-queue using ethtool.
> > >> But this would seem to also require exposing which queues are associated
> > >> with which PF/VF.
> > > 
> > > Just an idea since I don't know much about this stuff:
> > > 
> > > Since we now have the mq packet scheduler, which exposes the device
> > > queues as qdisc classes, how about adding driver-specific configuration
> > > attributes that are passed to the driver by the mq scheduler? This
> > > would allow to configure per-queue bandwidth limits using regular TC
> > > commands and also use those limits without VFs for any kind of traffic.
> > > Drivers not supporting this would refuse unsupported options.
> > 
> > Attached patch demonstrates the idea. Compile-tested only.
> > 
> 
> Thanks, that seems like a pretty good idea to me.
> I'll see if I can make it work.

I've been looking over this a little more closely. While using mq
does seem to be a good way to configure the hw bandwith allocation
for queues belonging to the PF I don't think it can be used
for queues belonging to any VFs. Primarily because the VFs will
belong be different devices, possibly in different OSes.

A further complication is that although the PF and VFs have different
devices they do share hw and the bandwidth allocation rules in the
datasheet seem to imply that they bandwidth allocations need to be
made together to ensure that the rules aren't violated. That is,
they need to be verified as a group at some point. Which is
what let to my original suggestion of using ethtool on the device
corresponding to the PF.

Re: igb bandwidth allocation configuration

[Or Gerlitz]

On 9/10/09, Simon Horman <horms@verge.net.au> wrote:
> I have been looking into adding support the 82586's per-PF/VF bandwidth allocation to
> the igb driver. It seems that the trickiest part is working out how to expose things to
> user-space.

Please note that there are bunch (not many, but more then 1-2) of
things to configure from user space in a PF/VF scheme, and I think it
would be best if we do that through one mechanism, which may be
netlink based or extension to ethtool as you suggested, we've started
to discuss this on the "L2 switching in igb" thread

The "82576 SR-IOV Driver Companion Guide" document, section 7.6
mentions "Transmit Bandwidth Allocation to VFs... define minimum
transmit bandwidth for individual VMs".

I'm not clear if one can program rate limiter (upper bound) per VF or
actually rate guarantee per VF, even  with these being  just details
of specific device, alex, I would be happy if you can clarify that.


Or.

Re: igb bandwidth allocation configuration

[Simon Horman]

On Mon, Sep 14, 2009 at 11:42:50AM +0300, Or Gerlitz wrote:
> On 9/10/09, Simon Horman <horms@verge.net.au> wrote:
> > I have been looking into adding support the 82586's per-PF/VF bandwidth allocation to
> > the igb driver. It seems that the trickiest part is working out how to expose things to
> > user-space.
> 
> Please note that there are bunch (not many, but more then 1-2) of
> things to configure from user space in a PF/VF scheme, and I think it
> would be best if we do that through one mechanism, which may be
> netlink based or extension to ethtool as you suggested, we've started
> to discuss this on the "L2 switching in igb" thread

Thanks, I've read that thread now. I'm certainly in favour
of discussing what mechanisms make sense.

It seems to me that the main problem is that from a driver point
of view the PF and VFs are independent. But from a hardware point
of view they aren't so its not always possible for their configuration
to be independent of each other. And I'm not sure what (existing) interfaces
can handle that nicely.

> The "82576 SR-IOV Driver Companion Guide" document, section 7.6
> mentions "Transmit Bandwidth Allocation to VFs... define minimum
> transmit bandwidth for individual VMs".
> 
> I'm not clear if one can program rate limiter (upper bound) per VF or
> actually rate guarantee per VF, even  with these being  just details
> of specific device, alex, I would be happy if you can clarify that.

Its not clear to me what you are asking.


------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf

Re: igb bandwidth allocation configuration

[Or Gerlitz]

Simon Horman wrote:
> It seems to me that the main problem is that from a driver point of view the PF and VFs are independent. But from a hardware point of view they aren't so its not always possible for their configuration to be independent of each other. And I'm not sure what (existing) interfaces can handle that nicely.

If the rate limiter is exposed as a feature of the VF, it doesn't matter 
who really enforces it, the "VF portion" of the HW or the PF itself. I 
agree that if you have to program the PF for the rate of a specific VF, 
then its more complex. Basically, I would expect that a VF can be 
configured with <mac, vlad-id, priority, rate> such that it can be done 
where the VF NIC is spawned, host kernel or guest kernel.

> Its not clear to me what you are asking.
I'm was asking/wondering if the Intel NICs have a rate limiter (i.e one 
can program the VF such that its rate doesn't exceed XX MB/s) or a "rate 
guarantee"  (i.e one can program the VF such that its guaranteed it will 
get YY MB/s in case it wants to xmit at least this bandwidth)

Or.


------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf

Re: igb bandwidth allocation configuration

[Alexander Duyck]

Or Gerlitz wrote:
> If the rate limiter is exposed as a feature of the VF, it doesn't matter 
> who really enforces it, the "VF portion" of the HW or the PF itself. I 
> agree that if you have to program the PF for the rate of a specific VF, 
> then its more complex. Basically, I would expect that a VF can be 
> configured with <mac, vlad-id, priority, rate> such that it can be done 
> where the VF NIC is spawned, host kernel or guest kernel.

Adding the rate limiter as a feature of the VF doesn't make much sense 
since the VF could be direct assigned to another OS for all we know so 
we won't have control over it from there.

The interface for all of this would make sense as part of a virtual 
ethernet switch control which is the way I am currently leaning on all 
this.  As such it is probably another thing we can bring up at the BOF 
session at the Linux Plumbers Conference.

> I'm was asking/wondering if the Intel NICs have a rate limiter (i.e one 
> can program the VF such that its rate doesn't exceed XX MB/s) or a "rate 
> guarantee"  (i.e one can program the VF such that its guaranteed it will 
> get YY MB/s in case it wants to xmit at least this bandwidth)

Based on the way I am reading the documentation I would say the all 
these registers do is guarantee a minimum percentage of the bandwidth. 
With these registers set you can repartition the traffic so that a 
percentage can be guaranteed to the PF/VFs if needed.  It works very 
similar to how DCB allows you to guarantee a certain amount of bandwidth 
for each of the traffic classes.  However any time the full tx bandwidth 
is not being used it will be reallocated to the other queues and then 
end up back in the default behavior.

The default behavior is to DMA descriptors from the rings in a round 
robin fashion.  Since this effectively guarantees that there will be 
packets being pulled off the rings I didn't really feel the necessity to 
add the additional overhead of doing this on a per PF/VF bandwidth basis.

Thanks,

Alex

RE: igb bandwidth allocation configuration

[Nelson, Shannon]

Alexander Duyck wrote: 
>Or Gerlitz wrote:
>> I'm was asking/wondering if the Intel NICs have a rate limiter (i.e one 
>> can program the VF such that its rate doesn't exceed XX MB/s) or a "rate 
>> guarantee"  (i.e one can program the VF such that its guaranteed it will 
>> get YY MB/s in case it wants to xmit at least this bandwidth)
>
>Based on the way I am reading the documentation I would say the all 
>these registers do is guarantee a minimum percentage of the bandwidth. 

FYI, the newer 10gig 82599 *does* have a rate limiting feature in hw.  This feature is not currently supported by the ixgbe driver.

sln

Re: igb bandwidth allocation configuration

[Simon Horman]

On Tue, Sep 15, 2009 at 11:01:52AM -0700, Alexander Duyck wrote:
> Or Gerlitz wrote:
> >If the rate limiter is exposed as a feature of the VF, it doesn't
> >matter who really enforces it, the "VF portion" of the HW or the
> >PF itself. I agree that if you have to program the PF for the rate
> >of a specific VF, then its more complex. Basically, I would expect
> >that a VF can be configured with <mac, vlad-id, priority, rate>
> >such that it can be done where the VF NIC is spawned, host kernel
> >or guest kernel.
> 
> Adding the rate limiter as a feature of the VF doesn't make much
> sense since the VF could be direct assigned to another OS for all we
> know so we won't have control over it from there.
> 
> The interface for all of this would make sense as part of a virtual
> ethernet switch control which is the way I am currently leaning on
> all this.  As such it is probably another thing we can bring up at
> the BOF session at the Linux Plumbers Conference.

Unfortunately I won't be able to make it to the BOF or Plumbers.
I look forward to hearing what is discussed.

> >I'm was asking/wondering if the Intel NICs have a rate limiter
> >(i.e one can program the VF such that its rate doesn't exceed XX
> >MB/s) or a "rate guarantee"  (i.e one can program the VF such that
> >its guaranteed it will get YY MB/s in case it wants to xmit at
> >least this bandwidth)
> 
> Based on the way I am reading the documentation I would say the all
> these registers do is guarantee a minimum percentage of the
> bandwidth. With these registers set you can repartition the traffic
> so that a percentage can be guaranteed to the PF/VFs if needed.  It
> works very similar to how DCB allows you to guarantee a certain
> amount of bandwidth for each of the traffic classes.  However any
> time the full tx bandwidth is not being used it will be reallocated
> to the other queues and then end up back in the default behavior.
> 
> The default behavior is to DMA descriptors from the rings in a round
> robin fashion.  Since this effectively guarantees that there will be
> packets being pulled off the rings I didn't really feel the
> necessity to add the additional overhead of doing this on a per
> PF/VF bandwidth basis.

My reading of the documentation is also that its a minimum percentage of
bandwidth. But after playing with the registers a bit it seems
that it is actually a ceiling expressed. And the percentage is of 1Gigabit,
regardless of the actual link speed.

Re: igb bandwidth allocation configuration

[Or Gerlitz]

Simon Horman wrote:
> My reading of the documentation is also that its a minimum percentage of bandwidth. But after playing with the registers a bit it seems that it is actually a ceiling expressed. And the percentage is of 1Gigabit, regardless of the actual link speed.
Hi,

Does the documentation you refer to is the 82576 one? if yes, can you 
send a pointer to the doc/section you refer to? also is there 82599 
(Niantic) documentation which is publicly avail and I can look at? 
specifically, I would love taking a look on the equivalent of the "Intel 
82576 SR-IOV Driver Companion Guide"


Or.

Re: igb bandwidth allocation configuration

[Simon Horman]

On Wed, Sep 16, 2009 at 09:47:28AM +0300, Or Gerlitz wrote:
> Simon Horman wrote:
> >My reading of the documentation is also that its a minimum percentage of bandwidth. But after playing with the registers a bit it seems that it is actually a ceiling expressed. And the percentage is of 1Gigabit, regardless of the actual link speed.
> Hi,
> 
> Does the documentation you refer to is the 82576 one? if yes, can
> you send a pointer to the doc/section you refer to?

Yes, its the 82576. I'm looking at the 82576 datasheet.  In particular
section 4.5.11.1.5.1 "Configuring Tx Bandwidth to VMs", although there are
also some other sections covering the topic in that document.

> also is there
> 82599 (Niantic) documentation which is publicly avail and I can look
> at? specifically, I would love taking a look on the equivalent of
> the "Intel 82576 SR-IOV Driver Companion Guide"

Sorry, I don't know anything about the 82599. But I am only working

with publicly available documentation.

Re: igb bandwidth allocation configuration

[Nelson, Shannon]

Simon Horman wrote:
>On Wed, Sep 16, 2009 at 09:47:28AM +0300, Or Gerlitz wrote:
>> also is there
>> 82599 (Niantic) documentation which is publicly avail and I can look
>> at? specifically, I would love taking a look on the equivalent of
>> the "Intel 82576 SR-IOV Driver Companion Guide"
>
>Sorry, I don't know anything about the 82599. But I am only working
>with publicly available documentation.

The "82599 Developer Manual" is available at http://sourceforge.net/projects/e1000/files/.

sln

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf

Re: igb bandwidth allocation configuration

[Simon Horman]

On Wed, Sep 16, 2009 at 08:29:26AM +1000, Simon Horman wrote:
> On Tue, Sep 15, 2009 at 11:01:52AM -0700, Alexander Duyck wrote:
> > Or Gerlitz wrote:
> > >If the rate limiter is exposed as a feature of the VF, it doesn't
> > >matter who really enforces it, the "VF portion" of the HW or the
> > >PF itself. I agree that if you have to program the PF for the rate
> > >of a specific VF, then its more complex. Basically, I would expect
> > >that a VF can be configured with <mac, vlad-id, priority, rate>
> > >such that it can be done where the VF NIC is spawned, host kernel
> > >or guest kernel.
> > 
> > Adding the rate limiter as a feature of the VF doesn't make much
> > sense since the VF could be direct assigned to another OS for all we
> > know so we won't have control over it from there.
> > 
> > The interface for all of this would make sense as part of a virtual
> > ethernet switch control which is the way I am currently leaning on
> > all this.  As such it is probably another thing we can bring up at
> > the BOF session at the Linux Plumbers Conference.
> 
> Unfortunately I won't be able to make it to the BOF or Plumbers.
> I look forward to hearing what is discussed.

Is there any chance of being able to participate in this remotely,
even just a listen-only feed of some sort would be quite valuable to me
and possibly others.

------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf

Re: igb bandwidth allocation configuration

[Or Gerlitz]

Alexander Duyck wrote:
> The interface for all of this would make sense as part of a virtual 
> ethernet switch control which is the way I am currently leaning on all 
> this.
Yes, you can say that out of the per VF <mac, vlan-id, priority, rate> 
tuple I mentioned, except for the mac, the other parameters actually 
belong to the egress flow of the virtual switch port this VF is 
connected to. So the vswitch actually signs the packet with vlan+pbits 
and enforces the rate. Now vswitch can be software based, or hardware 
NIC based.

Now, I assume there may be NICs which will let you configure the 
<vlan-id, priority, rate> as part of the their virtual switch config, 
but others, e.g
the 82576 as an example, and following our discussion, will let you do 
that for the VF, in the VF driver which as you said may run the guest OS 
where we can't control it...

Or.

Re: igb bandwidth allocation configuration

[Alexander Duyck]

Or Gerlitz wrote:
> Alexander Duyck wrote:
>> The interface for all of this would make sense as part of a virtual 
>> ethernet switch control which is the way I am currently leaning on all 
>> this.
> Yes, you can say that out of the per VF <mac, vlan-id, priority, rate> 
> tuple I mentioned, except for the mac, the other parameters actually 
> belong to the egress flow of the virtual switch port this VF is 
> connected to. So the vswitch actually signs the packet with vlan+pbits 
> and enforces the rate. Now vswitch can be software based, or hardware 
> NIC based.

Even something such as MAC address would make sense for a virtual 
ethernet switch configuration in that you could restrict unicast ingress 
traffic for the VF to a specific address much like you would do on a 
regular L2 switch.

> Now, I assume there may be NICs which will let you configure the 
> <vlan-id, priority, rate> as part of the their virtual switch config, 
> but others, e.g
> the 82576 as an example, and following our discussion, will let you do 
> that for the VF, in the VF driver which as you said may run the guest OS 
> where we can't control it...

I think you may be a bit confused.  The configuration for the VFs would 
be part of the PF via the virtual ethernet switch control.  As a result 
it is only the PF which needs to be running on the host.

Thanks,

Alex