* igb bandwidth allocation configuration
@ 2009-09-10 8:18 Simon Horman
2009-09-10 11:28 ` Patrick McHardy
2009-09-14 8:42 ` Or Gerlitz
0 siblings, 2 replies; 17+ messages in thread
From: Simon Horman @ 2009-09-10 8:18 UTC (permalink / raw)
To: e1000-devel, netdev
Hi,
I have been looking into adding support the 82586's per-PF/VF
bandwidth allocation to the igb driver. It seems that the trickiest
part is working out how to expose things to user-space.
I was thinking along the lines of an ethtool option as follows:
ethtool --bandwidth ethN LIMIT...
where:
* There is one LIMIT per PF/VF.
The 82576 can have up to 7 VFs per PF,
so there would be up to 8 LIMITS
* A keyword (none?) can be used to denote that
bandwidth allocation should be disabled for the
corresponding VM
* Otherwise LIMITS are in Megabits/s
This may get a bit combersome if there are a lot of VFs per PF,
perhaps a better syntax would be:
ethtool --bandwidth ethN M=LIMIT...
where:
* LIMIT is as above
* M is some key to denote which VF/PF is
having its limit set.
Internally it seems that actually the limits are applied to HW Tx queues
rather than directly VMs. There are 16 such queues. Accordingly it might
be useful to design an interface to set limits per-queue using ethtool.
But this would seem to also require exposing which queues are associated
with which PF/VF.
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: igb bandwidth allocation configuration
2009-09-10 8:18 igb bandwidth allocation configuration Simon Horman
@ 2009-09-10 11:28 ` Patrick McHardy
2009-09-10 11:55 ` Patrick McHardy
2009-09-14 8:42 ` Or Gerlitz
1 sibling, 1 reply; 17+ messages in thread
From: Patrick McHardy @ 2009-09-10 11:28 UTC (permalink / raw)
To: Simon Horman; +Cc: e1000-devel, netdev
Simon Horman wrote:
> Hi,
>
> I have been looking into adding support the 82586's per-PF/VF
> bandwidth allocation to the igb driver. It seems that the trickiest
> part is working out how to expose things to user-space.
>
> I was thinking along the lines of an ethtool option as follows:
>
> ethtool --bandwidth ethN LIMIT...
>
> where:
> * There is one LIMIT per PF/VF.
> The 82576 can have up to 7 VFs per PF,
> so there would be up to 8 LIMITS
> * A keyword (none?) can be used to denote that
> bandwidth allocation should be disabled for the
> corresponding VM
> * Otherwise LIMITS are in Megabits/s
>
> This may get a bit combersome if there are a lot of VFs per PF,
> perhaps a better syntax would be:
>
> ethtool --bandwidth ethN M=LIMIT...
>
> where:
> * LIMIT is as above
> * M is some key to denote which VF/PF is
> having its limit set.
>
> Internally it seems that actually the limits are applied to HW Tx queues
> rather than directly VMs. There are 16 such queues. Accordingly it might
> be useful to design an interface to set limits per-queue using ethtool.
> But this would seem to also require exposing which queues are associated
> with which PF/VF.
Just an idea since I don't know much about this stuff:
Since we now have the mq packet scheduler, which exposes the device
queues as qdisc classes, how about adding driver-specific configuration
attributes that are passed to the driver by the mq scheduler? This
would allow to configure per-queue bandwidth limits using regular TC
commands and also use those limits without VFs for any kind of traffic.
Drivers not supporting this would refuse unsupported options.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: igb bandwidth allocation configuration
2009-09-10 11:28 ` Patrick McHardy
@ 2009-09-10 11:55 ` Patrick McHardy
2009-09-11 0:38 ` Simon Horman
0 siblings, 1 reply; 17+ messages in thread
From: Patrick McHardy @ 2009-09-10 11:55 UTC (permalink / raw)
To: Simon Horman; +Cc: e1000-devel, netdev
[-- Attachment #1: Type: text/plain, Size: 1123 bytes --]
Patrick McHardy wrote:
> Simon Horman wrote:
>>
>> I have been looking into adding support the 82586's per-PF/VF
>> bandwidth allocation to the igb driver. It seems that the trickiest
>> part is working out how to expose things to user-space.
>>
>> ...
>> Internally it seems that actually the limits are applied to HW Tx queues
>> rather than directly VMs. There are 16 such queues. Accordingly it might
>> be useful to design an interface to set limits per-queue using ethtool.
>> But this would seem to also require exposing which queues are associated
>> with which PF/VF.
>
> Just an idea since I don't know much about this stuff:
>
> Since we now have the mq packet scheduler, which exposes the device
> queues as qdisc classes, how about adding driver-specific configuration
> attributes that are passed to the driver by the mq scheduler? This
> would allow to configure per-queue bandwidth limits using regular TC
> commands and also use those limits without VFs for any kind of traffic.
> Drivers not supporting this would refuse unsupported options.
Attached patch demonstrates the idea. Compile-tested only.
[-- Attachment #2: x --]
[-- Type: text/plain, Size: 3012 bytes --]
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index a44118b..388841c 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -178,6 +178,7 @@ enum {
struct neighbour;
struct neigh_parms;
struct sk_buff;
+struct nlattr;
struct netif_rx_stats
{
@@ -636,6 +637,12 @@ struct net_device_ops {
int (*ndo_fcoe_ddp_done)(struct net_device *dev,
u16 xid);
#endif
+ int (*ndo_queue_config)(struct net_device *dev,
+ unsigned int qnum,
+ const struct nlattr *nla[]);
+ int (*ndo_get_queue_config)(struct net_device *dev,
+ struct sk_buff *skb,
+ unsigned int qnum);
};
/*
diff --git a/include/linux/pkt_sched.h b/include/linux/pkt_sched.h
index d51a2b3..742db43 100644
--- a/include/linux/pkt_sched.h
+++ b/include/linux/pkt_sched.h
@@ -518,4 +518,14 @@ struct tc_drr_stats
__u32 deficit;
};
+/* MQ */
+
+enum
+{
+ TCA_MQ_UNSPEC,
+ __TCA_MQ_MAX
+};
+
+#define TCA_MQ_MAX (__TCA_MQ_MAX - 1)
+
#endif
diff --git a/net/sched/sch_mq.c b/net/sched/sch_mq.c
index dd5ee02..13132b9 100644
--- a/net/sched/sch_mq.c
+++ b/net/sched/sch_mq.c
@@ -171,15 +171,61 @@ static void mq_put(struct Qdisc *sch, unsigned long cl)
return;
}
+static const struct nla_policy mq_policy[TCA_MQ_MAX + 1] = {
+ /* nothing so far */
+};
+
+static int mq_change_class(struct Qdisc *sch, u32 classid, u32 parentid,
+ struct nlattr **tca, unsigned long *arg)
+{
+ struct net_device *dev = qdisc_dev(sch);
+ struct nlattr *tb[TCA_MQ_MAX + 1];
+ unsigned long ntx;
+ int err;
+
+ if (*arg == 0)
+ return -EOPNOTSUPP;
+ if (mq_queue_get(sch, *arg))
+ return -ENOENT;
+ ntx = *arg - 1;
+
+ if (tca == NULL)
+ return -EINVAL;
+
+ err = nla_parse_nested(tb, TCA_MQ_MAX, tca[TCA_OPTIONS], mq_policy);
+ if (err < 0)
+ return err;
+
+ if (dev->netdev_ops->ndo_queue_config == NULL)
+ return -EOPNOTSUPP;
+ return dev->netdev_ops->ndo_queue_config(dev, ntx, (void *)tb);
+}
+
static int mq_dump_class(struct Qdisc *sch, unsigned long cl,
struct sk_buff *skb, struct tcmsg *tcm)
{
struct netdev_queue *dev_queue = mq_queue_get(sch, cl);
+ struct net_device *dev = qdisc_dev(sch);
+ struct nlattr *nest;
tcm->tcm_parent = TC_H_ROOT;
tcm->tcm_handle |= TC_H_MIN(cl);
tcm->tcm_info = dev_queue->qdisc_sleeping->handle;
- return 0;
+
+ if (dev->netdev_ops->ndo_get_queue_config) {
+ nest = nla_nest_start(skb, TCA_OPTIONS);
+ if (nest == NULL)
+ goto nla_put_failure;
+ if (dev->netdev_ops->ndo_get_queue_config(dev, skb, cl - 1) < 0)
+ goto nla_put_failure;
+ nla_nest_end(skb, nest);
+ }
+
+ return skb->len;
+
+nla_put_failure:
+ nla_nest_cancel(skb, nest);
+ return -EMSGSIZE;
}
static int mq_dump_class_stats(struct Qdisc *sch, unsigned long cl,
@@ -214,6 +260,7 @@ static void mq_walk(struct Qdisc *sch, struct qdisc_walker *arg)
static const struct Qdisc_class_ops mq_class_ops = {
.select_queue = mq_select_queue,
+ .change = mq_change_class,
.graft = mq_graft,
.leaf = mq_leaf,
.get = mq_get,
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: igb bandwidth allocation configuration
2009-09-10 11:55 ` Patrick McHardy
@ 2009-09-11 0:38 ` Simon Horman
2009-09-15 11:32 ` Simon Horman
0 siblings, 1 reply; 17+ messages in thread
From: Simon Horman @ 2009-09-11 0:38 UTC (permalink / raw)
To: Patrick McHardy; +Cc: e1000-devel, netdev
On Thu, Sep 10, 2009 at 01:55:37PM +0200, Patrick McHardy wrote:
> Patrick McHardy wrote:
> > Simon Horman wrote:
> >>
> >> I have been looking into adding support the 82586's per-PF/VF
> >> bandwidth allocation to the igb driver. It seems that the trickiest
> >> part is working out how to expose things to user-space.
> >>
> >> ...
> >> Internally it seems that actually the limits are applied to HW Tx queues
> >> rather than directly VMs. There are 16 such queues. Accordingly it might
> >> be useful to design an interface to set limits per-queue using ethtool.
> >> But this would seem to also require exposing which queues are associated
> >> with which PF/VF.
> >
> > Just an idea since I don't know much about this stuff:
> >
> > Since we now have the mq packet scheduler, which exposes the device
> > queues as qdisc classes, how about adding driver-specific configuration
> > attributes that are passed to the driver by the mq scheduler? This
> > would allow to configure per-queue bandwidth limits using regular TC
> > commands and also use those limits without VFs for any kind of traffic.
> > Drivers not supporting this would refuse unsupported options.
>
> Attached patch demonstrates the idea. Compile-tested only.
>
Thanks, that seems like a pretty good idea to me.
I'll see if I can make it work.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: igb bandwidth allocation configuration
2009-09-10 8:18 igb bandwidth allocation configuration Simon Horman
2009-09-10 11:28 ` Patrick McHardy
@ 2009-09-14 8:42 ` Or Gerlitz
2009-09-15 11:36 ` Simon Horman
1 sibling, 1 reply; 17+ messages in thread
From: Or Gerlitz @ 2009-09-14 8:42 UTC (permalink / raw)
To: Simon Horman
Cc: e1000-devel, netdev, Alexander Duyck, Kirsher, Jeffrey T,
Or Gerlitz
On 9/10/09, Simon Horman <horms@verge.net.au> wrote:
> I have been looking into adding support the 82586's per-PF/VF bandwidth allocation to
> the igb driver. It seems that the trickiest part is working out how to expose things to
> user-space.
Please note that there are bunch (not many, but more then 1-2) of
things to configure from user space in a PF/VF scheme, and I think it
would be best if we do that through one mechanism, which may be
netlink based or extension to ethtool as you suggested, we've started
to discuss this on the "L2 switching in igb" thread
The "82576 SR-IOV Driver Companion Guide" document, section 7.6
mentions "Transmit Bandwidth Allocation to VFs... define minimum
transmit bandwidth for individual VMs".
I'm not clear if one can program rate limiter (upper bound) per VF or
actually rate guarantee per VF, even with these being just details
of specific device, alex, I would be happy if you can clarify that.
Or.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: igb bandwidth allocation configuration
2009-09-11 0:38 ` Simon Horman
@ 2009-09-15 11:32 ` Simon Horman
0 siblings, 0 replies; 17+ messages in thread
From: Simon Horman @ 2009-09-15 11:32 UTC (permalink / raw)
To: Patrick McHardy; +Cc: e1000-devel, netdev
On Fri, Sep 11, 2009 at 10:38:38AM +1000, Simon Horman wrote:
> On Thu, Sep 10, 2009 at 01:55:37PM +0200, Patrick McHardy wrote:
> > Patrick McHardy wrote:
> > > Simon Horman wrote:
> > >>
> > >> I have been looking into adding support the 82586's per-PF/VF
> > >> bandwidth allocation to the igb driver. It seems that the trickiest
> > >> part is working out how to expose things to user-space.
> > >>
> > >> ...
> > >> Internally it seems that actually the limits are applied to HW Tx queues
> > >> rather than directly VMs. There are 16 such queues. Accordingly it might
> > >> be useful to design an interface to set limits per-queue using ethtool.
> > >> But this would seem to also require exposing which queues are associated
> > >> with which PF/VF.
> > >
> > > Just an idea since I don't know much about this stuff:
> > >
> > > Since we now have the mq packet scheduler, which exposes the device
> > > queues as qdisc classes, how about adding driver-specific configuration
> > > attributes that are passed to the driver by the mq scheduler? This
> > > would allow to configure per-queue bandwidth limits using regular TC
> > > commands and also use those limits without VFs for any kind of traffic.
> > > Drivers not supporting this would refuse unsupported options.
> >
> > Attached patch demonstrates the idea. Compile-tested only.
> >
>
> Thanks, that seems like a pretty good idea to me.
> I'll see if I can make it work.
I've been looking over this a little more closely. While using mq
does seem to be a good way to configure the hw bandwith allocation
for queues belonging to the PF I don't think it can be used
for queues belonging to any VFs. Primarily because the VFs will
belong be different devices, possibly in different OSes.
A further complication is that although the PF and VFs have different
devices they do share hw and the bandwidth allocation rules in the
datasheet seem to imply that they bandwidth allocations need to be
made together to ensure that the rules aren't violated. That is,
they need to be verified as a group at some point. Which is
what let to my original suggestion of using ethtool on the device
corresponding to the PF.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: igb bandwidth allocation configuration
2009-09-14 8:42 ` Or Gerlitz
@ 2009-09-15 11:36 ` Simon Horman
2009-09-15 13:27 ` Or Gerlitz
0 siblings, 1 reply; 17+ messages in thread
From: Simon Horman @ 2009-09-15 11:36 UTC (permalink / raw)
To: Or Gerlitz
Cc: e1000-devel, netdev, Kirsher, Jeffrey T, Alexander Duyck,
Or Gerlitz
On Mon, Sep 14, 2009 at 11:42:50AM +0300, Or Gerlitz wrote:
> On 9/10/09, Simon Horman <horms@verge.net.au> wrote:
> > I have been looking into adding support the 82586's per-PF/VF bandwidth allocation to
> > the igb driver. It seems that the trickiest part is working out how to expose things to
> > user-space.
>
> Please note that there are bunch (not many, but more then 1-2) of
> things to configure from user space in a PF/VF scheme, and I think it
> would be best if we do that through one mechanism, which may be
> netlink based or extension to ethtool as you suggested, we've started
> to discuss this on the "L2 switching in igb" thread
Thanks, I've read that thread now. I'm certainly in favour
of discussing what mechanisms make sense.
It seems to me that the main problem is that from a driver point
of view the PF and VFs are independent. But from a hardware point
of view they aren't so its not always possible for their configuration
to be independent of each other. And I'm not sure what (existing) interfaces
can handle that nicely.
> The "82576 SR-IOV Driver Companion Guide" document, section 7.6
> mentions "Transmit Bandwidth Allocation to VFs... define minimum
> transmit bandwidth for individual VMs".
>
> I'm not clear if one can program rate limiter (upper bound) per VF or
> actually rate guarantee per VF, even with these being just details
> of specific device, alex, I would be happy if you can clarify that.
Its not clear to me what you are asking.
------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: igb bandwidth allocation configuration
2009-09-15 11:36 ` Simon Horman
@ 2009-09-15 13:27 ` Or Gerlitz
2009-09-15 18:01 ` Alexander Duyck
0 siblings, 1 reply; 17+ messages in thread
From: Or Gerlitz @ 2009-09-15 13:27 UTC (permalink / raw)
To: Simon Horman; +Cc: e1000-devel, netdev, Kirsher, Jeffrey T, Alexander Duyck
Simon Horman wrote:
> It seems to me that the main problem is that from a driver point of view the PF and VFs are independent. But from a hardware point of view they aren't so its not always possible for their configuration to be independent of each other. And I'm not sure what (existing) interfaces can handle that nicely.
If the rate limiter is exposed as a feature of the VF, it doesn't matter
who really enforces it, the "VF portion" of the HW or the PF itself. I
agree that if you have to program the PF for the rate of a specific VF,
then its more complex. Basically, I would expect that a VF can be
configured with <mac, vlad-id, priority, rate> such that it can be done
where the VF NIC is spawned, host kernel or guest kernel.
> Its not clear to me what you are asking.
I'm was asking/wondering if the Intel NICs have a rate limiter (i.e one
can program the VF such that its rate doesn't exceed XX MB/s) or a "rate
guarantee" (i.e one can program the VF such that its guaranteed it will
get YY MB/s in case it wants to xmit at least this bandwidth)
Or.
------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: igb bandwidth allocation configuration
2009-09-15 13:27 ` Or Gerlitz
@ 2009-09-15 18:01 ` Alexander Duyck
2009-09-15 18:25 ` Nelson, Shannon
` (2 more replies)
0 siblings, 3 replies; 17+ messages in thread
From: Alexander Duyck @ 2009-09-15 18:01 UTC (permalink / raw)
To: Or Gerlitz
Cc: Simon Horman, e1000-devel@lists.sourceforge.net,
netdev@vger.kernel.org, Alexander Duyck, Kirsher, Jeffrey T
Or Gerlitz wrote:
> If the rate limiter is exposed as a feature of the VF, it doesn't matter
> who really enforces it, the "VF portion" of the HW or the PF itself. I
> agree that if you have to program the PF for the rate of a specific VF,
> then its more complex. Basically, I would expect that a VF can be
> configured with <mac, vlad-id, priority, rate> such that it can be done
> where the VF NIC is spawned, host kernel or guest kernel.
Adding the rate limiter as a feature of the VF doesn't make much sense
since the VF could be direct assigned to another OS for all we know so
we won't have control over it from there.
The interface for all of this would make sense as part of a virtual
ethernet switch control which is the way I am currently leaning on all
this. As such it is probably another thing we can bring up at the BOF
session at the Linux Plumbers Conference.
> I'm was asking/wondering if the Intel NICs have a rate limiter (i.e one
> can program the VF such that its rate doesn't exceed XX MB/s) or a "rate
> guarantee" (i.e one can program the VF such that its guaranteed it will
> get YY MB/s in case it wants to xmit at least this bandwidth)
Based on the way I am reading the documentation I would say the all
these registers do is guarantee a minimum percentage of the bandwidth.
With these registers set you can repartition the traffic so that a
percentage can be guaranteed to the PF/VFs if needed. It works very
similar to how DCB allows you to guarantee a certain amount of bandwidth
for each of the traffic classes. However any time the full tx bandwidth
is not being used it will be reallocated to the other queues and then
end up back in the default behavior.
The default behavior is to DMA descriptors from the rings in a round
robin fashion. Since this effectively guarantees that there will be
packets being pulled off the rings I didn't really feel the necessity to
add the additional overhead of doing this on a per PF/VF bandwidth basis.
Thanks,
Alex
^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: igb bandwidth allocation configuration
2009-09-15 18:01 ` Alexander Duyck
@ 2009-09-15 18:25 ` Nelson, Shannon
2009-09-15 22:29 ` Simon Horman
2009-09-16 14:10 ` Or Gerlitz
2 siblings, 0 replies; 17+ messages in thread
From: Nelson, Shannon @ 2009-09-15 18:25 UTC (permalink / raw)
To: Duyck, Alexander H, Or Gerlitz
Cc: e1000-devel@lists.sourceforge.net, netdev@vger.kernel.org,
Simon Horman, Kirsher, Jeffrey T, Alexander Duyck
Alexander Duyck wrote:
>Or Gerlitz wrote:
>> I'm was asking/wondering if the Intel NICs have a rate limiter (i.e one
>> can program the VF such that its rate doesn't exceed XX MB/s) or a "rate
>> guarantee" (i.e one can program the VF such that its guaranteed it will
>> get YY MB/s in case it wants to xmit at least this bandwidth)
>
>Based on the way I am reading the documentation I would say the all
>these registers do is guarantee a minimum percentage of the bandwidth.
FYI, the newer 10gig 82599 *does* have a rate limiting feature in hw. This feature is not currently supported by the ixgbe driver.
sln
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: igb bandwidth allocation configuration
2009-09-15 18:01 ` Alexander Duyck
2009-09-15 18:25 ` Nelson, Shannon
@ 2009-09-15 22:29 ` Simon Horman
2009-09-16 6:47 ` Or Gerlitz
2009-09-17 1:09 ` Simon Horman
2009-09-16 14:10 ` Or Gerlitz
2 siblings, 2 replies; 17+ messages in thread
From: Simon Horman @ 2009-09-15 22:29 UTC (permalink / raw)
To: Alexander Duyck
Cc: Or Gerlitz, e1000-devel@lists.sourceforge.net,
netdev@vger.kernel.org, Alexander Duyck, Kirsher, Jeffrey T
On Tue, Sep 15, 2009 at 11:01:52AM -0700, Alexander Duyck wrote:
> Or Gerlitz wrote:
> >If the rate limiter is exposed as a feature of the VF, it doesn't
> >matter who really enforces it, the "VF portion" of the HW or the
> >PF itself. I agree that if you have to program the PF for the rate
> >of a specific VF, then its more complex. Basically, I would expect
> >that a VF can be configured with <mac, vlad-id, priority, rate>
> >such that it can be done where the VF NIC is spawned, host kernel
> >or guest kernel.
>
> Adding the rate limiter as a feature of the VF doesn't make much
> sense since the VF could be direct assigned to another OS for all we
> know so we won't have control over it from there.
>
> The interface for all of this would make sense as part of a virtual
> ethernet switch control which is the way I am currently leaning on
> all this. As such it is probably another thing we can bring up at
> the BOF session at the Linux Plumbers Conference.
Unfortunately I won't be able to make it to the BOF or Plumbers.
I look forward to hearing what is discussed.
> >I'm was asking/wondering if the Intel NICs have a rate limiter
> >(i.e one can program the VF such that its rate doesn't exceed XX
> >MB/s) or a "rate guarantee" (i.e one can program the VF such that
> >its guaranteed it will get YY MB/s in case it wants to xmit at
> >least this bandwidth)
>
> Based on the way I am reading the documentation I would say the all
> these registers do is guarantee a minimum percentage of the
> bandwidth. With these registers set you can repartition the traffic
> so that a percentage can be guaranteed to the PF/VFs if needed. It
> works very similar to how DCB allows you to guarantee a certain
> amount of bandwidth for each of the traffic classes. However any
> time the full tx bandwidth is not being used it will be reallocated
> to the other queues and then end up back in the default behavior.
>
> The default behavior is to DMA descriptors from the rings in a round
> robin fashion. Since this effectively guarantees that there will be
> packets being pulled off the rings I didn't really feel the
> necessity to add the additional overhead of doing this on a per
> PF/VF bandwidth basis.
My reading of the documentation is also that its a minimum percentage of
bandwidth. But after playing with the registers a bit it seems
that it is actually a ceiling expressed. And the percentage is of 1Gigabit,
regardless of the actual link speed.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: igb bandwidth allocation configuration
2009-09-15 22:29 ` Simon Horman
@ 2009-09-16 6:47 ` Or Gerlitz
2009-09-16 7:04 ` Simon Horman
2009-09-17 1:09 ` Simon Horman
1 sibling, 1 reply; 17+ messages in thread
From: Or Gerlitz @ 2009-09-16 6:47 UTC (permalink / raw)
To: Simon Horman
Cc: Alexander Duyck, e1000-devel@lists.sourceforge.net, netdev,
Alexander Duyck, Kirsher, Jeffrey T
Simon Horman wrote:
> My reading of the documentation is also that its a minimum percentage of bandwidth. But after playing with the registers a bit it seems that it is actually a ceiling expressed. And the percentage is of 1Gigabit, regardless of the actual link speed.
Hi,
Does the documentation you refer to is the 82576 one? if yes, can you
send a pointer to the doc/section you refer to? also is there 82599
(Niantic) documentation which is publicly avail and I can look at?
specifically, I would love taking a look on the equivalent of the "Intel
82576 SR-IOV Driver Companion Guide"
Or.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: igb bandwidth allocation configuration
2009-09-16 6:47 ` Or Gerlitz
@ 2009-09-16 7:04 ` Simon Horman
2009-09-16 16:10 ` Nelson, Shannon
0 siblings, 1 reply; 17+ messages in thread
From: Simon Horman @ 2009-09-16 7:04 UTC (permalink / raw)
To: Or Gerlitz
Cc: Alexander Duyck, e1000-devel@lists.sourceforge.net, netdev,
Alexander Duyck, Kirsher, Jeffrey T
On Wed, Sep 16, 2009 at 09:47:28AM +0300, Or Gerlitz wrote:
> Simon Horman wrote:
> >My reading of the documentation is also that its a minimum percentage of bandwidth. But after playing with the registers a bit it seems that it is actually a ceiling expressed. And the percentage is of 1Gigabit, regardless of the actual link speed.
> Hi,
>
> Does the documentation you refer to is the 82576 one? if yes, can
> you send a pointer to the doc/section you refer to?
Yes, its the 82576. I'm looking at the 82576 datasheet. In particular
section 4.5.11.1.5.1 "Configuring Tx Bandwidth to VMs", although there are
also some other sections covering the topic in that document.
> also is there
> 82599 (Niantic) documentation which is publicly avail and I can look
> at? specifically, I would love taking a look on the equivalent of
> the "Intel 82576 SR-IOV Driver Companion Guide"
Sorry, I don't know anything about the 82599. But I am only working
with publicly available documentation.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: igb bandwidth allocation configuration
2009-09-15 18:01 ` Alexander Duyck
2009-09-15 18:25 ` Nelson, Shannon
2009-09-15 22:29 ` Simon Horman
@ 2009-09-16 14:10 ` Or Gerlitz
2009-09-16 15:53 ` Alexander Duyck
2 siblings, 1 reply; 17+ messages in thread
From: Or Gerlitz @ 2009-09-16 14:10 UTC (permalink / raw)
To: Alexander Duyck
Cc: Simon Horman, e1000-devel@lists.sourceforge.net,
netdev@vger.kernel.org, Alexander Duyck, Kirsher, Jeffrey T
Alexander Duyck wrote:
> The interface for all of this would make sense as part of a virtual
> ethernet switch control which is the way I am currently leaning on all
> this.
Yes, you can say that out of the per VF <mac, vlan-id, priority, rate>
tuple I mentioned, except for the mac, the other parameters actually
belong to the egress flow of the virtual switch port this VF is
connected to. So the vswitch actually signs the packet with vlan+pbits
and enforces the rate. Now vswitch can be software based, or hardware
NIC based.
Now, I assume there may be NICs which will let you configure the
<vlan-id, priority, rate> as part of the their virtual switch config,
but others, e.g
the 82576 as an example, and following our discussion, will let you do
that for the VF, in the VF driver which as you said may run the guest OS
where we can't control it...
Or.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: igb bandwidth allocation configuration
2009-09-16 14:10 ` Or Gerlitz
@ 2009-09-16 15:53 ` Alexander Duyck
0 siblings, 0 replies; 17+ messages in thread
From: Alexander Duyck @ 2009-09-16 15:53 UTC (permalink / raw)
To: Or Gerlitz
Cc: Simon Horman, e1000-devel@lists.sourceforge.net,
netdev@vger.kernel.org, Alexander Duyck, Kirsher, Jeffrey T
Or Gerlitz wrote:
> Alexander Duyck wrote:
>> The interface for all of this would make sense as part of a virtual
>> ethernet switch control which is the way I am currently leaning on all
>> this.
> Yes, you can say that out of the per VF <mac, vlan-id, priority, rate>
> tuple I mentioned, except for the mac, the other parameters actually
> belong to the egress flow of the virtual switch port this VF is
> connected to. So the vswitch actually signs the packet with vlan+pbits
> and enforces the rate. Now vswitch can be software based, or hardware
> NIC based.
Even something such as MAC address would make sense for a virtual
ethernet switch configuration in that you could restrict unicast ingress
traffic for the VF to a specific address much like you would do on a
regular L2 switch.
> Now, I assume there may be NICs which will let you configure the
> <vlan-id, priority, rate> as part of the their virtual switch config,
> but others, e.g
> the 82576 as an example, and following our discussion, will let you do
> that for the VF, in the VF driver which as you said may run the guest OS
> where we can't control it...
I think you may be a bit confused. The configuration for the VFs would
be part of the PF via the virtual ethernet switch control. As a result
it is only the PF which needs to be running on the host.
Thanks,
Alex
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: igb bandwidth allocation configuration
2009-09-16 7:04 ` Simon Horman
@ 2009-09-16 16:10 ` Nelson, Shannon
0 siblings, 0 replies; 17+ messages in thread
From: Nelson, Shannon @ 2009-09-16 16:10 UTC (permalink / raw)
To: Simon Horman, Or Gerlitz
Cc: e1000-devel@lists.sourceforge.net, netdev, Kirsher, Jeffrey T,
Alexander Duyck
Simon Horman wrote:
>On Wed, Sep 16, 2009 at 09:47:28AM +0300, Or Gerlitz wrote:
>> also is there
>> 82599 (Niantic) documentation which is publicly avail and I can look
>> at? specifically, I would love taking a look on the equivalent of
>> the "Intel 82576 SR-IOV Driver Companion Guide"
>
>Sorry, I don't know anything about the 82599. But I am only working
>with publicly available documentation.
The "82599 Developer Manual" is available at http://sourceforge.net/projects/e1000/files/.
sln
------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: igb bandwidth allocation configuration
2009-09-15 22:29 ` Simon Horman
2009-09-16 6:47 ` Or Gerlitz
@ 2009-09-17 1:09 ` Simon Horman
1 sibling, 0 replies; 17+ messages in thread
From: Simon Horman @ 2009-09-17 1:09 UTC (permalink / raw)
To: Alexander Duyck
Cc: Or Gerlitz, e1000-devel@lists.sourceforge.net, Kirsher, Jeffrey T,
Alexander Duyck, netdev@vger.kernel.org
On Wed, Sep 16, 2009 at 08:29:26AM +1000, Simon Horman wrote:
> On Tue, Sep 15, 2009 at 11:01:52AM -0700, Alexander Duyck wrote:
> > Or Gerlitz wrote:
> > >If the rate limiter is exposed as a feature of the VF, it doesn't
> > >matter who really enforces it, the "VF portion" of the HW or the
> > >PF itself. I agree that if you have to program the PF for the rate
> > >of a specific VF, then its more complex. Basically, I would expect
> > >that a VF can be configured with <mac, vlad-id, priority, rate>
> > >such that it can be done where the VF NIC is spawned, host kernel
> > >or guest kernel.
> >
> > Adding the rate limiter as a feature of the VF doesn't make much
> > sense since the VF could be direct assigned to another OS for all we
> > know so we won't have control over it from there.
> >
> > The interface for all of this would make sense as part of a virtual
> > ethernet switch control which is the way I am currently leaning on
> > all this. As such it is probably another thing we can bring up at
> > the BOF session at the Linux Plumbers Conference.
>
> Unfortunately I won't be able to make it to the BOF or Plumbers.
> I look forward to hearing what is discussed.
Is there any chance of being able to participate in this remotely,
even just a listen-only feed of some sort would be quite valuable to me
and possibly others.
------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2009-09-17 1:09 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-09-10 8:18 igb bandwidth allocation configuration Simon Horman
2009-09-10 11:28 ` Patrick McHardy
2009-09-10 11:55 ` Patrick McHardy
2009-09-11 0:38 ` Simon Horman
2009-09-15 11:32 ` Simon Horman
2009-09-14 8:42 ` Or Gerlitz
2009-09-15 11:36 ` Simon Horman
2009-09-15 13:27 ` Or Gerlitz
2009-09-15 18:01 ` Alexander Duyck
2009-09-15 18:25 ` Nelson, Shannon
2009-09-15 22:29 ` Simon Horman
2009-09-16 6:47 ` Or Gerlitz
2009-09-16 7:04 ` Simon Horman
2009-09-16 16:10 ` Nelson, Shannon
2009-09-17 1:09 ` Simon Horman
2009-09-16 14:10 ` Or Gerlitz
2009-09-16 15:53 ` Alexander Duyck
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).