* Re: [PATCH 11/17] mlx4_en: Enabling new steering
From: David Miller @ 2011-03-08 22:26 UTC (permalink / raw)
To: roland; +Cc: or.gerlitz, yevgenyp, netdev, vlad, nirmu, alekseys
In-Reply-To: <AANLkTim8XDT5YKJc4wroyEDJc1X+xZOFL9y_090F5eLr@mail.gmail.com>
From: Roland Dreier <roland@kernel.org>
Date: Tue, 8 Mar 2011 13:30:25 -0800
> On Tue, Mar 8, 2011 at 1:10 PM, Or Gerlitz <or.gerlitz@gmail.com> wrote:
>>> + if (mdev->dev->caps.vep_mc_steering && ibqp->qp_type == IB_QPT_RAW_ETHERTYPE) {
>>
>> Guys,
>>
>> Didn't you want to refer here to the "raw ethernet QP" which is typed
>> to IB_QPT_RAW_PACKET by the series posted on December last year, see
>> http://marc.info/?l=linux-rdma&m=129162076818154 --
>> IB_QPT_RAW_ETHERTYPE is a rename of IB's IB_QPT_RAW_ETY done by
>> commit a2ebf07ae53e65bd073f96877e4818f2e89271ae
>
> Good catch. I think the drivers/infiniband part of this patch should
> just be dropped -- we don't have any raw QP support of any kind, right?
Once this is all sorted out, please someone respin this entire patch
set.
Thanks.
^ permalink raw reply
* Re: [PATCH] vxge: update MAINTAINERS
From: David Miller @ 2011-03-08 22:28 UTC (permalink / raw)
To: jdmason; +Cc: netdev
In-Reply-To: <1299517321-4567-1-git-send-email-jdmason@kudzu.us>
From: Jon Mason <jdmason@kudzu.us>
Date: Mon, 7 Mar 2011 11:02:01 -0600
> Exar has exited the 10G Ethernet NIC market, orphaning both the s2io and
> vxge drivers. With the promise of free hardware, I'll take over
> maintainership.
>
> Signed-off-by: Jon Mason <jdmason@kudzu.us>
Applied, thanks.
^ permalink raw reply
* [PATCH] inetpeer: Don't disable BH for initial fast RCU lookup.
From: David Miller @ 2011-03-08 22:59 UTC (permalink / raw)
To: netdev; +Cc: eric.dumazet
If modifications on other cpus are ok, then modifications to
the tree during lookup done by the local cpu are ok too.
Signed-off-by: David S. Miller <davem@davemloft.net>
---
net/ipv4/inetpeer.c | 18 +++++++++---------
1 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/net/ipv4/inetpeer.c b/net/ipv4/inetpeer.c
index f604ffd..6442c35 100644
--- a/net/ipv4/inetpeer.c
+++ b/net/ipv4/inetpeer.c
@@ -206,16 +206,16 @@ static int addr_compare(const struct inetpeer_addr *a,
})
/*
- * Called with rcu_read_lock_bh()
+ * Called with rcu_read_lock()
* Because we hold no lock against a writer, its quite possible we fall
* in an endless loop.
* But every pointer we follow is guaranteed to be valid thanks to RCU.
* We exit from this function if number of links exceeds PEER_MAXDEPTH
*/
-static struct inet_peer *lookup_rcu_bh(const struct inetpeer_addr *daddr,
- struct inet_peer_base *base)
+static struct inet_peer *lookup_rcu(const struct inetpeer_addr *daddr,
+ struct inet_peer_base *base)
{
- struct inet_peer *u = rcu_dereference_bh(base->root);
+ struct inet_peer *u = rcu_dereference(base->root);
int count = 0;
while (u != peer_avl_empty) {
@@ -231,9 +231,9 @@ static struct inet_peer *lookup_rcu_bh(const struct inetpeer_addr *daddr,
return u;
}
if (cmp == -1)
- u = rcu_dereference_bh(u->avl_left);
+ u = rcu_dereference(u->avl_left);
else
- u = rcu_dereference_bh(u->avl_right);
+ u = rcu_dereference(u->avl_right);
if (unlikely(++count == PEER_MAXDEPTH))
break;
}
@@ -470,11 +470,11 @@ struct inet_peer *inet_getpeer(struct inetpeer_addr *daddr, int create)
/* Look up for the address quickly, lockless.
* Because of a concurrent writer, we might not find an existing entry.
*/
- rcu_read_lock_bh();
+ rcu_read_lock();
sequence = read_seqbegin(&base->lock);
- p = lookup_rcu_bh(daddr, base);
+ p = lookup_rcu(daddr, base);
invalidated = read_seqretry(&base->lock, sequence);
- rcu_read_unlock_bh();
+ rcu_read_unlock();
if (p) {
/* The existing node has been found.
--
1.7.4.1
^ permalink raw reply related
* Re: [PATCH] Make CUBIC Hystart more robust to RTT variations
From: Stephen Hemminger @ 2011-03-08 23:21 UTC (permalink / raw)
To: David Miller; +Cc: rhee, lucas.nussbaum, xiyou.wangcong, netdev
In-Reply-To: <20110308.114346.48506864.davem@davemloft.net>
On Tue, 08 Mar 2011 11:43:46 -0800 (PST)
David Miller <davem@davemloft.net> wrote:
> From: Injong Rhee <rhee@ncsu.edu>
> Date: Tue, 08 Mar 2011 10:26:36 -0500
>
> > Thanks for updating CUBIC hystart. You might want to test the
> > cases with more background traffic and verify whether this
> > threshold is too conservative.
>
> So let's get down to basics.
>
> What does Hystart do specially that allows it to avoid all of the
> problems that TCP VEGAS runs into.
>
> Specifically, that if you use RTTs to make congestion control
> decisions it is impossible to notice new bandwidth becomming available
> fast enough.
>
> Again, it's impossible to react fast enough. No matter what you tweak
> all of your various settings to, this problem will still exist.
>
> This is a core issue, you cannot get around it.
>
> This is why I feel that Hystart is fundamentally flawed and we should
> turn it off by default if not flat-out remove it.
>
> Distributions are turning it off by default already, therefore it's
> stupid for the upstream kernel to behave differently if that's what
> %99 of the world is going to end up experiencing.
The assumption in Hystart that spacing between ACK's is solely due to
congestion is a bad. If you read the paper, this is why FreeBSD's
estimation logic is dismissed. The Hystart problem is different
than the Vegas issue.
Algorithms that look at min RTT are ok, since the lower bound is
fixed; additional queuing and variation in network only increases RTT
it never reduces it. With a min RTT it is possible to compute the
upper bound on available bandwidth. i.e If all packets were as good as
this estimate minRTT then the available bandwidth is X. But then using
an individual RTT sample to estimate unused bandwidth is flawed. To
quote paper.
"Thus, by checking whether ∆(N ) is larger than Dmin , we
can detect whether cwnd has reached the available capacity
of the path"
So what goes wrong:
1. Dmin can be too large because this connection always sees delays
due to other traffic or hardware. i.e buffer bloat. This would cause
the bandwidth estimate to be too low and therefore TCP would leave
slow start too early (and not get up to full bandwidth).
2. Dmin can be smaller than the clock resolution. This would cause
either sample to be ignored, or Dmin to be zero. If Dmin is zero,
the bandwidth estimate would in theory be infinite, which would
lead to TCP not leaving slow start because of Hystart. Instead
TCP would leave slow start at first loss.
Other possible problems:
3. ACK's could be nudged together by variations in delay.
This would cause HyStart to exit slow start prematurely. To false
think it is an ACK train.
Noise in network is not catastrophic, it just
causes TCP to exit slow-start early and have to go into normal
window growth phase. The problem is that the original non-Hystart
behavior of Cubic is unfair; the first flow dominates the link
and other flows are unable to get in. If you run tests with two
flows one will get a larger share of the bandwidth.
I think Hystart is okay in concept but there may be issues
on low RTT links as well as other corner cases that need bug
fixing.
1. Needs to use better resolution than HZ. Since HZ can be 100.
2. Hardcoding 2ms as spacing between ACK's as train is wrong
for local networks.
^ permalink raw reply
* Re: [rds-devel] [PATCH] rds: prevent BUG_ON triggering on congestion map updates
From: Venkat Venkatsubra @ 2011-03-08 23:36 UTC (permalink / raw)
To: nhorman; +Cc: netdev, rds-devel, chris.mason, davem
I was trying to recreate on our test systems. It's not a ppc.
So far not successful in crashing even though rds_ib_xmit hits this path:
/* Do not send cong updates to IB loopback */
if (conn->c_loopback
&& rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) {
rds_cong_map_updated(conn->c_fcong, ~(u64) 0);
return sizeof(struct rds_header) + RDS_CONG_MAP_BYTES;
}
For the BUG_ON condition in rds_send_xmit:
ret=4096 conn->c_xmit_sg=1 rm->data.op_nents=2
BUG_ON(ret != 0 && conn->c_xmit_sg == rm->data.op.nents);
Venkat
----- Original Message -----
From: nhorman@tuxdriver.com
To: davem@davemloft.net
Cc: chris.mason@oracle.com, netdev@vger.kernel.org, rds-devel@oss.oracle.com, venkat.x.venkatsubra@oracle.com
Sent: Tuesday, March 8, 2011 1:36:50 PM GMT -06:00 US/Canada Central
Subject: Re: [rds-devel] [PATCH] rds: prevent BUG_ON triggering on congestion map updates
On Tue, Mar 08, 2011 at 11:22:58AM -0800, David Miller wrote:
> From: Chris Mason <chris.mason@oracle.com>
> Date: Tue, 08 Mar 2011 13:58:57 -0500
>
> > Excerpts from David Miller's message of 2011-03-08 13:52:23 -0500:
> >> From: Chris Mason <chris.mason@oracle.com>
> >> Date: Mon, 07 Mar 2011 15:41:04 -0500
> >>
> >> > Excerpts from David Miller's message of 2011-03-07 15:27:53 -0500:
> >> >> From: Neil Horman <nhorman@tuxdriver.com>
> >> >> Date: Wed, 2 Mar 2011 11:28:22 -0500
> >> >>
> >> >> > Recently had this bug halt reported to me:
> >> >>
> >> >> Well, does anyone on the RDS team care about this bug fix at all?
> >> >>
> >> >> Stuff like this should not sit for nearly a week without any reply
> >> >> whatsoever.
> >> >>
> >> >
> >> > The patch looks good to me, but I'm surprised we haven't seen it here.
> >> > Venkat please take a look (link to the patch below) Has it only been seen on ppc?
> >> >
> >> > http://permalink.gmane.org/gmane.linux.network/187933
> >>
> >> Can I get an ACK or something, _please_?
> >
> > Sorry, wasn't clear:
> >
> > Acked-by: Chris Mason <chris.mason@oracle.com>
>
> Applied, thanks everyone.
Thanks, Chris, Dave.
Neil
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: linux-next: manual merge of the net tree with the net-current tree
From: Stephen Rothwell @ 2011-03-09 0:11 UTC (permalink / raw)
To: Dmitry Kravkov
Cc: David Miller, netdev@vger.kernel.org, linux-next@vger.kernel.org,
linux-kernel@vger.kernel.org, Eilon Greenstein,
Vladislav Zolotarov
In-Reply-To: <1299577473.17233.41.camel@lb-tlvb-dmitry>
[-- Attachment #1: Type: text/plain, Size: 1171 bytes --]
Hi Dmitry,
On Tue, 8 Mar 2011 11:44:33 +0200 "Dmitry Kravkov" <dmitry@broadcom.com> wrote:
>
> You are correct it's just an addition and it does not really matter
> where to do this. But the merge pushed new code in the middle
> of MAC configuration between MAC and multicast list.
> It's clearer to put it at the end of MAC/ML/UL block:
> ---
> drivers/net/bnx2x/bnx2x_cmn.c | 5 +++++
> 1 files changed, 5 insertions(+), 0 deletions(-)
>
> diff --git a/drivers/net/bnx2x/bnx2x_cmn.c b/drivers/net/bnx2x/bnx2x_cmn.c
> index b01b622..9e37675 100644
> --- a/drivers/net/bnx2x/bnx2x_cmn.c
> +++ b/drivers/net/bnx2x/bnx2x_cmn.c
> @@ -1498,6 +1498,11 @@ int bnx2x_nic_load(struct bnx2x *bp, int load_mode)
> /* Clear UC lists configuration */
> bnx2x_invalidate_uc_list(bp);
>
> + if (bp->pending_max) {
> + bnx2x_update_max_mf_config(bp, bp->pending_max);
> + bp->pending_max = 0;
> + }
> +
> if (bp->port.pmf)
> bnx2x_initial_phy_init(bp, load_mode);
OK, thanks. I have changed the conflict resolution to do that.
--
Cheers,
Stephen Rothwell sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/
[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]
^ permalink raw reply
* Re: [PATCH] ipv4: Cache source address in nexthop entries.
From: Julian Anastasov @ 2011-03-09 0:49 UTC (permalink / raw)
To: David Miller; +Cc: netdev
In-Reply-To: <20110308.103801.193702247.davem@davemloft.net>
Hello,
On Tue, 8 Mar 2011, David Miller wrote:
>> As the addresses have nothing to do with the link
>> state, I don't think it is correct to call fib_update_nh_saddrs
>> for DEV events.
>
> fib_update_nh_saddrs() gets called for "DEV" events since those
> are what are emitted when addresses are addded and removed.
But we still need to propagate the address events
to nexthops on all devices. Even if it is slow, I see it in
this way (not tested):
static inline void update_nh_saddr(const struct in_ifaddr *ifa,
struct net_device *dev,
struct fib_nh *nexthop_nh,
unsigned long event)
{
if (event == NETDEV_UP) {
if (ifa->ifa_local == nexthop_nh->nh_saddr)
return;
} else {
if (ifa->ifa_local != nexthop_nh->nh_saddr)
return;
}
if (ifa->ifa_scope > nexthop_nh->nh_cfg_scope)
return;
if (dev != nexthop_nh->nh_dev && ifa->ifa_scope >= RT_SCOPE_LINK)
return;
if (nexthop_nh->nh_flags & RTNH_F_DEAD)
return;
nexthop_nh->nh_saddr = inet_select_addr(nexthop_nh->nh_dev,
nexthop_nh->nh_gw,
nexthop_nh->nh_cfg_scope);
}
void fib_update_nh_saddrs(const struct in_ifaddr *ifa, unsigned long event)
{
struct net_device *dev = ifa->ifa_dev->dev;
int i;
for (i = 0; i < fib_hash_size; i++) {
struct hlist_head *head = &fib_info_hash[i];
struct hlist_node *node;
struct fib_info *fi;
hlist_for_each_entry(fi, node, head, fib_hash) {
if (fi->fib_prefsrc)
continue;
change_nexthops(fi) {
if (!nexthop_nh->nh_dev)
continue;
update_nh_saddr(ifa, dev, nexthop_nh, event);
} endfor_nexthops(fi);
}
}
}
Then may be fib_sync_up() should also refresh nh_saddr,
so that all alive nexthops get actual source address which
was not possible while device was down.
Regards
--
Julian Anastasov <ja@ssi.bg>
^ permalink raw reply
* Re: [PATCH] Make CUBIC Hystart more robust to RTT variations
From: Injong Rhee @ 2011-03-09 1:30 UTC (permalink / raw)
To: Stephen Hemminger
Cc: David Miller, lucas.nussbaum, xiyou.wangcong, netdev, sangtae.ha
In-Reply-To: <20110308152103.714f5f05@nehalam>
HyStart is a slow start algorithm, but not a congestion control
algorithm. So the difference between vegas and hystart is obvious. Yes.
Both hystart and vegas use delays for indication of congestion. But
hystart exits slow starts at the detection of congestion and enters
normal congestion avoidance; in some sense, it is much safer than vegas
as it does not change the regular behaviors of congestion control.
I think the main problem arising right now is not because it is using
noisy delays as congestion indication, but because of rather some
implementation issues like use of Hz, hardcoding 2ms, etc.
Then, you might ask why hystart can use delays while vegas can't. The
main motivation for use delays during slow start is that slow start
creates an environment where delay samples can be more trusted. That is
because it sends so many packets as a a burst because of doubling
windows, which can be used as packet train to estimate the available
capacity more reliably.
(tool 1) When many packets are sent in burst, the spacing in returning
ACKs can be a good indicator. Hystart also uses delays as an estimation.
(tool 2) If estimated avg delays increase beyond a certain threshold, it
sees that as a possible congestion.
Now, both tools can be wrong. But that is not catastrophic since
congestion avoidance can kick in to save the day. In a pipe where no
other flows are competing, then exiting slow start too early can slow
things down as the window can be still too small. But that is in fact
when delays are most reliable. So those tests that say bad performance
with hystart are in fact, where hystart is supposed to perform well.
Then why do we have a bad performance? I think the answer is again the
implementation flaws -- use different hardware, some hardwired codes,
etc, and also could be related to a few corner cases like very low RTT
links.
Let us examine Stephen's analysis in more detail.
1. Use of minRTT is ok. I agree.
2. Dmin can be too large at the beginning. But it is just like minRTT.
This cannot be too large. If you trust minRTT, then delay estimation
should say that there is a congestion. This is exactly the opposite case
to the cases we are seeing. If Dmin is too large, then hystart would not
exit the slow start as it does not detect the congestion. That is not
what we are seeing right now.
3. Dmin can be smaller than clock resolution. That is why we are using a
bunch of ACKs to get better accuracy. With a bunch of ACKs, we get
higher value of spacing so that we can take average.
4. If ACKs are nudged together, then hystart does not quit slow start.
Instead, it sees that there is no congestion. It is when it sees big
spacing between ACKs -- that is when it detects congestion.
On 3/8/11 6:21 PM, Stephen Hemminger wrote:
> On Tue, 08 Mar 2011 11:43:46 -0800 (PST)
> David Miller<davem@davemloft.net> wrote:
>
>> From: Injong Rhee<rhee@ncsu.edu>
>> Date: Tue, 08 Mar 2011 10:26:36 -0500
>>
>>> Thanks for updating CUBIC hystart. You might want to test the
>>> cases with more background traffic and verify whether this
>>> threshold is too conservative.
>> So let's get down to basics.
>>
>> What does Hystart do specially that allows it to avoid all of the
>> problems that TCP VEGAS runs into.
>>
>> Specifically, that if you use RTTs to make congestion control
>> decisions it is impossible to notice new bandwidth becomming available
>> fast enough.
>>
>> Again, it's impossible to react fast enough. No matter what you tweak
>> all of your various settings to, this problem will still exist.
>>
>> This is a core issue, you cannot get around it.
>>
>> This is why I feel that Hystart is fundamentally flawed and we should
>> turn it off by default if not flat-out remove it.
>>
>> Distributions are turning it off by default already, therefore it's
>> stupid for the upstream kernel to behave differently if that's what
>> %99 of the world is going to end up experiencing.
> The assumption in Hystart that spacing between ACK's is solely due to
> congestion is a bad. If you read the paper, this is why FreeBSD's
> estimation logic is dismissed. The Hystart problem is different
> than the Vegas issue.
>
> Algorithms that look at min RTT are ok, since the lower bound is
> fixed; additional queuing and variation in network only increases RTT
> it never reduces it. With a min RTT it is possible to compute the
> upper bound on available bandwidth. i.e If all packets were as good as
> this estimate minRTT then the available bandwidth is X. But then using
> an individual RTT sample to estimate unused bandwidth is flawed. To
> quote paper.
>
> "Thus, by checking whether ∆(N ) is larger than Dmin , we
> can detect whether cwnd has reached the available capacity
> of the path"
>
> So what goes wrong:
> 1. Dmin can be too large because this connection always sees delays
> due to other traffic or hardware. i.e buffer bloat. This would cause
> the bandwidth estimate to be too low and therefore TCP would leave
> slow start too early (and not get up to full bandwidth).
>
> 2. Dmin can be smaller than the clock resolution. This would cause
> either sample to be ignored, or Dmin to be zero. If Dmin is zero,
> the bandwidth estimate would in theory be infinite, which would
> lead to TCP not leaving slow start because of Hystart. Instead
> TCP would leave slow start at first loss.
>
> Other possible problems:
> 3. ACK's could be nudged together by variations in delay.
> This would cause HyStart to exit slow start prematurely. To false
> think it is an ACK train.
>
> Noise in network is not catastrophic, it just
> causes TCP to exit slow-start early and have to go into normal
> window growth phase. The problem is that the original non-Hystart
> behavior of Cubic is unfair; the first flow dominates the link
> and other flows are unable to get in. If you run tests with two
> flows one will get a larger share of the bandwidth.
>
> I think Hystart is okay in concept but there may be issues
> on low RTT links as well as other corner cases that need bug
> fixing.
>
> 1. Needs to use better resolution than HZ. Since HZ can be 100.
> 2. Hardcoding 2ms as spacing between ACK's as train is wrong
> for local networks.
>
>
>
>
>
^ permalink raw reply
* Re: [PATCH] Make CUBIC Hystart more robust to RTT variations
From: Sangtae Ha @ 2011-03-09 1:33 UTC (permalink / raw)
To: Stephen Hemminger
Cc: David Miller, rhee, lucas.nussbaum, xiyou.wangcong, netdev
In-Reply-To: <20110308152103.714f5f05@nehalam>
Hi Stephen,
Thank you for your feedback. Please see my answers below.
On Tue, Mar 8, 2011 at 6:21 PM, Stephen Hemminger <shemminger@vyatta.com> wrote:
> On Tue, 08 Mar 2011 11:43:46 -0800 (PST)
> David Miller <davem@davemloft.net> wrote:
>
>> From: Injong Rhee <rhee@ncsu.edu>
>> Date: Tue, 08 Mar 2011 10:26:36 -0500
>>
>> > Thanks for updating CUBIC hystart. You might want to test the
>> > cases with more background traffic and verify whether this
>> > threshold is too conservative.
>>
>> So let's get down to basics.
>>
>> What does Hystart do specially that allows it to avoid all of the
>> problems that TCP VEGAS runs into.
>>
>> Specifically, that if you use RTTs to make congestion control
>> decisions it is impossible to notice new bandwidth becomming available
>> fast enough.
>>
>> Again, it's impossible to react fast enough. No matter what you tweak
>> all of your various settings to, this problem will still exist.
>>
>> This is a core issue, you cannot get around it.
>>
>> This is why I feel that Hystart is fundamentally flawed and we should
>> turn it off by default if not flat-out remove it.
>>
>> Distributions are turning it off by default already, therefore it's
>> stupid for the upstream kernel to behave differently if that's what
>> %99 of the world is going to end up experiencing.
>
> The assumption in Hystart that spacing between ACK's is solely due to
> congestion is a bad. If you read the paper, this is why FreeBSD's
> estimation logic is dismissed. The Hystart problem is different
> than the Vegas issue.
>
> Algorithms that look at min RTT are ok, since the lower bound is
> fixed; additional queuing and variation in network only increases RTT
> it never reduces it. With a min RTT it is possible to compute the
> upper bound on available bandwidth. i.e If all packets were as good as
> this estimate minRTT then the available bandwidth is X. But then using
> an individual RTT sample to estimate unused bandwidth is flawed. To
> quote paper.
>
> "Thus, by checking whether ∆(N ) is larger than Dmin , we
> can detect whether cwnd has reached the available capacity
> of the path"
>
> So what goes wrong:
> 1. Dmin can be too large because this connection always sees delays
> due to other traffic or hardware. i.e buffer bloat. This would cause
> the bandwidth estimate to be too low and therefore TCP would leave
> slow start too early (and not get up to full bandwidth).
This is true. But the idea is that running the congestion avoidance
algorithm of CUBIC for this case is better than hurting other flows
with abrupt perturbation, since the growth of CUBIC is quite
responsive and grab the bandwidth quickly in normal network
conditions.
>
> 2. Dmin can be smaller than the clock resolution. This would cause
> either sample to be ignored, or Dmin to be zero. If Dmin is zero,
> the bandwidth estimate would in theory be infinite, which would
> lead to TCP not leaving slow start because of Hystart. Instead
> TCP would leave slow start at first loss.
True. But since HyStart didn't clamp the threshold, ca->delay_min>>4,
it can prematurely leave slow start for very small Dmin. I think this
needs to be fixed, along with the hard-coded 2ms you mentioned below.
>
> Other possible problems:
> 3. ACK's could be nudged together by variations in delay.
> This would cause HyStart to exit slow start prematurely. To false
> think it is an ACK train.
This doesn't happen when the delay is not too small (in typical WAN
including DSL), but it is possible with very small delays since the
code checking the valid ACK train uses the 2ms fixed value, which is
large for LAN.
>
> Noise in network is not catastrophic, it just
> causes TCP to exit slow-start early and have to go into normal
> window growth phase. The problem is that the original non-Hystart
> behavior of Cubic is unfair; the first flow dominates the link
> and other flows are unable to get in. If you run tests with two
> flows one will get a larger share of the bandwidth.
>
> I think Hystart is okay in concept but there may be issues
> on low RTT links as well as other corner cases that need bug
> fixing.
We do not use the delay as indication of congestion, but we use it for
improving the stability and overall performance.
Preventing burst losses quite help for mid to large BDP paths and the
performance results with non-TCP-SACK receivers are also encouraging.
I will work on the fix for the issues below.
>
> 1. Needs to use better resolution than HZ. Since HZ can be 100.
> 2. Hardcoding 2ms as spacing between ACK's as train is wrong
> for local networks.
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* [net-next-2.6 PATCH] enic: Support newer version of firmware devcmd CMD_MCPU_FW_INFO
From: Vasanthy Kolluri @ 2011-03-09 1:35 UTC (permalink / raw)
To: davem; +Cc: netdev
From: Vasanthy Kolluri <vkolluri@cisco.com>
This patch provides support to the newer version of firmware devcmd CMD_MCPU_FW_INFO
that returns additional information (ASIC type and revision) about the underlying hardware.
This knowledge is required by the driver to implement any hardware specific features.
Signed-off-by: Vasanthy Kolluri <vkolluri@cisco.com>
Signed-off-by: Roopa Prabhu <roprabhu@cisco.com>
Signed-off-by: David Wang <dwang2@cisco.com>
---
drivers/net/enic/enic.h | 2 +-
drivers/net/enic/vnic_dev.c | 7 +++++++
drivers/net/enic/vnic_devcmd.h | 38 ++++++++++++++++++++++++++++++++++++--
3 files changed, 44 insertions(+), 3 deletions(-)
diff --git a/drivers/net/enic/enic.h b/drivers/net/enic/enic.h
index e816bbb..3a3c3c8 100644
--- a/drivers/net/enic/enic.h
+++ b/drivers/net/enic/enic.h
@@ -32,7 +32,7 @@
#define DRV_NAME "enic"
#define DRV_DESCRIPTION "Cisco VIC Ethernet NIC Driver"
-#define DRV_VERSION "2.1.1.10"
+#define DRV_VERSION "2.1.1.12"
#define DRV_COPYRIGHT "Copyright 2008-2011 Cisco Systems, Inc"
#define ENIC_BARS_MAX 6
diff --git a/drivers/net/enic/vnic_dev.c b/drivers/net/enic/vnic_dev.c
index c489e72..c089b36 100644
--- a/drivers/net/enic/vnic_dev.c
+++ b/drivers/net/enic/vnic_dev.c
@@ -408,10 +408,17 @@ int vnic_dev_fw_info(struct vnic_dev *vdev,
if (!vdev->fw_info)
return -ENOMEM;
+ memset(vdev->fw_info, 0, sizeof(struct vnic_devcmd_fw_info));
+
a0 = vdev->fw_info_pa;
+ a1 = sizeof(struct vnic_devcmd_fw_info);
/* only get fw_info once and cache it */
err = vnic_dev_cmd(vdev, CMD_MCPU_FW_INFO, &a0, &a1, wait);
+ if (err == ERR_ECMDUNKNOWN) {
+ err = vnic_dev_cmd(vdev, CMD_MCPU_FW_INFO_OLD,
+ &a0, &a1, wait);
+ }
}
*fw_info = vdev->fw_info;
diff --git a/drivers/net/enic/vnic_devcmd.h b/drivers/net/enic/vnic_devcmd.h
index 9abb3d5..d833a07 100644
--- a/drivers/net/enic/vnic_devcmd.h
+++ b/drivers/net/enic/vnic_devcmd.h
@@ -80,8 +80,34 @@
enum vnic_devcmd_cmd {
CMD_NONE = _CMDC(_CMD_DIR_NONE, _CMD_VTYPE_NONE, 0),
- /* mcpu fw info in mem: (u64)a0=paddr to struct vnic_devcmd_fw_info */
- CMD_MCPU_FW_INFO = _CMDC(_CMD_DIR_WRITE, _CMD_VTYPE_ALL, 1),
+ /*
+ * mcpu fw info in mem:
+ * in:
+ * (u64)a0=paddr to struct vnic_devcmd_fw_info
+ * action:
+ * Fills in struct vnic_devcmd_fw_info (128 bytes)
+ * note:
+ * An old definition of CMD_MCPU_FW_INFO
+ */
+ CMD_MCPU_FW_INFO_OLD = _CMDC(_CMD_DIR_WRITE, _CMD_VTYPE_ALL, 1),
+
+ /*
+ * mcpu fw info in mem:
+ * in:
+ * (u64)a0=paddr to struct vnic_devcmd_fw_info
+ * (u16)a1=size of the structure
+ * out:
+ * (u16)a1=0 for in:a1 = 0,
+ * data size actually written for other values.
+ * action:
+ * Fills in first 128 bytes of vnic_devcmd_fw_info for in:a1 = 0,
+ * first in:a1 bytes for 0 < in:a1 <= 132,
+ * 132 bytes for other values of in:a1.
+ * note:
+ * CMD_MCPU_FW_INFO and CMD_MCPU_FW_INFO_OLD have the same enum 1
+ * for source compatibility.
+ */
+ CMD_MCPU_FW_INFO = _CMDC(_CMD_DIR_RW, _CMD_VTYPE_ALL, 1),
/* dev-specific block member:
* in: (u16)a0=offset,(u8)a1=size
@@ -291,11 +317,19 @@ enum vnic_devcmd_error {
ERR_EMAXRES = 10,
};
+/*
+ * note: hw_version and asic_rev refer to the same thing,
+ * but have different formats. hw_version is
+ * a 32-byte string (e.g. "A2") and asic_rev is
+ * a 16-bit integer (e.g. 0xA2).
+ */
struct vnic_devcmd_fw_info {
char fw_version[32];
char fw_build[32];
char hw_version[32];
char hw_serial_number[32];
+ u16 asic_type;
+ u16 asic_rev;
};
struct vnic_devcmd_notify {
^ permalink raw reply related
* Re: Network performance with small packets
From: Andrew Theurer @ 2011-03-09 2:21 UTC (permalink / raw)
To: Shirley Ma
Cc: Rusty Russell, Michael S. Tsirkin, Krishna Kumar2, David Miller,
kvm, netdev, steved, Tom Lendacky
In-Reply-To: <1299621444.25664.77.camel@localhost.localdomain>
On Tue, 2011-03-08 at 13:57 -0800, Shirley Ma wrote:
> On Wed, 2011-02-09 at 11:07 +1030, Rusty Russell wrote:
> > I've finally read this thread... I think we need to get more serious
> > with our stats gathering to diagnose these kind of performance issues.
> >
> > This is a start; it should tell us what is actually happening to the
> > virtio ring(s) without significant performance impact...
>
> Should we also add similar stat on vhost vq as well for monitoring
> vhost_signal & vhost_notify?
Tom L has started using Rusty's patches and found some interesting
results, sent yesterday:
http://marc.info/?l=kvm&m=129953710930124&w=2
-Andrew
>
> Shirley
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* linux-next: build failure after merge of the net tree
From: Stephen Rothwell @ 2011-03-09 2:42 UTC (permalink / raw)
To: David Miller, netdev
Cc: linux-next, linux-kernel, Emil Tantilov, Stephen Ko, Jeff Kirsher
[-- Attachment #1: Type: text/plain, Size: 602 bytes --]
Hi all,
After merging the net tree, today's linux-next build (x86_64 allmodconfig)
failed like this:
drivers/net/ixgbe/ixgbe_mbx.c: In function 'ixgbe_init_mbx_params_pf':
drivers/net/ixgbe/ixgbe_mbx.c:456: error: 'struct ixgbe_mbx_info' has no member named 'udelay'
Caused by commit d7c8a29fc8bd20ba45ec2f52577ed04a988a9500 ("ixgbe:
improve logic in ixgbe_init_mbx_params_pf").
I am wondering if that was ever built ...
I have used the net tree from next-20110308 for today.
--
Cheers,
Stephen Rothwell sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/
[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]
^ permalink raw reply
* Re: bonding can't change to another slave if you ifdown the active slave
From: Weiping Pan @ 2011-03-09 2:40 UTC (permalink / raw)
To: WANG Cong; +Cc: netdev
In-Reply-To: <il58p9$3ka$2@dough.gmane.org>
On 03/08/2011 08:51 PM, WANG Cong wrote:
> On Tue, 08 Mar 2011 14:52:52 +0800, Weiping Pan wrote:
>
>> ok, I use "Host-only mode" and get
>>
>>
>>
>> an conclusion, that if the first enslaved nic is pulled out, bonding
>> can't handle well.
>>
>> First test.
>> I first enslave eth6, then pull it out, bonding doesn't work. on host,
> ...
>>
>> Second test
>> I first enslave eth6, then pull eth7 out, bonding works well. on guest,
> Can you show me your /proc/net/bonding/bond0 before and after pulling down
> eth6 or eth7? And what does `ip link show` say?
>
Ok, let me repeat my test, and gather more information.
on host,
[root@localhost ~]# VBoxManage -v
4.0.2r69518
[root@localhost ~]# VBoxManage showvminfo
67b83c47-0ee2-46bc-b0ff-e0eb43edc1c2|grep ^NIC
NIC 1: MAC: 0800270481A8, Attachment: Host-only Interface
'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82540EM,
Reported speed: 0 Mbps, Boot priority: 0
NIC 2: MAC: 08002778F641, Attachment: Host-only Interface
'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82540EM,
Reported speed: 0 Mbps, Boot priority: 0
NIC 3: MAC: 080027C408BA, Attachment: Host-only Interface
'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82540EM,
Reported speed: 0 Mbps, Boot priority: 0
NIC 4: MAC: 080027DB339A, Attachment: Host-only Interface
'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82540EM,
Reported speed: 0 Mbps, Boot priority: 0
[root@localhost ~]# ifconfig
vboxnet0 Link encap:Ethernet HWaddr 0A:00:27:00:00:00
inet addr:192.168.56.1 Bcast:192.168.56.255 Mask:255.255.255.0
inet6 addr: fe80::800:27ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:3438 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:296716 (289.7 KiB)
vboxnet0:0 Link encap:Ethernet HWaddr 0A:00:27:00:00:00
inet addr:192.168.1.100 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
vboxnet0:1 Link encap:Ethernet HWaddr 0A:00:27:00:00:00
inet addr:192.168.1.101 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
[root@localhost ~]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
192.168.1.0 * 255.255.255.0 U 0 0 0
vboxnet0
192.168.56.0 * 255.255.255.0 U 0 0 0
vboxnet0
10.66.64.0 * 255.255.254.0 U 1 0 0 eth0
default corerouter.nay. 0.0.0.0 UG 0 0 0 eth0
[root@localhost ~]# ip neigh show
10.66.65.254 dev eth0 lladdr 00:1d:45:20:d5:ff REACHABLE
then restart guest,
[root@localhost ~]# ifconfig
eth6 Link encap:Ethernet HWaddr 08:00:27:78:F6:41
inet addr:192.168.56.101 Bcast:192.168.56.255
Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe78:f641/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:14 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6772 (6.6 KiB) TX bytes:1152 (1.1 KiB)
eth7 Link encap:Ethernet HWaddr 08:00:27:04:81:A8
inet addr:192.168.56.102 Bcast:192.168.56.255
Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe04:81a8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:14 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6772 (6.6 KiB) TX bytes:1152 (1.1 KiB)
eth8 Link encap:Ethernet HWaddr 08:00:27:DB:33:9A
inet addr:192.168.56.104 Bcast:192.168.56.255
Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fedb:339a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:14 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6772 (6.6 KiB) TX bytes:1152 (1.1 KiB)
eth9 Link encap:Ethernet HWaddr 08:00:27:C4:08:BA
inet addr:192.168.56.103 Bcast:192.168.56.255
Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fec4:8ba/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:14 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6772 (6.6 KiB) TX bytes:1152 (1.1 KiB)
so according to mac address,
NIC 1: eth7
NIC 2: eth6
NIC 3: eth9
NIC 4: eth8
[root@localhost ~]# ifconfig eth6 down
[root@localhost ~]# ifconfig eth7 down
[root@localhost ~]# ifconfig eth8 down
[root@localhost ~]# ifconfig eth9 down
[root@localhost ~]# modprobe bonding mode=0 miimon=100
[root@localhost ~]# ifconfig bond0 192.168.1.2 netmask 255.255.255.0 up
[root@localhost ~]# ifenslave bond0 eth6
[root@localhost ~]# ifenslave bond0 eth7
[root@localhost ~]# dmesg
[ 1436.344751] bonding: Ethernet Channel Bonding Driver: v3.6.0
(September 26, 2009)
[ 1436.344756] bonding: MII link monitoring set to 100 ms
[ 1480.485933] ADDRCONF(NETDEV_UP): bond0: link is not ready
[ 1490.087608] e1000: eth6 NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: RX
[ 1490.091795] bonding: bond0: enslaving eth6 as an active interface
with an up link.
[ 1490.092326] ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
[ 1492.676643] e1000: eth7 NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: RX
[ 1492.684839] bonding: bond0: enslaving eth7 as an active interface
with an up link.
[root@localhost ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth6
MII Status: up
Link Failure Count: 0
Permanent HW addr: 08:00:27:78:f6:41
Slave Interface: eth7
MII Status: up
Link Failure Count: 0
Permanent HW addr: 08:00:27:04:81:a8
[root@localhost ~]# ifconfig
bond0 Link encap:Ethernet HWaddr 08:00:27:78:F6:41
inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe78:f641/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:40 errors:0 dropped:0 overruns:0 frame:0
TX packets:25 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:19632 (19.1 KiB) TX bytes:4070 (3.9 KiB)
eth6 Link encap:Ethernet HWaddr 08:00:27:78:F6:41
inet addr:192.168.56.101 Bcast:192.168.56.255
Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:21 errors:0 dropped:0 overruns:0 frame:0
TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:10158 (9.9 KiB) TX bytes:1802 (1.7 KiB)
eth7 Link encap:Ethernet HWaddr 08:00:27:78:F6:41
inet addr:192.168.56.101 Bcast:192.168.56.255
Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:19 errors:0 dropped:0 overruns:0 frame:0
TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9474 (9.2 KiB) TX bytes:2268 (2.2 KiB)
[root@localhost ~]# ping 192.168.1.100 -c 5
PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data.
64 bytes from 192.168.1.100: icmp_req=1 ttl=64 time=3.69 ms
64 bytes from 192.168.1.100: icmp_req=2 ttl=64 time=0.242 ms
64 bytes from 192.168.1.100: icmp_req=3 ttl=64 time=0.154 ms
64 bytes from 192.168.1.100: icmp_req=4 ttl=64 time=0.270 ms
64 bytes from 192.168.1.100: icmp_req=5 ttl=64 time=0.278 ms
--- 192.168.1.100 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 0.154/0.928/3.696/1.384 ms
[root@localhost ~]# dmesg -c &>/dev/null
on host,
[root@localhost ~]# VBoxManage controlvm
67b83c47-0ee2-46bc-b0ff-e0eb43edc1c2 setlinkstate2 off
on guest,
[root@localhost ~]# dmesg
[ 1731.945272] e1000: eth6 NIC Link is Down
[ 1732.026207] bonding: bond0: link status definitely down for interface
eth6, disabling it
[root@localhost ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth6
MII Status: down
Link Failure Count: 1
Permanent HW addr: 08:00:27:78:f6:41
Slave Interface: eth7
MII Status: up
Link Failure Count: 0
Permanent HW addr: 08:00:27:04:81:a8
[root@localhost ~]# ping 192.168.1.100 -c 5
PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data.
From 192.168.1.2 icmp_seq=2 Destination Host Unreachable
From 192.168.1.2 icmp_seq=3 Destination Host Unreachable
From 192.168.1.2 icmp_seq=4 Destination Host Unreachable
From 192.168.1.2 icmp_seq=5 Destination Host Unreachable
--- 192.168.1.100 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4001ms
pipe 3
[root@localhost ~]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth7: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master bond0 state UP qlen 1000
link/ether 08:00:27:78:f6:41 brd ff:ff:ff:ff:ff:ff
3: eth6: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc
pfifo_fast master bond0 state DOWN qlen 1000
link/ether 08:00:27:78:f6:41 brd ff:ff:ff:ff:ff:ff
4: eth9: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen
1000
link/ether 08:00:27:c4:08:ba brd ff:ff:ff:ff:ff:ff
5: eth8: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen
1000
link/ether 08:00:27:db:33:9a brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP
link/ether 08:00:27:78:f6:41 brd ff:ff:ff:ff:ff:ff
[root@localhost ~]# ip neigh show
192.168.1.100 dev bond0 FAILED
on host,
[root@localhost ~]# ip neigh show
10.66.65.254 dev eth0 lladdr 00:1d:45:20:d5:ff STALE
192.168.1.2 dev vboxnet0 lladdr 08:00:27:78:f6:41 STALE
on guest, ping while tcpdump
[root@localhost ~]# ping 192.168.1.100
PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data.
From 192.168.1.2 icmp_seq=11 Destination Host Unreachable
From 192.168.1.2 icmp_seq=12 Destination Host Unreachable
From 192.168.1.2 icmp_seq=13 Destination Host Unreachable
From 192.168.1.2 icmp_seq=15 Destination Host Unreachable
From 192.168.1.2 icmp_seq=16 Destination Host Unreachable
From 192.168.1.2 icmp_seq=17 Destination Host Unreachable
From 192.168.1.2 icmp_seq=19 Destination Host Unreachable
From 192.168.1.2 icmp_seq=20 Destination Host Unreachable
From 192.168.1.2 icmp_seq=21 Destination Host Unreachable
From 192.168.1.2 icmp_seq=23 Destination Host Unreachable
From 192.168.1.2 icmp_seq=24 Destination Host Unreachable
From 192.168.1.2 icmp_seq=25 Destination Host Unreachable
From 192.168.1.2 icmp_seq=27 Destination Host Unreachable
From 192.168.1.2 icmp_seq=28 Destination Host Unreachable
From 192.168.1.2 icmp_seq=29 Destination Host Unreachable
^C
--- 192.168.1.100 ping statistics ---
30 packets transmitted, 0 received, +15 errors, 100% packet loss, time
29011ms
pipe 3
[root@localhost ~]# tcpdump -i bond0 -p arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 65535 bytes
21:22:40.694058 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:22:41.695875 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:22:42.698067 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:22:44.689068 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:22:45.689837 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:22:46.692076 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:22:48.691080 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:22:49.693828 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:22:50.696074 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:22:52.693070 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:22:53.693837 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:22:54.696075 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:22:56.694072 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:22:57.696063 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:22:58.698096 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:23:00.696065 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:23:01.698072 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:23:02.700063 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:23:04.698071 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:23:05.700504 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
21:23:06.702009 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
^C
21 packets captured
21 packets received by filter
0 packets dropped by kernel
meanwhile on host,
[root@localhost ~]# tcpdump -i vboxnet0 -p arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vboxnet0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:22:56.936833 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:22:56.936839 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:22:57.938415 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:22:57.938422 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:22:58.939875 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:22:58.939881 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:23:00.937292 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:23:00.937299 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:23:01.939242 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:23:01.939249 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:23:02.940559 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:23:02.940565 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:23:04.938148 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:23:04.938156 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:23:05.939688 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:23:05.939695 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:23:06.941288 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
10:23:06.941295 ARP, Request who-has 192.168.1.100 tell 192.168.1.2,
length 28
^C
18 packets captured
18 packets received by filter
0 packets dropped by kernel
Maybe this is the cause of the problem.
The guest can't receive correct ARP reply, maybe the virtual network of
VirtualBox doesn't transfer it.
on host,
[root@localhost ~]# VBoxManage controlvm
67b83c47-0ee2-46bc-b0ff-e0eb43edc1c2 setlinkstate2 on
on guest,
[root@localhost ~]# dmesg
[ 2392.304591] e1000: eth6 NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: RX
[ 2392.381223] bonding: bond0: link status definitely up for interface eth6.
[root@localhost ~]# ping 192.168.1.100 -c5
PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data.
64 bytes from 192.168.1.100: icmp_req=1 ttl=64 time=0.648 ms
64 bytes from 192.168.1.100: icmp_req=2 ttl=64 time=0.208 ms
64 bytes from 192.168.1.100: icmp_req=3 ttl=64 time=0.187 ms
64 bytes from 192.168.1.100: icmp_req=4 ttl=64 time=0.274 ms
64 bytes from 192.168.1.100: icmp_req=5 ttl=64 time=0.216 ms
--- 192.168.1.100 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4002ms
rtt min/avg/max/mdev = 0.187/0.306/0.648/0.174 ms
[root@localhost ~]# ip neigh show
192.168.1.100 dev bond0 lladdr 0a:00:27:00:00:00 REACHABLE
thanks
Weiping Pan
^ permalink raw reply
* Re: bonding can't change to another slave if you ifdown the active slave
From: Weiping Pan @ 2011-03-09 3:38 UTC (permalink / raw)
To: WANG Cong; +Cc: netdev
In-Reply-To: <il58p9$3ka$2@dough.gmane.org>
On 03/08/2011 08:51 PM, WANG Cong wrote:
> On Tue, 08 Mar 2011 14:52:52 +0800, Weiping Pan wrote:
>
>> ok, I use "Host-only mode" and get
>>
>>
>>
>> an conclusion, that if the first enslaved nic is pulled out, bonding
>> can't handle well.
>>
>> First test.
>> I first enslave eth6, then pull it out, bonding doesn't work. on host,
> ...
>>
>> Second test
>> I first enslave eth6, then pull eth7 out, bonding works well. on guest,
> Can you show me your /proc/net/bonding/bond0 before and after pulling down
> eth6 or eth7? And what does `ip link show` say?
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
And I test it again, there is no problem with kvm.
Maybe there is bug in virtualbox.
thanks
Weiping Pan
^ permalink raw reply
* [PATCH 1/2] net: Allow no-cache copy from user on transmit
From: Tom Herbert @ 2011-03-09 5:05 UTC (permalink / raw)
To: davem, netdev
This patch uses __copy_from_user_nocache (from skb_copy_to_page)
on transmit to bypass data cache for a performance improvement.
This functionality is configurable per device using ethtool, the
device must also be doing TX csum offload to enable. It seems
reasonable to set this when the device does not copy or
otherwise touch the data.
This patch was tested using 200 instances of netperf TCP_RR with
1400 byte request and one byte reply. Platform is 16 core AMD.
No-cache copy disabled:
672703 tps, 97.13% client utilization
50/90/99% latency 244.31 484.205 1028.41
No-cache copy enabled:
702113 tps, 96.16% client utilization,
50/90/99% latency 238.56 467.56 956.955
This seems to provide a nice little performance improvement and is
consistent in the tests I ran. Presumably, this would provide
the greatest benfits in the presence of an application workload
stressing the cache and a lot of transmit data happening. I don't
yet see a downside to using this.
Signed-off-by: Tom Herbert <therbert@google.com>
---
include/linux/ethtool.h | 4 ++++
include/linux/netdevice.h | 4 +++-
include/net/sock.h | 5 +++++
net/core/ethtool.c | 11 ++++++++++-
4 files changed, 22 insertions(+), 2 deletions(-)
diff --git a/include/linux/ethtool.h b/include/linux/ethtool.h
index aac3e2e..2cf3cc9 100644
--- a/include/linux/ethtool.h
+++ b/include/linux/ethtool.h
@@ -833,6 +833,10 @@ struct ethtool_ops {
#define ETHTOOL_GFEATURES 0x0000003a /* Get device offload settings */
#define ETHTOOL_SFEATURES 0x0000003b /* Change device offload settings */
+#define ETHTOOL_GNCCOPY 0x0000003c /* Get no-cache-copy enable
+ * (ethtool_value). */
+#define ETHTOOL_SNCCOPY 0x0000003d /* Set no-cache-copy enable
+ * (ethtool_value). */
/* compatibility with older code */
#define SPARC_ETH_GSET ETHTOOL_GSET
#define SPARC_ETH_SSET ETHTOOL_SSET
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 71563e7..1b72f29 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -978,6 +978,7 @@ struct net_device {
#define NETIF_F_NTUPLE (1 << 27) /* N-tuple filters supported */
#define NETIF_F_RXHASH (1 << 28) /* Receive hashing offload */
#define NETIF_F_RXCSUM (1 << 29) /* Receive checksumming offload */
+#define NETIF_F_NOCACHE_COPY (1 << 30) /* Use no-cache copyfromuser */
/* Segmentation offload features */
#define NETIF_F_GSO_SHIFT 16
@@ -1020,7 +1021,8 @@ struct net_device {
NETIF_F_FRAGLIST)
/* changeable features with no special hardware requirements */
-#define NETIF_F_SOFT_FEATURES (NETIF_F_GSO | NETIF_F_GRO)
+#define NETIF_F_SOFT_FEATURES (NETIF_F_GSO | NETIF_F_GRO | \
+ NETIF_F_NOCACHE_COPY)
/* Interface index. Unique device identifier */
int ifindex;
diff --git a/include/net/sock.h b/include/net/sock.h
index da0534d..55b1385 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -1401,6 +1401,11 @@ static inline int skb_copy_to_page(struct sock *sk, char __user *from,
if (err)
return err;
skb->csum = csum_block_add(skb->csum, csum, skb->len);
+ } else if (sk->sk_route_caps & NETIF_F_NOCACHE_COPY) {
+ if (!access_ok(VERIFY_READ, from, copy)
+ || __copy_from_user_nocache(page_address(page) + off,
+ from, copy))
+ return -EFAULT;
} else if (copy_from_user(page_address(page) + off, from, copy))
return -EFAULT;
diff --git a/net/core/ethtool.c b/net/core/ethtool.c
index c1a71bb..a8b2638 100644
--- a/net/core/ethtool.c
+++ b/net/core/ethtool.c
@@ -328,6 +328,7 @@ static const char netdev_features_strings[ETHTOOL_DEV_FEATURE_WORDS * 32][ETH_GS
/* NETIF_F_NETNS_LOCAL */ "netns-local",
/* NETIF_F_GRO */ "rx-gro",
/* NETIF_F_LRO */ "rx-lro",
+ /* NETIF_F_NOCACHE_COPY */ "tx-nocache-copy",
/* NETIF_F_TSO */ "tx-tcp-segmentation",
/* NETIF_F_UFO */ "tx-udp-fragmentation",
@@ -336,7 +337,6 @@ static const char netdev_features_strings[ETHTOOL_DEV_FEATURE_WORDS * 32][ETH_GS
/* NETIF_F_TSO6 */ "tx-tcp6-segmentation",
/* NETIF_F_FSO */ "tx-fcoe-segmentation",
"",
- "",
/* NETIF_F_FCOE_CRC */ "tx-checksum-fcoe-crc",
/* NETIF_F_SCTP_CSUM */ "tx-checksum-sctp",
@@ -400,6 +400,9 @@ static u32 ethtool_get_feature_mask(u32 eth_cmd)
case ETHTOOL_GGRO:
case ETHTOOL_SGRO:
return NETIF_F_GRO;
+ case ETHTOOL_GNCCOPY:
+ case ETHTOOL_SNCCOPY:
+ return NETIF_F_NOCACHE_COPY;
default:
BUG();
}
@@ -1487,6 +1490,9 @@ static int __ethtool_set_tx_csum(struct net_device *dev, u32 data)
return err;
}
+ if (!data)
+ dev->features &= ~NETIF_F_NOCACHE_COPY;
+
return dev->ethtool_ops->set_tx_csum(dev, data);
}
@@ -1768,6 +1774,7 @@ int dev_ethtool(struct net *net, struct ifreq *ifr)
case ETHTOOL_GUFO:
case ETHTOOL_GGSO:
case ETHTOOL_GGRO:
+ case ETHTOOL_GNCCOPY:
case ETHTOOL_GFLAGS:
case ETHTOOL_GPFLAGS:
case ETHTOOL_GRXFH:
@@ -1924,6 +1931,7 @@ int dev_ethtool(struct net *net, struct ifreq *ifr)
case ETHTOOL_GUFO:
case ETHTOOL_GGSO:
case ETHTOOL_GGRO:
+ case ETHTOOL_GNCCOPY:
rc = ethtool_get_one_feature(dev, useraddr, ethcmd);
break;
case ETHTOOL_STXCSUM:
@@ -1933,6 +1941,7 @@ int dev_ethtool(struct net *net, struct ifreq *ifr)
case ETHTOOL_SUFO:
case ETHTOOL_SGSO:
case ETHTOOL_SGRO:
+ case ETHTOOL_SNCCOPY:
rc = ethtool_set_one_feature(dev, useraddr, ethcmd);
break;
default:
--
1.7.3.1
^ permalink raw reply related
* [PATCH 1/2] ethtool: Support no-cache copy in config
From: Tom Herbert @ 2011-03-09 5:05 UTC (permalink / raw)
To: davem, netdev
Add support for nccopy configuration as an offload option. This allows
enabling and disabling no-cache copies in copy-from-user on
transmission for a device.
Signed-off-by: Tom Herbert <therbert@google.com>
---
ethtool-copy.h | 4 ++++
ethtool.c | 37 ++++++++++++++++++++++++++++++++-----
2 files changed, 36 insertions(+), 5 deletions(-)
diff --git a/ethtool-copy.h b/ethtool-copy.h
index 75c3ae7..98529d8 100644
--- a/ethtool-copy.h
+++ b/ethtool-copy.h
@@ -585,6 +585,10 @@ struct ethtool_flash {
#define ETHTOOL_GRXFHINDIR 0x00000038 /* Get RX flow hash indir'n table */
#define ETHTOOL_SRXFHINDIR 0x00000039 /* Set RX flow hash indir'n table */
+#define ETHTOOL_GNCCOPY 0x0000003c /* Get no-cache-copy enable
+ * (ethtool_value). */
+#define ETHTOOL_SNCCOPY 0x0000003d /* Set no-cache-copy enable
+ * (ethtool_value). */
/* compatibility with older code */
#define SPARC_ETH_GSET ETHTOOL_GSET
#define SPARC_ETH_SSET ETHTOOL_SSET
diff --git a/ethtool.c b/ethtool.c
index 1afdfe4..7be8270 100644
--- a/ethtool.c
+++ b/ethtool.c
@@ -210,6 +210,7 @@ static struct option {
" [ txvlan on|off ]\n"
" [ ntuple on|off ]\n"
" [ rxhash on|off ]\n"
+ " [ nccopy on|off ]\n"
},
{ "-i", "--driver", MODE_GDRV, "Show driver information" },
{ "-d", "--register-dump", MODE_GREGS, "Do a register dump",
@@ -308,6 +309,7 @@ static int off_gso_wanted = -1;
static u32 off_flags_wanted = 0;
static u32 off_flags_mask = 0;
static int off_gro_wanted = -1;
+static int off_nccopy_wanted = -1;
static struct ethtool_pauseparam epause;
static int gpause_changed = 0;
@@ -472,6 +474,7 @@ static struct cmdline_info cmdline_offload[] = {
{ "lro", CMDL_FLAG, &off_flags_wanted, NULL,
ETH_FLAG_LRO, &off_flags_mask },
{ "gro", CMDL_BOOL, &off_gro_wanted, NULL },
+ { "nccopy", CMDL_BOOL, &off_nccopy_wanted, NULL },
{ "rxvlan", CMDL_FLAG, &off_flags_wanted, NULL,
ETH_FLAG_RXVLAN, &off_flags_mask },
{ "txvlan", CMDL_FLAG, &off_flags_wanted, NULL,
@@ -1874,7 +1877,7 @@ static int dump_coalesce(void)
static int dump_offload(int rx, int tx, int sg, int tso, int ufo, int gso,
int gro, int lro, int rxvlan, int txvlan, int ntuple,
- int rxhash)
+ int rxhash, int nccopy)
{
fprintf(stdout,
"rx-checksumming: %s\n"
@@ -1888,7 +1891,8 @@ static int dump_offload(int rx, int tx, int sg, int tso, int ufo, int gso,
"rx-vlan-offload: %s\n"
"tx-vlan-offload: %s\n"
"ntuple-filters: %s\n"
- "receive-hashing: %s\n",
+ "receive-hashing: %s\n"
+ "no-cache-copy: %s\n",
rx ? "on" : "off",
tx ? "on" : "off",
sg ? "on" : "off",
@@ -1900,7 +1904,8 @@ static int dump_offload(int rx, int tx, int sg, int tso, int ufo, int gso,
rxvlan ? "on" : "off",
txvlan ? "on" : "off",
ntuple ? "on" : "off",
- rxhash ? "on" : "off");
+ rxhash ? "on" : "off",
+ nccopy ? "on" : "off");
return 0;
}
@@ -2224,7 +2229,7 @@ static int do_goffload(int fd, struct ifreq *ifr)
struct ethtool_value eval;
int err, allfail = 1, rx = 0, tx = 0, sg = 0;
int tso = 0, ufo = 0, gso = 0, gro = 0, lro = 0, rxvlan = 0, txvlan = 0,
- ntuple = 0, rxhash = 0;
+ ntuple = 0, rxhash = 0, nccopy = 0;
fprintf(stdout, "Offload parameters for %s:\n", devname);
@@ -2312,13 +2317,23 @@ static int do_goffload(int fd, struct ifreq *ifr)
allfail = 0;
}
+ eval.cmd = ETHTOOL_GNCCOPY;
+ ifr->ifr_data = (caddr_t)&eval;
+ err = ioctl(fd, SIOCETHTOOL, ifr);
+ if (err)
+ perror("Cannot get device no-cache copy settings");
+ else {
+ nccopy = eval.data;
+ allfail = 0;
+ }
+
if (allfail) {
fprintf(stdout, "no offload info available\n");
return 83;
}
return dump_offload(rx, tx, sg, tso, ufo, gso, gro, lro, rxvlan, txvlan,
- ntuple, rxhash);
+ ntuple, rxhash, nccopy);
}
static int do_soffload(int fd, struct ifreq *ifr)
@@ -2428,6 +2443,18 @@ static int do_soffload(int fd, struct ifreq *ifr)
}
}
+ if (off_nccopy_wanted >= 0) {
+ changed = 1;
+ eval.cmd = ETHTOOL_SNCCOPY;
+ eval.data = (off_nccopy_wanted == 1);
+ ifr->ifr_data = (caddr_t)&eval;
+ err = ioctl(fd, SIOCETHTOOL, ifr);
+ if (err) {
+ perror("Cannot set device no-cache copy settings");
+ return 94;
+ }
+ }
+
if (!changed) {
fprintf(stdout, "no offload settings changed\n");
}
--
1.7.3.1
^ permalink raw reply related
* Re: [PATCH 2/3] [RFC] Changes for MQ virtio-net
From: Krishna Kumar2 @ 2011-03-09 6:01 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: anthony, arnd, avi, davem, eric.dumazet, horms, kvm, netdev,
rusty
In-Reply-To: <20110308154104.GA13931@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 510 bytes --]
"Michael S. Tsirkin" <mst@redhat.com> wrote on 03/08/2011 09:11:04 PM:
> Also, could you post your current version of the qemu code pls?
> It's useful for testing and to see the whole picture.
Sorry for the delay on this.
I am attaching the qemu changes. Some parts of the patch are
completely redundant, eg MAX_TUN_DEVICES and I will remove it
later.
It works with latest qemu and the kernel patch sent earlier.
Please let me know if there are any issues.
thanks,
- KK
(See attached file: qemu.patch)
[-- Attachment #2: qemu.patch --]
[-- Type: application/octet-stream, Size: 33116 bytes --]
diff -ruNp org/hw/vhost.c new/hw/vhost.c
--- org/hw/vhost.c 2011-02-28 14:53:34.000000000 +0530
+++ new/hw/vhost.c 2011-03-09 09:09:27.000000000 +0530
@@ -581,7 +581,7 @@ static void vhost_virtqueue_cleanup(stru
0, virtio_queue_get_desc_size(vdev, idx));
}
-int vhost_dev_init(struct vhost_dev *hdev, int devfd, bool force)
+int vhost_dev_init(struct vhost_dev *hdev, int devfd, bool force, int numtxqs)
{
uint64_t features;
int r;
@@ -593,11 +593,13 @@ int vhost_dev_init(struct vhost_dev *hde
return -errno;
}
}
- r = ioctl(hdev->control, VHOST_SET_OWNER, NULL);
+ r = ioctl(hdev->control, VHOST_SET_OWNER, numtxqs);
if (r < 0) {
goto fail;
}
+ hdev->nvqs = numtxqs * 2;
+
r = ioctl(hdev->control, VHOST_GET_FEATURES, &features);
if (r < 0) {
goto fail;
diff -ruNp org/hw/vhost.h new/hw/vhost.h
--- org/hw/vhost.h 2011-02-03 15:02:20.000000000 +0530
+++ new/hw/vhost.h 2011-03-09 09:09:44.000000000 +0530
@@ -41,7 +41,7 @@ struct vhost_dev {
bool force;
};
-int vhost_dev_init(struct vhost_dev *hdev, int devfd, bool force);
+int vhost_dev_init(struct vhost_dev *hdev, int devfd, bool force, int numtxqs);
void vhost_dev_cleanup(struct vhost_dev *hdev);
bool vhost_dev_query(struct vhost_dev *hdev, VirtIODevice *vdev);
int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev);
diff -ruNp org/hw/vhost_net.c new/hw/vhost_net.c
--- org/hw/vhost_net.c 2011-02-03 15:02:20.000000000 +0530
+++ new/hw/vhost_net.c 2011-03-09 10:52:03.000000000 +0530
@@ -36,8 +36,9 @@
struct vhost_net {
struct vhost_dev dev;
- struct vhost_virtqueue vqs[2];
- int backend;
+ struct vhost_virtqueue *vqs;
+ int nvqs;
+ int *backend;
VLANClientState *vc;
};
@@ -70,11 +71,11 @@ void vhost_net_ack_features(struct vhost
}
}
-static int vhost_net_get_fd(VLANClientState *backend)
+static int vhost_net_get_fd(VLANClientState *backend, int index)
{
switch (backend->info->type) {
case NET_CLIENT_TYPE_TAP:
- return tap_get_fd(backend);
+ return tap_get_fd(backend, index);
default:
fprintf(stderr, "vhost-net requires tap backend\n");
return -EBADFD;
@@ -82,27 +83,36 @@ static int vhost_net_get_fd(VLANClientSt
}
struct vhost_net *vhost_net_init(VLANClientState *backend, int devfd,
- bool force)
+ bool force, int numtxqs)
{
- int r;
+ int i, r;
struct vhost_net *net = qemu_malloc(sizeof *net);
if (!backend) {
fprintf(stderr, "vhost-net requires backend to be setup\n");
goto fail;
}
- r = vhost_net_get_fd(backend);
- if (r < 0) {
- goto fail;
+
+ net->backend = qemu_malloc(numtxqs * (sizeof *net->backend));
+ for (i = 0; i < numtxqs; i++) {
+ r = vhost_net_get_fd(backend, i);
+ if (r < 0) {
+ goto fail;
+ }
+ net->backend[i] = r;
}
+
net->vc = backend;
net->dev.backend_features = tap_has_vnet_hdr(backend) ? 0 :
(1 << VHOST_NET_F_VIRTIO_NET_HDR);
- net->backend = r;
- r = vhost_dev_init(&net->dev, devfd, force);
+ r = vhost_dev_init(&net->dev, devfd, force, numtxqs);
if (r < 0) {
goto fail;
}
+
+ net->nvqs = numtxqs * 2;
+ net->vqs = qemu_malloc(net->nvqs * (sizeof *net->vqs));
+
if (!tap_has_vnet_hdr_len(backend,
sizeof(struct virtio_net_hdr_mrg_rxbuf))) {
net->dev.features &= ~(1 << VIRTIO_NET_F_MRG_RXBUF);
@@ -137,7 +147,6 @@ int vhost_net_start(struct vhost_net *ne
sizeof(struct virtio_net_hdr_mrg_rxbuf));
}
- net->dev.nvqs = 2;
net->dev.vqs = net->vqs;
r = vhost_dev_start(&net->dev, dev);
if (r < 0) {
@@ -145,9 +154,10 @@ int vhost_net_start(struct vhost_net *ne
}
net->vc->info->poll(net->vc, false);
- qemu_set_fd_handler(net->backend, NULL, NULL, NULL);
- file.fd = net->backend;
for (file.index = 0; file.index < net->dev.nvqs; ++file.index) {
+ if (file.index < net->dev.nvqs / 2)
+ qemu_set_fd_handler(net->backend[file.index], NULL, NULL, NULL);
+ file.fd = net->backend[file.index % (net->dev.nvqs / 2)];
r = ioctl(net->dev.control, VHOST_NET_SET_BACKEND, &file);
if (r < 0) {
r = -errno;
@@ -195,7 +205,7 @@ void vhost_net_cleanup(struct vhost_net
}
#else
struct vhost_net *vhost_net_init(VLANClientState *backend, int devfd,
- bool force)
+ bool force, int numtxqs)
{
return NULL;
}
diff -ruNp org/hw/vhost_net.h new/hw/vhost_net.h
--- org/hw/vhost_net.h 2011-02-03 15:02:20.000000000 +0530
+++ new/hw/vhost_net.h 2011-03-09 09:16:21.000000000 +0530
@@ -6,7 +6,8 @@
struct vhost_net;
typedef struct vhost_net VHostNetState;
-VHostNetState *vhost_net_init(VLANClientState *backend, int devfd, bool force);
+VHostNetState *vhost_net_init(VLANClientState *backend, int devfd, bool force,
+ int numtxqs);
bool vhost_net_query(VHostNetState *net, VirtIODevice *dev);
int vhost_net_start(VHostNetState *net, VirtIODevice *dev);
diff -ruNp org/hw/virtio-net.c new/hw/virtio-net.c
--- org/hw/virtio-net.c 2011-02-03 15:02:20.000000000 +0530
+++ new/hw/virtio-net.c 2011-03-09 09:26:30.000000000 +0530
@@ -31,8 +31,8 @@ typedef struct VirtIONet
VirtIODevice vdev;
uint8_t mac[ETH_ALEN];
uint16_t status;
- VirtQueue *rx_vq;
- VirtQueue *tx_vq;
+ VirtQueue **rx_vq;
+ VirtQueue **tx_vq;
VirtQueue *ctrl_vq;
NICState *nic;
QEMUTimer *tx_timer;
@@ -63,6 +63,7 @@ typedef struct VirtIONet
} mac_table;
uint32_t *vlans;
DeviceState *qdev;
+ uint16_t numtxqs;
} VirtIONet;
/* TODO
@@ -80,6 +81,7 @@ static void virtio_net_get_config(VirtIO
struct virtio_net_config netcfg;
netcfg.status = lduw_p(&n->status);
+ netcfg.numtxqs = n->numtxqs;
memcpy(netcfg.mac, n->mac, ETH_ALEN);
memcpy(config, &netcfg, sizeof(netcfg));
}
@@ -227,6 +229,9 @@ static uint32_t virtio_net_get_features(
VirtIONet *n = to_virtio_net(vdev);
features |= (1 << VIRTIO_NET_F_MAC);
+ if (n->numtxqs > 1)
+ features |= (1 << VIRTIO_NET_F_NUMTXQS);
+
if (peer_has_vnet_hdr(n)) {
tap_using_vnet_hdr(n->nic->nc.peer, 1);
@@ -459,7 +464,10 @@ static int virtio_net_can_receive(VLANCl
return 0;
}
- if (!virtio_queue_ready(n->rx_vq) ||
+ /*
+ * If this function executes, we are single RX and hence use only rxq[0]
+ */
+ if (!virtio_queue_ready(n->rx_vq[0]) ||
!(n->vdev.status & VIRTIO_CONFIG_S_DRIVER_OK))
return 0;
@@ -468,22 +476,22 @@ static int virtio_net_can_receive(VLANCl
static int virtio_net_has_buffers(VirtIONet *n, int bufsize)
{
- if (virtio_queue_empty(n->rx_vq) ||
+ if (virtio_queue_empty(n->rx_vq[0]) ||
(n->mergeable_rx_bufs &&
- !virtqueue_avail_bytes(n->rx_vq, bufsize, 0))) {
- virtio_queue_set_notification(n->rx_vq, 1);
+ !virtqueue_avail_bytes(n->rx_vq[0], bufsize, 0))) {
+ virtio_queue_set_notification(n->rx_vq[0], 1);
/* To avoid a race condition where the guest has made some buffers
* available after the above check but before notification was
* enabled, check for available buffers again.
*/
- if (virtio_queue_empty(n->rx_vq) ||
+ if (virtio_queue_empty(n->rx_vq[0]) ||
(n->mergeable_rx_bufs &&
- !virtqueue_avail_bytes(n->rx_vq, bufsize, 0)))
+ !virtqueue_avail_bytes(n->rx_vq[0], bufsize, 0)))
return 0;
}
- virtio_queue_set_notification(n->rx_vq, 0);
+ virtio_queue_set_notification(n->rx_vq[0], 0);
return 1;
}
@@ -622,7 +630,7 @@ static ssize_t virtio_net_receive(VLANCl
total = 0;
- if (virtqueue_pop(n->rx_vq, &elem) == 0) {
+ if (virtqueue_pop(n->rx_vq[0], &elem) == 0) {
if (i == 0)
return -1;
error_report("virtio-net unexpected empty queue: "
@@ -674,15 +682,15 @@ static ssize_t virtio_net_receive(VLANCl
}
/* signal other side */
- virtqueue_fill(n->rx_vq, &elem, total, i++);
+ virtqueue_fill(n->rx_vq[0], &elem, total, i++);
}
if (mhdr) {
mhdr->num_buffers = lduw_p(&i);
}
- virtqueue_flush(n->rx_vq, i);
- virtio_notify(&n->vdev, n->rx_vq);
+ virtqueue_flush(n->rx_vq[0], i);
+ virtio_notify(&n->vdev, n->rx_vq[0]);
return size;
}
@@ -693,13 +701,16 @@ static void virtio_net_tx_complete(VLANC
{
VirtIONet *n = DO_UPCAST(NICState, nc, nc)->opaque;
- virtqueue_push(n->tx_vq, &n->async_tx.elem, n->async_tx.len);
- virtio_notify(&n->vdev, n->tx_vq);
+ /*
+ * If this function executes, we are single TX and hence use only txq[0]
+ */
+ virtqueue_push(n->tx_vq[0], &n->async_tx.elem, n->async_tx.len);
+ virtio_notify(&n->vdev, n->tx_vq[0]);
n->async_tx.elem.out_num = n->async_tx.len = 0;
- virtio_queue_set_notification(n->tx_vq, 1);
- virtio_net_flush_tx(n, n->tx_vq);
+ virtio_queue_set_notification(n->tx_vq[0], 1);
+ virtio_net_flush_tx(n, n->tx_vq[0]);
}
/* TX */
@@ -714,7 +725,7 @@ static int32_t virtio_net_flush_tx(VirtI
assert(n->vdev.vm_running);
if (n->async_tx.elem.out_num) {
- virtio_queue_set_notification(n->tx_vq, 0);
+ virtio_queue_set_notification(n->tx_vq[0], 0);
return num_packets;
}
@@ -749,7 +760,7 @@ static int32_t virtio_net_flush_tx(VirtI
ret = qemu_sendv_packet_async(&n->nic->nc, out_sg, out_num,
virtio_net_tx_complete);
if (ret == 0) {
- virtio_queue_set_notification(n->tx_vq, 0);
+ virtio_queue_set_notification(n->tx_vq[0], 0);
n->async_tx.elem = elem;
n->async_tx.len = len;
return -EBUSY;
@@ -817,8 +828,8 @@ static void virtio_net_tx_timer(void *op
if (!(n->vdev.status & VIRTIO_CONFIG_S_DRIVER_OK))
return;
- virtio_queue_set_notification(n->tx_vq, 1);
- virtio_net_flush_tx(n, n->tx_vq);
+ virtio_queue_set_notification(n->tx_vq[0], 1);
+ virtio_net_flush_tx(n, n->tx_vq[0]);
}
static void virtio_net_tx_bh(void *opaque)
@@ -834,7 +845,7 @@ static void virtio_net_tx_bh(void *opaqu
if (unlikely(!(n->vdev.status & VIRTIO_CONFIG_S_DRIVER_OK)))
return;
- ret = virtio_net_flush_tx(n, n->tx_vq);
+ ret = virtio_net_flush_tx(n, n->tx_vq[0]);
if (ret == -EBUSY) {
return; /* Notification re-enable handled by tx_complete */
}
@@ -850,9 +861,9 @@ static void virtio_net_tx_bh(void *opaqu
/* If less than a full burst, re-enable notification and flush
* anything that may have come in while we weren't looking. If
* we find something, assume the guest is still active and reschedule */
- virtio_queue_set_notification(n->tx_vq, 1);
- if (virtio_net_flush_tx(n, n->tx_vq) > 0) {
- virtio_queue_set_notification(n->tx_vq, 0);
+ virtio_queue_set_notification(n->tx_vq[0], 1);
+ if (virtio_net_flush_tx(n, n->tx_vq[0]) > 0) {
+ virtio_queue_set_notification(n->tx_vq[0], 0);
qemu_bh_schedule(n->tx_bh);
n->tx_waiting = 1;
}
@@ -868,6 +879,7 @@ static void virtio_net_save(QEMUFile *f,
virtio_save(&n->vdev, f);
qemu_put_buffer(f, n->mac, ETH_ALEN);
+ qemu_put_be16(f, n->numtxqs);
qemu_put_be32(f, n->tx_waiting);
qemu_put_be32(f, n->mergeable_rx_bufs);
qemu_put_be16(f, n->status);
@@ -897,6 +909,7 @@ static int virtio_net_load(QEMUFile *f,
virtio_load(&n->vdev, f);
qemu_get_buffer(f, n->mac, ETH_ALEN);
+ n->numtxqs = qemu_get_be32(f);
n->tx_waiting = qemu_get_be32(f);
n->mergeable_rx_bufs = qemu_get_be32(f);
@@ -995,11 +1008,13 @@ VirtIODevice *virtio_net_init(DeviceStat
virtio_net_conf *net)
{
VirtIONet *n;
+ int i;
n = (VirtIONet *)virtio_common_init("virtio-net", VIRTIO_ID_NET,
sizeof(struct virtio_net_config),
sizeof(VirtIONet));
+ n->numtxqs = conf->peer->numtxqs;
n->vdev.get_config = virtio_net_get_config;
n->vdev.set_config = virtio_net_set_config;
n->vdev.get_features = virtio_net_get_features;
@@ -1007,7 +1022,11 @@ VirtIODevice *virtio_net_init(DeviceStat
n->vdev.bad_features = virtio_net_bad_features;
n->vdev.reset = virtio_net_reset;
n->vdev.set_status = virtio_net_set_status;
- n->rx_vq = virtio_add_queue(&n->vdev, 256, virtio_net_handle_rx);
+
+ /* Allocate per rx vq's */
+ n->rx_vq = qemu_mallocz(n->numtxqs * sizeof(*n->rx_vq));
+ for (i = 0; i < n->numtxqs; i++)
+ n->rx_vq[i] = virtio_add_queue(&n->vdev, 256, virtio_net_handle_rx);
if (net->tx && strcmp(net->tx, "timer") && strcmp(net->tx, "bh")) {
error_report("virtio-net: "
@@ -1016,12 +1035,22 @@ VirtIODevice *virtio_net_init(DeviceStat
error_report("Defaulting to \"bh\"");
}
+ /* Allocate per tx vq's */
+ n->tx_vq = qemu_mallocz(n->numtxqs * sizeof(*n->tx_vq));
+ for (i = 0; i < n->numtxqs; i++) {
+ if (net->tx && !strcmp(net->tx, "timer")) {
+ n->tx_vq[i] = virtio_add_queue(&n->vdev, 256,
+ virtio_net_handle_tx_timer);
+ } else {
+ n->tx_vq[i] = virtio_add_queue(&n->vdev, 256,
+ virtio_net_handle_tx_bh);
+ }
+ }
+
if (net->tx && !strcmp(net->tx, "timer")) {
- n->tx_vq = virtio_add_queue(&n->vdev, 256, virtio_net_handle_tx_timer);
n->tx_timer = qemu_new_timer(vm_clock, virtio_net_tx_timer, n);
n->tx_timeout = net->txtimer;
} else {
- n->tx_vq = virtio_add_queue(&n->vdev, 256, virtio_net_handle_tx_bh);
n->tx_bh = qemu_bh_new(virtio_net_tx_bh, n);
}
n->ctrl_vq = virtio_add_queue(&n->vdev, 64, virtio_net_handle_ctrl);
diff -ruNp org/hw/virtio-net.h new/hw/virtio-net.h
--- org/hw/virtio-net.h 2010-12-13 12:02:44.000000000 +0530
+++ new/hw/virtio-net.h 2011-03-09 09:27:24.000000000 +0530
@@ -44,6 +44,7 @@
#define VIRTIO_NET_F_CTRL_RX 18 /* Control channel RX mode support */
#define VIRTIO_NET_F_CTRL_VLAN 19 /* Control channel VLAN filtering */
#define VIRTIO_NET_F_CTRL_RX_EXTRA 20 /* Extra RX mode control support */
+#define VIRTIO_NET_F_NUMTXQS 21 /* Supports multiple TX queues */
#define VIRTIO_NET_S_LINK_UP 1 /* Link is up */
@@ -72,6 +73,7 @@ struct virtio_net_config
uint8_t mac[ETH_ALEN];
/* See VIRTIO_NET_F_STATUS and VIRTIO_NET_S_* above */
uint16_t status;
+ uint16_t numtxqs; /* number of transmit/receive queues */
} __attribute__((packed));
/* This is the first element of the scatter-gather list. If you don't
diff -ruNp org/hw/virtio-pci.c new/hw/virtio-pci.c
--- org/hw/virtio-pci.c 2011-02-03 15:02:20.000000000 +0530
+++ new/hw/virtio-pci.c 2011-03-09 09:31:15.000000000 +0530
@@ -103,6 +103,7 @@ typedef struct {
uint32_t addr;
uint32_t class_code;
uint32_t nvectors;
+ uint32_t mq;
BlockConf block;
NICConf nic;
uint32_t host_features;
@@ -965,6 +966,7 @@ static PCIDeviceInfo virtio_info[] = {
DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags,
VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, false),
DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 3),
+ DEFINE_PROP_UINT32("mq", VirtIOPCIProxy, mq, 1),
DEFINE_VIRTIO_NET_FEATURES(VirtIOPCIProxy, host_features),
DEFINE_NIC_PROPERTIES(VirtIOPCIProxy, nic),
DEFINE_PROP_UINT32("x-txtimer", VirtIOPCIProxy,
diff -ruNp org/net/tap.c new/net/tap.c
--- org/net/tap.c 2011-02-03 15:02:20.000000000 +0530
+++ new/net/tap.c 2011-03-09 11:17:36.000000000 +0530
@@ -49,16 +49,20 @@
*/
#define TAP_BUFSIZE (4096 + 65536)
+#define MAX_TUN_DEVICES 8
+
typedef struct TAPState {
VLANClientState nc;
- int fd;
+ int *fds;
+ int numfds;
char down_script[1024];
- char down_script_arg[128];
+ char down_script_arg[MAX_TUN_DEVICES][128];
uint8_t buf[TAP_BUFSIZE];
unsigned int read_poll : 1;
unsigned int write_poll : 1;
unsigned int using_vnet_hdr : 1;
unsigned int has_ufo: 1;
+ unsigned int do_script: 1;
VHostNetState *vhost_net;
unsigned host_vnet_hdr_len;
} TAPState;
@@ -71,11 +75,17 @@ static void tap_writable(void *opaque);
static void tap_update_fd_handler(TAPState *s)
{
- qemu_set_fd_handler2(s->fd,
- s->read_poll ? tap_can_send : NULL,
- s->read_poll ? tap_send : NULL,
- s->write_poll ? tap_writable : NULL,
- s);
+ int i;
+
+ for (i = 0; i < s->numfds; i++) {
+ if (i < MAX_TUN_DEVICES || s->fds[i] != s->fds[i % MAX_TUN_DEVICES]) {
+ qemu_set_fd_handler2(s->fds[i],
+ s->read_poll ? tap_can_send : NULL,
+ s->read_poll ? tap_send : NULL,
+ s->write_poll ? tap_writable : NULL,
+ s);
+ }
+ }
}
static void tap_read_poll(TAPState *s, int enable)
@@ -104,7 +114,7 @@ static ssize_t tap_write_packet(TAPState
ssize_t len;
do {
- len = writev(s->fd, iov, iovcnt);
+ len = writev(s->fds[0], iov, iovcnt);
} while (len == -1 && errno == EINTR);
if (len == -1 && errno == EAGAIN) {
@@ -197,7 +207,7 @@ static void tap_send(void *opaque)
do {
uint8_t *buf = s->buf;
- size = tap_read_packet(s->fd, s->buf, sizeof(s->buf));
+ size = tap_read_packet(s->fds[0], s->buf, sizeof(s->buf));
if (size <= 0) {
break;
}
@@ -238,18 +248,21 @@ int tap_has_vnet_hdr_len(VLANClientState
assert(nc->info->type == NET_CLIENT_TYPE_TAP);
- return tap_probe_vnet_hdr_len(s->fd, len);
+ return tap_probe_vnet_hdr_len(s->fds[0], len);
}
void tap_set_vnet_hdr_len(VLANClientState *nc, int len)
{
TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ int i;
assert(nc->info->type == NET_CLIENT_TYPE_TAP);
assert(len == sizeof(struct virtio_net_hdr_mrg_rxbuf) ||
len == sizeof(struct virtio_net_hdr));
- tap_fd_set_vnet_hdr_len(s->fd, len);
+ for (i = 0; i < s->numfds; i++)
+ if (i < MAX_TUN_DEVICES || s->fds[i] != s->fds[i % MAX_TUN_DEVICES])
+ tap_fd_set_vnet_hdr_len(s->fds[i], len);
s->host_vnet_hdr_len = len;
}
@@ -269,16 +282,29 @@ void tap_set_offload(VLANClientState *nc
int tso6, int ecn, int ufo)
{
TAPState *s = DO_UPCAST(TAPState, nc, nc);
- if (s->fd < 0) {
- return;
+ int i;
+
+ for (i = 0; i < s->numfds; i++) {
+ if (s->fds[i] >= 0 && (i < MAX_TUN_DEVICES ||
+ s->fds[i] != s->fds[i % MAX_TUN_DEVICES]))
+ tap_fd_set_offload(s->fds[i], csum, tso4, tso6, ecn, ufo);
}
+}
- tap_fd_set_offload(s->fd, csum, tso4, tso6, ecn, ufo);
+static void close_tap_fds(int *fds, int numtxqs)
+{
+ int i;
+
+ for (i = 0; i < numtxqs; i++) {
+ if (i < MAX_TUN_DEVICES || fds[i] != fds[i % MAX_TUN_DEVICES])
+ close(fds[i]);
+ }
}
static void tap_cleanup(VLANClientState *nc)
{
TAPState *s = DO_UPCAST(TAPState, nc, nc);
+ int i;
if (s->vhost_net) {
vhost_net_cleanup(s->vhost_net);
@@ -287,13 +313,17 @@ static void tap_cleanup(VLANClientState
qemu_purge_queued_packets(nc);
- if (s->down_script[0])
- launch_script(s->down_script, s->down_script_arg, s->fd);
+ for (i = 0; i < s->numfds; i++) {
+ if (i < MAX_TUN_DEVICES || s->fds[i] != s->fds[i % MAX_TUN_DEVICES]) {
+ if (s->down_script[0])
+ launch_script(s->down_script, s->down_script_arg[i], s->fds[i]);
+ }
+ }
tap_read_poll(s, 0);
tap_write_poll(s, 0);
- close(s->fd);
- s->fd = -1;
+
+ close_tap_fds(s->fds, s->numfds);
}
static void tap_poll(VLANClientState *nc, bool enable)
@@ -303,11 +333,12 @@ static void tap_poll(VLANClientState *nc
tap_write_poll(s, enable);
}
-int tap_get_fd(VLANClientState *nc)
+int tap_get_fd(VLANClientState *nc, int index)
{
TAPState *s = DO_UPCAST(TAPState, nc, nc);
assert(nc->info->type == NET_CLIENT_TYPE_TAP);
- return s->fd;
+ assert(index < s->numfds);
+ return s->fds[index];
}
/* fd support */
@@ -325,20 +356,26 @@ static NetClientInfo net_tap_info = {
static TAPState *net_tap_fd_init(VLANState *vlan,
const char *model,
const char *name,
- int fd,
+ int *fds, int numtxqs,
int vnet_hdr)
{
VLANClientState *nc;
TAPState *s;
+ int i;
nc = qemu_new_net_client(&net_tap_info, vlan, NULL, model, name);
+ nc->numtxqs = numtxqs;
s = DO_UPCAST(TAPState, nc, nc);
- s->fd = fd;
+ s->fds = fds;
+ s->numfds = numtxqs;
s->host_vnet_hdr_len = vnet_hdr ? sizeof(struct virtio_net_hdr) : 0;
s->using_vnet_hdr = 0;
- s->has_ufo = tap_probe_has_ufo(s->fd);
+ for (i = 0; i < s->numfds; i++) {
+ if (i < MAX_TUN_DEVICES || fds[i] != fds[i % MAX_TUN_DEVICES])
+ s->has_ufo = tap_probe_has_ufo(s->fds[i]);
+ }
tap_set_offload(&s->nc, 0, 0, 0, 0, 0);
tap_read_poll(s, 1);
s->vhost_net = NULL;
@@ -389,11 +426,28 @@ static int launch_script(const char *set
return -1;
}
-static int net_tap_init(QemuOpts *opts, int *vnet_hdr)
+static int net_tap_init(QemuOpts *opts, int *vnet_hdr, int *fds, int numtxqs,
+ int *script)
{
- int fd, vnet_hdr_required;
+ int i, vnet_hdr_required;
char ifname[128] = {0,};
const char *setup_script;
+ int launch = 0;
+ const char *dev;
+
+ if (qemu_opt_get(opts, "vtap")) {
+ *vnet_hdr = 1;
+ *script = 0; /* we don't need start/stop script */
+ dev = qemu_opt_get(opts, "vtap");
+ for (i = 0; i < numtxqs; i++) {
+ TFR(fds[i] = vtap_open(dev, vnet_hdr, 1));
+ if (fds[i] < 0)
+ goto err;
+ fcntl(fds[i], F_SETFL, O_NONBLOCK);
+ }
+ *vnet_hdr = !!tap_probe_vnet_hdr(fds[0]);
+ return 0;
+ }
if (qemu_opt_get(opts, "ifname")) {
pstrcpy(ifname, sizeof(ifname), qemu_opt_get(opts, "ifname"));
@@ -406,29 +460,81 @@ static int net_tap_init(QemuOpts *opts,
vnet_hdr_required = 0;
}
- TFR(fd = tap_open(ifname, sizeof(ifname), vnet_hdr, vnet_hdr_required));
- if (fd < 0) {
- return -1;
- }
-
setup_script = qemu_opt_get(opts, "script");
if (setup_script &&
setup_script[0] != '\0' &&
- strcmp(setup_script, "no") != 0 &&
- launch_script(setup_script, ifname, fd)) {
- close(fd);
- return -1;
+ strcmp(setup_script, "no") != 0) {
+ launch = 1;
+ *script = 1;
+ }
+
+ if (numtxqs == 1) {
+ fprintf(stderr, "Device: %s\n", ifname);
+ TFR(fds[0] = tap_open(ifname, sizeof(ifname), vnet_hdr,
+ vnet_hdr_required));
+ if (fds[0] < 0) {
+ goto err;
+ }
+
+ if (launch && launch_script(setup_script, ifname, fds[0]))
+ goto err;
+ } else {
+ char alt_name[128];
+
+ for (i = 0; i < numtxqs; i++) {
+ if (i < MAX_TUN_DEVICES) {
+ sprintf(alt_name, "%s.%d", ifname, i);
+ fprintf(stderr, "Device: %s\n", alt_name);
+ TFR(fds[i] = tap_open(alt_name, sizeof(alt_name), vnet_hdr,
+ vnet_hdr_required));
+ if (fds[i] < 0) {
+ goto err;
+ }
+
+ if (launch && launch_script(setup_script, alt_name, fds[i]))
+ goto err;
+ } else {
+ fds[i] = fds[i % MAX_TUN_DEVICES];
+ }
+ }
}
qemu_opt_set(opts, "ifname", ifname);
- return fd;
+ return 0;
+
+err:
+ close_tap_fds(fds, numtxqs);
+ return -1;
}
int net_init_tap(QemuOpts *opts, Monitor *mon, const char *name, VLANState *vlan)
{
TAPState *s;
- int fd, vnet_hdr = 0;
+ int *fds, vnet_hdr = 0;
+ int i, vhost;
+ int script = 0, numtxqs = 1;
+
+ vhost = qemu_opt_get_bool(opts, "vhost", 0);
+
+ /*
+ * We support multiple tx queues if:
+ * 1. smp > 1
+ * 2. vhost=on
+ * 3. mq=on
+ * In this case, #txqueues = #cpus. This value can be changed by
+ * using the "numtxqs" option.
+ */
+ if (vhost && smp_cpus > 1) {
+ if (qemu_opt_get_bool(opts, "mq", 0)) {
+#define VIRTIO_MAX_TXQS 8
+ int dflt = MIN(smp_cpus, VIRTIO_MAX_TXQS);
+
+ numtxqs = qemu_opt_get_number(opts, "numtxqs", dflt);
+ }
+ }
+
+ fds = qemu_mallocz(numtxqs * sizeof(*fds));
if (qemu_opt_get(opts, "fd")) {
if (qemu_opt_get(opts, "ifname") ||
@@ -439,14 +545,14 @@ int net_init_tap(QemuOpts *opts, Monitor
return -1;
}
- fd = net_handle_fd_param(mon, qemu_opt_get(opts, "fd"));
- if (fd == -1) {
+ fds[0] = net_handle_fd_param(mon, qemu_opt_get(opts, "fd"));
+ if (fds[0] == -1) {
return -1;
}
- fcntl(fd, F_SETFL, O_NONBLOCK);
+ fcntl(fds[0], F_SETFL, O_NONBLOCK);
- vnet_hdr = tap_probe_vnet_hdr(fd);
+ vnet_hdr = tap_probe_vnet_hdr(fds[0]);
} else {
if (!qemu_opt_get(opts, "script")) {
qemu_opt_set(opts, "script", DEFAULT_NETWORK_SCRIPT);
@@ -456,24 +562,30 @@ int net_init_tap(QemuOpts *opts, Monitor
qemu_opt_set(opts, "downscript", DEFAULT_NETWORK_DOWN_SCRIPT);
}
- fd = net_tap_init(opts, &vnet_hdr);
- if (fd == -1) {
+ if (net_tap_init(opts, &vnet_hdr, fds, numtxqs, &script) == -1) {
return -1;
}
}
- s = net_tap_fd_init(vlan, "tap", name, fd, vnet_hdr);
+ s = net_tap_fd_init(vlan, "tap", name, fds, numtxqs, vnet_hdr);
if (!s) {
- close(fd);
+ close_tap_fds(fds, numtxqs);
return -1;
}
- if (tap_set_sndbuf(s->fd, opts) < 0) {
- return -1;
+ s->do_script = script;
+
+ for (i = 0; i < s->numfds; i++) {
+ if (i < MAX_TUN_DEVICES || fds[i] != fds[i % MAX_TUN_DEVICES]) {
+ if (tap_set_sndbuf(s->fds[i], opts) < 0) {
+ close_tap_fds(fds, numtxqs);
+ return -1;
+ }
+ }
}
if (qemu_opt_get(opts, "fd")) {
- snprintf(s->nc.info_str, sizeof(s->nc.info_str), "fd=%d", fd);
+ snprintf(s->nc.info_str, sizeof(s->nc.info_str), "fd=%d", fds[0]);
} else {
const char *ifname, *script, *downscript;
@@ -487,12 +599,20 @@ int net_init_tap(QemuOpts *opts, Monitor
if (strcmp(downscript, "no") != 0) {
snprintf(s->down_script, sizeof(s->down_script), "%s", downscript);
- snprintf(s->down_script_arg, sizeof(s->down_script_arg), "%s", ifname);
+ for (i = 0; i < s->numfds; i++) {
+ char alt_name[128];
+
+ if (s->numfds == 1) {
+ pstrcpy(alt_name, sizeof(ifname), ifname);
+ } else {
+ sprintf(alt_name, "%s.%d", ifname, i);
+ }
+ snprintf(s->down_script_arg[i], sizeof(s->down_script_arg[i]), "%s", alt_name);
+ }
}
}
- if (qemu_opt_get_bool(opts, "vhost", !!qemu_opt_get(opts, "vhostfd") ||
- qemu_opt_get_bool(opts, "vhostforce", false))) {
+ if (vhost) {
int vhostfd, r;
bool force = qemu_opt_get_bool(opts, "vhostforce", false);
if (qemu_opt_get(opts, "vhostfd")) {
@@ -504,9 +624,13 @@ int net_init_tap(QemuOpts *opts, Monitor
} else {
vhostfd = -1;
}
- s->vhost_net = vhost_net_init(&s->nc, vhostfd, force);
+ s->vhost_net = vhost_net_init(&s->nc, vhostfd, force, numtxqs);
if (!s->vhost_net) {
error_report("vhost-net requested but could not be initialized");
+ if (numtxqs > 1) {
+ error_report("Need vhost support for numtxqs > 1, exiting...");
+ exit(1);
+ }
return -1;
}
} else if (qemu_opt_get(opts, "vhostfd")) {
diff -ruNp org/net/tap.h new/net/tap.h
--- org/net/tap.h 2010-12-13 12:02:44.000000000 +0530
+++ new/net/tap.h 2011-03-09 10:09:39.000000000 +0530
@@ -35,6 +35,7 @@
int net_init_tap(QemuOpts *opts, Monitor *mon, const char *name, VLANState *vlan);
int tap_open(char *ifname, int ifname_size, int *vnet_hdr, int vnet_hdr_required);
+int vtap_open(const char *devname, int *vnet_hdr, int vnet_hdr_required);
ssize_t tap_read_packet(int tapfd, uint8_t *buf, int maxlen);
@@ -52,7 +53,7 @@ int tap_probe_has_ufo(int fd);
void tap_fd_set_offload(int fd, int csum, int tso4, int tso6, int ecn, int ufo);
void tap_fd_set_vnet_hdr_len(int fd, int len);
-int tap_get_fd(VLANClientState *vc);
+int tap_get_fd(VLANClientState *vc, int index);
struct vhost_net;
struct vhost_net *tap_get_vhost_net(VLANClientState *vc);
diff -ruNp org/net/tap-linux.c new/net/tap-linux.c
--- org/net/tap-linux.c 2011-02-03 15:02:20.000000000 +0530
+++ new/net/tap-linux.c 2011-03-09 10:11:03.000000000 +0530
@@ -82,6 +82,48 @@ int tap_open(char *ifname, int ifname_si
return fd;
}
+int vtap_open(const char *devname, int *vnet_hdr, int vnet_hdr_required)
+{
+ struct ifreq ifr;
+ int fd, ret;
+
+ TFR(fd = open(devname, O_RDWR));
+ if (fd < 0) {
+ fprintf(stderr, "warning: could not open %s: no virtual network emulation\n", devname);
+ return -1;
+ }
+ memset(&ifr, 0, sizeof(ifr));
+ ifr.ifr_flags = IFF_TAP | IFF_NO_PI;
+
+ if (*vnet_hdr) {
+ unsigned int features;
+
+ if (ioctl(fd, TUNGETFEATURES, &features) == 0 &&
+ features & IFF_VNET_HDR) {
+ *vnet_hdr = 1;
+ ifr.ifr_flags |= IFF_VNET_HDR;
+ } else {
+ *vnet_hdr = 0;
+ }
+
+ if (vnet_hdr_required && !*vnet_hdr) {
+ error_report("vnet_hdr=1 requested, but no kernel "
+ "support for IFF_VNET_HDR available");
+ close(fd);
+ return -1;
+ }
+ }
+
+ ret = ioctl(fd, TUNSETIFF, (void *) &ifr);
+ if (ret != 0) {
+ fprintf(stderr, "warning: could not configure %s: no virtual network emulation\n", devname);
+ close(fd);
+ return -1;
+ }
+ fcntl(fd, F_SETFL, O_NONBLOCK);
+ return fd;
+}
+
/* sndbuf implements a kind of flow control for tap.
* Unfortunately when it's enabled, and packets are sent
* to other guests on the same host, the receiver
diff -ruNp org/net.c new/net.c
--- org/net.c 2011-01-10 10:19:13.000000000 +0530
+++ new/net.c 2011-03-09 10:14:10.000000000 +0530
@@ -855,6 +855,16 @@ static int net_init_nic(QemuOpts *opts,
return -1;
}
+ if (nd->netdev->numtxqs > 1 && nd->nvectors == DEV_NVECTORS_UNSPECIFIED) {
+ /*
+ * User specified mq for guest, but no "vectors=", tune
+ * it automatically to 'numtxqs' TX + 'numtxqs' RX + 1 controlq.
+ */
+ nd->nvectors = nd->netdev->numtxqs * 2 + 1;
+ monitor_printf(mon, "nvectors tuned to %d\n", nd->nvectors);
+ }
+
+
nd->used = 1;
nb_nics++;
@@ -998,6 +1008,18 @@ static const struct {
},
#ifndef _WIN32
{
+ .name = "vtap",
+ .type = QEMU_OPT_STRING,
+ .help = "name of macvtap device to use",
+ }, {
+ .name = "mq",
+ .type = QEMU_OPT_BOOL,
+ .help = "enable multiqueue on network i/f",
+ }, {
+ .name = "numtxqs",
+ .type = QEMU_OPT_NUMBER,
+ .help = "optional number of RX/TX queues, if mq is enabled",
+ }, {
.name = "fd",
.type = QEMU_OPT_STRING,
.help = "file descriptor of an already opened tap",
diff -ruNp org/net.h new/net.h
--- org/net.h 2011-01-10 10:19:13.000000000 +0530
+++ new/net.h 2011-03-09 10:14:57.000000000 +0530
@@ -64,6 +64,7 @@ struct VLANClientState {
struct VLANState *vlan;
VLANClientState *peer;
NetQueue *send_queue;
+ int numtxqs;
char *model;
char *name;
char info_str[256];
^ permalink raw reply
* Re: bonding can't change to another slave if you ifdown the active slave
From: Américo Wang @ 2011-03-09 6:02 UTC (permalink / raw)
To: Weiping Pan; +Cc: WANG Cong, netdev
In-Reply-To: <4D76E8A4.6080801@gmail.com>
On Wed, Mar 09, 2011 at 10:40:36AM +0800, Weiping Pan wrote:
<...>
>[root@localhost ~]# ifconfig
>bond0 Link encap:Ethernet HWaddr 08:00:27:78:F6:41
> inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0
> inet6 addr: fe80::a00:27ff:fe78:f641/64 Scope:Link
> UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
> RX packets:40 errors:0 dropped:0 overruns:0 frame:0
> TX packets:25 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:19632 (19.1 KiB) TX bytes:4070 (3.9 KiB)
>
>eth6 Link encap:Ethernet HWaddr 08:00:27:78:F6:41
> inet addr:192.168.56.101 Bcast:192.168.56.255
>Mask:255.255.255.0
> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
> RX packets:21 errors:0 dropped:0 overruns:0 frame:0
> TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:10158 (9.9 KiB) TX bytes:1802 (1.7 KiB)
>
>eth7 Link encap:Ethernet HWaddr 08:00:27:78:F6:41
> inet addr:192.168.56.101 Bcast:192.168.56.255
>Mask:255.255.255.0
> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
> RX packets:19 errors:0 dropped:0 overruns:0 frame:0
> TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:9474 (9.2 KiB) TX bytes:2268 (2.2 KiB)
It's weird that you bond two nic's into another different subnet.
>
>[root@localhost ~]# ping 192.168.1.100 -c 5
>PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data.
>64 bytes from 192.168.1.100: icmp_req=1 ttl=64 time=3.69 ms
>64 bytes from 192.168.1.100: icmp_req=2 ttl=64 time=0.242 ms
>64 bytes from 192.168.1.100: icmp_req=3 ttl=64 time=0.154 ms
>64 bytes from 192.168.1.100: icmp_req=4 ttl=64 time=0.270 ms
>64 bytes from 192.168.1.100: icmp_req=5 ttl=64 time=0.278 ms
>
What is your route table in your guest?
>[root@localhost ~]# ip link show
>1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>2: eth7: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc
>pfifo_fast master bond0 state UP qlen 1000
> link/ether 08:00:27:78:f6:41 brd ff:ff:ff:ff:ff:ff
>3: eth6: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc
>pfifo_fast master bond0 state DOWN qlen 1000
Clearly this equals to pulling off the cable from eth6,
which is your first slave.
^ permalink raw reply
* Re: [PATCH 1/2] net: Allow no-cache copy from user on transmit
From: Stephen Hemminger @ 2011-03-09 6:12 UTC (permalink / raw)
To: Tom Herbert; +Cc: davem, netdev
In-Reply-To: <alpine.DEB.2.00.1103082053350.1433@pokey.mtv.corp.google.com>
On Tue, 8 Mar 2011 21:05:06 -0800 (PST)
Tom Herbert <therbert@google.com> wrote:
> This patch uses __copy_from_user_nocache (from skb_copy_to_page)
> on transmit to bypass data cache for a performance improvement.
> This functionality is configurable per device using ethtool, the
> device must also be doing TX csum offload to enable. It seems
> reasonable to set this when the device does not copy or
> otherwise touch the data.
>
> This patch was tested using 200 instances of netperf TCP_RR with
> 1400 byte request and one byte reply. Platform is 16 core AMD.
>
> No-cache copy disabled:
> 672703 tps, 97.13% client utilization
> 50/90/99% latency 244.31 484.205 1028.41
>
> No-cache copy enabled:
> 702113 tps, 96.16% client utilization,
> 50/90/99% latency 238.56 467.56 956.955
>
> This seems to provide a nice little performance improvement and is
> consistent in the tests I ran. Presumably, this would provide
> the greatest benfits in the presence of an application workload
> stressing the cache and a lot of transmit data happening. I don't
> yet see a downside to using this.
>
> Signed-off-by: Tom Herbert <therbert@google.com>
Since this is not an ethernet only optimization or device
specific, I would prefer it be a property of device not
something in ethtool.
^ permalink raw reply
* Re: linux-next: build failure after merge of the net tree
From: David Miller @ 2011-03-09 6:45 UTC (permalink / raw)
To: sfr
Cc: netdev, linux-next, linux-kernel, emil.s.tantilov, stephen.s.ko,
jeffrey.t.kirsher
In-Reply-To: <20110309134234.32cd09b7.sfr@canb.auug.org.au>
From: Stephen Rothwell <sfr@canb.auug.org.au>
Date: Wed, 9 Mar 2011 13:42:34 +1100
> Hi all,
>
> After merging the net tree, today's linux-next build (x86_64 allmodconfig)
> failed like this:
>
> drivers/net/ixgbe/ixgbe_mbx.c: In function 'ixgbe_init_mbx_params_pf':
> drivers/net/ixgbe/ixgbe_mbx.c:456: error: 'struct ixgbe_mbx_info' has no member named 'udelay'
>
> Caused by commit d7c8a29fc8bd20ba45ec2f52577ed04a988a9500 ("ixgbe:
> improve logic in ixgbe_init_mbx_params_pf").
>
> I am wondering if that was ever built ...
>
> I have used the net tree from next-20110308 for today.
Probably two minutes after you pulled this a fix went in:
commit 5217e8794619ac0a29151f29be20c7d6188303ba
Author: Andy Gospodarek <andy@greyhouse.net>
Date: Tue Mar 8 14:26:00 2011 -0800
ixgbe: fix compile failure in ixgbe_init_mbx_params_pf
This commit:
commit d7c8a29fc8bd20ba45ec2f52577ed04a988a9500
Author: Emil Tantilov <emil.s.tantilov@intel.com>
Date: Thu Mar 3 09:25:02 2011 +0000
ixgbe: improve logic in ixgbe_init_mbx_params_pf
incorrectly added a line that accessed mbx->udelay. I'm sure the intent
was mbx->usec_delay. This patch fixes the compilation error.
Signed-off-by: Andy Gospodarek <andy@greyhouse.net>
Signed-off-by: David S. Miller <davem@davemloft.net>
diff --git a/drivers/net/ixgbe/ixgbe_mbx.c b/drivers/net/ixgbe/ixgbe_mbx.c
index 3cf8aec..c7ed82e 100644
--- a/drivers/net/ixgbe/ixgbe_mbx.c
+++ b/drivers/net/ixgbe/ixgbe_mbx.c
@@ -453,7 +453,7 @@ void ixgbe_init_mbx_params_pf(struct ixgbe_hw *hw)
return;
mbx->timeout = 0;
- mbx->udelay = 0;
+ mbx->usec_delay = 0;
mbx->stats.msgs_tx = 0;
mbx->stats.msgs_rx = 0;
^ permalink raw reply related
* Re: [PATCH] Make CUBIC Hystart more robust to RTT variations
From: Lucas Nussbaum @ 2011-03-09 6:53 UTC (permalink / raw)
To: Injong Rhee
Cc: Stephen Hemminger, David Miller, xiyou.wangcong, netdev,
sangtae.ha
In-Reply-To: <4D76D851.4050600@ncsu.edu>
On 08/03/11 at 20:30 -0500, Injong Rhee wrote:
> Now, both tools can be wrong. But that is not catastrophic since
> congestion avoidance can kick in to save the day. In a pipe where no
> other flows are competing, then exiting slow start too early can
> slow things down as the window can be still too small. But that is
> in fact when delays are most reliable. So those tests that say bad
> performance with hystart are in fact, where hystart is supposed to
> perform well.
Hi,
In my setup, there is no congestion at all (except the buffer bloat).
Without Hystart, transferring 8 Gb of data takes 9s, with CUBIC exiting
slow start at ~2000 packets.
With Hystart, transferring 8 Gb of data takes 19s, with CUBIC exiting
slow start at ~20 packets.
I don't think that this is "hystart performing well". We could just as
well remove slow start completely, and only do congestion avoidance,
then.
While I see the value in Hystart, it's clear that there are some flaws
in the current implementation. It probably makes sense to disable
hystart by default until those problems are fixed.
--
| Lucas Nussbaum MCF Université Nancy 2 |
| lucas.nussbaum@loria.fr LORIA / AlGorille |
| http://www.loria.fr/~lnussbau/ +33 3 54 95 86 19 |
^ permalink raw reply
* Re: Network performance with small packets - continued
From: Michael S. Tsirkin @ 2011-03-09 7:15 UTC (permalink / raw)
To: Shirley Ma, Rusty Russell, Michael S. Tsirkin, Krishna Kumar2,
David Miller
In-Reply-To: <201103071631.41964.tahm@linux.vnet.ibm.com>
On Mon, Mar 07, 2011 at 04:31:41PM -0600, Tom Lendacky wrote:
> We've been doing some more experimenting with the small packet network
> performance problem in KVM. I have a different setup than what Steve D. was
> using so I re-baselined things on the kvm.git kernel on both the host and
> guest with a 10GbE adapter. I also made use of the virtio-stats patch.
>
> The virtual machine has 2 vCPUs, 8GB of memory and two virtio network adapters
> (the first connected to a 1GbE adapter and a LAN, the second connected to a
> 10GbE adapter that is direct connected to another system with the same 10GbE
> adapter) running the kvm.git kernel. The test was a TCP_RR test with 100
> connections from a baremetal client to the KVM guest using a 256 byte message
> size in both directions.
>
> I used the uperf tool to do this after verifying the results against netperf.
> Uperf allows the specification of the number of connections as a parameter in
> an XML file as opposed to launching, in this case, 100 separate instances of
> netperf.
>
> Here is the baseline for baremetal using 2 physical CPUs:
> Txn Rate: 206,389.59 Txn/Sec, Pkt Rate: 410,048 Pkts/Sec
> TxCPU: 7.88% RxCPU: 99.41%
>
> To be sure to get consistent results with KVM I disabled the hyperthreads,
> pinned the qemu-kvm process, vCPUs, vhost thread and ethernet adapter
> interrupts (this resulted in runs that differed by only about 2% from lowest
> to highest). The fact that pinning is required to get consistent results is a
> different problem that we'll have to look into later...
>
> Here is the KVM baseline (average of six runs):
> Txn Rate: 87,070.34 Txn/Sec, Pkt Rate: 172,992 Pkts/Sec
> Exits: 148,444.58 Exits/Sec
> TxCPU: 2.40% RxCPU: 99.35%
> About 42% of baremetal.
>
Can you add interrupt stats as well please?
> empty. So I coded a quick patch to delay freeing of the used Tx buffers until
> more than half the ring was used (I did not test this under a stream condition
> so I don't know if this would have a negative impact). Here are the results
> from delaying the freeing of used Tx buffers (average of six runs):
> Txn Rate: 90,886.19 Txn/Sec, Pkt Rate: 180,571 Pkts/Sec
> Exits: 142,681.67 Exits/Sec
> TxCPU: 2.78% RxCPU: 99.36%
> About a 4% increase over baseline and about 44% of baremetal.
Hmm, I am not sure what you mean by delaying freeing.
I think we do have a problem that free_old_xmit_skbs
tries to flush out the ring aggressively:
it always polls until the ring is empty,
so there could be bursts of activity where
we spend a lot of time flushing the old entries
before e.g. sending an ack, resulting in
latency bursts.
Generally we'll need some smarter logic,
but with indirect at the moment we can just poll
a single packet after we post a new one, and be done with it.
Is your patch something like the patch below?
Could you try mine as well please?
> This spread out the kick_notify but still resulted in alot of them. I decided
> to build on the delayed Tx buffer freeing and code up an "ethtool" like
> coalescing patch in order to delay the kick_notify until there were at least 5
> packets on the ring or 2000 usecs, whichever occurred first. Here are the
> results of delaying the kick_notify (average of six runs):
> Txn Rate: 107,106.36 Txn/Sec, Pkt Rate: 212,796 Pkts/Sec
> Exits: 102,587.28 Exits/Sec
> TxCPU: 3.03% RxCPU: 99.33%
> About a 23% increase over baseline and about 52% of baremetal.
>
> Running the perf command against the guest I noticed almost 19% of the time
> being spent in _raw_spin_lock. Enabling lockstat in the guest showed alot of
> contention in the "irq_desc_lock_class". Pinning the virtio1-input interrupt
> to a single cpu in the guest and re-running the last test resulted in
> tremendous gains (average of six runs):
> Txn Rate: 153,696.59 Txn/Sec, Pkt Rate: 305,358 Pkgs/Sec
> Exits: 62,603.37 Exits/Sec
> TxCPU: 3.73% RxCPU: 98.52%
> About a 77% increase over baseline and about 74% of baremetal.
>
> Vhost is receiving a lot of notifications for packets that are to be
> transmitted (over 60% of the packets generate a kick_notify). Also, it looks
> like vhost is sending a lot of notifications for packets it has received
> before the guest can get scheduled to disable notifications and begin
> processing the packets
Hmm, is this really what happens to you? The effect would be that guest
gets an interrupt while notifications are disabled in guest, right? Could
you add a counter and check this please?
Another possible thing to try would be these old patches to publish used index
from guest to make sure this double interrupt does not happen:
[PATCHv2] virtio: put last seen used index into ring itself
[PATCHv2] vhost-net: utilize PUBLISH_USED_IDX feature
> resulting in some lock contention in the guest (and
> high interrupt rates).
>
> Some thoughts for the transmit path... can vhost be enhanced to do some
> adaptive polling so that the number of kick_notify events are reduced and
> replaced by kick_no_notify events?
Worth a try.
>
> Comparing the transmit path to the receive path, the guest disables
> notifications after the first kick and vhost re-enables notifications after
> completing processing of the tx ring.
Is this really what happens? I though the host disables notifications
after the first kick.
> Can a similar thing be done for the
> receive path? Once vhost sends the first notification for a received packet
> it can disable notifications and let the guest re-enable notifications when it
> has finished processing the receive ring. Also, can the virtio-net driver do
> some adaptive polling (or does napi take care of that for the guest)?
Worth a try. I don't think napi does anything like this.
> Running the same workload on the same configuration with a different
> hypervisor results in performance that is almost equivalent to baremetal
> without doing any pinning.
>
> Thanks,
> Tom Lendacky
There's no need to flush out all used buffers
before we post more for transmit: with indirect,
just a single one is enough. Without indirect we'll
need more possibly, but just for testing this should
be enough.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
Note: untested.
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 82dba5a..ebe3337 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -514,11 +514,11 @@ static unsigned int free_old_xmit_skbs(struct virtnet_info *vi)
struct sk_buff *skb;
unsigned int len, tot_sgs = 0;
- while ((skb = virtqueue_get_buf(vi->svq, &len)) != NULL) {
+ if ((skb = virtqueue_get_buf(vi->svq, &len)) != NULL) {
pr_debug("Sent skb %p\n", skb);
vi->dev->stats.tx_bytes += skb->len;
vi->dev->stats.tx_packets++;
- tot_sgs += skb_vnet_hdr(skb)->num_sg;
+ tot_sgs = 2+MAX_SKB_FRAGS;
dev_kfree_skb_any(skb);
}
return tot_sgs;
@@ -576,9 +576,6 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev)
struct virtnet_info *vi = netdev_priv(dev);
int capacity;
- /* Free up any pending old buffers before queueing new ones. */
- free_old_xmit_skbs(vi);
-
/* Try to transmit */
capacity = xmit_skb(vi, skb);
@@ -605,6 +602,10 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev)
skb_orphan(skb);
nf_reset(skb);
+ /* Free up any old buffers so we can queue new ones. */
+ if (capacity < 2+MAX_SKB_FRAGS)
+ capacity += free_old_xmit_skbs(vi);
+
/* Apparently nice girls don't return TX_BUSY; stop the queue
* before it gets out of hand. Naturally, this wastes entries. */
if (capacity < 2+MAX_SKB_FRAGS) {
^ permalink raw reply related
* Re: [patch net-next-2.6] net: reinject arps into bonding slave instead of master
From: Jiri Pirko @ 2011-03-09 7:45 UTC (permalink / raw)
To: Nicolas de Pesloüan
Cc: Andy Gospodarek, netdev, davem, shemminger, kaber, fubar,
eric.dumazet
In-Reply-To: <4D76A345.9040200@gmail.com>
Tue, Mar 08, 2011 at 10:44:37PM CET, nicolas.2p.debian@gmail.com wrote:
>Le 08/03/2011 14:42, Andy Gospodarek a écrit :
>>I'm pretty sure this patch will have the same catastrophic problem your
>>last one did. By cloning and setting skb2->dev = orig_dev you just
>>inserted a frame identical to the one we received right back into the
>>stack. It only took a few minutes for my box to melt as one frame on
>>the wire will cause an infinite number of frames to be received by the
>>stack.
>
>I agree with Andy. We still keep one reinject (netif_rx), which is
>probably better that two (__netif_receive_skb), but not enough.
>
>I really think we need a general framework for late delivery of final
>packets to packet handler registered somewhere in the rx_handler
>path.
>
>Jiri, is this patch the one you announced as "I have some kind nice
>solution in mind and I'm going to submit that as a patch later (too
>many patches are in the wind atm)" ?
I did not had time to verify my thought yet but I think that the only
think needed against my original patch (bonding: move processing of recv
handlers into handle_frame()) is ro remove vlan_on_bond_hook, period.
Because all incoming arps are seen by bond_handle_frame =>
bond->recv_probe , even vlan ones - that would make eth0-bond0-bond0.5
work and eth0-bond0-br0-bond0.5 as well. But again, need to verify this.
Jirka
^ permalink raw reply
* [PATCHv4 net-next 0/8] Phonet: pipe protocol cleanup
From: Rémi Denis-Courmont @ 2011-03-09 8:43 UTC (permalink / raw)
To: netdev
Hello,
As "promised", here is the cleaned up and unified Phonet pipe code.
The same kernel can now support both old (e.g. Nokia N900) and new
(e.g. ST-Ericsson U8500 SoC) modems.
The following changes since commit 7b46ac4e77f3224a1befe032c77f1df31d1b42c4:
inetpeer: Don't disable BH for initial fast RCU lookup. (2011-03-08 14:59:28 -0800)
are available in the git repository at:
git://git.remlab.net/linux-phonet.git master
Rémi Denis-Courmont (8):
Phonet: fix NULL dereference on TX path with implicit source
Phonet: return an error when packet TX fails
Phonet: correct pipe backlog callback return values
Phonet: factor common code to send control messages
Phonet: allocate sock from accept syscall rather than soft IRQ
Phonet: provide pipe socket option to retrieve the pipe identifier
Phonet: support active connection without pipe controller on modem
Phonet: kill the ST-Ericsson pipe controller Kconfig
Documentation/networking/phonet.txt | 67 ++---
include/linux/phonet.h | 4 +-
include/net/phonet/pep.h | 1 -
net/phonet/Kconfig | 12 -
net/phonet/af_phonet.c | 13 +-
net/phonet/pep.c | 682 ++++++++++++++---------------------
net/phonet/socket.c | 112 +++----
7 files changed, 352 insertions(+), 539 deletions(-)
--
Rémi Denis-Courmont
http://www.remlab.net/
^ permalink raw reply
* [PATCH 1/8] Phonet: fix NULL dereference on TX path with implicit source
From: Rémi Denis-Courmont @ 2011-03-09 8:44 UTC (permalink / raw)
To: netdev
In-Reply-To: <201103091043.32391.remi.denis-courmont@nokia.com>
The previous Phonet patch series introduced per-socket implicit
destination (i.e. connect()). In that case, the destination
socket address is NULL in the transmit function.
However commit a8059512b120362b15424f152b2548fe8b11bd0c
("Phonet: implement per-socket destination/peer address")
is incomplete and would trigger a NULL dereference.
(Fortunately, the code is not in released kernel, and in fact
currently not reachable.)
Signed-off-by: Rémi Denis-Courmont <remi.denis-courmont@nokia.com>
---
net/phonet/af_phonet.c | 5 ++---
1 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/net/phonet/af_phonet.c b/net/phonet/af_phonet.c
index 30cc676..4706b77 100644
--- a/net/phonet/af_phonet.c
+++ b/net/phonet/af_phonet.c
@@ -262,10 +262,9 @@ int pn_skb_send(struct sock *sk, struct sk_buff *skb,
else if (phonet_address_lookup(net, daddr) == 0) {
dev = phonet_device_get(net);
skb->pkt_type = PACKET_LOOPBACK;
- } else if (pn_sockaddr_get_object(target) == 0) {
+ } else if (dst == 0) {
/* Resource routing (small race until phonet_rcv()) */
- struct sock *sk = pn_find_sock_by_res(net,
- target->spn_resource);
+ struct sock *sk = pn_find_sock_by_res(net, res);
if (sk) {
sock_put(sk);
dev = phonet_device_get(net);
--
1.7.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox