* [PATCH 0/2] ieee80211: Update generic definitions to latest specs - take #2
From: Gertjan van Wingerde @ 2005-06-03 20:31 UTC (permalink / raw)
To: netdev; +Cc: jgarzik
Hi,
Following patches update the definitions of the generic ieee80211 stack
to the latest versions
of the published 802.11x specification suite, and cleans up the long
list of defines.
The set of patches is a resubmittal of my earlier patch, with the
comments of Jiri Benc and Stephen
Hemminger fixed.
The patches need to be applied in order.
Thanks,
Gertjan van Wingerde
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: Ben Greear @ 2005-06-03 20:30 UTC (permalink / raw)
To: Mitch Williams
Cc: David S. Miller, hadi, john.ronciak, jdmason, shemminger, netdev,
Robert.Olsson, ganesh.venkatesan, jesse.brandeburg
In-Reply-To: <Pine.CYG.4.58.0506031202280.3344@mawilli1-desk2.amr.corp.intel.com>
Mitch Williams wrote:
>
> On Fri, 3 Jun 2005, David S. Miller wrote:
>
>
>>From: jamal <hadi@cyberus.ca>
>>Date: Fri, 03 Jun 2005 14:42:30 -0400
>>
>>
>>>When you reduce the weight, the system is spending less time in the
>>>softirq processing packets before softirq yields. If this gives more
>>>opportunity to your app to run, then the performance will go up.
>>>Is this what you are seeing?
>>
>>Jamal, this is my current theory as well, we hit the jiffies
>>check.
>
>
> Well, I hate to mess up your guys' theories, but the real reason is
> simpler: hardware receive resources, specifically descriptors and
> buffers.
>
> In a typical NAPI polling loop, the driver processes receive packets until
> it either hits the quota or runs out of packets. Then, at the end of the
> loop, it returns all of those now-free receive resources back to the
> hardware.
>
> With a heavy receive load, the hardware will run out of receive
> descriptors in the time it takes the driver/NAPI/stack to process 64
> packets. So it drops them on the floor. And, as we know, dropped packets
> are A Bad Thing.
If it can fill up more than 190 RX descriptors in the time it takes NAPI
to pull 64, then there is no possible way to not drop packets! How could
NAPI ever keep up if what you say is true?
> By reducing the driver weight, we cause the driver to give receive
> resources back to the hardware more often, which prevents dropped packets.
>
> As Ben Greer noticed, increasing the number of descriptors can help with
> this issue. But it really can't eliminate the problem -- once the ring
> is full, it doesn't matter how big it is, it's still full.
If you have 1024 rx descriptors, and the NAPI poll pulls off 64 at one
time, I do not see how pulling off 20 could be any more useful. Either way,
you have more than 900 other RX descriptors to be received.
Even if you only have the default of 256 the NIC should be able to continue
receiving packets with the other 190 or so descriptors while NAPI is doing
it's receive poll. If the buffers are often nearly used up, then the problem
is that the NAPI poll cannot pull the packets fast enough, and again, I do not
see how making it do more polls could make it able to pull packets from the
NIC more efficiently.
Maybe you could instrument the NAPI receive logic to
see if there is some horrible waste of CPU and/or time when it tries to pull
larger amounts of packets at once? A linear increase in work cannot explain
what you are describing.
> In my testing (Dual 2.8GHz Xeon, PCI-X bus, Gigabit network, 10 clients),
> I was able to completely eliminate dropped packets in most cases by
> reducing the driver weight down to about 20.
At least tell us what type of traffic you are using? TCP with MTU sized
packets, traffic-generator with 60 byte packets? Actual speed that you
are running (aggregate)? Full-duplex traffic, or mostly uni-directional?
packets-per-second you are receiving & transmitting when the drops occur?
On a dual 2.8Ghz xeon system with PCI-X bus, with a quad-port Intel pro/1000
NIC I can run about 950Mbps of traffic, bi-directional, on two ports at the
same time, and drop few or no packets. (MTU sized packets here).
This is using a modified version of pktgen, btw. So, if you are seeing
any amount of dropped pkts on a single NIC, especially if you are mostly
doing uni-directional traffic, then I think the problem might be elsewhere,
because the stock 2.6.11 and similar kernels can easily handle this amount
of network traffic.
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: David S. Miller @ 2005-06-03 20:30 UTC (permalink / raw)
To: Robert.Olsson
Cc: john.ronciak, jdmason, shemminger, hadi, mitch.a.williams, netdev,
ganesh.venkatesan, jesse.brandeburg
In-Reply-To: <17056.47835.583602.151291@robur.slu.se>
From: Robert Olsson <Robert.Olsson@data.slu.se>
Date: Fri, 3 Jun 2005 22:17:31 +0200
> It is possible that a lower weight forced your driver to disable interrupts
> and do packet reception w/o interrupts often this is more efficient as
> we get rid intr. latency etc.
>
> Again I think weight should only used for fairness and not control the
> threshold when to disable interrupts.
>
> You can test with a new policy in e1000_clean so you schedule for a new
> poll if work_done (any pkts received) or tx_cleaned is true.
I don't think this is it. What's happening is that E1000
pulls up to a full dev->quota of packets off the ring,
and _THEN_ goes back and does RX buffer replenishing.
It is very clear why E1000 runs out of RX descriptors with
this kind of policy.
I outlined a way to fix this in the E1000 driver in another
email.
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: David S. Miller @ 2005-06-03 20:29 UTC (permalink / raw)
To: mitch.a.williams
Cc: hadi, john.ronciak, jdmason, shemminger, netdev, Robert.Olsson,
ganesh.venkatesan, jesse.brandeburg
In-Reply-To: <20050603.132257.23013342.davem@davemloft.net>
From: "David S. Miller" <davem@davemloft.net>
Date: Fri, 03 Jun 2005 13:22:57 -0700 (PDT)
> This is why you should replenish RX packets _IN_ your
> RX packet receive processing, not via some tasklet
> or other seperate work processing context.
>
> No wonder I never see this on tg3.
Actually, the problem is slightly different.
E1000 processes the full QUOTA of RX packets,
_THEN_ replenishes with new RX buffers. No wonder
the chip runs out of RX descriptors.
You should replenish _AS_ you grab RX packets
off the receive queue, just as tg3 does. This
allows you to accomplish two things:
1) Keep up with the chip so that it does not starve,
regardless of dev->weight setting or system load.
2) Make intelligent decisions when RX buffer allocation
fails. When we look at a RX descriptor in tg3 we
never leave the descriptor empty.
If replacement RX buffer fails, we simply ignore the
RX packet we're looking at and give it back to the
chip. Every driver should implement this policy.
Drivers that do not do things this way run into all
kinds of RX ring chip starvation issues like the ones
you are seeing here.
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: Mitch Williams @ 2005-06-03 20:28 UTC (permalink / raw)
To: jamal
Cc: David S. Miller, Williams, Mitch A, Ronciak, John, jdmason,
shemminger, netdev, Robert.Olsson, Venkatesan, Ganesh,
Brandeburg, Jesse
In-Reply-To: <1117830205.6071.81.camel@localhost.localdomain>
On Fri, 3 Jun 2005, jamal wrote:
>
> Sorry, Its already there as Dave said in his email.
> Look for time_squeeze. Its the column i labeled XXXX below.
>
> -----
> $ cat /proc/net/softnet_stat
> 0000f938 00000000 XXXXXXX 00000000 00000000 00000000 00000000 00000000
> 00000000
> ------
I might not be able to get into the lab today (they keep making me do
work!), but I should be able to pop in Monday and take a look. Shouldn't
take too long.
Thanks,
Mitch
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: jamal @ 2005-06-03 20:23 UTC (permalink / raw)
To: David S. Miller
Cc: mitch.a.williams, john.ronciak, jdmason, shemminger, netdev,
Robert.Olsson, ganesh.venkatesan, jesse.brandeburg
In-Reply-To: <1117827650.6071.59.camel@localhost.localdomain>
On Fri, 2005-03-06 at 15:40 -0400, jamal wrote:
> On Fri, 2005-03-06 at 12:01 -0700, David S. Miller wrote:
>
> I think you are more than likely right. If we can instrument it Mitch
> could check it out. Mitch would you like to try something that will
> instrument this? I know i have seen this behavior but it was when i was
> playing with some system that had a real small HZ.
>
Sorry, Its already there as Dave said in his email.
Look for time_squeeze. Its the column i labeled XXXX below.
-----
$ cat /proc/net/softnet_stat
0000f938 00000000 XXXXXXX 00000000 00000000 00000000 00000000 00000000
00000000
------
cheers,
jamal
^ permalink raw reply
* Re: [PATCH]: Tigon3 new NAPI locking v2
From: Jeff Garzik @ 2005-06-03 20:23 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, mchan
In-Reply-To: <20050603.122558.88474819.davem@davemloft.net>
David S. Miller wrote:
> [TG3]: Eliminate all hw IRQ handler spinlocks.
>
> Move all driver spinlocks to be taken at sw IRQ
> context only.
>
> This fixes the skb_copy() we were doing with hw
> IRQs disabled (which is illegal and triggers a
> BUG() with HIGHMEM enabled). It also simplifies
> the locking all over the driver tremendously.
>
> We accomplish this feat by creating a special
> sequence to synchronize with the hw IRQ handler
> using a 2-bit atomic state.
>
> Signed-off-by: David S. Miller <davem@davemloft.net>
overall, pretty spiffy :)
As further work, I would like to see how much (alot? all?) of the timer
code could be moved into a workqueue, where we could kill the last of
the horrible-udelay loops in the driver. Particularly awful is
while (++tick < 195000) {
status = tg3_fiber_aneg_smachine(tp, &aninfo);
if (status == ANEG_DONE || status == ANEG_FAILED)
break;
udelay(1);
}
where you could freeze a uniprocess box (lock out everything but
interrupts) for over 1 second. IOW, the slower the phy, the more these
slow-path delays can affect the overall system.
This is a MINOR, low priority issue; but long delays are uglies that
should be fixed, if its relatively painless.
> +static void tg3_irq_quiesce(struct tg3 *tp)
> +{
> + BUG_ON(test_bit(TG3_IRQSTATE_SYNC, &tp->irq_state));
> +
> + set_bit(TG3_IRQSTATE_SYNC, &tp->irq_state);
> + smp_mb();
> + tw32(GRC_LOCAL_CTRL,
> + tp->grc_local_ctrl | GRC_LCLCTRL_SETINT);
> +
> + while (!test_bit(TG3_IRQSTATE_COMPLETE, &tp->irq_state)) {
> + u32 val = tr32(MAILBOX_INTERRUPT_0 + TG3_64BIT_REG_LOW);
> +
> + if (val == 0x00000001)
> + break;
> +
> + cpu_relax();
> + }
> +}
* This loop makes me nervous... If there's a fault on the PCI bus or
the hardware is unplugged, val will equal 0xffffffff.
* A few comments for normal humans like "force an interrupt" and "wait
for interrupt handler to complete" might be nice.
* a BUG_ON(if-interrupts-are-disabled) line might be nice
> +static inline int tg3_irq_sync(struct tg3 *tp)
> +{
> + if (test_bit(TG3_IRQSTATE_SYNC, &tp->irq_state)) {
> + set_bit(TG3_IRQSTATE_COMPLETE, &tp->irq_state);
> + return 1;
> + }
> + return 0;
> +}
> +
> +/* Fully shutdown all tg3 driver activity elsewhere in the system.
> + * If irq_sync is non-zero, then the IRQ handler must be synchronized
> + * with as well. Most of the time, this is not necessary except when
> + * shutting down the device.
> + */
> +static inline void tg3_full_lock(struct tg3 *tp, int irq_sync)
> +{
> + if (irq_sync)
> + tg3_irq_quiesce(tp);
> + spin_lock_bh(&tp->lock);
> + spin_lock(&tp->tx_lock);
> +}
Rather than an 'irq_sync' arg, my instinct would have been to create
tg3_full_lock() and tg3_full_lock_sync(). This makes the action -much-
more obvious to the reader, and since its inline doesn't cost anything
(compiler's optimizer even does a tiny bit less work my way).
Jeff
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: David S. Miller @ 2005-06-03 20:22 UTC (permalink / raw)
To: mitch.a.williams
Cc: hadi, john.ronciak, jdmason, shemminger, netdev, Robert.Olsson,
ganesh.venkatesan, jesse.brandeburg
In-Reply-To: <Pine.CYG.4.58.0506031202280.3344@mawilli1-desk2.amr.corp.intel.com>
From: Mitch Williams <mitch.a.williams@intel.com>
Date: Fri, 3 Jun 2005 12:28:10 -0700
> In a typical NAPI polling loop, the driver processes receive packets until
> it either hits the quota or runs out of packets. Then, at the end of the
> loop, it returns all of those now-free receive resources back to the
> hardware.
>
> With a heavy receive load, the hardware will run out of receive
> descriptors in the time it takes the driver/NAPI/stack to process 64
> packets. So it drops them on the floor. And, as we know, dropped packets
> are A Bad Thing.
This is why you should replenish RX packets _IN_ your
RX packet receive processing, not via some tasklet
or other seperate work processing context.
No wonder I never see this on tg3.
It is the only way to do this cleanly.
^ permalink raw reply
* RE: RFC: NAPI packet weighting patch
From: Robert Olsson @ 2005-06-03 20:17 UTC (permalink / raw)
To: Ronciak, John
Cc: Robert Olsson, David S. Miller, jdmason, shemminger, hadi,
Williams, Mitch A, netdev, Venkatesan, Ganesh, Brandeburg, Jesse
In-Reply-To: <468F3FDA28AA87429AD807992E22D07E0450BFE8@orsmsx408>
Ronciak, John writes:
> With the same system (fairly high end with nothing major running on it)
> we got rid of the dropped frames by just reducing the weight for 64. So
> the weight did have something to do with the dropped frames. Maybe
> other factors as well, but in static tests like this it sure looks like
> the 64 value is wrong is some cases.
It is possible that a lower weight forced your driver to disable interrupts
and do packet reception w/o interrupts often this is more efficient as
we get rid intr. latency etc.
Again I think weight should only used for fairness and not control the
threshold when to disable interrupts.
You can test with a new policy in e1000_clean so you schedule for a new
poll if work_done (any pkts received) or tx_cleaned is true.
Cheers.
--ro
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: jamal @ 2005-06-03 19:59 UTC (permalink / raw)
To: Mitch Williams
Cc: David S. Miller, john.ronciak, jdmason, shemminger, netdev,
Robert.Olsson, ganesh.venkatesan, jesse.brandeburg
In-Reply-To: <Pine.CYG.4.58.0506031202280.3344@mawilli1-desk2.amr.corp.intel.com>
On Fri, 2005-03-06 at 12:28 -0700, Mitch Williams wrote:
>
> On Fri, 3 Jun 2005, David S. Miller wrote:
>
> > From: jamal <hadi@cyberus.ca>
> > Date: Fri, 03 Jun 2005 14:42:30 -0400
> >
> > > When you reduce the weight, the system is spending less time in the
> > > softirq processing packets before softirq yields. If this gives more
> > > opportunity to your app to run, then the performance will go up.
> > > Is this what you are seeing?
> >
> > Jamal, this is my current theory as well, we hit the jiffies
> > check.
>
> Well, I hate to mess up your guys' theories, but the real reason is
> simpler: hardware receive resources, specifically descriptors and
> buffers.
>
> In a typical NAPI polling loop, the driver processes receive packets until
> it either hits the quota or runs out of packets. Then, at the end of the
> loop, it returns all of those now-free receive resources back to the
> hardware.
>
> With a heavy receive load, the hardware will run out of receive
> descriptors in the time it takes the driver/NAPI/stack to process 64
> packets. So it drops them on the floor. And, as we know, dropped packets
> are A Bad Thing.
>
> By reducing the driver weight, we cause the driver to give receive
> resources back to the hardware more often, which prevents dropped packets.
>
> As Ben Greer noticed, increasing the number of descriptors can help with
> this issue. But it really can't eliminate the problem -- once the ring
> is full, it doesn't matter how big it is, it's still full.
>
> In my testing (Dual 2.8GHz Xeon, PCI-X bus, Gigabit network, 10 clients),
> I was able to completely eliminate dropped packets in most cases by
> reducing the driver weight down to about 20.
>
> Now for some speculation:
>
What you said above is unfortunately also speculation ;->
But one that you could validate by putting proper hooks. As an example,
try to restore a descriptor every time you pick one - for an example of
this look at the sb1250 driver.
cheers,
jamal
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: Michael Chan @ 2005-06-03 19:49 UTC (permalink / raw)
To: David S. Miller
Cc: mitch.a.williams, hadi, john.ronciak, jdmason, shemminger, netdev,
Robert.Olsson, ganesh.venkatesan, jesse.brandeburg
In-Reply-To: <20050603.132922.63997492.davem@davemloft.net>
On Fri, 2005-06-03 at 13:29 -0700, David S. Miller wrote:
> E1000 processes the full QUOTA of RX packets,
> _THEN_ replenishes with new RX buffers. No wonder
> the chip runs out of RX descriptors.
>
> You should replenish _AS_ you grab RX packets
> off the receive queue, just as tg3 does.
Yes, in tg3, rx buffers are replenished and put back into the ring as
completed packets are taken off the ring. But we don't tell the chip
about these new buffers until we get to the end of the loop, potentially
after a full quota of packets. Doesn't this make the end result the same
as e1000?
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: jamal @ 2005-06-03 19:40 UTC (permalink / raw)
To: David S. Miller
Cc: mitch.a.williams, john.ronciak, jdmason, shemminger, netdev,
Robert.Olsson, ganesh.venkatesan, jesse.brandeburg
In-Reply-To: <20050603.120126.41874584.davem@davemloft.net>
On Fri, 2005-03-06 at 12:01 -0700, David S. Miller wrote:
> From: jamal <hadi@cyberus.ca>
> Date: Fri, 03 Jun 2005 14:42:30 -0400
>
> > When you reduce the weight, the system is spending less time in the
> > softirq processing packets before softirq yields. If this gives more
> > opportunity to your app to run, then the performance will go up.
> > Is this what you are seeing?
>
> Jamal, this is my current theory as well, we hit the jiffies
> check.
>
I think you are more than likely right. If we can instrument it Mitch
could check it out. Mitch would you like to try something that will
instrument this? I know i have seen this behavior but it was when i was
playing with some system that had a real small HZ.
> It it the only logical explanation I can come up with for the
> single adapter case.
>
> There are some ways we can mitigate this. Here is one idea
> off the top of my head.
>
> When the jiffies check is hit, lower the weight of the most recently
> polled device towards some minimum (perhaps divide by two). If we
> successfully poll without hitting the jiffies check, make a small
> increment of the weight up to some limit.
>
You probably wanna start high up first until you hit congestion
and then start lowering.
> It is Van Jacobson TCP congestion avoidance applied to NAPI :-)
>
> Just a simple AIMD (Additive Increase, Multiplicative Decrease).
> So, hitting the jiffies work limit is congestion, and the cause
> of the congestion is the most recently polled device.
>
> In this regime, what the driver currently specifies as "->weight"
> is actually the maximum we'll use in the congestion control
> algorithm. And we can choose some constant minimum, something
> like "8" ought to work well.
>
> Comments?
>
In theory it looks good - but i think you end up defeating the fairness
factor. If you can narrow it down to which driver is causing congestion,
and only penalize that driver i think it would work well.
cheers,
jamal
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: Mitch Williams @ 2005-06-03 19:28 UTC (permalink / raw)
To: David S. Miller
Cc: hadi, mitch.a.williams, john.ronciak, jdmason, shemminger, netdev,
Robert.Olsson, ganesh.venkatesan, jesse.brandeburg
In-Reply-To: <20050603.120126.41874584.davem@davemloft.net>
On Fri, 3 Jun 2005, David S. Miller wrote:
> From: jamal <hadi@cyberus.ca>
> Date: Fri, 03 Jun 2005 14:42:30 -0400
>
> > When you reduce the weight, the system is spending less time in the
> > softirq processing packets before softirq yields. If this gives more
> > opportunity to your app to run, then the performance will go up.
> > Is this what you are seeing?
>
> Jamal, this is my current theory as well, we hit the jiffies
> check.
Well, I hate to mess up your guys' theories, but the real reason is
simpler: hardware receive resources, specifically descriptors and
buffers.
In a typical NAPI polling loop, the driver processes receive packets until
it either hits the quota or runs out of packets. Then, at the end of the
loop, it returns all of those now-free receive resources back to the
hardware.
With a heavy receive load, the hardware will run out of receive
descriptors in the time it takes the driver/NAPI/stack to process 64
packets. So it drops them on the floor. And, as we know, dropped packets
are A Bad Thing.
By reducing the driver weight, we cause the driver to give receive
resources back to the hardware more often, which prevents dropped packets.
As Ben Greer noticed, increasing the number of descriptors can help with
this issue. But it really can't eliminate the problem -- once the ring
is full, it doesn't matter how big it is, it's still full.
In my testing (Dual 2.8GHz Xeon, PCI-X bus, Gigabit network, 10 clients),
I was able to completely eliminate dropped packets in most cases by
reducing the driver weight down to about 20.
Now for some speculation:
Aside from dropped packets, I saw continued performance gain with even
lower weights, with the sweet spot (on a single adapter) being about 8 to
10. I don't have a definite answer for why this is happening, but my
theory is that it's latency. Packets are processed more often, meaning
they spend less time sitting in hardware-owned buffers, which means they
get to the stack quicker, which means less latency.
But I'm happy to admit I might be wrong with this theory. Nevertheless,
the effect exists, and I've seen it on drivers other than just e1000.
(And, no, I'm not allowed to say which other drivers I've used, or give
specific numbers, or our lawyers will string me up by my toes.)
Anybody else got a theory?
-Mitch
^ permalink raw reply
* [PATCH]: Tigon3 new NAPI locking v2
From: David S. Miller @ 2005-06-03 19:25 UTC (permalink / raw)
To: netdev; +Cc: mchan
This version incorporates two bug fixes from Michael.
1) Check the mailbox register for 0x1 while polling on the COMPLETE
state bit.
2) Remove the BUG_ON() check in tg3_restart_ints(), it can legally and
harmlessly occur.
Point #2 may want some refinements, but this patch below is good
enough for testing.
If someone (please please, pretty please) could be adventurous enough
to attempt this kind of change for e1000, that would be great.
Thanks.
[TG3]: Eliminate all hw IRQ handler spinlocks.
Move all driver spinlocks to be taken at sw IRQ
context only.
This fixes the skb_copy() we were doing with hw
IRQs disabled (which is illegal and triggers a
BUG() with HIGHMEM enabled). It also simplifies
the locking all over the driver tremendously.
We accomplish this feat by creating a special
sequence to synchronize with the hw IRQ handler
using a 2-bit atomic state.
Signed-off-by: David S. Miller <davem@davemloft.net>
--- 1/drivers/net/tg3.c.~1~ 2005-06-03 12:11:40.000000000 -0700
+++ 2/drivers/net/tg3.c 2005-06-03 12:15:34.000000000 -0700
@@ -337,12 +337,10 @@ static struct {
static void tg3_write_indirect_reg32(struct tg3 *tp, u32 off, u32 val)
{
if ((tp->tg3_flags & TG3_FLAG_PCIX_TARGET_HWBUG) != 0) {
- unsigned long flags;
-
- spin_lock_irqsave(&tp->indirect_lock, flags);
+ spin_lock_bh(&tp->indirect_lock);
pci_write_config_dword(tp->pdev, TG3PCI_REG_BASE_ADDR, off);
pci_write_config_dword(tp->pdev, TG3PCI_REG_DATA, val);
- spin_unlock_irqrestore(&tp->indirect_lock, flags);
+ spin_unlock_bh(&tp->indirect_lock);
} else {
writel(val, tp->regs + off);
if ((tp->tg3_flags & TG3_FLAG_5701_REG_WRITE_BUG) != 0)
@@ -353,12 +351,10 @@ static void tg3_write_indirect_reg32(str
static void _tw32_flush(struct tg3 *tp, u32 off, u32 val)
{
if ((tp->tg3_flags & TG3_FLAG_PCIX_TARGET_HWBUG) != 0) {
- unsigned long flags;
-
- spin_lock_irqsave(&tp->indirect_lock, flags);
+ spin_lock_bh(&tp->indirect_lock);
pci_write_config_dword(tp->pdev, TG3PCI_REG_BASE_ADDR, off);
pci_write_config_dword(tp->pdev, TG3PCI_REG_DATA, val);
- spin_unlock_irqrestore(&tp->indirect_lock, flags);
+ spin_unlock_bh(&tp->indirect_lock);
} else {
void __iomem *dest = tp->regs + off;
writel(val, dest);
@@ -398,28 +394,24 @@ static inline void _tw32_tx_mbox(struct
static void tg3_write_mem(struct tg3 *tp, u32 off, u32 val)
{
- unsigned long flags;
-
- spin_lock_irqsave(&tp->indirect_lock, flags);
+ spin_lock_bh(&tp->indirect_lock);
pci_write_config_dword(tp->pdev, TG3PCI_MEM_WIN_BASE_ADDR, off);
pci_write_config_dword(tp->pdev, TG3PCI_MEM_WIN_DATA, val);
/* Always leave this as zero. */
pci_write_config_dword(tp->pdev, TG3PCI_MEM_WIN_BASE_ADDR, 0);
- spin_unlock_irqrestore(&tp->indirect_lock, flags);
+ spin_unlock_bh(&tp->indirect_lock);
}
static void tg3_read_mem(struct tg3 *tp, u32 off, u32 *val)
{
- unsigned long flags;
-
- spin_lock_irqsave(&tp->indirect_lock, flags);
+ spin_lock_bh(&tp->indirect_lock);
pci_write_config_dword(tp->pdev, TG3PCI_MEM_WIN_BASE_ADDR, off);
pci_read_config_dword(tp->pdev, TG3PCI_MEM_WIN_DATA, val);
/* Always leave this as zero. */
pci_write_config_dword(tp->pdev, TG3PCI_MEM_WIN_BASE_ADDR, 0);
- spin_unlock_irqrestore(&tp->indirect_lock, flags);
+ spin_unlock_bh(&tp->indirect_lock);
}
static void tg3_disable_ints(struct tg3 *tp)
@@ -443,7 +435,7 @@ static void tg3_enable_ints(struct tg3 *
tw32_mailbox(MAILBOX_INTERRUPT_0 + TG3_64BIT_REG_LOW,
(tp->last_tag << 24));
tr32(MAILBOX_INTERRUPT_0 + TG3_64BIT_REG_LOW);
-
+ tp->irq_state = 0;
tg3_cond_int(tp);
}
@@ -2578,7 +2570,7 @@ static void tg3_tx(struct tg3 *tp)
sw_idx = NEXT_TX(sw_idx);
}
- dev_kfree_skb_irq(skb);
+ dev_kfree_skb(skb);
}
tp->tx_cons = sw_idx;
@@ -2884,11 +2876,8 @@ static int tg3_poll(struct net_device *n
{
struct tg3 *tp = netdev_priv(netdev);
struct tg3_hw_status *sblk = tp->hw_status;
- unsigned long flags;
int done;
- spin_lock_irqsave(&tp->lock, flags);
-
/* handle link change and other phy events */
if (!(tp->tg3_flags &
(TG3_FLAG_USE_LINKCHG_REG |
@@ -2896,7 +2885,9 @@ static int tg3_poll(struct net_device *n
if (sblk->status & SD_STATUS_LINK_CHG) {
sblk->status = SD_STATUS_UPDATED |
(sblk->status & ~SD_STATUS_LINK_CHG);
+ spin_lock(&tp->lock);
tg3_setup_phy(tp, 0);
+ spin_unlock(&tp->lock);
}
}
@@ -2907,8 +2898,6 @@ static int tg3_poll(struct net_device *n
spin_unlock(&tp->tx_lock);
}
- spin_unlock_irqrestore(&tp->lock, flags);
-
/* run RX thread, within the bounds set by NAPI.
* All RX "locking" is done by ensuring outside
* code synchronizes with dev->poll()
@@ -2933,15 +2922,62 @@ static int tg3_poll(struct net_device *n
/* if no more work, tell net stack and NIC we're done */
done = !tg3_has_work(tp);
if (done) {
- spin_lock_irqsave(&tp->lock, flags);
+ spin_lock(&tp->lock);
__netif_rx_complete(netdev);
tg3_restart_ints(tp);
- spin_unlock_irqrestore(&tp->lock, flags);
+ spin_unlock(&tp->lock);
}
return (done ? 0 : 1);
}
+static void tg3_irq_quiesce(struct tg3 *tp)
+{
+ BUG_ON(test_bit(TG3_IRQSTATE_SYNC, &tp->irq_state));
+
+ set_bit(TG3_IRQSTATE_SYNC, &tp->irq_state);
+ smp_mb();
+ tw32(GRC_LOCAL_CTRL,
+ tp->grc_local_ctrl | GRC_LCLCTRL_SETINT);
+
+ while (!test_bit(TG3_IRQSTATE_COMPLETE, &tp->irq_state)) {
+ u32 val = tr32(MAILBOX_INTERRUPT_0 + TG3_64BIT_REG_LOW);
+
+ if (val == 0x00000001)
+ break;
+
+ cpu_relax();
+ }
+}
+
+static inline int tg3_irq_sync(struct tg3 *tp)
+{
+ if (test_bit(TG3_IRQSTATE_SYNC, &tp->irq_state)) {
+ set_bit(TG3_IRQSTATE_COMPLETE, &tp->irq_state);
+ return 1;
+ }
+ return 0;
+}
+
+/* Fully shutdown all tg3 driver activity elsewhere in the system.
+ * If irq_sync is non-zero, then the IRQ handler must be synchronized
+ * with as well. Most of the time, this is not necessary except when
+ * shutting down the device.
+ */
+static inline void tg3_full_lock(struct tg3 *tp, int irq_sync)
+{
+ if (irq_sync)
+ tg3_irq_quiesce(tp);
+ spin_lock_bh(&tp->lock);
+ spin_lock(&tp->tx_lock);
+}
+
+static inline void tg3_full_unlock(struct tg3 *tp)
+{
+ spin_unlock(&tp->tx_lock);
+ spin_unlock_bh(&tp->lock);
+}
+
/* MSI ISR - No need to check for interrupt sharing and no need to
* flush status block and interrupt mailbox. PCI ordering rules
* guarantee that MSI will arrive after the status block.
@@ -2951,9 +2987,6 @@ static irqreturn_t tg3_msi(int irq, void
struct net_device *dev = dev_id;
struct tg3 *tp = netdev_priv(dev);
struct tg3_hw_status *sblk = tp->hw_status;
- unsigned long flags;
-
- spin_lock_irqsave(&tp->lock, flags);
/*
* Writing any value to intr-mbox-0 clears PCI INTA# and
@@ -2964,6 +2997,8 @@ static irqreturn_t tg3_msi(int irq, void
*/
tw32_mailbox(MAILBOX_INTERRUPT_0 + TG3_64BIT_REG_LOW, 0x00000001);
tp->last_tag = sblk->status_tag;
+ if (tg3_irq_sync(tp))
+ goto out;
sblk->status &= ~SD_STATUS_UPDATED;
if (likely(tg3_has_work(tp)))
netif_rx_schedule(dev); /* schedule NAPI poll */
@@ -2972,9 +3007,7 @@ static irqreturn_t tg3_msi(int irq, void
tw32_mailbox(MAILBOX_INTERRUPT_0 + TG3_64BIT_REG_LOW,
tp->last_tag << 24);
}
-
- spin_unlock_irqrestore(&tp->lock, flags);
-
+out:
return IRQ_RETVAL(1);
}
@@ -2983,11 +3016,8 @@ static irqreturn_t tg3_interrupt(int irq
struct net_device *dev = dev_id;
struct tg3 *tp = netdev_priv(dev);
struct tg3_hw_status *sblk = tp->hw_status;
- unsigned long flags;
unsigned int handled = 1;
- spin_lock_irqsave(&tp->lock, flags);
-
/* In INTx mode, it is possible for the interrupt to arrive at
* the CPU before the status block posted prior to the interrupt.
* Reading the PCI State register will confirm whether the
@@ -3004,6 +3034,8 @@ static irqreturn_t tg3_interrupt(int irq
*/
tw32_mailbox(MAILBOX_INTERRUPT_0 + TG3_64BIT_REG_LOW,
0x00000001);
+ if (tg3_irq_sync(tp))
+ goto out;
sblk->status &= ~SD_STATUS_UPDATED;
if (likely(tg3_has_work(tp)))
netif_rx_schedule(dev); /* schedule NAPI poll */
@@ -3018,9 +3050,7 @@ static irqreturn_t tg3_interrupt(int irq
} else { /* shared interrupt */
handled = 0;
}
-
- spin_unlock_irqrestore(&tp->lock, flags);
-
+out:
return IRQ_RETVAL(handled);
}
@@ -3029,11 +3059,8 @@ static irqreturn_t tg3_interrupt_tagged(
struct net_device *dev = dev_id;
struct tg3 *tp = netdev_priv(dev);
struct tg3_hw_status *sblk = tp->hw_status;
- unsigned long flags;
unsigned int handled = 1;
- spin_lock_irqsave(&tp->lock, flags);
-
/* In INTx mode, it is possible for the interrupt to arrive at
* the CPU before the status block posted prior to the interrupt.
* Reading the PCI State register will confirm whether the
@@ -3051,6 +3078,8 @@ static irqreturn_t tg3_interrupt_tagged(
tw32_mailbox(MAILBOX_INTERRUPT_0 + TG3_64BIT_REG_LOW,
0x00000001);
tp->last_tag = sblk->status_tag;
+ if (tg3_irq_sync(tp))
+ goto out;
sblk->status &= ~SD_STATUS_UPDATED;
if (likely(tg3_has_work(tp)))
netif_rx_schedule(dev); /* schedule NAPI poll */
@@ -3065,9 +3094,7 @@ static irqreturn_t tg3_interrupt_tagged(
} else { /* shared interrupt */
handled = 0;
}
-
- spin_unlock_irqrestore(&tp->lock, flags);
-
+out:
return IRQ_RETVAL(handled);
}
@@ -3106,8 +3133,7 @@ static void tg3_reset_task(void *_data)
tg3_netif_stop(tp);
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 1);
restart_timer = tp->tg3_flags2 & TG3_FLG2_RESTART_TIMER;
tp->tg3_flags2 &= ~TG3_FLG2_RESTART_TIMER;
@@ -3117,8 +3143,7 @@ static void tg3_reset_task(void *_data)
tg3_netif_start(tp);
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
if (restart_timer)
mod_timer(&tp->timer, jiffies + 1);
@@ -3224,39 +3249,21 @@ static int tg3_start_xmit(struct sk_buff
unsigned int i;
u32 len, entry, base_flags, mss;
int would_hit_hwbug;
- unsigned long flags;
len = skb_headlen(skb);
/* No BH disabling for tx_lock here. We are running in BH disabled
* context and TX reclaim runs via tp->poll inside of a software
- * interrupt. Rejoice!
- *
- * Actually, things are not so simple. If we are to take a hw
- * IRQ here, we can deadlock, consider:
- *
- * CPU1 CPU2
- * tg3_start_xmit
- * take tp->tx_lock
- * tg3_timer
- * take tp->lock
- * tg3_interrupt
- * spin on tp->lock
- * spin on tp->tx_lock
- *
- * So we really do need to disable interrupts when taking
- * tx_lock here.
+ * interrupt. Furthermore, IRQ processing runs lockless so we have
+ * no IRQ context deadlocks to worry about either. Rejoice!
*/
- local_irq_save(flags);
- if (!spin_trylock(&tp->tx_lock)) {
- local_irq_restore(flags);
+ if (!spin_trylock(&tp->tx_lock))
return NETDEV_TX_LOCKED;
- }
/* This is a hard error, log it. */
if (unlikely(TX_BUFFS_AVAIL(tp) <= (skb_shinfo(skb)->nr_frags + 1))) {
netif_stop_queue(dev);
- spin_unlock_irqrestore(&tp->tx_lock, flags);
+ spin_unlock(&tp->tx_lock);
printk(KERN_ERR PFX "%s: BUG! Tx Ring full when queue awake!\n",
dev->name);
return NETDEV_TX_BUSY;
@@ -3421,7 +3428,7 @@ static int tg3_start_xmit(struct sk_buff
out_unlock:
mmiowb();
- spin_unlock_irqrestore(&tp->tx_lock, flags);
+ spin_unlock(&tp->tx_lock);
dev->trans_start = jiffies;
@@ -3455,8 +3462,8 @@ static int tg3_change_mtu(struct net_dev
}
tg3_netif_stop(tp);
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+
+ tg3_full_lock(tp, 1);
tg3_halt(tp, RESET_KIND_SHUTDOWN, 1);
@@ -3466,8 +3473,7 @@ static int tg3_change_mtu(struct net_dev
tg3_netif_start(tp);
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
return 0;
}
@@ -5088,9 +5094,9 @@ static int tg3_set_mac_addr(struct net_d
memcpy(dev->dev_addr, addr->sa_data, dev->addr_len);
- spin_lock_irq(&tp->lock);
+ spin_lock_bh(&tp->lock);
__tg3_set_mac_addr(tp);
- spin_unlock_irq(&tp->lock);
+ spin_unlock_bh(&tp->lock);
return 0;
}
@@ -5802,10 +5808,8 @@ static void tg3_periodic_fetch_stats(str
static void tg3_timer(unsigned long __opaque)
{
struct tg3 *tp = (struct tg3 *) __opaque;
- unsigned long flags;
- spin_lock_irqsave(&tp->lock, flags);
- spin_lock(&tp->tx_lock);
+ spin_lock(&tp->lock);
if (!(tp->tg3_flags & TG3_FLAG_TAGGED_STATUS)) {
/* All of this garbage is because when using non-tagged
@@ -5822,8 +5826,7 @@ static void tg3_timer(unsigned long __op
if (!(tr32(WDMAC_MODE) & WDMAC_MODE_ENABLE)) {
tp->tg3_flags2 |= TG3_FLG2_RESTART_TIMER;
- spin_unlock(&tp->tx_lock);
- spin_unlock_irqrestore(&tp->lock, flags);
+ spin_unlock(&tp->lock);
schedule_work(&tp->reset_task);
return;
}
@@ -5891,8 +5894,7 @@ static void tg3_timer(unsigned long __op
tp->asf_counter = tp->asf_multiplier;
}
- spin_unlock(&tp->tx_lock);
- spin_unlock_irqrestore(&tp->lock, flags);
+ spin_unlock(&tp->lock);
tp->timer.expires = jiffies + tp->timer_offset;
add_timer(&tp->timer);
@@ -6007,14 +6009,12 @@ static int tg3_test_msi(struct tg3 *tp)
/* Need to reset the chip because the MSI cycle may have terminated
* with Master Abort.
*/
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 1);
tg3_halt(tp, RESET_KIND_SHUTDOWN, 1);
err = tg3_init_hw(tp);
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
if (err)
free_irq(tp->pdev->irq, dev);
@@ -6027,14 +6027,12 @@ static int tg3_open(struct net_device *d
struct tg3 *tp = netdev_priv(dev);
int err;
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 0);
tg3_disable_ints(tp);
tp->tg3_flags &= ~TG3_FLAG_INIT_COMPLETE;
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
/* The placement of this call is tied
* to the setup and use of Host TX descriptors.
@@ -6081,8 +6079,7 @@ static int tg3_open(struct net_device *d
return err;
}
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 0);
err = tg3_init_hw(tp);
if (err) {
@@ -6106,8 +6103,7 @@ static int tg3_open(struct net_device *d
tp->timer.function = tg3_timer;
}
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
if (err) {
free_irq(tp->pdev->irq, dev);
@@ -6123,8 +6119,7 @@ static int tg3_open(struct net_device *d
err = tg3_test_msi(tp);
if (err) {
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 0);
if (tp->tg3_flags2 & TG3_FLG2_USING_MSI) {
pci_disable_msi(tp->pdev);
@@ -6134,22 +6129,19 @@ static int tg3_open(struct net_device *d
tg3_free_rings(tp);
tg3_free_consistent(tp);
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
return err;
}
}
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 0);
add_timer(&tp->timer);
tp->tg3_flags |= TG3_FLAG_INIT_COMPLETE;
tg3_enable_ints(tp);
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
netif_start_queue(dev);
@@ -6395,8 +6387,7 @@ static int tg3_close(struct net_device *
del_timer_sync(&tp->timer);
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 1);
#if 0
tg3_dump_state(tp);
#endif
@@ -6410,8 +6401,7 @@ static int tg3_close(struct net_device *
TG3_FLAG_GOT_SERDES_FLOWCTL);
netif_carrier_off(tp->dev);
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
free_irq(tp->pdev->irq, dev);
if (tp->tg3_flags2 & TG3_FLG2_USING_MSI) {
@@ -6448,16 +6438,15 @@ static unsigned long calc_crc_errors(str
if (!(tp->tg3_flags2 & TG3_FLG2_PHY_SERDES) &&
(GET_ASIC_REV(tp->pci_chip_rev_id) == ASIC_REV_5700 ||
GET_ASIC_REV(tp->pci_chip_rev_id) == ASIC_REV_5701)) {
- unsigned long flags;
u32 val;
- spin_lock_irqsave(&tp->lock, flags);
+ spin_lock_bh(&tp->lock);
if (!tg3_readphy(tp, 0x1e, &val)) {
tg3_writephy(tp, 0x1e, val | 0x8000);
tg3_readphy(tp, 0x14, &val);
} else
val = 0;
- spin_unlock_irqrestore(&tp->lock, flags);
+ spin_unlock_bh(&tp->lock);
tp->phy_crc_errors += val;
@@ -6719,11 +6708,9 @@ static void tg3_set_rx_mode(struct net_d
{
struct tg3 *tp = netdev_priv(dev);
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 0);
__tg3_set_rx_mode(dev);
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
}
#define TG3_REGDUMP_LEN (32 * 1024)
@@ -6745,8 +6732,7 @@ static void tg3_get_regs(struct net_devi
memset(p, 0, TG3_REGDUMP_LEN);
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 0);
#define __GET_REG32(reg) (*(p)++ = tr32(reg))
#define GET_REG32_LOOP(base,len) \
@@ -6796,8 +6782,7 @@ do { p = (u32 *)(orig_p + (reg)); \
#undef GET_REG32_LOOP
#undef GET_REG32_1
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
}
static int tg3_get_eeprom_len(struct net_device *dev)
@@ -6973,8 +6958,7 @@ static int tg3_set_settings(struct net_d
return -EINVAL;
}
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 0);
tp->link_config.autoneg = cmd->autoneg;
if (cmd->autoneg == AUTONEG_ENABLE) {
@@ -6990,8 +6974,7 @@ static int tg3_set_settings(struct net_d
if (netif_running(dev))
tg3_setup_phy(tp, 1);
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
return 0;
}
@@ -7027,12 +7010,12 @@ static int tg3_set_wol(struct net_device
!(tp->tg3_flags & TG3_FLAG_SERDES_WOL_CAP))
return -EINVAL;
- spin_lock_irq(&tp->lock);
+ spin_lock_bh(&tp->lock);
if (wol->wolopts & WAKE_MAGIC)
tp->tg3_flags |= TG3_FLAG_WOL_ENABLE;
else
tp->tg3_flags &= ~TG3_FLAG_WOL_ENABLE;
- spin_unlock_irq(&tp->lock);
+ spin_unlock_bh(&tp->lock);
return 0;
}
@@ -7072,7 +7055,7 @@ static int tg3_nway_reset(struct net_dev
if (!netif_running(dev))
return -EAGAIN;
- spin_lock_irq(&tp->lock);
+ spin_lock_bh(&tp->lock);
r = -EINVAL;
tg3_readphy(tp, MII_BMCR, &bmcr);
if (!tg3_readphy(tp, MII_BMCR, &bmcr) &&
@@ -7080,7 +7063,7 @@ static int tg3_nway_reset(struct net_dev
tg3_writephy(tp, MII_BMCR, bmcr | BMCR_ANRESTART);
r = 0;
}
- spin_unlock_irq(&tp->lock);
+ spin_unlock_bh(&tp->lock);
return r;
}
@@ -7111,8 +7094,7 @@ static int tg3_set_ringparam(struct net_
if (netif_running(dev))
tg3_netif_stop(tp);
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 0);
tp->rx_pending = ering->rx_pending;
@@ -7128,8 +7110,7 @@ static int tg3_set_ringparam(struct net_
tg3_netif_start(tp);
}
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
return 0;
}
@@ -7150,8 +7131,8 @@ static int tg3_set_pauseparam(struct net
if (netif_running(dev))
tg3_netif_stop(tp);
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 1);
+
if (epause->autoneg)
tp->tg3_flags |= TG3_FLAG_PAUSE_AUTONEG;
else
@@ -7170,8 +7151,8 @@ static int tg3_set_pauseparam(struct net
tg3_init_hw(tp);
tg3_netif_start(tp);
}
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+
+ tg3_full_unlock(tp);
return 0;
}
@@ -7192,12 +7173,12 @@ static int tg3_set_rx_csum(struct net_de
return 0;
}
- spin_lock_irq(&tp->lock);
+ spin_lock_bh(&tp->lock);
if (data)
tp->tg3_flags |= TG3_FLAG_RX_CHECKSUMS;
else
tp->tg3_flags &= ~TG3_FLAG_RX_CHECKSUMS;
- spin_unlock_irq(&tp->lock);
+ spin_unlock_bh(&tp->lock);
return 0;
}
@@ -7719,8 +7700,7 @@ static void tg3_self_test(struct net_dev
if (netif_running(dev))
tg3_netif_stop(tp);
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 1);
tg3_halt(tp, RESET_KIND_SUSPEND, 1);
tg3_nvram_lock(tp);
@@ -7742,14 +7722,14 @@ static void tg3_self_test(struct net_dev
data[4] = 1;
}
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
+
if (tg3_test_interrupt(tp) != 0) {
etest->flags |= ETH_TEST_FL_FAILED;
data[5] = 1;
}
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+
+ tg3_full_lock(tp, 0);
tg3_halt(tp, RESET_KIND_SHUTDOWN, 1);
if (netif_running(dev)) {
@@ -7757,8 +7737,8 @@ static void tg3_self_test(struct net_dev
tg3_init_hw(tp);
tg3_netif_start(tp);
}
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+
+ tg3_full_unlock(tp);
}
}
@@ -7779,9 +7759,9 @@ static int tg3_ioctl(struct net_device *
if (tp->tg3_flags2 & TG3_FLG2_PHY_SERDES)
break; /* We have no PHY */
- spin_lock_irq(&tp->lock);
+ spin_lock_bh(&tp->lock);
err = tg3_readphy(tp, data->reg_num & 0x1f, &mii_regval);
- spin_unlock_irq(&tp->lock);
+ spin_unlock_bh(&tp->lock);
data->val_out = mii_regval;
@@ -7795,9 +7775,9 @@ static int tg3_ioctl(struct net_device *
if (!capable(CAP_NET_ADMIN))
return -EPERM;
- spin_lock_irq(&tp->lock);
+ spin_lock_bh(&tp->lock);
err = tg3_writephy(tp, data->reg_num & 0x1f, data->val_in);
- spin_unlock_irq(&tp->lock);
+ spin_unlock_bh(&tp->lock);
return err;
@@ -7813,28 +7793,24 @@ static void tg3_vlan_rx_register(struct
{
struct tg3 *tp = netdev_priv(dev);
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 0);
tp->vlgrp = grp;
/* Update RX_MODE_KEEP_VLAN_TAG bit in RX_MODE register. */
__tg3_set_rx_mode(dev);
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
}
static void tg3_vlan_rx_kill_vid(struct net_device *dev, unsigned short vid)
{
struct tg3 *tp = netdev_priv(dev);
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 0);
if (tp->vlgrp)
tp->vlgrp->vlan_devices[vid] = NULL;
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
}
#endif
@@ -10141,24 +10117,19 @@ static int tg3_suspend(struct pci_dev *p
del_timer_sync(&tp->timer);
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 1);
tg3_disable_ints(tp);
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
netif_device_detach(dev);
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 0);
tg3_halt(tp, RESET_KIND_SHUTDOWN, 1);
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
err = tg3_set_power_state(tp, pci_choose_state(pdev, state));
if (err) {
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 0);
tg3_init_hw(tp);
@@ -10168,8 +10139,7 @@ static int tg3_suspend(struct pci_dev *p
netif_device_attach(dev);
tg3_netif_start(tp);
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
}
return err;
@@ -10192,8 +10162,7 @@ static int tg3_resume(struct pci_dev *pd
netif_device_attach(dev);
- spin_lock_irq(&tp->lock);
- spin_lock(&tp->tx_lock);
+ tg3_full_lock(tp, 0);
tg3_init_hw(tp);
@@ -10204,8 +10173,7 @@ static int tg3_resume(struct pci_dev *pd
tg3_netif_start(tp);
- spin_unlock(&tp->tx_lock);
- spin_unlock_irq(&tp->lock);
+ tg3_full_unlock(tp);
return 0;
}
--- 1/drivers/net/tg3.h.~1~ 2005-06-03 12:11:44.000000000 -0700
+++ 2/drivers/net/tg3.h 2005-06-03 12:12:03.000000000 -0700
@@ -2006,17 +2006,33 @@ struct tg3_ethtool_stats {
struct tg3 {
/* begin "general, frequently-used members" cacheline section */
+ /* If the IRQ handler (which runs lockless) needs to be
+ * quiesced, the following bitmask state is used. The
+ * SYNC bit is set by non-IRQ context code to initiate
+ * the quiescence. The setter of this bit also forces
+ * an interrupt to run via the GRC misc host control
+ * register.
+ *
+ * The IRQ handler notes this, disables interrupts, and
+ * sets the COMPLETE bit. At this point the SYNC bit
+ * setter can be assured that interrupts will no longer
+ * get run.
+ *
+ * In this way all SMP driver locks are never acquired
+ * in hw IRQ context, only sw IRQ context or lower.
+ */
+ unsigned long irq_state;
+#define TG3_IRQSTATE_SYNC 0
+#define TG3_IRQSTATE_COMPLETE 1
+
/* SMP locking strategy:
*
* lock: Held during all operations except TX packet
* processing.
*
- * tx_lock: Held during tg3_start_xmit{,_4gbug} and tg3_tx
+ * tx_lock: Held during tg3_start_xmit and tg3_tx
*
- * If you want to shut up all asynchronous processing you must
- * acquire both locks, 'lock' taken before 'tx_lock'. IRQs must
- * be disabled to take 'lock' but only softirq disabling is
- * necessary for acquisition of 'tx_lock'.
+ * Both of these locks are to be held with BH safety.
*/
spinlock_t lock;
spinlock_t indirect_lock;
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: David S. Miller @ 2005-06-03 19:02 UTC (permalink / raw)
To: greearb
Cc: john.ronciak, Robert.Olsson, jdmason, shemminger, hadi,
mitch.a.williams, netdev, ganesh.venkatesan, jesse.brandeburg
In-Reply-To: <42A0A897.5080006@candelatech.com>
From: Ben Greear <greearb@candelatech.com>
Date: Fri, 03 Jun 2005 11:59:35 -0700
> David S. Miller wrote:
> > In a reply I just sent out to this thread, I postulate that the
> > jiffies check is hitting earlier with a lower weight value, a quick
> > look at /proc/net/softnet_stat during their testing will confirm or
> > deny this theory.
>
> That would basically just decrease the work done in the NAPI poll though,
> so I don't see how that could be the problem, since the 'solution' was to
> force less work to be done.
It allows his application to get onto the CPU faster.
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: David S. Miller @ 2005-06-03 19:01 UTC (permalink / raw)
To: hadi
Cc: mitch.a.williams, john.ronciak, jdmason, shemminger, netdev,
Robert.Olsson, ganesh.venkatesan, jesse.brandeburg
In-Reply-To: <1117824150.6071.34.camel@localhost.localdomain>
From: jamal <hadi@cyberus.ca>
Date: Fri, 03 Jun 2005 14:42:30 -0400
> When you reduce the weight, the system is spending less time in the
> softirq processing packets before softirq yields. If this gives more
> opportunity to your app to run, then the performance will go up.
> Is this what you are seeing?
Jamal, this is my current theory as well, we hit the jiffies
check.
It it the only logical explanation I can come up with for the
single adapter case.
There are some ways we can mitigate this. Here is one idea
off the top of my head.
When the jiffies check is hit, lower the weight of the most recently
polled device towards some minimum (perhaps divide by two). If we
successfully poll without hitting the jiffies check, make a small
increment of the weight up to some limit.
It is Van Jacobson TCP congestion avoidance applied to NAPI :-)
Just a simple AIMD (Additive Increase, Multiplicative Decrease).
So, hitting the jiffies work limit is congestion, and the cause
of the congestion is the most recently polled device.
In this regime, what the driver currently specifies as "->weight"
is actually the maximum we'll use in the congestion control
algorithm. And we can choose some constant minimum, something
like "8" ought to work well.
Comments?
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: Ben Greear @ 2005-06-03 18:59 UTC (permalink / raw)
To: David S. Miller
Cc: john.ronciak, Robert.Olsson, jdmason, shemminger, hadi,
mitch.a.williams, netdev, ganesh.venkatesan, jesse.brandeburg
In-Reply-To: <20050603.114950.119242486.davem@davemloft.net>
David S. Miller wrote:
> From: Ben Greear <greearb@candelatech.com>
>>Maybe the poll is disabling the IRQs on the NIC for too long, or something
>>like that?
>
>
> In a reply I just sent out to this thread, I postulate that the
> jiffies check is hitting earlier with a lower weight value, a quick
> look at /proc/net/softnet_stat during their testing will confirm or
> deny this theory.
That would basically just decrease the work done in the NAPI poll though,
so I don't see how that could be the problem, since the 'solution' was to
force less work to be done.
> It could also just be a simple bug in the dev->quota accounting
> somewhere.
>
> Note that, in all of this, I do not have any objections to providing
> a way to configure the dev->weight values. I will be applying Stephen
> Hemminger's patches.
Good. The more knobs the merrier, so long as they are at least somewhat
documented and default to good sane values :)
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: David S. Miller @ 2005-06-03 18:49 UTC (permalink / raw)
To: greearb
Cc: john.ronciak, Robert.Olsson, jdmason, shemminger, hadi,
mitch.a.williams, netdev, ganesh.venkatesan, jesse.brandeburg
In-Reply-To: <42A0A25C.8000503@candelatech.com>
From: Ben Greear <greearb@candelatech.com>
Date: Fri, 03 Jun 2005 11:33:00 -0700
> Is this implying that having the NAPI poll do less work per poll
> of the driver actually increases performance? I would have guessed that
> the opposite would be true.
Exactly my thoughts as well :)
> Maybe the poll is disabling the IRQs on the NIC for too long, or something
> like that?
In a reply I just sent out to this thread, I postulate that the
jiffies check is hitting earlier with a lower weight value, a quick
look at /proc/net/softnet_stat during their testing will confirm or
deny this theory.
It could also just be a simple bug in the dev->quota accounting
somewhere.
Note that, in all of this, I do not have any objections to providing
a way to configure the dev->weight values. I will be applying Stephen
Hemminger's patches.
But I think we MUST find out the reason for the observed behavior,
especially in the single-adapter case since the result is so illogical.
We could find an important bug in the NAPI implementation, or learn
something new about how NAPI behaves.
^ permalink raw reply
* [PATCH 2.6.12-rc5] bonding: documentation update
From: Jay Vosburgh @ 2005-06-03 18:48 UTC (permalink / raw)
To: netdev, bonding-devel
Documentation update: added some more configuration info,
(hopefully) better examples, updated some out of date info, and a bonus
pass through ispell to banish the "paramters."
-J
---
-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com
Signed-off-by: Jay Vosburgh <fubar@us.ibm.com>
diff -ur linux-2.6.12-rc5/Documentation/networking/bonding.txt linux-doc/Documentation/networking/bonding.txt
--- linux-2.6.12-rc5/Documentation/networking/bonding.txt 2005-06-03 11:29:04.394823672 -0700
+++ linux-doc/Documentation/networking/bonding.txt 2005-06-03 11:29:41.143237064 -0700
@@ -1,5 +1,7 @@
- Linux Ethernet Bonding Driver HOWTO
+ Linux Ethernet Bonding Driver HOWTO
+
+ Latest update: 2 June 2005
Initial release : Thomas Davis <tadavis at lbl.gov>
Corrections, HA extensions : 2000/10/03-15 :
@@ -11,15 +13,22 @@
Reorganized and updated Feb 2005 by Jay Vosburgh
-Note :
-------
+Introduction
+============
+
+ The Linux bonding driver provides a method for aggregating
+multiple network interfaces into a single logical "bonded" interface.
+The behavior of the bonded interfaces depends upon the mode; generally
+speaking, modes provide either hot standby or load balancing services.
+Additionally, link integrity monitoring may be performed.
-The bonding driver originally came from Donald Becker's beowulf patches for
-kernel 2.0. It has changed quite a bit since, and the original tools from
-extreme-linux and beowulf sites will not work with this version of the driver.
+ The bonding driver originally came from Donald Becker's
+beowulf patches for kernel 2.0. It has changed quite a bit since, and
+the original tools from extreme-linux and beowulf sites will not work
+with this version of the driver.
-For new versions of the driver, patches for older kernels and the updated
-userspace tools, please follow the links at the end of this file.
+ For new versions of the driver, updated userspace tools, and
+who to ask for help, please follow the links at the end of this file.
Table of Contents
=================
@@ -30,9 +39,13 @@
3. Configuring Bonding Devices
3.1 Configuration with sysconfig support
+3.1.1 Using DHCP with sysconfig
+3.1.2 Configuring Multiple Bonds with sysconfig
3.2 Configuration with initscripts support
+3.2.1 Using DHCP with initscripts
+3.2.2 Configuring Multiple Bonds with initscripts
3.3 Configuring Bonding Manually
-3.4 Configuring Multiple Bonds
+3.3.1 Configuring Multiple Bonds Manually
5. Querying Bonding Configuration
5.1 Bonding Configuration
@@ -56,21 +69,28 @@
11. Promiscuous mode
-12. High Availability Information
+12. Configuring Bonding for High Availability
12.1 High Availability in a Single Switch Topology
-12.1.1 Bonding Mode Selection for Single Switch Topology
-12.1.2 Link Monitoring for Single Switch Topology
12.2 High Availability in a Multiple Switch Topology
-12.2.1 Bonding Mode Selection for Multiple Switch Topology
-12.2.2 Link Monitoring for Multiple Switch Topology
-12.3 Switch Behavior Issues for High Availability
+12.2.1 HA Bonding Mode Selection for Multiple Switch Topology
+12.2.2 HA Link Monitoring for Multiple Switch Topology
+
+13. Configuring Bonding for Maximum Throughput
+13.1 Maximum Throughput in a Single Switch Topology
+13.1.1 MT Bonding Mode Selection for Single Switch Topology
+13.1.2 MT Link Monitoring for Single Switch Topology
+13.2 Maximum Throughput in a Multiple Switch Topology
+13.2.1 MT Bonding Mode Selection for Multiple Switch Topology
+13.2.2 MT Link Monitoring for Multiple Switch Topology
-13. Hardware Specific Considerations
-13.1 IBM BladeCenter
+14. Switch Behavior Issues
-14. Frequently Asked Questions
+15. Hardware Specific Considerations
+15.1 IBM BladeCenter
-15. Resources and Links
+16. Frequently Asked Questions
+
+17. Resources and Links
1. Bonding Driver Installation
@@ -86,16 +106,10 @@
1.1 Configure and build the kernel with bonding
-----------------------------------------------
- The latest version of the bonding driver is available in the
+ The current version of the bonding driver is available in the
drivers/net/bonding subdirectory of the most recent kernel source
-(which is available on http://kernel.org).
-
- Prior to the 2.4.11 kernel, the bonding driver was maintained
-largely outside the kernel tree; patches for some earlier kernels are
-available on the bonding sourceforge site, although those patches are
-still several years out of date. Most users will want to use either
-the most recent kernel from kernel.org or whatever kernel came with
-their distro.
+(which is available on http://kernel.org). Most users "rolling their
+own" will want to use the most recent kernel from kernel.org.
Configure kernel with "make menuconfig" (or "make xconfig" or
"make config"), then select "Bonding driver support" in the "Network
@@ -103,8 +117,8 @@
driver as module since it is currently the only way to pass parameters
to the driver or configure more than one bonding device.
- Build and install the new kernel and modules, then proceed to
-step 2.
+ Build and install the new kernel and modules, then continue
+below to install ifenslave.
1.2 Install ifenslave Control Utility
-------------------------------------
@@ -147,9 +161,9 @@
Options for the bonding driver are supplied as parameters to
the bonding module at load time. They may be given as command line
arguments to the insmod or modprobe command, but are usually specified
-in either the /etc/modprobe.conf configuration file, or in a
-distro-specific configuration file (some of which are detailed in the
-next section).
+in either the /etc/modules.conf or /etc/modprobe.conf configuration
+file, or in a distro-specific configuration file (some of which are
+detailed in the next section).
The available bonding driver parameters are listed below. If a
parameter is not specified the default value is used. When initially
@@ -162,34 +176,34 @@
support at least miimon, so there is really no reason not to use it.
Options with textual values will accept either the text name
- or, for backwards compatibility, the option value. E.g.,
- "mode=802.3ad" and "mode=4" set the same mode.
+or, for backwards compatibility, the option value. E.g.,
+"mode=802.3ad" and "mode=4" set the same mode.
The parameters are as follows:
arp_interval
- Specifies the ARP monitoring frequency in milli-seconds. If
- ARP monitoring is used in a load-balancing mode (mode 0 or 2),
- the switch should be configured in a mode that evenly
- distributes packets across all links - such as round-robin. If
- the switch is configured to distribute the packets in an XOR
+ Specifies the ARP link monitoring frequency in milliseconds.
+ If ARP monitoring is used in an etherchannel compatible mode
+ (modes 0 and 2), the switch should be configured in a mode
+ that evenly distributes packets across all links. If the
+ switch is configured to distribute the packets in an XOR
fashion, all replies from the ARP targets will be received on
the same link which could cause the other team members to
- fail. ARP monitoring should not be used in conjunction with
- miimon. A value of 0 disables ARP monitoring. The default
+ fail. ARP monitoring should not be used in conjunction with
+ miimon. A value of 0 disables ARP monitoring. The default
value is 0.
arp_ip_target
- Specifies the ip addresses to use when arp_interval is > 0.
- These are the targets of the ARP request sent to determine the
- health of the link to the targets. Specify these values in
- ddd.ddd.ddd.ddd format. Multiple ip adresses must be
- seperated by a comma. At least one IP address must be given
- for ARP monitoring to function. The maximum number of targets
- that can be specified is 16. The default value is no IP
- addresses.
+ Specifies the IP addresses to use as ARP monitoring peers when
+ arp_interval is > 0. These are the targets of the ARP request
+ sent to determine the health of the link to the targets.
+ Specify these values in ddd.ddd.ddd.ddd format. Multiple IP
+ addresses must be separated by a comma. At least one IP
+ address must be given for ARP monitoring to function. The
+ maximum number of targets that can be specified is 16. The
+ default value is no IP addresses.
downdelay
@@ -207,11 +221,13 @@
are:
slow or 0
- Request partner to transmit LACPDUs every 30 seconds (default)
+ Request partner to transmit LACPDUs every 30 seconds
fast or 1
Request partner to transmit LACPDUs every 1 second
+ The default is slow.
+
max_bonds
Specifies the number of bonding devices to create for this
@@ -221,10 +237,11 @@
miimon
- Specifies the frequency in milli-seconds that MII link
- monitoring will occur. A value of zero disables MII link
- monitoring. A value of 100 is a good starting point. The
- use_carrier option, below, affects how the link state is
+ Specifies the MII link monitoring frequency in milliseconds.
+ This determines how often the link state of each slave is
+ inspected for link failures. A value of zero disables MII
+ link monitoring. A value of 100 is a good starting point.
+ The use_carrier option, below, affects how the link state is
determined. See the High Availability section for additional
information. The default value is 0.
@@ -270,7 +287,7 @@
duplex settings. Utilizes all slaves in the active
aggregator according to the 802.3ad specification.
- Pre-requisites:
+ Prerequisites:
1. Ethtool support in the base drivers for retrieving
the speed and duplex of each slave.
@@ -333,7 +350,7 @@
When a link is reconnected or a new slave joins the
bond the receive traffic is redistributed among all
- active slaves in the bond by intiating ARP Replies
+ active slaves in the bond by initiating ARP Replies
with the selected mac address to each of the
clients. The updelay parameter (detailed below) must
be set to a value equal or greater than the switch's
@@ -448,8 +465,9 @@
slave devices. On SLES 9, this is most easily done by running the
yast2 sysconfig configuration utility. The goal is for to create an
ifcfg-id file for each slave device. The simplest way to accomplish
-this is to configure the devices for DHCP. The name of the
-configuration file for each device will be of the form:
+this is to configure the devices for DHCP (this is only to get the
+file ifcfg-id file created; see below for some issues with DHCP). The
+name of the configuration file for each device will be of the form:
ifcfg-id-xx:xx:xx:xx:xx:xx
@@ -459,7 +477,7 @@
Once the set of ifcfg-id-xx:xx:xx:xx:xx:xx files has been
created, it is necessary to edit the configuration files for the slave
devices (the MAC addresses correspond to those of the slave devices).
-Before editing, the file will contain muliple lines, and will look
+Before editing, the file will contain multiple lines, and will look
something like this:
BOOTPROTO='dhcp'
@@ -501,11 +519,6 @@
Replace the sample BROADCAST, IPADDR, NETMASK and NETWORK
values with the appropriate values for your network.
- Note that configuring the bonding device with BOOTPROTO='dhcp'
-does not work; the scripts attempt to obtain the device address from
-DHCP prior to adding any of the slave devices. Without active slaves,
-the DHCP requests are not sent to the network.
-
The STARTMODE specifies when the device is brought online.
The possible values are:
@@ -544,7 +557,7 @@
Note that the network control script (/sbin/ifdown) will
remove the bonding module as part of the network shutdown processing,
so it is not necessary to remove the module by hand if, e.g., the
-module paramters have changed.
+module parameters have changed.
Also, at this writing, YaST/YaST2 will not manage bonding
devices (they do not show bonding interfaces on its list of network
@@ -559,12 +572,37 @@
Note that the template does not document the various BONDING_
settings described above, but does describe many of the other options.
+3.1.1 Using DHCP with sysconfig
+-------------------------------
+
+ Under sysconfig, configuring a device with BOOTPROTO='dhcp'
+will cause it to query DHCP for its IP address information. At this
+writing, this does not function for bonding devices; the scripts
+attempt to obtain the device address from DHCP prior to adding any of
+the slave devices. Without active slaves, the DHCP requests are not
+sent to the network.
+
+3.1.2 Configuring Multiple Bonds with sysconfig
+-----------------------------------------------
+
+ The sysconfig network initialization system is capable of
+handling multiple bonding devices. All that is necessary is for each
+bonding instance to have an appropriately configured ifcfg-bondX file
+(as described above). Do not specify the "max_bonds" parameter to any
+instance of bonding, as this will confuse sysconfig. If you require
+multiple bonding devices with identical parameters, create multiple
+ifcfg-bondX files.
+
+ Because the sysconfig scripts supply the bonding module
+options in the ifcfg-bondX file, it is not necessary to add them to
+the system /etc/modules.conf or /etc/modprobe.conf configuration file.
+
3.2 Configuration with initscripts support
------------------------------------------
This section applies to distros using a version of initscripts
with bonding support, for example, Red Hat Linux 9 or Red Hat
-Enterprise Linux version 3. On these systems, the network
+Enterprise Linux version 3 or 4. On these systems, the network
initialization scripts have some knowledge of bonding, and can be
configured to control bonding devices.
@@ -614,10 +652,11 @@
Be sure to change the networking specific lines (IPADDR,
NETMASK, NETWORK and BROADCAST) to match your network configuration.
- Finally, it is necessary to edit /etc/modules.conf to load the
-bonding module when the bond0 interface is brought up. The following
-sample lines in /etc/modules.conf will load the bonding module, and
-select its options:
+ Finally, it is necessary to edit /etc/modules.conf (or
+/etc/modprobe.conf, depending upon your distro) to load the bonding
+module with your desired options when the bond0 interface is brought
+up. The following lines in /etc/modules.conf (or modprobe.conf) will
+load the bonding module, and select its options:
alias bond0 bonding
options bond0 mode=balance-alb miimon=100
@@ -629,6 +668,33 @@
will restart the networking subsystem and your bond link should be now
up and running.
+3.2.1 Using DHCP with initscripts
+---------------------------------
+
+ Recent versions of initscripts (the version supplied with
+Fedora Core 3 and Red Hat Enterprise Linux 4 is reported to work) do
+have support for assigning IP information to bonding devices via DHCP.
+
+ To configure bonding for DHCP, configure it as described
+above, except replace the line "BOOTPROTO=none" with "BOOTPROTO=dhcp"
+and add a line consisting of "TYPE=Bonding". Note that the TYPE value
+is case sensitive.
+
+3.2.2 Configuring Multiple Bonds with initscripts
+-------------------------------------------------
+
+ At this writing, the initscripts package does not directly
+support loading the bonding driver multiple times, so the process for
+doing so is the same as described in the "Configuring Multiple Bonds
+Manually" section, below.
+
+ NOTE: It has been observed that some Red Hat supplied kernels
+are apparently unable to rename modules at load time (the "-obonding1"
+part). Attempts to pass that option to modprobe will produce an
+"Operation not permitted" error. This has been reported on some
+Fedora Core kernels, and has been seen on RHEL 4 as well. On kernels
+exhibiting this problem, it will be impossible to configure multiple
+bonds with differing parameters.
3.3 Configuring Bonding Manually
--------------------------------
@@ -638,10 +704,11 @@
knowledge of bonding. One such distro is SuSE Linux Enterprise Server
version 8.
- The general methodology for these systems is to place the
-bonding module parameters into /etc/modprobe.conf, then add modprobe
-and/or ifenslave commands to the system's global init script. The
-name of the global init script differs; for sysconfig, it is
+ The general method for these systems is to place the bonding
+module parameters into /etc/modules.conf or /etc/modprobe.conf (as
+appropriate for the installed distro), then add modprobe and/or
+ifenslave commands to the system's global init script. The name of
+the global init script differs; for sysconfig, it is
/etc/init.d/boot.local and for initscripts it is /etc/rc.d/rc.local.
For example, if you wanted to make a simple bond of two e100
@@ -649,7 +716,7 @@
reboots, edit the appropriate file (/etc/init.d/boot.local or
/etc/rc.d/rc.local), and add the following:
-modprobe bonding -obond0 mode=balance-alb miimon=100
+modprobe bonding mode=balance-alb miimon=100
modprobe e100
ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up
ifenslave bond0 eth0
@@ -657,11 +724,7 @@
Replace the example bonding module parameters and bond0
network configuration (IP address, netmask, etc) with the appropriate
-values for your configuration. The above example loads the bonding
-module with the name "bond0," this simplifies the naming if multiple
-bonding modules are loaded (each successive instance of the module is
-given a different name, and the module instance names match the
-bonding interface names).
+values for your configuration.
Unfortunately, this method will not provide support for the
ifup and ifdown scripts on the bond devices. To reload the bonding
@@ -684,20 +747,23 @@
the following:
# ifconfig bond0 down
-# rmmod bond0
+# rmmod bonding
# rmmod e100
Again, for convenience, it may be desirable to create a script
with these commands.
-3.4 Configuring Multiple Bonds
-------------------------------
+3.3.1 Configuring Multiple Bonds Manually
+-----------------------------------------
This section contains information on configuring multiple
-bonding devices with differing options. If you require multiple
-bonding devices, but all with the same options, see the "max_bonds"
-module paramter, documented above.
+bonding devices with differing options for those systems whose network
+initialization scripts lack support for configuring multiple bonds.
+
+ If you require multiple bonding devices, but all with the same
+options, you may wish to use the "max_bonds" module parameter,
+documented above.
To create multiple bonding devices with differing options, it
is necessary to load the bonding driver multiple times. Note that
@@ -724,11 +790,16 @@
miimon of 100. The second instance is named "bond1" and creates the
bond1 device in balance-alb mode with an miimon of 50.
+ In some circumstances (typically with older distributions),
+the above does not work, and the second bonding instance never sees
+its options. In that case, the second options line can be substituted
+as follows:
+
+install bonding1 /sbin/modprobe bonding -obond1 mode=balance-alb miimon=50
+
This may be repeated any number of times, specifying a new and
-unique name in place of bond0 or bond1 for each instance.
+unique name in place of bond1 for each subsequent instance.
- When the appropriate module paramters are in place, then
-configure bonding according to the instructions for your distro.
5. Querying Bonding Configuration
=================================
@@ -846,8 +917,8 @@
self generated packets.
For reasons of simplicity, and to support the use of adapters
-that can do VLAN hardware acceleration offloding, the bonding
-interface declares itself as fully hardware offloaing capable, it gets
+that can do VLAN hardware acceleration offloading, the bonding
+interface declares itself as fully hardware offloading capable, it gets
the add_vid/kill_vid notifications to gather the necessary
information, and it propagates those actions to the slaves. In case
of mixed adapter types, hardware accelerated tagged packets that
@@ -880,7 +951,7 @@
matches the hardware address of the VLAN interfaces.
Note that changing a VLAN interface's HW address would set the
-underlying device -- i.e. the bonding interface -- to promiscouos
+underlying device -- i.e. the bonding interface -- to promiscuous
mode, which might not be what you want.
@@ -923,7 +994,7 @@
an additional target (or several) increases the reliability of the ARP
monitoring.
- Multiple ARP targets must be seperated by commas as follows:
+ Multiple ARP targets must be separated by commas as follows:
# example options for ARP monitoring with three targets
alias bond0 bonding
@@ -1045,7 +1116,7 @@
This will, when loading the bonding module, rather than
performing the normal action, instead execute the provided command.
This command loads the device drivers in the order needed, then calls
-modprobe with --ingore-install to cause the normal action to then take
+modprobe with --ignore-install to cause the normal action to then take
place. Full documentation on this can be found in the modprobe.conf
and modprobe manual pages.
@@ -1130,14 +1201,14 @@
common to enable promiscuous mode on the device, so that all traffic
is seen (instead of seeing only traffic destined for the local host).
The bonding driver handles promiscuous mode changes to the bonding
-master device (e.g., bond0), and propogates the setting to the slave
+master device (e.g., bond0), and propagates the setting to the slave
devices.
For the balance-rr, balance-xor, broadcast, and 802.3ad modes,
-the promiscuous mode setting is propogated to all slaves.
+the promiscuous mode setting is propagated to all slaves.
For the active-backup, balance-tlb and balance-alb modes, the
-promiscuous mode setting is propogated only to the active slave.
+promiscuous mode setting is propagated only to the active slave.
For balance-tlb mode, the active slave is the slave currently
receiving inbound traffic.
@@ -1148,46 +1219,182 @@
For the active-backup, balance-tlb and balance-alb modes, when
the active slave changes (e.g., due to a link failure), the
-promiscuous setting will be propogated to the new active slave.
+promiscuous setting will be propagated to the new active slave.
-12. High Availability Information
-=================================
+12. Configuring Bonding for High Availability
+=============================================
High Availability refers to configurations that provide
maximum network availability by having redundant or backup devices,
-links and switches between the host and the rest of the world.
-
- There are currently two basic methods for configuring to
-maximize availability. They are dependent on the network topology and
-the primary goal of the configuration, but in general, a configuration
-can be optimized for maximum available bandwidth, or for maximum
-network availability.
+links or switches between the host and the rest of the world. The
+goal is to provide the maximum availability of network connectivity
+(i.e., the network always works), even though other configurations
+could provide higher throughput.
12.1 High Availability in a Single Switch Topology
--------------------------------------------------
- If two hosts (or a host and a switch) are directly connected
-via multiple physical links, then there is no network availability
-penalty for optimizing for maximum bandwidth: there is only one switch
-(or peer), so if it fails, you have no alternative access to fail over
-to.
-
-Example 1 : host to switch (or other host)
-
- +----------+ +----------+
- | |eth0 eth0| switch |
- | Host A +--------------------------+ or |
- | +--------------------------+ other |
- | |eth1 eth1| host |
- +----------+ +----------+
+ If two hosts (or a host and a single switch) are directly
+connected via multiple physical links, then there is no availability
+penalty to optimizing for maximum bandwidth. In this case, there is
+only one switch (or peer), so if it fails, there is no alternative
+access to fail over to. Additionally, the bonding load balance modes
+support link monitoring of their members, so if individual links fail,
+the load will be rebalanced across the remaining devices.
+
+ See Section 13, "Configuring Bonding for Maximum Throughput"
+for information on configuring bonding with one peer device.
+12.2 High Availability in a Multiple Switch Topology
+----------------------------------------------------
-12.1.1 Bonding Mode Selection for single switch topology
---------------------------------------------------------
+ With multiple switches, the configuration of bonding and the
+network changes dramatically. In multiple switch topologies, there is
+a trade off between network availability and usable bandwidth.
+
+ Below is a sample network, configured to maximize the
+availability of the network:
+
+ | |
+ |port3 port3|
+ +-----+----+ +-----+----+
+ | |port2 ISL port2| |
+ | switch A +--------------------------+ switch B |
+ | | | |
+ +-----+----+ +-----++---+
+ |port1 port1|
+ | +-------+ |
+ +-------------+ host1 +---------------+
+ eth0 +-------+ eth1
+
+ In this configuration, there is a link between the two
+switches (ISL, or inter switch link), and multiple ports connecting to
+the outside world ("port3" on each switch). There is no technical
+reason that this could not be extended to a third switch.
+
+12.2.1 HA Bonding Mode Selection for Multiple Switch Topology
+-------------------------------------------------------------
+
+ In a topology such as the example above, the active-backup and
+broadcast modes are the only useful bonding modes when optimizing for
+availability; the other modes require all links to terminate on the
+same peer for them to behave rationally.
+
+active-backup: This is generally the preferred mode, particularly if
+ the switches have an ISL and play together well. If the
+ network configuration is such that one switch is specifically
+ a backup switch (e.g., has lower capacity, higher cost, etc),
+ then the primary option can be used to insure that the
+ preferred link is always used when it is available.
+
+broadcast: This mode is really a special purpose mode, and is suitable
+ only for very specific needs. For example, if the two
+ switches are not connected (no ISL), and the networks beyond
+ them are totally independent. In this case, if it is
+ necessary for some specific one-way traffic to reach both
+ independent networks, then the broadcast mode may be suitable.
+
+12.2.2 HA Link Monitoring Selection for Multiple Switch Topology
+----------------------------------------------------------------
+
+ The choice of link monitoring ultimately depends upon your
+switch. If the switch can reliably fail ports in response to other
+failures, then either the MII or ARP monitors should work. For
+example, in the above example, if the "port3" link fails at the remote
+end, the MII monitor has no direct means to detect this. The ARP
+monitor could be configured with a target at the remote end of port3,
+thus detecting that failure without switch support.
+
+ In general, however, in a multiple switch topology, the ARP
+monitor can provide a higher level of reliability in detecting end to
+end connectivity failures (which may be caused by the failure of any
+individual component to pass traffic for any reason). Additionally,
+the ARP monitor should be configured with multiple targets (at least
+one for each switch in the network). This will insure that,
+regardless of which switch is active, the ARP monitor has a suitable
+target to query.
+
+
+13. Configuring Bonding for Maximum Throughput
+==============================================
+
+13.1 Maximizing Throughput in a Single Switch Topology
+------------------------------------------------------
+
+ In a single switch configuration, the best method to maximize
+throughput depends upon the application and network environment. The
+various load balancing modes each have strengths and weaknesses in
+different environments, as detailed below.
+
+ For this discussion, we will break down the topologies into
+two categories. Depending upon the destination of most traffic, we
+categorize them into either "gatewayed" or "local" configurations.
+
+ In a gatewayed configuration, the "switch" is acting primarily
+as a router, and the majority of traffic passes through this router to
+other networks. An example would be the following:
+
+
+ +----------+ +----------+
+ | |eth0 port1| | to other networks
+ | Host A +---------------------+ router +------------------->
+ | +---------------------+ | Hosts B and C are out
+ | |eth1 port2| | here somewhere
+ +----------+ +----------+
+
+ The router may be a dedicated router device, or another host
+acting as a gateway. For our discussion, the important point is that
+the majority of traffic from Host A will pass through the router to
+some other network before reaching its final destination.
+
+ In a gatewayed network configuration, although Host A may
+communicate with many other systems, all of its traffic will be sent
+and received via one other peer on the local network, the router.
+
+ Note that the case of two systems connected directly via
+multiple physical links is, for purposes of configuring bonding, the
+same as a gatewayed configuration. In that case, it happens that all
+traffic is destined for the "gateway" itself, not some other network
+beyond the gateway.
+
+ In a local configuration, the "switch" is acting primarily as
+a switch, and the majority of traffic passes through this switch to
+reach other stations on the same network. An example would be the
+following:
+
+ +----------+ +----------+ +--------+
+ | |eth0 port1| +-------+ Host B |
+ | Host A +------------+ switch |port3 +--------+
+ | +------------+ | +--------+
+ | |eth1 port2| +------------------+ Host C |
+ +----------+ +----------+port4 +--------+
+
+
+ Again, the switch may be a dedicated switch device, or another
+host acting as a gateway. For our discussion, the important point is
+that the majority of traffic from Host A is destined for other hosts
+on the same local network (Hosts B and C in the above example).
+
+ In summary, in a gatewayed configuration, traffic to and from
+the bonded device will be to the same MAC level peer on the network
+(the gateway itself, i.e., the router), regardless of its final
+destination. In a local configuration, traffic flows directly to and
+from the final destinations, thus, each destination (Host B, Host C)
+will be addressed directly by their individual MAC addresses.
+
+ This distinction between a gatewayed and a local network
+configuration is important because many of the load balancing modes
+available use the MAC addresses of the local network source and
+destination to make load balancing decisions. The behavior of each
+mode is described below.
+
+
+13.1.1 MT Bonding Mode Selection for Single Switch Topology
+-----------------------------------------------------------
This configuration is the easiest to set up and to understand,
although you will have to decide which bonding mode best suits your
-needs. The tradeoffs for each mode are detailed below:
+needs. The trade offs for each mode are detailed below:
balance-rr: This mode is the only mode that will permit a single
TCP/IP connection to stripe traffic across multiple
@@ -1206,6 +1413,23 @@
interface's worth of throughput, even after adjusting
tcp_reordering.
+ Note that this out of order delivery occurs when both the
+ sending and receiving systems are utilizing a multiple
+ interface bond. Consider a configuration in which a
+ balance-rr bond feeds into a single higher capacity network
+ channel (e.g., multiple 100Mb/sec ethernets feeding a single
+ gigabit ethernet via an etherchannel capable switch). In this
+ configuration, traffic sent from the multiple 100Mb devices to
+ a destination connected to the gigabit device will not see
+ packets out of order. However, traffic sent from the gigabit
+ device to the multiple 100Mb devices may or may not see
+ traffic out of order, depending upon the balance policy of the
+ switch. Many switches do not support any modes that stripe
+ traffic (instead choosing a port based upon IP or MAC level
+ addresses); for those devices, traffic flowing from the
+ gigabit device to the many 100Mb devices will only utilize one
+ interface.
+
If you are utilizing protocols other than TCP/IP, UDP for
example, and your application can tolerate out of order
delivery, then this mode can allow for single stream datagram
@@ -1220,16 +1444,21 @@
connected to the same peer as the primary. In this case, a
load balancing mode (with link monitoring) will provide the
same level of network availability, but with increased
- available bandwidth. On the plus side, it does not require
- any configuration of the switch.
+ available bandwidth. On the plus side, active-backup mode
+ does not require any configuration of the switch, so it may
+ have value if the hardware available does not support any of
+ the load balance modes.
balance-xor: This mode will limit traffic such that packets destined
for specific peers will always be sent over the same
interface. Since the destination is determined by the MAC
- addresses involved, this may be desirable if you have a large
- network with many hosts. It is likely to be suboptimal if all
- your traffic is passed through a single router, however. As
- with balance-rr, the switch ports need to be configured for
+ addresses involved, this mode works best in a "local" network
+ configuration (as described above), with destinations all on
+ the same local network. This mode is likely to be suboptimal
+ if all your traffic is passed through a single router (i.e., a
+ "gatewayed" network configuration, as described above).
+
+ As with balance-rr, the switch ports need to be configured for
"etherchannel" or "trunking."
broadcast: Like active-backup, there is not much advantage to this
@@ -1241,122 +1470,128 @@
protocol includes automatic configuration of the aggregates,
so minimal manual configuration of the switch is needed
(typically only to designate that some set of devices is
- usable for 802.3ad). The 802.3ad standard also mandates that
- frames be delivered in order (within certain limits), so in
- general single connections will not see misordering of
+ available for 802.3ad). The 802.3ad standard also mandates
+ that frames be delivered in order (within certain limits), so
+ in general single connections will not see misordering of
packets. The 802.3ad mode does have some drawbacks: the
standard mandates that all devices in the aggregate operate at
the same speed and duplex. Also, as with all bonding load
balance modes other than balance-rr, no single connection will
be able to utilize more than a single interface's worth of
- bandwidth. Additionally, the linux bonding 802.3ad
- implementation distributes traffic by peer (using an XOR of
- MAC addresses), so in general all traffic to a particular
- destination will use the same interface. Finally, the 802.3ad
- mode mandates the use of the MII monitor, therefore, the ARP
- monitor is not available in this mode.
-
-balance-tlb: This mode is also a good choice for this type of
- topology. It has no special switch configuration
- requirements, and balances outgoing traffic by peer, in a
- vaguely intelligent manner (not a simple XOR as in balance-xor
- or 802.3ad mode), so that unlucky MAC addresses will not all
- "bunch up" on a single interface. Interfaces may be of
- differing speeds. On the down side, in this mode all incoming
- traffic arrives over a single interface, this mode requires
- certain ethtool support in the network device driver of the
- slave interfaces, and the ARP monitor is not available.
-
-balance-alb: This mode is everything that balance-tlb is, and more. It
- has all of the features (and restrictions) of balance-tlb, and
- will also balance incoming traffic from peers (as described in
- the Bonding Module Options section, above). The only extra
- down side to this mode is that the network device driver must
- support changing the hardware address while the device is
- open.
+ bandwidth.
-12.1.2 Link Monitoring for Single Switch Topology
--------------------------------------------------
+ Additionally, the linux bonding 802.3ad implementation
+ distributes traffic by peer (using an XOR of MAC addresses),
+ so in a "gatewayed" configuration, all outgoing traffic will
+ generally use the same device. Incoming traffic may also end
+ up on a single device, but that is dependent upon the
+ balancing policy of the peer's 8023.ad implementation. In a
+ "local" configuration, traffic will be distributed across the
+ devices in the bond.
+
+ Finally, the 802.3ad mode mandates the use of the MII monitor,
+ therefore, the ARP monitor is not available in this mode.
+
+balance-tlb: The balance-tlb mode balances outgoing traffic by peer.
+ Since the balancing is done according to MAC address, in a
+ "gatewayed" configuration (as described above), this mode will
+ send all traffic across a single device. However, in a
+ "local" network configuration, this mode balances multiple
+ local network peers across devices in a vaguely intelligent
+ manner (not a simple XOR as in balance-xor or 802.3ad mode),
+ so that mathematically unlucky MAC addresses (i.e., ones that
+ XOR to the same value) will not all "bunch up" on a single
+ interface.
+
+ Unlike 802.3ad, interfaces may be of differing speeds, and no
+ special switch configuration is required. On the down side,
+ in this mode all incoming traffic arrives over a single
+ interface, this mode requires certain ethtool support in the
+ network device driver of the slave interfaces, and the ARP
+ monitor is not available.
+
+balance-alb: This mode is everything that balance-tlb is, and more.
+ It has all of the features (and restrictions) of balance-tlb,
+ and will also balance incoming traffic from local network
+ peers (as described in the Bonding Module Options section,
+ above).
+
+ The only additional down side to this mode is that the network
+ device driver must support changing the hardware address while
+ the device is open.
+
+13.1.2 MT Link Monitoring for Single Switch Topology
+----------------------------------------------------
The choice of link monitoring may largely depend upon which
mode you choose to use. The more advanced load balancing modes do not
support the use of the ARP monitor, and are thus restricted to using
-the MII monitor (which does not provide as high a level of assurance
-as the ARP monitor).
-
-
-12.2 High Availability in a Multiple Switch Topology
-----------------------------------------------------
-
- With multiple switches, the configuration of bonding and the
-network changes dramatically. In multiple switch topologies, there is
-a tradeoff between network availability and usable bandwidth.
-
- Below is a sample network, configured to maximize the
-availability of the network:
-
- | |
- |port3 port3|
- +-----+----+ +-----+----+
- | |port2 ISL port2| |
- | switch A +--------------------------+ switch B |
- | | | |
- +-----+----+ +-----++---+
- |port1 port1|
- | +-------+ |
- +-------------+ host1 +---------------+
- eth0 +-------+ eth1
-
- In this configuration, there is a link between the two
-switches (ISL, or inter switch link), and multiple ports connecting to
-the outside world ("port3" on each switch). There is no technical
-reason that this could not be extended to a third switch.
+the MII monitor (which does not provide as high a level of end to end
+assurance as the ARP monitor).
-12.2.1 Bonding Mode Selection for Multiple Switch Topology
-----------------------------------------------------------
+13.2 Maximum Throughput in a Multiple Switch Topology
+-----------------------------------------------------
- In a topology such as this, the active-backup and broadcast
-modes are the only useful bonding modes; the other modes require all
-links to terminate on the same peer for them to behave rationally.
-
-active-backup: This is generally the preferred mode, particularly if
- the switches have an ISL and play together well. If the
- network configuration is such that one switch is specifically
- a backup switch (e.g., has lower capacity, higher cost, etc),
- then the primary option can be used to insure that the
- preferred link is always used when it is available.
+ Multiple switches may be utilized to optimize for throughput
+when they are configured in parallel as part of an isolated network
+between two or more systems, for example:
+
+ +-----------+
+ | Host A |
+ +-+---+---+-+
+ | | |
+ +--------+ | +---------+
+ | | |
+ +------+---+ +-----+----+ +-----+----+
+ | Switch A | | Switch B | | Switch C |
+ +------+---+ +-----+----+ +-----+----+
+ | | |
+ +--------+ | +---------+
+ | | |
+ +-+---+---+-+
+ | Host B |
+ +-----------+
+
+ In this configuration, the switches are isolated from one
+another. One reason to employ a topology such as this is for an
+isolated network with many hosts (a cluster configured for high
+performance, for example), using multiple smaller switches can be more
+cost effective than a single larger switch, e.g., on a network with 24
+hosts, three 24 port switches can be significantly less expensive than
+a single 72 port switch.
+
+ If access beyond the network is required, an individual host
+can be equipped with an additional network device connected to an
+external network; this host then additionally acts as a gateway.
-broadcast: This mode is really a special purpose mode, and is suitable
- only for very specific needs. For example, if the two
- switches are not connected (no ISL), and the networks beyond
- them are totally independant. In this case, if it is
- necessary for some specific one-way traffic to reach both
- independent networks, then the broadcast mode may be suitable.
-
-12.2.2 Link Monitoring Selection for Multiple Switch Topology
+13.2.1 MT Bonding Mode Selection for Multiple Switch Topology
-------------------------------------------------------------
- The choice of link monitoring ultimately depends upon your
-switch. If the switch can reliably fail ports in response to other
-failures, then either the MII or ARP monitors should work. For
-example, in the above example, if the "port3" link fails at the remote
-end, the MII monitor has no direct means to detect this. The ARP
-monitor could be configured with a target at the remote end of port3,
-thus detecting that failure without switch support.
-
- In general, however, in a multiple switch topology, the ARP
-monitor can provide a higher level of reliability in detecting link
-failures. Additionally, it should be configured with multiple targets
-(at least one for each switch in the network). This will insure that,
-regardless of which switch is active, the ARP monitor has a suitable
-target to query.
-
+ In actual practice, the bonding mode typically employed in
+configurations of this type is balance-rr. Historically, in this
+network configuration, the usual caveats about out of order packet
+delivery are mitigated by the use of network adapters that do not do
+any kind of packet coalescing (via the use of NAPI, or because the
+device itself does not generate interrupts until some number of
+packets has arrived). When employed in this fashion, the balance-rr
+mode allows individual connections between two hosts to effectively
+utilize greater than one interface's bandwidth.
+
+13.2.2 MT Link Monitoring for Multiple Switch Topology
+------------------------------------------------------
+
+ Again, in actual practice, the MII monitor is most often used
+in this configuration, as performance is given preference over
+availability. The ARP monitor will function in this topology, but its
+advantages over the MII monitor are mitigated by the volume of probes
+needed as the number of systems involved grows (remember that each
+host in the network is configured with bonding).
-12.3 Switch Behavior Issues for High Availability
--------------------------------------------------
+14. Switch Behavior Issues
+--------------------------
- You may encounter issues with the timing of link up and down
-reporting by the switch.
+ Some switches exhibit undesirable behavior with regard to the
+timing of link up and down reporting by the switch.
First, when a link comes up, some switches may indicate that
the link is up (carrier available), but not pass traffic over the
@@ -1370,30 +1605,31 @@
Second, some switches may "bounce" the link state one or more
times while a link is changing state. This occurs most commonly while
the switch is initializing. Again, an appropriate updelay value may
-help, but note that if all links are down, then updelay is ignored
-when any link becomes active (the slave closest to completing its
-updelay is chosen).
+help.
Note that when a bonding interface has no active links, the
-driver will immediately reuse the first link that goes up, even if
-updelay parameter was specified. If there are slave interfaces
-waiting for the updelay timeout to expire, the interface that first
-went into that state will be immediately reused. This reduces down
-time of the network if the value of updelay has been overestimated.
+driver will immediately reuse the first link that goes up, even if the
+updelay parameter has been specified (the updelay is ignored in this
+case). If there are slave interfaces waiting for the updelay timeout
+to expire, the interface that first went into that state will be
+immediately reused. This reduces down time of the network if the
+value of updelay has been overestimated, and since this occurs only in
+cases with no connectivity, there is no additional penalty for
+ignoring the updelay.
In addition to the concerns about switch timings, if your
switches take a long time to go into backup mode, it may be desirable
to not activate a backup interface immediately after a link goes down.
Failover may be delayed via the downdelay bonding module option.
-13. Hardware Specific Considerations
+15. Hardware Specific Considerations
====================================
This section contains additional information for configuring
bonding on specific hardware platforms, or for interfacing bonding
with particular switches or other devices.
-13.1 IBM BladeCenter
+15.1 IBM BladeCenter
--------------------
This applies to the JS20 and similar systems.
@@ -1407,12 +1643,12 @@
--------------------------------
All JS20s come with two Broadcom Gigabit Ethernet ports
-integrated on the planar. In the BladeCenter chassis, the eth0 port
-of all JS20 blades is hard wired to I/O Module #1; similarly, all eth1
-ports are wired to I/O Module #2. An add-on Broadcom daughter card
-can be installed on a JS20 to provide two more Gigabit Ethernet ports.
-These ports, eth2 and eth3, are wired to I/O Modules 3 and 4,
-respectively.
+integrated on the planar (that's "motherboard" in IBM-speak). In the
+BladeCenter chassis, the eth0 port of all JS20 blades is hard wired to
+I/O Module #1; similarly, all eth1 ports are wired to I/O Module #2.
+An add-on Broadcom daughter card can be installed on a JS20 to provide
+two more Gigabit Ethernet ports. These ports, eth2 and eth3, are
+wired to I/O Modules 3 and 4, respectively.
Each I/O Module may contain either a switch or a passthrough
module (which allows ports to be directly connected to an external
@@ -1432,29 +1668,30 @@
of ways, this discussion will be confined to describing basic
configurations.
- Normally, Ethernet Switch Modules (ESM) are used in I/O
+ Normally, Ethernet Switch Modules (ESMs) are used in I/O
modules 1 and 2. In this configuration, the eth0 and eth1 ports of a
JS20 will be connected to different internal switches (in the
respective I/O modules).
- An optical passthru module (OPM) connects the I/O module
-directly to an external switch. By using OPMs in I/O module #1 and
-#2, the eth0 and eth1 interfaces of a JS20 can be redirected to the
-outside world and connected to a common external switch.
-
- Depending upon the mix of ESM and OPM modules, the network
-will appear to bonding as either a single switch topology (all OPM
-modules) or as a multiple switch topology (one or more ESM modules,
-zero or more OPM modules). It is also possible to connect ESM modules
-together, resulting in a configuration much like the example in "High
-Availability in a multiple switch topology."
-
-Requirements for specifc modes
-------------------------------
-
- The balance-rr mode requires the use of OPM modules for
-devices in the bond, all connected to an common external switch. That
-switch must be configured for "etherchannel" or "trunking" on the
+ A passthrough module (OPM or CPM, optical or copper,
+passthrough module) connects the I/O module directly to an external
+switch. By using PMs in I/O module #1 and #2, the eth0 and eth1
+interfaces of a JS20 can be redirected to the outside world and
+connected to a common external switch.
+
+ Depending upon the mix of ESMs and PMs, the network will
+appear to bonding as either a single switch topology (all PMs) or as a
+multiple switch topology (one or more ESMs, zero or more PMs). It is
+also possible to connect ESMs together, resulting in a configuration
+much like the example in "High Availability in a Multiple Switch
+Topology," above.
+
+Requirements for specific modes
+-------------------------------
+
+ The balance-rr mode requires the use of passthrough modules
+for devices in the bond, all connected to an common external switch.
+That switch must be configured for "etherchannel" or "trunking" on the
appropriate ports, as is usual for balance-rr.
The balance-alb and balance-tlb modes will function with
@@ -1484,17 +1721,18 @@
Other concerns
--------------
- The Serial Over LAN link is established over the primary
+ The Serial Over LAN (SoL) link is established over the primary
ethernet (eth0) only, therefore, any loss of link to eth0 will result
in losing your SoL connection. It will not fail over with other
-network traffic.
+network traffic, as the SoL system is beyond the control of the
+bonding driver.
It may be desirable to disable spanning tree on the switch
(either the internal Ethernet Switch Module, or an external switch) to
-avoid fail-over delays issues when using bonding.
+avoid fail-over delay issues when using bonding.
-14. Frequently Asked Questions
+16. Frequently Asked Questions
==============================
1. Is it SMP safe?
@@ -1505,8 +1743,8 @@
2. What type of cards will work with it?
Any Ethernet type cards (you can even mix cards - a Intel
-EtherExpress PRO/100 and a 3com 3c905b, for example). They need not
-be of the same speed.
+EtherExpress PRO/100 and a 3com 3c905b, for example). For most modes,
+devices need not be of the same speed.
3. How many bonding devices can I have?
@@ -1524,11 +1762,12 @@
disabled. The active-backup mode will fail over to a backup link, and
other modes will ignore the failed link. The link will continue to be
monitored, and should it recover, it will rejoin the bond (in whatever
-manner is appropriate for the mode). See the section on High
-Availability for additional information.
+manner is appropriate for the mode). See the sections on High
+Availability and the documentation for each mode for additional
+information.
Link monitoring can be enabled via either the miimon or
-arp_interval paramters (described in the module paramters section,
+arp_interval parameters (described in the module parameters section,
above). In general, miimon monitors the carrier state as sensed by
the underlying network device, and the arp monitor (arp_interval)
monitors connectivity to another host on the local network.
@@ -1536,7 +1775,7 @@
If no link monitoring is configured, the bonding driver will
be unable to detect link failures, and will assume that all links are
always available. This will likely result in lost packets, and a
-resulting degredation of performance. The precise performance loss
+resulting degradation of performance. The precise performance loss
depends upon the bonding mode and network configuration.
6. Can bonding be used for High Availability?
@@ -1550,12 +1789,12 @@
In the basic balance modes (balance-rr and balance-xor), it
works with any system that supports etherchannel (also called
trunking). Most managed switches currently available have such
-support, and many unmananged switches as well.
+support, and many unmanaged switches as well.
The advanced balance modes (balance-tlb and balance-alb) do
not have special switch requirements, but do need device drivers that
support specific features (described in the appropriate section under
-module paramters, above).
+module parameters, above).
In 802.3ad mode, it works with with systems that support IEEE
802.3ad Dynamic Link Aggregation. Most managed and many unmanaged
@@ -1565,17 +1804,19 @@
8. Where does a bonding device get its MAC address from?
- If not explicitly configured with ifconfig, the MAC address of
-the bonding device is taken from its first slave device. This MAC
-address is then passed to all following slaves and remains persistent
-(even if the the first slave is removed) until the bonding device is
-brought down or reconfigured.
+ If not explicitly configured (with ifconfig or ip link), the
+MAC address of the bonding device is taken from its first slave
+device. This MAC address is then passed to all following slaves and
+remains persistent (even if the the first slave is removed) until the
+bonding device is brought down or reconfigured.
If you wish to change the MAC address, you can set it with
-ifconfig:
+ifconfig or ip link:
# ifconfig bond0 hw ether 00:11:22:33:44:55
+# ip link set bond0 address 66:77:88:99:aa:bb
+
The MAC address can be also changed by bringing down/up the
device and then changing its slaves (or their order):
@@ -1591,23 +1832,28 @@
then restore the MAC addresses that the slaves had before they were
enslaved.
-15. Resources and Links
+16. Resources and Links
=======================
The latest version of the bonding driver can be found in the latest
version of the linux kernel, found on http://kernel.org
+The latest version of this document can be found in either the latest
+kernel source (named Documentation/networking/bonding.txt), or on the
+bonding sourceforge site:
+
+http://www.sourceforge.net/projects/bonding
+
Discussions regarding the bonding driver take place primarily on the
bonding-devel mailing list, hosted at sourceforge.net. If you have
-questions or problems, post them to the list.
+questions or problems, post them to the list. The list address is:
bonding-devel@lists.sourceforge.net
-https://lists.sourceforge.net/lists/listinfo/bonding-devel
-
-There is also a project site on sourceforge.
+ The administrative interface (to subscribe or unsubscribe) can
+be found at:
-http://www.sourceforge.net/projects/bonding
+https://lists.sourceforge.net/lists/listinfo/bonding-devel
Donald Becker's Ethernet Drivers and diag programs may be found at :
- http://www.scyld.com/network/
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: jamal @ 2005-06-03 18:42 UTC (permalink / raw)
To: Mitch Williams
Cc: David S. Miller, Ronciak, John, jdmason, shemminger, netdev,
Robert.Olsson, Venkatesan, Ganesh, Brandeburg, Jesse
In-Reply-To: <Pine.CYG.4.58.0506030929300.2788@mawilli1-desk2.amr.corp.intel.com>
On Fri, 2005-03-06 at 10:43 -0700, Mitch Williams wrote:
>
> On Thu, 2 Jun 2005, jamal wrote:
> >
> > Heres what i think i saw as a flow of events:
> > Someone posted a theory that if you happen to reduce the weight
> > (iirc the reduction was via a shift) then the DRR would give less CPU
> > time cycle to the driver - Whats the big suprise there? thats DRR design
> > intent.
>
> Well, that was me. Or at least I was the original poster on this thread.
> But my theory (if you can call it that) really wasn't about CPU time. I
> spent several weeks in our lab with the somewhat nebulous task of "look at
> Linux performance". And what I found was, to me, counterintuitive:
> reducing weight improved performance, sometimes significantly.
>
When you reduce the weight, the system is spending less time in the
softirq processing packets before softirq yields. If this gives more
opportunity to your app to run, then the performance will go up.
Is this what you are seeing?
> OK, well, call me a blasphemer (against whom?).
> I'm not really saying
> that the DRR algorithm is not real-world, but rather that NAPI as
> currently implemented has some significant performance limitations.
>
And we need to be fair and investigate why.
> In my mind, there are two major problems with NAPI as it stands today.
> First, at Gigabit and higher speeds, the default settings don't allow the
> driver to process received packets in a timely manner.
What do you mean by timely?
> This causes
> dropped packets due to lack of receive resources. Lowering the weight can
> fix this, at least in a single-adapter environment.
>
If your know your workload you could tune the weight. Additionaly you
could tune the softirq using nice.
> Second, at 10Mbps and 100Mbps, modern processors are just too fast for the
> network. The NAPI polling loop runs so much quicker than the wire speed
> that only one or two packets are processed per softirq -- which
> effectively puts the adapter back in interrupt mode. Because of this, you
> can easily bog down a very fast box with relatively slow traffic, just due
> to the massive number of interrupts generated.
>
Massive is an overstatement. The issue is really IO. If you process one
packet in each interupt then NAPI does add extra IO costs at "low"
traffic levels.
Note that this is also a known issue - reference the threads from waay
back from people like Manfred Spraul and recently from the SGI folks.
IO unfortunately hasnt kept up with CPU speeds; hardware vendors such as
your company have been busy making processors faster but forgetting
about IO and RAM latencies. PCI-E seems promising from what i have
heard, interim PCI-E bridging to PCI-X is form what i have heard on its
IO performance worse.
> My original post (and patch) were to address the first issue. By using
> the shift value on the quota, I effectively lowered the weight for every
> driver in the system. Stephen sent out a patch that allowed you to
> adjust each driver's weight individually. My testing has shown that, as
> expected, you can achieve the same performance gain either way.
>
Ok, glad to hear thats resolved.
> In a multiple-adapter environment, you need to adjust the weight of all
> drivers together to fix the dropped packets issue. Lowering the weight on
> one adapter won't help it if the other interfaces are still taking up a
> lot of time in their receive loops.
>
> My patch gave you one knob to twiddle that would correct this issue.
> Stephen's patch gave you one knob for each adapter, but now you need to
> twiddle them all to see any benefit.
>
makes sense
> The second issue currently has no fix. What is needed is a way for the
> driver to request a delayed poll, possibly based on line speed. If we
> could wait, say, 8 packet times before polling, we could significantly
> reduce the number of interrupts the system has to deal with, at the cost
> of higher latency. We haven't had time to investigate this at all, but
> the need is clearly present -- we've had customer calls about this issue.
>
I can believe you (note it has to do with IO costs though) having seen
how horrific MMIO numbers are on faster processors. Talk to Jesse, he
has seen a little program from Lennert/Robert/Harald that does MMIO
measurements.
It seems the trend is that as CPUs get faster, IO gets more expensive in
both cpu cycles as well as absolute time.
The solution to this issue is to be found in mitigation at the moment in
conjunction with NAPI.
The SGI folks have made some real progress with recent patches from
Davem and Michael Chan on tg3.
I have been experimenting with some patches but they introduce
unacceptable jitter in latency.
So lets summarize it this way: This is something that needs to be
resolved - but whatever solution needs to be generic.
> Either way, I think the netdev community needs to look critically at NAPI,
> and make some changes.
I think what you call as the second issue needs a solution. Mitigation
is the only generic solution at the moment.
> Network performance in 2.6.12-rcWhatever is
> pretty poor. 2.4.30 beats it handily, and it really shouldn't be that
> way.
>
Are you using NAPI as well on 2.4.30?
cheers,
jamal
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: David S. Miller @ 2005-06-03 18:38 UTC (permalink / raw)
To: mitch.a.williams
Cc: hadi, john.ronciak, jdmason, shemminger, netdev, Robert.Olsson,
ganesh.venkatesan, jesse.brandeburg
In-Reply-To: <Pine.CYG.4.58.0506030929300.2788@mawilli1-desk2.amr.corp.intel.com>
From: Mitch Williams <mitch.a.williams@intel.com>
Date: Fri, 3 Jun 2005 10:43:32 -0700
> In my mind, there are two major problems with NAPI as it stands today.
> First, at Gigabit and higher speeds, the default settings don't allow the
> driver to process received packets in a timely manner. This causes
> dropped packets due to lack of receive resources. Lowering the weight can
> fix this, at least in a single-adapter environment.
I really don't see how changing the weight can change things
in the single adapter case.
When we hit the quota, we just loop and process more packets.
It doesn't fundamentally change anything about how the NAPI
code operates.
Please investigate what exactly is happening. I have a few
theories. First, is it the case that with a lower weight we
drop out of the loop because 'jiffies' advanced one tick?
Some simply instrumentation in net/core/dev.c:net_rx_action()
would show what's going on. Actually, we keep this statistic
via netdev_rx_stat, so just cat /proc/net/softnet_stat to
get a look at if "time_squeeze" is being incremented when
dev->weight is 64 in your tests.
Next, I don't think "budget" in that function is going down to zero,
that's set to 300 by default.
If the quota is consumed, the device is just added right back
to the tail of the poll_list, and if it's the only device
active we jump right back into it's ->poll() routine over
and over until there is no more pending work in the device
or we hit the "jiffies - start_time > 1" test.
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: Ben Greear @ 2005-06-03 18:33 UTC (permalink / raw)
To: Ronciak, John
Cc: Robert Olsson, David S. Miller, jdmason, shemminger, hadi,
Williams, Mitch A, netdev, Venkatesan, Ganesh, Brandeburg, Jesse
In-Reply-To: <468F3FDA28AA87429AD807992E22D07E0450BFE8@orsmsx408>
Ronciak, John wrote:
>> It's not obvious that weight is to blame for frames dropped. I would
>> look into RX ring size in relation to HW mitigation.
>> And of course if you system is very loaded the RX softirq gives room
>> for other jobs and frames get dropped
>>
>
> With the same system (fairly high end with nothing major running on it)
> we got rid of the dropped frames by just reducing the weight for 64. So
> the weight did have something to do with the dropped frames. Maybe
> other factors as well, but in static tests like this it sure looks like
> the 64 value is wrong is some cases.
Is this implying that having the NAPI poll do less work per poll
of the driver actually increases performance? I would have guessed that
the opposite would be true.
Maybe the poll is disabling the IRQs on the NIC for too long, or something
like that?
For e1000, are you using larger than the default 256 receive descriptors?
I have seen that increasing these descriptors helps decrease drops by
a small percentage.
Have you tried increasing the netdev-backlog setting to see if that
fixes the problem (while leaving the weight at the default)?
What packet sizes and speeds are you using for your tests?
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply
* RE: RFC: NAPI packet weighting patch
From: Ronciak, John @ 2005-06-03 18:19 UTC (permalink / raw)
To: Robert Olsson
Cc: David S. Miller, jdmason, shemminger, hadi, Williams, Mitch A,
netdev, Venkatesan, Ganesh, Brandeburg, Jesse
> It's not obvious that weight is to blame for frames dropped. I would
> look into RX ring size in relation to HW mitigation.
> And of course if you system is very loaded the RX softirq gives room
> for other jobs and frames get dropped
>
With the same system (fairly high end with nothing major running on it)
we got rid of the dropped frames by just reducing the weight for 64. So
the weight did have something to do with the dropped frames. Maybe
other factors as well, but in static tests like this it sure looks like
the 64 value is wrong is some cases.
Cheers,
John
^ permalink raw reply
* RE: RFC: NAPI packet weighting patch
From: Robert Olsson @ 2005-06-03 18:08 UTC (permalink / raw)
To: Ronciak, John
Cc: David S. Miller, jdmason, shemminger, hadi, Williams, Mitch A,
netdev, Robert.Olsson, Venkatesan, Ganesh, Brandeburg, Jesse
In-Reply-To: <468F3FDA28AA87429AD807992E22D07E0450BFE6@orsmsx408>
Ronciak, John writes:
> > What more do you need other than checking the statistics counter? The
> > drop statistics (the ones we care about) are incremented in real time
> > by the ->poll() code, so it's not like we have to trigger some
> > asynchronous event to get a current version of the number.
> >
>
> I think that there is some more confusion here. I'm talking about
> frames dropped by the Ethernet controller at the hardware level (no
> descriptor available). This for example is happening now with our
> driver with the weight set to 64. This is also what started us looking
> into what was going on with the weight. I don't see how the NAPI code
> to dynamically adjust the weight could easily get the hardware stats
> number to know if frames are being dropped or not. Sorry if I caused
> the confusion here.
It's not obvious that weight is to blame for frames dropped. I would
look into RX ring size in relation to HW mitigation.
And of course if you system is very loaded the RX softirq gives room
for other jobs and frames get dropped
Cheers.
--ro
^ permalink raw reply
* Re: RFC: NAPI packet weighting patch
From: Mitch Williams @ 2005-06-03 17:43 UTC (permalink / raw)
To: jamal
Cc: David S. Miller, Ronciak, John, jdmason, shemminger, netdev,
Robert.Olsson, Venkatesan, Ganesh, Brandeburg, Jesse
In-Reply-To: <1117765954.6095.49.camel@localhost.localdomain>
On Thu, 2 Jun 2005, jamal wrote:
>
> Heres what i think i saw as a flow of events:
> Someone posted a theory that if you happen to reduce the weight
> (iirc the reduction was via a shift) then the DRR would give less CPU
> time cycle to the driver - Whats the big suprise there? thats DRR design
> intent.
Well, that was me. Or at least I was the original poster on this thread.
But my theory (if you can call it that) really wasn't about CPU time. I
spent several weeks in our lab with the somewhat nebulous task of "look at
Linux performance". And what I found was, to me, counterintuitive:
reducing weight improved performance, sometimes significantly.
>
> Stephen has a patch which allows people to reduce the weight.
> DRR provides fairness. If you have 10 NICs coming at different wire
> rates, the weights provide a fairness quota without caring about what
> those speeds are. So it doesnt make any sense IMO to have the weight
> based on what the NIC speed is. Infact i claim it is _nonsense_. You
> dont need to factor speed. And the claim that DRR is not real world
> is blasphemous.
OK, well, call me a blasphemer (against whom?). I'm not really saying
that the DRR algorithm is not real-world, but rather that NAPI as
currently implemented has some significant performance limitations.
In my mind, there are two major problems with NAPI as it stands today.
First, at Gigabit and higher speeds, the default settings don't allow the
driver to process received packets in a timely manner. This causes
dropped packets due to lack of receive resources. Lowering the weight can
fix this, at least in a single-adapter environment.
Second, at 10Mbps and 100Mbps, modern processors are just too fast for the
network. The NAPI polling loop runs so much quicker than the wire speed
that only one or two packets are processed per softirq -- which
effectively puts the adapter back in interrupt mode. Because of this, you
can easily bog down a very fast box with relatively slow traffic, just due
to the massive number of interrupts generated.
My original post (and patch) were to address the first issue. By using
the shift value on the quota, I effectively lowered the weight for every
driver in the system. Stephen sent out a patch that allowed you to
adjust each driver's weight individually. My testing has shown that, as
expected, you can achieve the same performance gain either way.
In a multiple-adapter environment, you need to adjust the weight of all
drivers together to fix the dropped packets issue. Lowering the weight on
one adapter won't help it if the other interfaces are still taking up a
lot of time in their receive loops.
My patch gave you one knob to twiddle that would correct this issue.
Stephen's patch gave you one knob for each adapter, but now you need to
twiddle them all to see any benefit.
The second issue currently has no fix. What is needed is a way for the
driver to request a delayed poll, possibly based on line speed. If we
could wait, say, 8 packet times before polling, we could significantly
reduce the number of interrupts the system has to deal with, at the cost
of higher latency. We haven't had time to investigate this at all, but
the need is clearly present -- we've had customer calls about this issue.
>
> Having said that:
> I have a feeling that issue which is which is being waded around is the
> amount that the softirq chews in the CPU (unfortunately a well known
> issue) and to some extent the packet flow a specific driver chews
> depending on the path it takes.
I fiddled with this concept a little bit, but didn't see much performance
gain by doing so. But it may be something that we can go back and look
at.
Either way, I think the netdev community needs to look critically at NAPI,
and make some changes. Network performance in 2.6.12-rcWhatever is
pretty poor. 2.4.30 beats it handily, and it really shouldn't be that
way.
> This, however, does not eradicate the need for DRR and is absolutely not
> driver specific.
Agreed. All of the changes I've experimented with at the NAPI level have
affected performance similarly on multiple drivers.
-Mitch
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox