[Bridge] Problem bridging frames with bridge and real interface MTU > 1527

[Bridge] Problem bridging frames with bridge and real interface MTU > 1527

[Mark Zipp]

Hi,

I'm doing some fault finding where we suspect that the (non-linux)
device with the fault is generating frames with an MTU / maximum size
greater than 1500 / 1518 bytes. This device is performing GRE
tunneling, and it seems that it isn't performing fragmentation or
PMTUD before it attempts to send a normal, 1500 byte IP packet with
the GRE header out a GigE interface with a 1500 byte MTU.

I've been asked by the vendor of this device to perform some packet
captures. As I have a Linux box with a couple of interfaces that
support > 1500 byte MTUs (an Intel GigE, and a Netgear FA311 (that
supports an MTU up to 2024 bytes)), I figured I could bridge this
traffic through this Linux box, and then perform some packet captures
on the bridge virtual interface.

I set that all up, and it seemed to all be ok. I set both the real
interface MTUs to 2024 bytes, and when I added them into a bridge, the
bridge inherited this MTU.

Unfortunately I found that when I attempted to forward frames through
this bridge configuration, certain sized ones were silently dropped.
As the devices attached to the bridge shouldn't be generating frames
with an MTU larger than 1500 (this is our work around solution as a
reference), it seems that with this 2024 bridge MTU, even normal 1500
byte MTU frames are dropped.

Through experimentation, by sending packets  and changing the MTU of
the real and bridge interfaces, it seems that the bridge will only
forward frames up to 1527 bytes MTU. Fortunately I think this will be
enough to diagnose the fault we are having.

Is this expected behaviour ? A maximum of 1527 bytes seems to be an
unsual value for the maxiumum supported bridge MTU.

The kernel in question is the Arch Linux 2.6.16.17 kernel. Having a
brief look at the package details
(http://cvs.archlinux.org/cgi-bin/viewcvs.cgi/kernels/kernel26/PKGBUILD.diff?r1=text&tr1=1.129&r2=text&tr2=1.128&diff_format=h),
 it doesn't seem that there have been any bridge patches, so I'd guess
that the bridge code is vanilla kernel.org.

Thanks,
Mark.

Re: [Bridge] Problem bridging frames with bridge and real interface MTU > 1527

[Stephen Hemminger]

On Tue, 23 May 2006 10:07:13 +0930
"Mark Zipp" <mark.r.zipp@gmail.com> wrote:

> Hi,
> 
> I'm doing some fault finding where we suspect that the (non-linux)
> device with the fault is generating frames with an MTU / maximum size
> greater than 1500 / 1518 bytes. This device is performing GRE
> tunneling, and it seems that it isn't performing fragmentation or
> PMTUD before it attempts to send a normal, 1500 byte IP packet with
> the GRE header out a GigE interface with a 1500 byte MTU.
> 
> I've been asked by the vendor of this device to perform some packet
> captures. As I have a Linux box with a couple of interfaces that
> support > 1500 byte MTUs (an Intel GigE, and a Netgear FA311 (that
> supports an MTU up to 2024 bytes)), I figured I could bridge this
> traffic through this Linux box, and then perform some packet captures
> on the bridge virtual interface.
> 
> I set that all up, and it seemed to all be ok. I set both the real
> interface MTUs to 2024 bytes, and when I added them into a bridge, the
> bridge inherited this MTU.
> 
> Unfortunately I found that when I attempted to forward frames through
> this bridge configuration, certain sized ones were silently dropped.
> As the devices attached to the bridge shouldn't be generating frames
> with an MTU larger than 1500 (this is our work around solution as a
> reference), it seems that with this 2024 bridge MTU, even normal 1500
> byte MTU frames are dropped.
> 
> Through experimentation, by sending packets  and changing the MTU of
> the real and bridge interfaces, it seems that the bridge will only
> forward frames up to 1527 bytes MTU. Fortunately I think this will be
> enough to diagnose the fault we are having.
> 
> Is this expected behaviour ? A maximum of 1527 bytes seems to be an
> unsual value for the maxiumum supported bridge MTU.

The only bridge related dropping is here in br_forward.c

int br_dev_queue_push_xmit(struct sk_buff *skb)
{
	/* drop mtu oversized packets except tso */
	if (packet_length(skb) > skb->dev->mtu && !skb_shinfo(skb)->tso_size)
		kfree_skb(skb);

Are you doing VLAN's?  There was a bug not fixed until 2.6.17 that didn't
account for VLAN header properly.

Are you doing filtering?  Some netfilter module might be dropping the packets.

Re: [Bridge] Problem bridging frames with bridge and real interface MTU > 1527

[Mark Zipp]

Hi Stephen,

Thanks very much for getting back to me.

On 24/05/06, Stephen Hemminger <shemminger@osdl.org> wrote:
> On Tue, 23 May 2006 10:07:13 +0930
> "Mark Zipp" <mark.r.zipp@gmail.com> wrote:
>

<snip>
>
> The only bridge related dropping is here in br_forward.c
>
> int br_dev_queue_push_xmit(struct sk_buff *skb)
> {
>         /* drop mtu oversized packets except tso */
>         if (packet_length(skb) > skb->dev->mtu && !skb_shinfo(skb)->tso_size)
>                 kfree_skb(skb);
>
> Are you doing VLAN's?  There was a bug not fixed until 2.6.17 that didn't
> account for VLAN header properly.
>

No I'm not doing VLANs, just plain IP inside ethernet frames.

There was a new ArchLinux kernel package available, which, going by
its version number, corresponds to the 2.6.16.18 kernel. I've
installed that, and been able to send 2000 byte IP pings through the
bridge, so my problem seems to have disappeared.

> Are you doing filtering?  Some netfilter module might be dropping the packets.
>

I did have some filtering enabled originally, I've just tested my 2K
pings with and without having blanket permit lines in the incoming and
outgoing iptables rules, and have had complete success in either case.

I haven't tested it with the traffic IP/GRE traffic I originally
encountered it with yesterday, I doubt I'll now have a problem with
it, I'll let you know if I do.

(I'll create a new email to the list to start a new thread about
another thing that I think could be another bug in the bridge code -
ports go through some, if not all of the STP port states even when STP
is not running)

Thanks for your help,
Mark.

Re: [Bridge] Problem bridging frames with bridge and real interface MTU > 1527

[Stephen Hemminger]

Mark Zipp wrote:
> Hi Stephen,
>
> Thanks very much for getting back to me.
>
> On 24/05/06, Stephen Hemminger <shemminger@osdl.org> wrote:
>> On Tue, 23 May 2006 10:07:13 +0930
>> "Mark Zipp" <mark.r.zipp@gmail.com> wrote:
>>
>
> <snip>
>>
>> The only bridge related dropping is here in br_forward.c
>>
>> int br_dev_queue_push_xmit(struct sk_buff *skb)
>> {
>>         /* drop mtu oversized packets except tso */
>>         if (packet_length(skb) > skb->dev->mtu && 
>> !skb_shinfo(skb)->tso_size)
>>                 kfree_skb(skb);
>>
>> Are you doing VLAN's?  There was a bug not fixed until 2.6.17 that 
>> didn't
>> account for VLAN header properly.
>>
>
> No I'm not doing VLANs, just plain IP inside ethernet frames.
>
> There was a new ArchLinux kernel package available, which, going by
> its version number, corresponds to the 2.6.16.18 kernel. I've
> installed that, and been able to send 2000 byte IP pings through the
> bridge, so my problem seems to have disappeared.
>
>> Are you doing filtering?  Some netfilter module might be dropping the 
>> packets.
>>
>
> I did have some filtering enabled originally, I've just tested my 2K
> pings with and without having blanket permit lines in the incoming and
> outgoing iptables rules, and have had complete success in either case.
>
> I haven't tested it with the traffic IP/GRE traffic I originally
> encountered it with yesterday, I doubt I'll now have a problem with
> it, I'll let you know if I do.
>
> (I'll create a new email to the list to start a new thread about
> another thing that I think could be another bug in the bridge code -
> ports go through some, if not all of the STP port states even when STP
> is not running)
>
> Thanks for your help,
> Mark.
The problem (as I remember it) was often with ip_conntrack. It 
reassembles fragmented
IP packets to look at them, then didn't fragment on output.

Re: [Bridge] Problem bridging frames with bridge and real interface MTU > 1527

[Mark Zipp]

Hi Stephen,


On 24/05/06, Stephen Hemminger <shemminger@osdl.org> wrote:
<snip>

> > Thanks for your help,
> > Mark.
> The problem (as I remember it) was often with ip_conntrack. It
> reassembles fragmented
> IP packets to look at them, then didn't fragment on output.
>

Strange thing was that even after I was able to successfully do 2024
byte pings through the bridge, with and without iptables connection
tracking rules in place, I still encountered the problem with the
oversized IP/GRE traffic that I'm trying to capture. What I think is
also interesting is that even though when I set the MTUs on the
ethernet interfaces and the bridge to 1527, not only does it all work,
the bridge is also forwarding frames that have a 1528 byte MTU (IP +
GRE + original 1500 byte IP packet), which I think should make the
bridge drop the frames as too large, yet setting the MTU 1528 on the
bridge and interfaces makes it fail.

Maybe it is the different ways IP connection tracking treats ICMP /
Ping traffice verses IP/GRE traffic that is causing the issue.

I've got some more testing to do again, this time around I'll try the
2024 byte MTU on the bridge, and a blind iptables permit all to and
from the bridge interfaces, which should hopefully eliminate ip
connection tracking from the equation, therefore identifying it as the
cause.

Thanks for your time,
Mark.

Re: [Bridge] Problem bridging frames with bridge and real interface MTU > 1527

[Stephen Hemminger]

On Thu, 1 Jun 2006 10:10:28 +0930
"Mark Zipp" <mark.r.zipp@gmail.com> wrote:

> Hi Stephen,
> 
> 
> On 24/05/06, Stephen Hemminger <shemminger@osdl.org> wrote:
> <snip>
> 
> > > Thanks for your help,
> > > Mark.
> > The problem (as I remember it) was often with ip_conntrack. It
> > reassembles fragmented
> > IP packets to look at them, then didn't fragment on output.
> >
> 
> Strange thing was that even after I was able to successfully do 2024
> byte pings through the bridge, with and without iptables connection
> tracking rules in place, I still encountered the problem with the
> oversized IP/GRE traffic that I'm trying to capture. What I think is
> also interesting is that even though when I set the MTUs on the
> ethernet interfaces and the bridge to 1527, not only does it all work,
> the bridge is also forwarding frames that have a 1528 byte MTU (IP +
> GRE + original 1500 byte IP packet), which I think should make the
> bridge drop the frames as too large, yet setting the MTU 1528 on the
> bridge and interfaces makes it fail.
> 
> Maybe it is the different ways IP connection tracking treats ICMP /
> Ping traffice verses IP/GRE traffic that is causing the issue.
> 
> I've got some more testing to do again, this time around I'll try the
> 2024 byte MTU on the bridge, and a blind iptables permit all to and
> from the bridge interfaces, which should hopefully eliminate ip
> connection tracking from the equation, therefore identifying it as the
> cause.
> 
> Thanks for your time,
> Mark.

You may want to either add some debug tracing or just try capturing packets
to see the resulting traffic.  I suspect connection tracking may be the
culprit since it often does higher layer filtering.