Re: e1000 and 802.1ad/stacked vlan tagging

Re: e1000 and 802.1ad/stacked vlan tagging

[Stephan von Krawczynski]

Hello Jesse,

thank you for answering anyway. Though I think your answer covers only the
obvious half of the problem.
Indeed one might think that this solves the issue - as long as there are only
linux kernels involved. Unfortunately my setup is a bit more complicated in
terms of hardware. So I should have probably clarified the question this way:
how do you configure the interface in a manner that packets with data length
of 1500 get transferred, and not only 1496 ?
I tried enlarging both real-device and first vlan interface mtu but that does
not work out. I really thought that the visible device setting of mtu=1500
should have worked out and that the driver (or some code in between) should
have corrected the allowed frame size to reflect the actual setup, not?

Regards,
Stephan

PS: crossposted to both lists, list-members keep in mind I am not subscribed
when answering! Thank you.



On Mon, 28 Aug 2006 10:23:09 -0700
"Brandeburg, Jesse" <jesse.brandeburg@intel.com> wrote:

> Stephan von Krawczynski wrote:
> > Hello Jesse,
> > 
> > sorry to bother you directly, but since you did the patch for my e1000
> > interrupt problem last time (February) I hope you have a short-hand
> > idea for my current issue, too.
> > 
> > I am trying to make stacked vlan tagging work under kernel 2.4 with
> > e1000. Generally I do this on two boxes connected back-to-back:
> > 
> > ifconfig eth0 up
> > vconfig add eth0 4094
> > ifconfig eth0.4094 up
> > vconfig add eth0.4094 1
> > ifconfig eth0.4094.1 192.168.1.1 netmask 255.255.255.0 broadcast
> > 192.168.1.255 up
> > 
> > (of course the second box gets another ip, lets assume .2).
> > 
> > if you do a
> > 
> > ping -s 1472 192.168.1.2
> > 
> > through the stacked vlan you see the packets vanish.
> > With
> > 
> > ping -s 1468 192.168.1.2
> > 
> > everything seems ok.
> > 
> > I have the impression that the stacked vlans show some problem with
> > mtu handling inside the e1000 driver. Mtu is set to 1500 but because
> > stacking tags uses 4 bytes more the packets cannot use the full mtu.
> > Any ideas what happens here?
> 
> The packet is being dropped because it is longer than the allowed frame
> size for 1500 MTU.  check ethtool -S eth0
> 
> mine shows 
> rx_long_length_errors: 169
> 
> which indicates that you need to change your mtu on the stacked
> interface to 1496, at which point after I did:
> 
> ip l s eth1.4094.1 mtu 1496
> 
> on both sides of my connection, everything was working.  I think in this
> case it is just a configuration problem.  When you stack vlans you have
> to account for the extra inserted length someplace and that place is by
> reducing the MTU.
> 
> I'd appreciate it in the future if you could use
> e1000-devel@lists.sf.net or netdev@vger.kernel.org for support questions
> like this because I'm not the only one who can answer questions (and I
> might have been on vacation! :-) )
> 
> Jesse
> 


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: e1000 and 802.1ad/stacked vlan tagging

[Ben Greear]

Stephan von Krawczynski wrote:
> Hello Jesse,
> 
> thank you for answering anyway. Though I think your answer covers only the
> obvious half of the problem.
> Indeed one might think that this solves the issue - as long as there are only
> linux kernels involved. Unfortunately my setup is a bit more complicated in
> terms of hardware. So I should have probably clarified the question this way:
> how do you configure the interface in a manner that packets with data length
> of 1500 get transferred, and not only 1496 ?
> I tried enlarging both real-device and first vlan interface mtu but that does
> not work out. I really thought that the visible device setting of mtu=1500
> should have worked out and that the driver (or some code in between) should
> have corrected the allowed frame size to reflect the actual setup, not?

Unless you are patching the VLAN code, stacked VLANs are not going to 
work anyway.  Search the archives of the VLAN mailing list for reasons 
why..and at least a few patches that 'fix' the problem for a few types 
of uses.

Ben

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: e1000 and 802.1ad/stacked vlan tagging

[Brandeburg, Jesse]

Stephan von Krawczynski wrote:
> thank you for answering anyway. Though I think your answer covers
> only the obvious half of the problem.
> Indeed one might think that this solves the issue - as long as there
> are only linux kernels involved. Unfortunately my setup is a bit more
> complicated in terms of hardware. So I should have probably clarified
> the question this way: how do you configure the interface in a manner
> that packets with data length of 1500 get transferred, and not only
> 1496 ? 

I tried setting mtu 2000 and everything worked with the virtual device
(both) mtu at 1500.
If you are getting long_packet errors at the mtu settings you tried (you
didn't mention which ones) then the hardware is dropping the packets due
to being over 1522 bytes in length (including CRC).

> I tried enlarging both real-device and first vlan interface mtu but
> that does not work out. I really thought that the visible device
> setting of mtu=1500 should have worked out and that the driver (or
> some code in between) should have corrected the allowed frame size to
> reflect the actual setup, not? 

unfortunately I believe that your hardware MTU on the base interface
MUST be adjusted in order to do stacked vlans because the vlan code
doesn't fragment packets, it just inserts tags.  The e1000 hardware is
capable of inserting/stripping 1 level of tags without dropping overlong
frames, but cannot seamlessly handle 1+n levels of inserted tags.
Transmit, we don't care how long the frames are that are given to us
(the driver doesn't enforce MTU on transmit) but on receive we have
limited space so it is important that each frame fit into the allocated
buffer (including CRC).

Please try MTU 1508 on eth0 (base interface only), as that configuration
worked for me.

Jesse

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: e1000 and 802.1ad/stacked vlan tagging

[Stephan von Krawczynski]

On Mon, 28 Aug 2006 13:53:36 -0700
"Brandeburg, Jesse" <jesse.brandeburg@intel.com> wrote:

> > I tried enlarging both real-device and first vlan interface mtu but
> > that does not work out. I really thought that the visible device
> > setting of mtu=1500 should have worked out and that the driver (or
> > some code in between) should have corrected the allowed frame size to
> > reflect the actual setup, not? 
> 
> unfortunately I believe that your hardware MTU on the base interface
> MUST be adjusted in order to do stacked vlans because the vlan code
> doesn't fragment packets, it just inserts tags.  The e1000 hardware is
> capable of inserting/stripping 1 level of tags without dropping overlong
> frames, but cannot seamlessly handle 1+n levels of inserted tags.
> Transmit, we don't care how long the frames are that are given to us
> (the driver doesn't enforce MTU on transmit) but on receive we have
> limited space so it is important that each frame fit into the allocated
> buffer (including CRC).
> 
> Please try MTU 1508 on eth0 (base interface only), as that configuration
> worked for me.

Hello Jesse,

I can confirm that the setup seems to work if enlarging the hw if mtu to 1508.
This is good.
Nevertheless I think that the receive buffer size of a physical device should
always be MAX(all virtual device mtu + tags and the like), you know what I
mean.
If I enlarge the basic mtu this may or may not have drawbacks concerning the
traffic routed directly via this interface.
Obviously the setup is somehow not consistent, as everything works well
without tuning mtus as long as you use only 1 tag. I always thought that the
driver uses at least hard_header_len + mtu as buffer size which should be
correct even in stacked setup. Only this information doesn't make it to the
driver level, right?

-- 
Regards,
Stephan von Krawczynski

RE: e1000 and 802.1ad/stacked vlan tagging

[Brandeburg, Jesse]

Stephan von Krawczynski wrote:
> I can confirm that the setup seems to work if enlarging the hw if mtu
> to 1508. This is good.

Okay, good to hear.

> Nevertheless I think that the receive buffer size of a
> physical device should always be MAX(all virtual device mtu +
> tags and the like), you know what I mean.

Unfortunately that would require us to currently allocate 16kB for every
receive descriptor, which would slow down the general case
significantly, not to mention bring up all sorts of errors as the system
memory fragmentation increases.

> If I enlarge the basic mtu this may or may not have drawbacks
> concerning the traffic routed directly via this interface.

I think that routed traffic won't care at all unless something checks
the path MTU, and even then it should take the lowest MTU.

> Obviously the setup is somehow not consistent, as everything
> works well without tuning mtus as long as you use only 1 tag.
1 tag is the way that things are supposed to work.  In fact, the
hardware takes care of all insertion and stripping of a single tag, so
that the data actually handed to the stack never even has the tag in the
packet data.

> I always thought that the driver uses at least
> hard_header_len + mtu as buffer size which should be correct
> even in stacked setup. Only this information doesn't make it
> to the driver level, right?

Actually I think that first statement isn't correct.  With the stacked
setup your maximum frame length is 1500 (MTU) + 14 enet + 4 (tag1) + 4
(tag2) + CRC = 1526.  The vlan code with stacked vlans is broken (from
the maintainer who mentioned it in this thread) probably because in
order to do stacked vlans correctly the vlan layer must support IP
fragmentation of some kind.

Seems like stacked vlans wouldn't work anyway, at least not very well.
The only way I see to get even close is the MTU increase on the base
interface.

Sorry to not offer much,
  Jesse