RE: e1000 performance hack for ppc64 (Power4)

RE: e1000 performance hack for ppc64 (Power4)

[Feldman, Scott]

> > Why not instead find out if it's possible to have the e1000 
> > fetch the entire cache line where the first byte of the
> > packet resides?  Even ancient designes like SunHME do that.
> 
> Rusty and I were wondering why the e1000 didnt do that exact thing.
> 
> Scott: is it possible to enable such a thing?

I thought the answer was no, so I double checked with a couple of
hardware guys, and the answer is still no.

-scott

Re: e1000 performance hack for ppc64 (Power4)

[David S. Miller]

   From: "Feldman, Scott" <scott.feldman@intel.com>
   Date: Fri, 13 Jun 2003 16:52:18 -0700

   > > Why not instead find out if it's possible to have the e1000 
   > > fetch the entire cache line where the first byte of the
   > > packet resides?  Even ancient designes like SunHME do that.
   > 
   > Rusty and I were wondering why the e1000 didnt do that exact thing.
   > 
   > Scott: is it possible to enable such a thing?
   
   I thought the answer was no, so I double checked with a couple of
   hardware guys, and the answer is still no.

Sigh...

So Anton, when the PCI controller gets a set of sub-cacheline word
reads from the device, it reads the value from memory once for every
one of those words?  ROFL, if so...  I can't believe they wouldn't
put caches on the PCI controller for this, at least a one-behind that
snoops the bus :(

Re: e1000 performance hack for ppc64 (Power4)

[Anton Blanchard]

 
> So Anton, when the PCI controller gets a set of sub-cacheline word
> reads from the device, it reads the value from memory once for every
> one of those words?  ROFL, if so...  I can't believe they wouldn't
> put caches on the PCI controller for this, at least a one-behind that
> snoops the bus :(

There is a cache in the host bridge and the PCI-PCI bridge. I dont
think we go back to memory for sub cacheline reads.

What I think is happening is that we arent tripping the prefetch logic.
We should take a latency hit for only the first cacheline at which point
the host bridge decides to start prefetching for us. If not then we take
take the latency hit on each transaction.

Anton

Re: e1000 performance hack for ppc64 (Power4)

[David S. Miller]

   From: Anton Blanchard <anton@samba.org>
   Date: Sat, 14 Jun 2003 10:55:34 +1000

   What I think is happening is that we arent tripping the prefetch
   logic.  We should take a latency hit for only the first cacheline
   at which point the host bridge decides to start prefetching for
   us. If not then we take take the latency hit on each transaction.

It sounds like what happens is that the sub-cacheline word reads
don't trigger the prefetch, but the first PCI read multiple
transaction does.

Re: e1000 performance hack for ppc64 (Power4)

[Anton Blanchard]

 
> I thought the answer was no, so I double checked with a couple of
> hardware guys, and the answer is still no.

Hi Scott,

Thats a pity, the e100 docs on sourceforge show it can do what we want,
it would be nice if e1000 had this feature too :)

4.2.2 Read Align

The Read Align feature is aimed to enhance performance in cache line
oriented systems. Starting a PCI transaction in these systems on a
non-cache line aligned address may result in low  performance. To solve
this performance problem, the controller can be configured to terminate
Transmit DMA cycles on a cache line boundary, and start the next
transaction on a cache line aligned address. This  feature is enabled
when the Read Align Enable bit is set in device Configure command
(Section 6.4.2.3, "Configure (010b)").

If this bit is set, the device operates as follows:

* When the device is close to running out of resources on the Transmit
* DMA (in other words, the Transmit FIFO is almost full), it attempts to
* terminate the read transaction on the nearest cache line boundary when 
* possible.

* When the arbitration counters feature is enabled (maximum Transmit DMA
* byte count value is set in configuration space), the device switches
* to other pending DMAs on cache line boundary only.

RE: e1000 performance hack for ppc64 (Power4)

[Feldman, Scott]

> Scott, would you be pleased if something was implemented out 
> of the driver, in generic net code?  Something that all the 
> drivers could use, even if nothing but e1000 used it for now.

I suppose the driver could unconditionally call something like
skb_realign_for_broken_hw, which is a nop on non-broken archs, but would
it make more sense to not have the driver mess with the skb at all?

-scott

RE: e1000 performance hack for ppc64 (Power4)

[Feldman, Scott]

Herman wrote:
> Its only the MTU 1500 case with non-TSO that we are 
> discussing here so copying a few bytes is really not a big 
> deal as the data is already in cache from copying into 
> kernel.  If it lets the adapter run at speed, thats what 
> customers want and what we need. Granted, if the HW could 
> deal with this we would not have to, but thats not the case 
> today so I want to spend a few CPU cycles to get best 
> performance. Again, if this is not done on other platforms, I 
> don't understand why you care.

I care because adding the arch-specific hack creates a maintenance issue
for me.

-scott

RE: e1000 performance hack for ppc64 (Power4)

[Dave Hansen]

On Mon, 2003-06-16 at 11:21, Feldman, Scott wrote:
> Herman wrote:
> > Its only the MTU 1500 case with non-TSO that we are 
> > discussing here so copying a few bytes is really not a big 
> > deal as the data is already in cache from copying into 
> > kernel.  If it lets the adapter run at speed, thats what 
> > customers want and what we need. Granted, if the HW could 
> > deal with this we would not have to, but thats not the case 
> > today so I want to spend a few CPU cycles to get best 
> > performance. Again, if this is not done on other platforms, I 
> > don't understand why you care.
> 
> I care because adding the arch-specific hack creates a maintenance issue
> for 

Scott, would you be pleased if something was implemented out of the
driver, in generic net code?  Something that all the drivers could use,
even if nothing but e1000 used it for now.

-- 
Dave Hansen
haveblue@us.ibm.com

Re: e1000 performance hack for ppc64 (Power4)

[Herman Dierks]

Look folks,   we run 40 to 48  GigE adapters in a p690 32 way on AIX and
they basically all run at full speed  so let me se you try that on most of
these other boxes you are talking about.     Same adapter, same hardware
logic.
I have also seen what many of these other boxes you talk about do when data
or structures are not aligned on 64 bit boundaries.
The PPC HW does not have those 64bit alignment  issues.   So, each machine
has some warts.  Have yet to see a perfect one.

If you want a lot of PCI adapters on a box, it takes a number of bridge
chips and other IO links to do that.
Memory controllers like to deal with cache lines.
For larger packets, like jumbo frames or large send (TSO), the few added
DMA's is not an issue as the packets are so large the DMA soon get aligned
and are not an issue.   With TSO being the default,   the small packet case
becomes less important anyway.   Its more an issue on 2.4 where TSO is not
provided.  We also want this to run well if someone does not want to use
TSO.

Its only the MTU 1500 case with non-TSO that we are discussing here so
copying a few bytes is really not a big deal as the data is already in
cache from copying into kernel.  If it lets the adapter run at speed, thats
what customers want and what we need.
Granted, if the HW could deal with this we would not have to, but thats not
the case today so I want to spend a few CPU cycles to get best performance.
Again, if this is not done on other platforms, I don't understand why you
care.

If we have to do this for PPC port, fine.   I have not seen any of you
suggest a better solution that works and will not be a worse hack to TCP or
other code.  Anton tried various other ideas before we fell back to doing
it the same way we did this in AIX.   This code is very localized and is
only used by platforms that need it.  Thus I don't see the big issue here.

Herman


"David S. Miller" <davem@redhat.com> on 06/14/2003 01:08:50 AM

To:    ltd@cisco.com
cc:    anton@samba.org, haveblue@us.ltcfwd.linux.ibm.com, Herman
       Dierks/Austin/IBM@IBMUS, scott.feldman@intel.com, dwg@au1.ibm.com,
       linux-kernel@vger.kernel.org, Nancy J Milliner/Austin/IBM@IBMUS,
       Ricardo C Gonzalez/Austin/IBM@ibmus, Brian
       Twichell/Austin/IBM@IBMUS, netdev@oss.sgi.com
Subject:    Re: e1000 performance hack for ppc64 (Power4)



   From: Lincoln Dale <ltd@cisco.com>
   Date: Sat, 14 Jun 2003 15:52:35 +1000

   can we have the TCP retransmit side take a performance hit if it needs
   to
   realign buffers?

You don't understand, the person who mangles the packet
must make the copy, not the person not doing the packet
modifications.

   for a "high performance app" requiring gigabit-type speeds,

...we probably won't be using ppc64 and e1000 cards, yes, I agree
:-)

Anton, go to the local computer store and pick up some tg3
 cards or a bunch of Taiwan specials :-)

Re: e1000 performance hack for ppc64 (Power4)

[David S. Miller]

   From: "Herman Dierks" <hdierks@us.ibm.com>
   Date: Sun, 15 Jun 2003 09:40:41 -0500
   
   With TSO being the default,   the small packet case
   becomes less important anyway.

This is a very narrow and unrealistic view of the situation.

Every third packet your system will process for any connection will be
an ACK, a small packet.  Most database and web and database
transactions happen using small packets for the transaction request.

Look, if you're gonna sit here and just rant justifying this bogus
behavior of your hardware, it is likely to go in one ear and out the
other.  Nobody wants to hear excuses. :)

The fact is, this system handles sub-cacheline reads inefficiently
even if a sequences of transactions are consequetive and to the same
cache line and no coherency transactions occur to that cache line.

That is dumb, and there is no arguing around this.  You would be
sensible to realize this, and accept it whilst others try to help you
find a solution for your problem.

Re: e1000 performance hack for ppc64 (Power4)

[Nivedita Singhvi]

Herman Dierks wrote:
> Look folks,   we run 40 to 48  GigE adapters in a p690 32 way on AIX and
> they basically all run at full speed  so let me se you try that on most of
> these other boxes you are talking about.     Same adapter, same hardware
> logic.

FWIW, I think that's pretty cool. Not easy to do. :)

> For larger packets, like jumbo frames or large send (TSO), the few added
> DMA's is not an issue as the packets are so large the DMA soon get aligned
> and are not an issue.   With TSO being the default,   the small packet case
> becomes less important anyway.   Its more an issue on 2.4 where TSO is not
> provided.  We also want this to run well if someone does not want to use
> TSO.

Slightly off-topic, but TSO being enabled and TSO being used are
two different things, right? Ditto jumbo frames..How often is this
the actual env in real world situations? I'm concerned that this
is more typical of development testing/performance testing
environments.

> Its only the MTU 1500 case with non-TSO that we are discussing here so

Which still is the pretty important case, I think..

> copying a few bytes is really not a big deal as the data is already in
> cache from copying into kernel.  If it lets the adapter run at speed, thats
> what customers want and what we need.

Yep.

> Granted, if the HW could deal with this we would not have to, but thats not
> the case today so I want to spend a few CPU cycles to get best performance.
> Again, if this is not done on other platforms, I don't understand why you
> care.

Still would be nice to put in the best solution possible, i.e.
address it for the broadest set of affected pieces and minimizing
the impact..

> If we have to do this for PPC port, fine.   I have not seen any of you

Hope it doesn't have to come to that..It would be nice to
see it in the mainline kernel. Regardless of platform, distro, etc..
these users are still people who are taking the time, the effort to
adopt Linux and sometimes in environments and situations that are
pretty critical. Change and innovation are difficult activities
to engage in some places, and anything we can do to make this
a no-brainer solution for them, and their decisions shine, thats
gotta be worth something to go the extra mile for :)

thanks,
Nivedita

Re: e1000 performance hack for ppc64 (Power4)

[Herman Dierks]

Anton, I think the option described below is intended to cause the adapter
to "get off on a cache line boundary" so when it restarts the DMA it will
be aligned.   This is for cases when the adapter has to get off, for exampe
due to FIFO full, etc.
Some adapters would get off on any boundary and that then causes perf
issues when the DMA is restarted.
This is a good option, but I don't think it addresses what we need here as
the host needs to ensure a DMA starts on a cache line.
Different adapter anyway, but  I am just pointing out that even if e1000
had this it would not be the solution.


Anton Blanchard <anton@samba.org> on 06/13/2003 07:03:42 PM

To:    "Feldman, Scott" <scott.feldman@intel.com>
cc:    "David S. Miller" <davem@redhat.com>,
       haveblue@us.ltcfwd.linux.ibm.com, Herman Dierks/Austin/IBM@IBMUS,
       dwg@au1.ibm.com, linux-kernel@vger.kernel.org, Nancy J
       Milliner/Austin/IBM@IBMUS, Ricardo C Gonzalez/Austin/IBM@ibmus,
       Brian Twichell/Austin/IBM@IBMUS, netdev@oss.sgi.com
Subject:    Re: e1000 performance hack for ppc64 (Power4)




> I thought the answer was no, so I double checked with a couple of
> hardware guys, and the answer is still no.

Hi Scott,

Thats a pity, the e100 docs on sourceforge show it can do what we want,
it would be nice if e1000 had this feature too :)

4.2.2 Read Align

The Read Align feature is aimed to enhance performance in cache line
oriented systems. Starting a PCI transaction in these systems on a
non-cache line aligned address may result in low  performance. To solve
this performance problem, the controller can be configured to terminate
Transmit DMA cycles on a cache line boundary, and start the next
transaction on a cache line aligned address. This  feature is enabled
when the Read Align Enable bit is set in device Configure command
(Section 6.4.2.3, "Configure (010b)").

If this bit is set, the device operates as follows:

* When the device is close to running out of resources on the Transmit
* DMA (in other words, the Transmit FIFO is almost full), it attempts to
* terminate the read transaction on the nearest cache line boundary when
* possible.

* When the arbitration counters feature is enabled (maximum Transmit DMA
* byte count value is set in configuration space), the device switches
 * to other pending DMAs on cache line boundary only.

RE: e1000 performance hack for ppc64 (Power4)

[Herman Dierks]

I will let Anton respond to this.   I think he may have tried this some
time back in his early prototypes to fix this.
I think the problem was not where the buffer started but where the  packet
ended up within the buffer.
Due to varying sizes of TCP and IP headers the packet ended up at some
non-cache aligned address.
What we need for the DMA to work well is to have the final packet  (with
datalink headers)  starting on a cache line as its the final packet that
must be DMA'd. In fact it may need to to be aligned to a higher level than
that (not sure).


haveblue@us.ltcfwd.linux.ibm.com on 06/13/2003 11:21:03 AM

To:    Herman Dierks/Austin/IBM@IBMUS
cc:    "Feldman, Scott" <scott.feldman@intel.com>, David Gibson
       <dwg@au1.ibm.com>, Linux Kernel Mailing List
       <linux-kernel@vger.kernel.org>, Anton Blanchard <anton@samba.org>,
       Nancy J Milliner/Austin/IBM@IBMUS, Ricardo C
       Gonzalez/Austin/IBM@ibmus, Brian Twichell/Austin/IBM@IBMUS,
       netdev@oss.sgi.com
Subject:    RE: e1000 performance hack for ppc64 (Power4)



Too long to quote:
http://marc.theaimsgroup.com/?t=105538879600001&r=1&w=2

Wouldn't you get most of the benefit from copying that stuff around in
the driver if you allocated the skb->data aligned in the first place?

There's already code to align them on CPU cache boundaries:
#define SKB_DATA_ALIGN(X)       (((X) + (SMP_CACHE_BYTES - 1)) & \
                                 ~(SMP_CACHE_BYTES - 1))

So, do something like this:
#ifdef ARCH_ALIGN_SKB_BYTES
#define SKB_ALIGN_BYTES ARCH_ALIGN_SKB_BYTES
#else
#define SKB_ALIGN_BYTES SMP_CACHE_BYTES
#endif
#define SKB_DATA_ALIGN(X)       (((X) + (ARCH_ALIGN_SKB - 1)) & \
                                 ~(SKB_ALIGN_BYTES - 1))

You could easily make this adaptive to no align on th arch size when the
request is bigger than that, just like in the e1000 patch you posted.
--
Dave Hansen
haveblue@us.ibm.com

RE: e1000 performance hack for ppc64 (Power4)

[Dave Hansen]

Too long to quote:
http://marc.theaimsgroup.com/?t=105538879600001&r=1&w=2

Wouldn't you get most of the benefit from copying that stuff around in
the driver if you allocated the skb->data aligned in the first place? 

There's already code to align them on CPU cache boundaries:
#define SKB_DATA_ALIGN(X)       (((X) + (SMP_CACHE_BYTES - 1)) & \
                                 ~(SMP_CACHE_BYTES - 1))

So, do something like this:
#ifdef ARCH_ALIGN_SKB_BYTES
#define SKB_ALIGN_BYTES ARCH_ALIGN_SKB_BYTES
#else
#define SKB_ALIGN_BYTES SMP_CACHE_BYTES
#endif
#define SKB_DATA_ALIGN(X)       (((X) + (ARCH_ALIGN_SKB - 1)) & \
                                 ~(SKB_ALIGN_BYTES - 1))

You could easily make this adaptive to no align on th arch size when the
request is bigger than that, just like in the e1000 patch you posted.  
-- 
Dave Hansen
haveblue@us.ibm.com

Re: e1000 performance hack for ppc64 (Power4)

[Anton Blanchard]

> Wouldn't you get most of the benefit from copying that stuff around in
> the driver if you allocated the skb->data aligned in the first place? 

Nice try, but my understanding is that on the transmit path we reserve
the maximum sized TCP header, copy the data in then form our TCP header
backwards from that point. Since the TCP header size changes with
various options, its not an easy task.

One thing I thought of doing was to cache the current TCP header size
and align the next packet based on it, with an extra cacheline at the
start for it to spill into if the TCP header grew.

This is only worth it if most packets will have the same sized header.
Networking guys: is this a valid assumption?

Anton

Re: e1000 performance hack for ppc64 (Power4)

[David S. Miller]

   From: Anton Blanchard <anton@samba.org>
   Date: Sat, 14 Jun 2003 08:38:41 +1000
   
   This is only worth it if most packets will have the same sized header.
   Networking guys: is this a valid assumption?

Not really... one retransmit and the TCP header size grows
due to the SACK options.

I find it truly bletcherous what you're trying to do here.

Why not instead find out if it's possible to have the e1000
fetch the entire cache line where the first byte of the packet
resides?  Even ancient designes like SunHME do that.

Re: e1000 performance hack for ppc64 (Power4)

[Anton Blanchard]

> Not really... one retransmit and the TCP header size grows
> due to the SACK options.

OK scratch that idea.

> I find it truly bletcherous what you're trying to do here.

I think so too, but its hard to ignore ~100Mbit/sec in performance.

> Why not instead find out if it's possible to have the e1000
> fetch the entire cache line where the first byte of the packet
> resides?  Even ancient designes like SunHME do that.

Rusty and I were wondering why the e1000 didnt do that exact thing.

Scott: is it possible to enable such a thing?

Anton

Re: e1000 performance hack for ppc64 (Power4)

[Lincoln Dale]

At 09:18 AM 14/06/2003 +1000, Anton Blanchard wrote:

> > Not really... one retransmit and the TCP header size grows
> > due to the SACK options.
>
>OK scratch that idea.

why not have a performance option that is a tradeoff between optimum 
payload size versus efficiency.

unless i misunderstand the problem, you can certainly pad the TCP options 
with NOPs ...

> > I find it truly bletcherous what you're trying to do here.
>
>I think so too, but its hard to ignore ~100Mbit/sec in performance.

another option is for the write() path is for instantant-send TCP sockets 
to delay the copy_from_user() until the IP+TCP header size is known.
i wouldn't expect the net folks to like that, however ..


cheers,

lincoln.

Re: e1000 performance hack for ppc64 (Power4)

[David S. Miller]

   From: Lincoln Dale <ltd@cisco.com>
   Date: Sat, 14 Jun 2003 11:52:53 +1000
   
   unless i misunderstand the problem, you can certainly pad the TCP
   options with NOPs ...

You may not mangle packet if it is not your's alone.

And every TCP packet is shared with TCP retransmit
queue and therefore would need to be copied before
being mangled.

Re: e1000 performance hack for ppc64 (Power4)

[Lincoln Dale]

At 10:41 PM 13/06/2003 -0700, David S. Miller wrote:
>    From: Lincoln Dale <ltd@cisco.com>
>    Date: Sat, 14 Jun 2003 11:52:53 +1000
>
>    unless i misunderstand the problem, you can certainly pad the TCP
>    options with NOPs ...
>
>You may not mangle packet if it is not your's alone.
>
>And every TCP packet is shared with TCP retransmit
>queue and therefore would need to be copied before
>being mangled.

ok, so lets take this a step further.

can we have the TCP retransmit side take a performance hit if it needs to 
realign buffers?

once again, for a "high performance app" requiring gigabit-type speeds, its 
probably fair to say that this is mostly in the realm of applications on a 
LAN rather than across a WAN or internet.
on a switched LAN, i'd expect TCP retransmissions to be far fewer ...


cheers,

lincoln.

Re: e1000 performance hack for ppc64 (Power4)

[David S. Miller]

   From: Lincoln Dale <ltd@cisco.com>
   Date: Sat, 14 Jun 2003 15:52:35 +1000
   
   can we have the TCP retransmit side take a performance hit if it needs to 
   realign buffers?
   
You don't understand, the person who mangles the packet
must make the copy, not the person not doing the packet
modifications.

   for a "high performance app" requiring gigabit-type speeds,

...we probably won't be using ppc64 and e1000 cards, yes, I agree
:-)

Anton, go to the local computer store and pick up some tg3
cards or a bunch of Taiwan specials :-)

Re: e1000 performance hack for ppc64 (Power4)

[David S. Miller]

Folks, can we remove whatever member of this CC: list creates
bounces that say:

Your message to Linux_news awaits moderator approval

Ok?  I can't guess which one it is because these all look
like normal people's email addresses (except possibly the
haveblue@us.ibm.com thing, maybe that's the one)

Re: e1000 performance hack for ppc64 (Power4)

[William Lee Irwin III]

On Fri, Jun 13, 2003 at 11:14:18PM -0700, David S. Miller wrote:
> Folks, can we remove whatever member of this CC: list creates
> bounces that say:
> Your message to Linux_news awaits moderator approval
> Ok?  I can't guess which one it is because these all look
> like normal people's email addresses (except possibly the
> haveblue@us.ibm.com thing, maybe that's the one)

That's legitimate, but I'm still trying to convince the BKL brigade
that he should have chosen something more eponymous e.g. dhansen. =)


-- wli

Re: e1000 performance hack for ppc64 (Power4)

[Greg KH]

On Fri, Jun 13, 2003 at 11:14:18PM -0700, David S. Miller wrote:
> 
> Folks, can we remove whatever member of this CC: list creates
> bounces that say:
> 
> Your message to Linux_news awaits moderator approval

It's someone subscribed to linux-kernel@vger.kernel.org that causes
this.

I've complained to the admin of that mail-news gateway that is barfing
on too many CC: members in the past to not do this, but it doesn't seem
like they are listening...

greg k-h

Re: e1000 performance hack for ppc64 (Power4)

[David S. Miller]

   From: Greg KH <greg@kroah.com>
   Date: Sat, 14 Jun 2003 10:08:49 -0700

   It's someone subscribed to linux-kernel@vger.kernel.org that causes
   this.
   
Thanks, I've nuked linux_news@nextphere.com, if they resubscribe
without contacting me, I'll block future subscription attempts from
them.

Re: e1000 performance hack for ppc64 (Power4)

[Nivedita Singhvi]

David S. Miller wrote:
>    From: Anton Blanchard <anton@samba.org>
>    Date: Sat, 14 Jun 2003 08:38:41 +1000
>    
>    This is only worth it if most packets will have the same sized header.
>    Networking guys: is this a valid assumption?
> 
> Not really... one retransmit and the TCP header size grows
> due to the SACK options.

Yep, but it really doesn't have too many options (sic pun ;))..
i.e. The max the options can add are 40 bytes, speaking
strictly TCP, not IP. This really should fit into one extra
cacheline for most architectures, at most, right?

[The TCP options have to end and the data start on a 32
bit boundary. For established connections, we're
principally talking SACK options and v. likely timestamp.
(Ignoring those egregious benchmark guys who turn everything
useful off ;)). SYNs wont have data in any case.

So its going to grow by (SACK = 8*n + 2)+ (TS = 10) bytes,
with n = number of sack options, with a max of n = 3
if timestamps are enabled. Adding that to the standard
length of 20 bytes, the total len of a TCP header is thus
very likely one of:
	20  + [ 0 | 20 |32 | 36] bytes
	= 20 | 40 | 52 | 56 bytes.

If cachelines were 64 bytes, we wouldnt be wasting a
whole lot of space if we aligned data start or some
other scheme as was suggested. Even given the larger
cachelines, it might be worth it, or is this totally
not an option (cough,sic ;))?

> I find it truly bletcherous what you're trying to do here.

yep

thanks,
Nivedita

Re: e1000 performance hack for ppc64 (Power4)

[David S. Miller]

   From: Nivedita Singhvi <niv@us.ibm.com>
   Date: Fri, 13 Jun 2003 22:16:22 -0700

   Yep, but it really doesn't have too many options (sic pun ;))..
   i.e. The max the options can add are 40 bytes, speaking
   strictly TCP, not IP. This really should fit into one extra
   cacheline for most architectures, at most, right?
   
It's what the bottom of the header is aligned to, but
we build the packet top to bottom not the other way around.