TCP/UDP checksum in hardware

TCP/UDP checksum in hardware

[Alexander Sirotkin]

It appears that Netfilter does not exploit the hardware TCP/UDP
checksum functionality present in some network cards, i.e.
NETIF_F_HW_CSUM. Is there any particular reason why it is not used ?

The reason I'm asking is that computing checksum (in case of NAT, for
instance) becomes a real problem on embedded devices

Re: TCP/UDP checksum in hardware

[Patrick McHardy]

Alexander Sirotkin wrote:
> It appears that Netfilter does not exploit the hardware TCP/UDP
> checksum functionality present in some network cards, i.e.
> NETIF_F_HW_CSUM. Is there any particular reason why it is not used ?

With pure NAT we do incremental checksumming, which shouldn't be
very expensive. One thing I want to do is offload checksumming in
the cases where we recalculate the entire checksum, but AFAIK thats
only when helpers mangle the packet.

> The reason I'm asking is that computing checksum (in case of NAT, for
> instance) becomes a real problem on embedded devices

Do you have any data to show this?

Re: TCP/UDP checksum in hardware

[Alexander Sirotkin]

On 3/4/07, Patrick McHardy <kaber@trash.net> wrote:
> Alexander Sirotkin wrote:
> > It appears that Netfilter does not exploit the hardware TCP/UDP
> > checksum functionality present in some network cards, i.e.
> > NETIF_F_HW_CSUM. Is there any particular reason why it is not used ?
>
> With pure NAT we do incremental checksumming, which shouldn't be
> very expensive. One thing I want to do is offload checksumming in
> the cases where we recalculate the entire checksum, but AFAIK thats
> only when helpers mangle the packet.
Oops. That's correct, as it turned out the problem was in the driver -
it did not report to the OS that it supports checksum offloading.
Incremental checksum of pure NAT is indeed not CPU intensive.
>
> > The reason I'm asking is that computing checksum (in case of NAT, for
> > instance) becomes a real problem on embedded devices
>
> Do you have any data to show this?
>
I don't know how relevant this is for netfilter, but yes - if the
device does not support checksum offloading my benchmark which I ran
on 266Mhz MIPS 24K (which is a pretty common processor for residential
gateways) showed that under 80Mbps UDP traffic, with NAT enabled,
checksum check takes about 15% CPU.

BTW, while we are on the subject, the overhead of netfilter itself,
i.e. the difference in CPU utilization of kernel with and without
netfilter on the above platform is more than 5%. Is there anybody hear
willing to discuss this ?

Re: TCP/UDP checksum in hardware

[Patrick McHardy]

Alexander Sirotkin wrote:
> On 3/4/07, Patrick McHardy <kaber@trash.net> wrote:
> 
>> Alexander Sirotkin wrote:
>> > The reason I'm asking is that computing checksum (in case of NAT, for
>> > instance) becomes a real problem on embedded devices
>>
>> Do you have any data to show this?
>>
> I don't know how relevant this is for netfilter, but yes - if the
> device does not support checksum offloading my benchmark which I ran
> on 266Mhz MIPS 24K (which is a pretty common processor for residential
> gateways) showed that under 80Mbps UDP traffic, with NAT enabled,
> checksum check takes about 15% CPU.

The first question would be whether this is really checksumming
done by netfilter or by the UDP code. Since enabling checksum
offloading seems to help, this points to the UDP code. In case
it is netfilter, the second question would be whether its
checksum verification or updates.

> BTW, while we are on the subject, the overhead of netfilter itself,
> i.e. the difference in CPU utilization of kernel with and without
> netfilter on the above platform is more than 5%. Is there anybody hear
> willing to discuss this ?

Is this with netfilter modules (like iptables, conntrack, NAT, ...)
loaded or just by enabling netfilter in the configuration?

BTW, which kernel version are you talking about?

Re: TCP/UDP checksum in hardware

[Alexander Sirotkin]

On 3/5/07, Patrick McHardy <kaber@trash.net> wrote:
> Alexander Sirotkin wrote:
> > On 3/4/07, Patrick McHardy <kaber@trash.net> wrote:
> >
> >> Alexander Sirotkin wrote:
> >> > The reason I'm asking is that computing checksum (in case of NAT, for
> >> > instance) becomes a real problem on embedded devices
> >>
> >> Do you have any data to show this?
> >>
> > I don't know how relevant this is for netfilter, but yes - if the
> > device does not support checksum offloading my benchmark which I ran
> > on 266Mhz MIPS 24K (which is a pretty common processor for residential
> > gateways) showed that under 80Mbps UDP traffic, with NAT enabled,
> > checksum check takes about 15% CPU.
>
> The first question would be whether this is really checksumming
> done by netfilter or by the UDP code. Since enabling checksum
> offloading seems to help, this points to the UDP code. In case
> it is netfilter, the second question would be whether its
> checksum verification or updates.
>
> > BTW, while we are on the subject, the overhead of netfilter itself,
> > i.e. the difference in CPU utilization of kernel with and without
> > netfilter on the above platform is more than 5%. Is there anybody hear
> > willing to discuss this ?
>
> Is this with netfilter modules (like iptables, conntrack, NAT, ...)
> loaded or just by enabling netfilter in the configuration?
>
Just netfilter, no modules.
Strangely enough, this only happens with bridge configuration. With
router configuration the difference in CPU utilization is minor,
however with bridge it is huge - about 25% on my system.

> BTW, which kernel version are you talking about?
>
2.6.16.22

Re: TCP/UDP checksum in hardware

[Patrick McHardy]

Alexander Sirotkin wrote:
> On 3/5/07, Patrick McHardy <kaber@trash.net> wrote:
> 
>> > BTW, while we are on the subject, the overhead of netfilter itself,
>> > i.e. the difference in CPU utilization of kernel with and without
>> > netfilter on the above platform is more than 5%. Is there anybody hear
>> > willing to discuss this ?
>>
>> Is this with netfilter modules (like iptables, conntrack, NAT, ...)
>> loaded or just by enabling netfilter in the configuration?
>>
> Just netfilter, no modules.
> Strangely enough, this only happens with bridge configuration. With
> router configuration the difference in CPU utilization is minor,
> however with bridge it is huge - about 25% on my system.


The bridge-netfilter code includes a number of hooks of its own,
additionally it passes all packets though all IP netfilter hooks
and duplicates quite a bit of the IP layer handling. I'm not very
surprised that its efficiency sucks, although 25% really is a lot.
If you could get profiles we could see if there are possibly some
simple optimizations.

If you don't need IP netfilter on bridged packets you could also
simply disable it through /proc/sys/net/bridge/bridge-nf-call-*.