How to do TCP tx checksums

How to do TCP tx checksums

[Ben Greear]

Hello!

I'm hacking on (my already hacked) pktgen, trying to get it to send TCP frames.

And, having issues getting checksums to work properly.

I'm trying this:
	struct iphdr *iph = ip_hdr(skb);
	struct net_device *odev = pkt_dev->odev;

	if (pkt_dev->flags & F_TCP) {
		if (odev->features & NETIF_F_V4_CSUM) {
			skb->ip_summed = CHECKSUM_PARTIAL;
		} else {
			skb->ip_summed = CHECKSUM_NONE;
		}
		skb->csum = 0;
		__tcp_v4_send_check(skb, iph->saddr, iph->daddr);

I added an export so I could call that __tcp_v4_send_check method
w/out having to put a fake socket struct on the stack.

But, the receiving NIC reports 100% checksum failure, so obviously
I'm not doing it correct.

Any suggestions on what I might be doing wrong or the proper method(s)
to call?

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: How to do TCP tx checksums

[Tom Herbert]

On Mon, Nov 16, 2015 at 4:20 PM, Ben Greear <greearb@candelatech.com> wrote:
> Hello!
>
> I'm hacking on (my already hacked) pktgen, trying to get it to send TCP
> frames.
>
> And, having issues getting checksums to work properly.
>
> I'm trying this:
>         struct iphdr *iph = ip_hdr(skb);
>         struct net_device *odev = pkt_dev->odev;
>
>         if (pkt_dev->flags & F_TCP) {
>                 if (odev->features & NETIF_F_V4_CSUM) {
>                         skb->ip_summed = CHECKSUM_PARTIAL;
>                 } else {
>                         skb->ip_summed = CHECKSUM_NONE;
>                 }
>                 skb->csum = 0;
>                 __tcp_v4_send_check(skb, iph->saddr, iph->daddr);
>
> I added an export so I could call that __tcp_v4_send_check method
> w/out having to put a fake socket struct on the stack.
>
> But, the receiving NIC reports 100% checksum failure, so obviously
> I'm not doing it correct.
>
> Any suggestions on what I might be doing wrong or the proper method(s)
> to call?
>
It's really hard to tell what is happening without seeing the full
patch your using. Maybe you're not setting the TCP correctly or
transport header is not set right in skb.

Tom

> Thanks,
> Ben
>
>
> --
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc  http://www.candelatech.com
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to do TCP tx checksums

[Eric Dumazet]

On Mon, 2015-11-16 at 21:02 -0800, Tom Herbert wrote:

> It's really hard to tell what is happening without seeing the full
> patch your using. Maybe you're not setting the TCP correctly or
> transport header is not set right in skb.

BTW, using trafgen from netsniff-ng would save you Ben lots of
headaches.

No need for kernel patches...

Re: How to do TCP tx checksums

[Ben Greear]

On 11/16/2015 09:02 PM, Tom Herbert wrote:
> On Mon, Nov 16, 2015 at 4:20 PM, Ben Greear <greearb@candelatech.com> wrote:
>> Hello!
>>
>> I'm hacking on (my already hacked) pktgen, trying to get it to send TCP
>> frames.
>>
>> And, having issues getting checksums to work properly.
>>
>> I'm trying this:
>>          struct iphdr *iph = ip_hdr(skb);
>>          struct net_device *odev = pkt_dev->odev;
>>
>>          if (pkt_dev->flags & F_TCP) {
>>                  if (odev->features & NETIF_F_V4_CSUM) {
>>                          skb->ip_summed = CHECKSUM_PARTIAL;
>>                  } else {
>>                          skb->ip_summed = CHECKSUM_NONE;
>>                  }
>>                  skb->csum = 0;
>>                  __tcp_v4_send_check(skb, iph->saddr, iph->daddr);
>>
>> I added an export so I could call that __tcp_v4_send_check method
>> w/out having to put a fake socket struct on the stack.
>>
>> But, the receiving NIC reports 100% checksum failure, so obviously
>> I'm not doing it correct.
>>
>> Any suggestions on what I might be doing wrong or the proper method(s)
>> to call?
>>
> It's really hard to tell what is happening without seeing the full
> patch your using. Maybe you're not setting the TCP correctly or
> transport header is not set right in skb.

Unfortunately, my pktgen has diverged significantly from upstream, so
a patch is probably not so useful.

The full pktgen.[ch] files are here:

http://www.candelatech.com/~greearb/patches/

If you look for 'F_TCP' you can find the pertinent bits
that I'm mucking with.  I have changed it a bit since I last posted
the patch above...but still the receiving NIC is reporting 100%
csum failure, so I must still be doing something wrong.

I have also tweaked how UDP is done, but that seems to be working.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: How to do TCP tx checksums

[Ben Greear]

On 11/16/2015 09:02 PM, Tom Herbert wrote:
> On Mon, Nov 16, 2015 at 4:20 PM, Ben Greear <greearb@candelatech.com> wrote:
>> Hello!
>>
>> I'm hacking on (my already hacked) pktgen, trying to get it to send TCP
>> frames.
>>
>> And, having issues getting checksums to work properly.
>>
>> I'm trying this:
>>          struct iphdr *iph = ip_hdr(skb);
>>          struct net_device *odev = pkt_dev->odev;
>>
>>          if (pkt_dev->flags & F_TCP) {
>>                  if (odev->features & NETIF_F_V4_CSUM) {
>>                          skb->ip_summed = CHECKSUM_PARTIAL;
>>                  } else {
>>                          skb->ip_summed = CHECKSUM_NONE;
>>                  }
>>                  skb->csum = 0;
>>                  __tcp_v4_send_check(skb, iph->saddr, iph->daddr);
>>
>> I added an export so I could call that __tcp_v4_send_check method
>> w/out having to put a fake socket struct on the stack.
>>
>> But, the receiving NIC reports 100% checksum failure, so obviously
>> I'm not doing it correct.
>>
>> Any suggestions on what I might be doing wrong or the proper method(s)
>> to call?
>>
> It's really hard to tell what is happening without seeing the full
> patch your using. Maybe you're not setting the TCP correctly or
> transport header is not set right in skb.

And in case it helps, here is the output from dmesg.

This is an Ethernet frame, and in this case csum-start is the
index of the first byte of the TCP header as far as I can tell.

[64967.194251] check: 0xf51b  csum-start: 34  offset: 16 summed: 0x3  saddr: 0x201010a  daddr: 0x101010a len: 1514


Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: How to do TCP tx checksums

[Eric Dumazet]

On Tue, 2015-11-17 at 09:44 -0800, Ben Greear wrote:

> And in case it helps, here is the output from dmesg.
> 
> This is an Ethernet frame, and in this case csum-start is the
> index of the first byte of the TCP header as far as I can tell.
> 
> [64967.194251] check: 0xf51b  csum-start: 34  offset: 16 summed: 0x3  saddr: 0x201010a  daddr: 0x101010a len: 1514
> 

You forgot to clear tcp header @check before calling
__tcp_v4_send_check(), I think.

( th->check = 0; )


Anyway, just use trafgen and get ~6 Mpps, more easily than with pktgen.

Re: How to do TCP tx checksums

[Ben Greear]

On 11/17/2015 10:26 AM, Eric Dumazet wrote:
> On Tue, 2015-11-17 at 09:44 -0800, Ben Greear wrote:
>
>> And in case it helps, here is the output from dmesg.
>>
>> This is an Ethernet frame, and in this case csum-start is the
>> index of the first byte of the TCP header as far as I can tell.
>>
>> [64967.194251] check: 0xf51b  csum-start: 34  offset: 16 summed: 0x3  saddr: 0x201010a  daddr: 0x101010a len: 1514
>>
>
> You forgot to clear tcp header @check before calling
> __tcp_v4_send_check(), I think.
>
> ( th->check = 0; )

 From printk I added to e1000e, it looks like I have bad offsets
somehow.  I'll try clearing check as well, however.

> Anyway, just use trafgen and get ~6 Mpps, more easily than with pktgen.

I do plan to look at this...but need to make sure I can get similar
features before I ditch pktgen.  It's not just raw throughput that is
of interest to me, but also randomizing, sequential, etc header fields
to test various network equipment (routers, MAC tables in switches, etc).

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: How to do TCP tx checksums

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Tue, 17 Nov 2015 10:26:47 -0800

> Anyway, just use trafgen and get ~6 Mpps, more easily than with
> pktgen.

+1

Re: How to do TCP tx checksums

[Daniel Borkmann]

On 11/17/2015 07:32 PM, Ben Greear wrote:
> On 11/17/2015 10:26 AM, Eric Dumazet wrote:
>> On Tue, 2015-11-17 at 09:44 -0800, Ben Greear wrote:
>>
>>> And in case it helps, here is the output from dmesg.
>>>
>>> This is an Ethernet frame, and in this case csum-start is the
>>> index of the first byte of the TCP header as far as I can tell.
>>>
>>> [64967.194251] check: 0xf51b  csum-start: 34  offset: 16 summed: 0x3  saddr: 0x201010a  daddr: 0x101010a len: 1514
>>>
>>
>> You forgot to clear tcp header @check before calling
>> __tcp_v4_send_check(), I think.
>>
>> ( th->check = 0; )
>
>  From printk I added to e1000e, it looks like I have bad offsets
> somehow.  I'll try clearing check as well, however.
>
>> Anyway, just use trafgen and get ~6 Mpps, more easily than with pktgen.
>
> I do plan to look at this...but need to make sure I can get similar
> features before I ditch pktgen.  It's not just raw throughput that is
> of interest to me, but also randomizing, sequential, etc header fields
> to test various network equipment (routers, MAC tables in switches, etc).

trafgen has a number of these things you mentioned. It started out as a
hobby project longer time ago as a simple way to tx raw frames, which was
needed to test some kernel project back in the days. Of course, nothing
is perfect and patches are always welcome. And if we can also get AF_PACKET
into a better shape by this, even better. ;)

Re: How to do TCP tx checksums

[Ben Greear]

On 11/16/2015 09:02 PM, Tom Herbert wrote:
> On Mon, Nov 16, 2015 at 4:20 PM, Ben Greear <greearb@candelatech.com> wrote:
>> Hello!
>>
>> I'm hacking on (my already hacked) pktgen, trying to get it to send TCP
>> frames.
>>
>> And, having issues getting checksums to work properly.
>>
>> I'm trying this:
>>          struct iphdr *iph = ip_hdr(skb);
>>          struct net_device *odev = pkt_dev->odev;
>>
>>          if (pkt_dev->flags & F_TCP) {
>>                  if (odev->features & NETIF_F_V4_CSUM) {
>>                          skb->ip_summed = CHECKSUM_PARTIAL;
>>                  } else {
>>                          skb->ip_summed = CHECKSUM_NONE;
>>                  }
>>                  skb->csum = 0;
>>                  __tcp_v4_send_check(skb, iph->saddr, iph->daddr);
>>
>> I added an export so I could call that __tcp_v4_send_check method
>> w/out having to put a fake socket struct on the stack.
>>
>> But, the receiving NIC reports 100% checksum failure, so obviously
>> I'm not doing it correct.
>>
>> Any suggestions on what I might be doing wrong or the proper method(s)
>> to call?
>>
> It's really hard to tell what is happening without seeing the full
> patch your using. Maybe you're not setting the TCP correctly or
> transport header is not set right in skb.

Ok, I have this working now.. at least in my current setup.

Main problem is that TCP stack uses skb->len before the other transport
headers are counted in skb->len, it appears.

So, in pktgen I need to subtract out the pkt prefix.  This at least
makes the hw-accel path work...not sure about software csum path yet...


static void pg_do_csum(struct pktgen_dev *pkt_dev, struct sk_buff *skb) {
	struct iphdr *iph = ip_hdr(skb);
	struct udphdr *uh;
	struct net_device *odev = pkt_dev->odev;

	if (pkt_dev->flags & F_TCP) {
		struct tcphdr *th = tcp_hdr(skb);
		unsigned int prefix_len = (unsigned int)((unsigned char*)th - skb->data);

		if (odev->features & NETIF_F_V4_CSUM) {
			skb->ip_summed = CHECKSUM_PARTIAL;
			/* Subtract out IP hdr and before */
			th->check = ~tcp_v4_check(skb->len - prefix_len, iph->saddr, iph->daddr, 0);
			skb->csum_start = skb_transport_header(skb) - skb->head;
			skb->csum_offset = offsetof(struct tcphdr, check);
		} else {
			skb->ip_summed = CHECKSUM_NONE;
			th->check = 0;
			skb->csum = 0;
			th->check = tcp_v4_check(skb->len - prefix_len, iph->saddr, iph->daddr,
						 csum_partial(th, th->doff << 2, skb->csum));
		}
	} else {

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com