Packet duplication in kernels 2.6 onwards

Packet duplication in kernels 2.6 onwards

[Richard Stearn]

Just put togther a packet system to test wireshark on and
discovered that packets are being duplicated at the ax0 interface.

My tests so far show that the issue is not present in kernels
2.4.29 & 2.4.37.11 and is present in kernel 2.6.33.3.
I also have reports of the issue being in kernel 3.x.x.x

Is this a known issue?

If so is there any idea as to when it appeared? My guess is during
the 2.5 kernel series.

Has anybody on the list attempted to trace the cause?

-- 
Regards
	Richard

Re: Packet duplication in kernels 2.6 onwards

[David Ranch]

Hello Richard,

Can you give us some more details?  Are the duplicates on every packet?  
Is this RX or TX duplicates  Only some?  Is this unproto or connected?  
Are you using a HW TNC or a SW TNC?
I'm using the 3.17.5 kernel w/o issues.

--David
KI6ZHD



On 05/04/2015 11:30 AM, Richard Stearn wrote:
> Just put togther a packet system to test wireshark on and
> discovered that packets are being duplicated at the ax0 interface.
>
> My tests so far show that the issue is not present in kernels
> 2.4.29 & 2.4.37.11 and is present in kernel 2.6.33.3.
> I also have reports of the issue being in kernel 3.x.x.x
>
> Is this a known issue?
>
> If so is there any idea as to when it appeared? My guess is during
> the 2.5 kernel series.
>
> Has anybody on the list attempted to trace the cause?
>

Re: Packet duplication in kernels 2.6 onwards

[Patrick Ouellette]

This has just recently come up in Debian's bug system at
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=783160https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=783160

Pat

Re: Packet duplication in kernels 2.6 onwards

[Richard Stearn]

David Ranch wrote:
>   Hello Richard,
> 
> Can you give us some more details?  Are the duplicates on every packet?  Is this 
> RX or TX duplicates  Only some?  Is this unproto or connected?  Are you using a 
> HW TNC or a SW TNC?
> I'm using the 3.17.5 kernel w/o issues.

Refined the conditions a little.

	1) Only occurs on ax25 packets containing IP payloads
	2) Occurs in both UI and VC packets
	3) Only occurs on packets being sent
	4) Does not appear to be sent "to air" by mkiss module,
	   i.e. the duplicate packet does not appear on the serial line.
	5) Is visible in "listen" output

I discovered this as I am documenting the amateur radio protocols in
wireshark.  It was also reported to Ubuntu (thank you Patrick) by
Ugo Poddine.  In the discussion on the Ubuntu bug report it would appear
that this is known about, hence my posting here to see what was known.
Apparently nothing.


Other things raised by this, in the process of rebuilding my test system
I discovered that the ROSE kernel module did not unload correctly (actually
not at all).  I traced this to the device usage counts not being maintained
correctly.  I have a patch for this if anybody wants it.

-- 
Regards
	Richard

Re: Packet duplication in kernels 2.6 onwards

[Ugo Poddine]

Hello everyone, hi Richard,

a couple of remarks, regarding this issue.

a) "duplication" on the air

Indeed I discovered the problem exactly "on the air", at least using 
software "TNCs".
I have some logs taken from Direwolf or MIXw2 showing duplication being 
sent on the radio channel (I can share, if somebody needs), and also 
some reports from Canada (thanks to VE1JOT) reporing practical 
impossibility to transmit TCP over AX25 on HF channels due to the 
duplication.

b) 6PACK problem

I was curious to understand if the problem affects also 6PACK, but I was 
unable to test it : on Debian Wheezy, switching kissattack in 6pack mode 
always crashes the kernel, on Jessie it doesn’t, but no data exchange is 
possible and - worst - sniffing the interface always show "kiss" frames 
instead of 6PACK frames (at least in the ARP dialogue). This happens 
also on Fedora21. I opened a ticket on the Jessie side on Saturday. 

Thank-you
Regards
73, Ugo

Re: Packet duplication in kernels 2.6 onwards

[Richard Stearn]

I think I have got to the bottom of this issue.  The good news is that it is
not a bug in the AX25 protocol or driver code, the bad news is that it is
the result of an architectural change in the kernel.

One correction to my original post, the duplication is on receive not
transmit.

In kernels upto and including 2.4 where an AX25 payload was IP or ARP the
packet was handed off by calling the ip or arp protocol driver directly.
In kernels 2.6 onwards after adjusting the packet details the payload is
handed back to the network interface for that interface to reroute the
packet into the IP or ARP protocol drivers.

A couple of code fragments:

from 2.4.37.11:
net/ax25/ax25_in.c
287                          case AX25_P_IP:
288                                  skb_pull(skb,2);                /* drop PID/CTRL */
289                                  skb->h.raw    = skb->data;
290                                  skb->nh.raw   = skb->data;
291                                  skb->dev      = dev;
292                                  skb->pkt_type = PACKET_HOST;
293                                  skb->protocol = htons(ETH_P_IP);
294                                  ip_rcv(skb, dev, ptype);        /* Note ptype here is 
the wrong one, fix me later */
295                                  break;
296

from 2.6.32.65:
net/ax25/ax25_in.c
244                  case AX25_P_IP:
245                          skb_pull(skb,2);                /* drop PID/CTRL */
246                          skb_reset_transport_header(skb);
247                          skb_reset_network_header(skb);
248                          skb->dev      = dev;
249                          skb->pkt_type = PACKET_HOST;
250                          skb->protocol = htons(ETH_P_IP);
251                          netif_rx(skb);
252                          break;

I was able to revert kernel 2.6.32.65 to the 2.4 method as ip_rcv & arp_rcv still
exist and it was a simple matter to expose the functions (exporting the symbols and
adding to the header files).  Once reverted the duplicated packets in the 2.6 kernel
disappeared.

All the testing has been done using virtual machines linked by a virtual serial link.
Testing was done to establish connections using AX25-UI, AX25-VC, NETROM, ROSE IP/ICMP,
IP/UDP & IP/TCP.
The first setup was of two 2.4.37.11 kernels.
The second setup was of one 2.4.37.11 kernel and one 2.6.32.65 kernel.
The third setup was of one 2.4.37.11 kernel and one 3.2.69 kernel.

For the first setup no duplicates were seen at any point.

For the second setup, duplicates of received packets were seen on the 2.6 kernel
wherever IP was involved. No duplicates were seen on the 2.4 kernel.

For the third setup, duplicates of received packets were seen on the 3.2 kernel
wherever IP was involved. No duplicates were seen on the 2.4 kernel.

These duplicate packets are seen by both "listen" and libpcap (and thus tcpdump or
wireshark).  From the protocol analysis point of view any kernel 2.6 onwards is
suspect where encapsulated protocols, such as IP-over-AX25, are concerned.

Notably NETROM and ROSE are not affected, this may be related to the manner in
which those protocols "register" their handlers with AX25.  The AX25 protocol
driver considers AX25_P_TEXT, AX25_P_SEGMENT, AX25_P_IP or AX25_P_ARP as
"internal" protocols (ax25/ax25_iface.c).  The solution may be to "register"
IP & ARP with AX25 although how to achieve this I can not immediate see.

The above explains why the amateur radio protocols work however when one starts
to look at the packets analytically what is seen at the network interface
for IP & ARP is false.

Of course I could have it totally wrong.

-- 
Regards
	Richard

Re: Packet duplication in kernels 2.6 onwards

[Ralf Baechle DL5RB]

On Wed, May 20, 2015 at 01:33:42PM +0100, Richard Stearn wrote:

> I think I have got to the bottom of this issue.  The good news is that it is
> not a bug in the AX25 protocol or driver code, the bad news is that it is
> the result of an architectural change in the kernel.
> 
> One correction to my original post, the duplication is on receive not
> transmit.
> 
> In kernels upto and including 2.4 where an AX25 payload was IP or ARP the
> packet was handed off by calling the ip or arp protocol driver directly.
> In kernels 2.6 onwards after adjusting the packet details the payload is
> handed back to the network interface for that interface to reroute the
> packet into the IP or ARP protocol drivers.
> 
> A couple of code fragments:
> 
> from 2.4.37.11:
> net/ax25/ax25_in.c
> 287                          case AX25_P_IP:
> 288                                  skb_pull(skb,2);                /* drop PID/CTRL */
> 289                                  skb->h.raw    = skb->data;
> 290                                  skb->nh.raw   = skb->data;
> 291                                  skb->dev      = dev;
> 292                                  skb->pkt_type = PACKET_HOST;
> 293                                  skb->protocol = htons(ETH_P_IP);
> 294                                  ip_rcv(skb, dev, ptype);        /* Note
> ptype here is the wrong one, fix me later */
> 295                                  break;
> 296
> 
> from 2.6.32.65:
> net/ax25/ax25_in.c
> 244                  case AX25_P_IP:
> 245                          skb_pull(skb,2);                /* drop PID/CTRL */
> 246                          skb_reset_transport_header(skb);
> 247                          skb_reset_network_header(skb);
> 248                          skb->dev      = dev;
> 249                          skb->pkt_type = PACKET_HOST;
> 250                          skb->protocol = htons(ETH_P_IP);
> 251                          netif_rx(skb);
> 252                          break;
> 
> I was able to revert kernel 2.6.32.65 to the 2.4 method as ip_rcv & arp_rcv still
> exist and it was a simple matter to expose the functions (exporting the symbols and
> adding to the header files).  Once reverted the duplicated packets in the 2.6 kernel
> disappeared.
> 
> All the testing has been done using virtual machines linked by a virtual serial link.
> Testing was done to establish connections using AX25-UI, AX25-VC, NETROM, ROSE IP/ICMP,
> IP/UDP & IP/TCP.
> The first setup was of two 2.4.37.11 kernels.
> The second setup was of one 2.4.37.11 kernel and one 2.6.32.65 kernel.
> The third setup was of one 2.4.37.11 kernel and one 3.2.69 kernel.
> 
> For the first setup no duplicates were seen at any point.
> 
> For the second setup, duplicates of received packets were seen on the 2.6 kernel
> wherever IP was involved. No duplicates were seen on the 2.4 kernel.
> 
> For the third setup, duplicates of received packets were seen on the 3.2 kernel
> wherever IP was involved. No duplicates were seen on the 2.4 kernel.
> 
> These duplicate packets are seen by both "listen" and libpcap (and thus tcpdump or
> wireshark).  From the protocol analysis point of view any kernel 2.6 onwards is
> suspect where encapsulated protocols, such as IP-over-AX25, are concerned.
> 
> Notably NETROM and ROSE are not affected, this may be related to the manner in
> which those protocols "register" their handlers with AX25.  The AX25 protocol
> driver considers AX25_P_TEXT, AX25_P_SEGMENT, AX25_P_IP or AX25_P_ARP as
> "internal" protocols (ax25/ax25_iface.c).  The solution may be to "register"
> IP & ARP with AX25 although how to achieve this I can not immediate see.
> 
> The above explains why the amateur radio protocols work however when one starts
> to look at the packets analytically what is seen at the network interface
> for IP & ARP is false.
> 
> Of course I could have it totally wrong.

I think you're if not spot on at least very close.

In fact when somebody else earlier in this thread said it did only affect
IP traffic it was quite clear where the issue must be.  I've only ever
tried to replicate this with non-IP traffic so never got to see the
duplication.

Most of the IP-over-{AX25.NETROM,ROSE} is quite forward.  Except ARP which
qualifies as an atrocity to the network stack.

I have hopes that Eric Biederman's patch series may have resolved the issue
but that.  It's currently sitting in linux-next and should probably make
4.2.

This might also fix the hard to properly fix ARP-over-ROSE issue that
resulted in a kernel crash.

73 de DL5RB op Ralf

--
Loc. JN47BS / CQ 14 / ITU 28 / DOK A21

Re: Packet duplication in kernels 2.6 onwards

[Richard Stearn]

Ralf Baechle DL5RB wrote:
> In fact when somebody else earlier in this thread said it did only affect
> IP traffic it was quite clear where the issue must be.  I've only ever
> tried to replicate this with non-IP traffic so never got to see the
> duplication.

I was working from reports, in the form of capture files, coming from a
different direction, Wireshark.


> This might also fix the hard to properly fix ARP-over-ROSE issue that
> resulted in a kernel crash.

Post 2.5 kernels there is nothing in the ROSE module that is IP or ARP
related.  Could this be related to the lack of socket locking and device
use count tracking in the ROSE module?  I have published a patch for that.
This weeks job is to raise a kernel bug report and attach the patch to
that.

The packet "duplication" affects more than IP-over-AX25/NETROM/ROSE, I
think it affects IP-over-IP, AX25-over-IP and other packet encapsulating
protocols.  Probably anywhere where "netif_rx()" is used because that call
puts the packet back onto the network interface for further processing.

Note: although this is being called "packet duplication" it appears to be
a packet being seen twice by promiscuous tools (listen & libpcap) as it
is sequentially processed, firstly on it's way to the AX25 handler and
then on it's way to the IP handler.  It is probably necessary to setup
filtering for "listen" and libpcap to select only AX25 packets rather
than all packets on an AX25 interface.

-- 
Regards
	Richard

Re: Packet duplication in kernels 2.6 onwards

[Ralf Baechle DL5RB]

On Mon, May 04, 2015 at 07:30:36PM +0100, Richard Stearn wrote:

> Just put togther a packet system to test wireshark on and
> discovered that packets are being duplicated at the ax0 interface.
> 
> My tests so far show that the issue is not present in kernels
> 2.4.29 & 2.4.37.11 and is present in kernel 2.6.33.3.
> I also have reports of the issue being in kernel 3.x.x.x
> 
> Is this a known issue?
> 
> If so is there any idea as to when it appeared? My guess is during
> the 2.5 kernel series.
> 
> Has anybody on the list attempted to trace the cause?

There has been a number of pretty vague bug reports on the issue that
have been insufficient for me to replicate the issue on my system.

  Ralf