Kernel Oops when using the Netfilter QUEUE target

Kernel Oops when using the Netfilter QUEUE target

[Martin Clausen]

[-- Attachment #1: Type: text/plain, Size: 3232 bytes --]

Hi there!

I have encountered a problem (perhaps a bug)! The attached code makes my kernel oops
in some cases when injecting new packets through Netfilter's QUEUE target. The problem 
only appears when the original packet is a TCP packet; i have tried with ICMP and UDP packets 
also but this does not trigger any oops. I have tried to code on several computers and they 
all oops. The following description regards the case when submitting new packets instead 
of TCP packets.

It seems that new packets can not have a length greater than 92 bytes under 2.4.2-ac21
and 76 under 2.4.3; these sizes may vary but the oops can be triggered by choosing
a larger packet size.

Netfilter is configured the following way:

[root@lwb7 ipsecd]# modprobe iptable_filter
[root@lwb7 ipsecd]# modprobe ip_queue      
[root@lwb7 ipsecd]# iptables -t mangle -A OUTPUT -d lwb5 -j LOG
[root@lwb7 ipsecd]# iptables -t mangle -A OUTPUT -d lwb5 -j QUEUE
[root@lwb7 ipsecd]# lsmod
Module                  Size  Used by
ipt_LOG                 4063   1  (autoclean)
iptable_mangle          2542   0  (autoclean) (unused)
ip_queue                5946   0  (unused)
iptable_filter          2533   0  (unused)
ip_tables              14936   3  [ipt_LOG iptable_mangle iptable_filter]
NVdriver              688003  12  (autoclean)
8139too                16845   1  (autoclean)
[root@lwb7 ipsecd]# iptables -t mangle -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
LOG        all  --  anywhere             lwb5.it.dtu.dk     LOG level warning 
QUEUE      all  --  anywhere             lwb5.it.dtu.dk  

I have added some printk's in net/code/netfilter.c in nf_reinject() and i seems that
the kernel oops' in info->okfn(skb) (i added printk before and after):

IN= OUT=eth0 SRC=130.225.76.37 DST=130.225.76.35 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=173 PROTO=TCP SPT=1025 DPT=23 WINDOW=5840 RES=0x00 SYN URGP=0
nf_hook: Verdict = QUEUE.                                                     
In nf_reinject() before info->okfn(skb) line 521                              
Unable to handle kernel NULL pointer dereference at virtual address 000002b4   
printing eip:                                                                
c01e7456                                                                      
*pde = 00000000
                                                                                                                                             
Entering kdb (current=0xc68f6000, pid 884) Oops: Oops                         
due to oops @ 0xc01e7456                                                      
eax = 0x000005dc ebx = 0xc7acf224 ecx = 0x0000000e edx = 0xc72f8440           
esi = 0xc7cee740 edi = 0x00000000 esp = 0xc68f7c90 eip = 0xc01e7456           
ebp = 0xc68f7cb0 xss = 0x00000018 xcs = 0x00000010 eflags = 0x00010287        
xds = 0x00000018 xes = 0x00000018 origeax = 0xffffffff &regs = 0xc68f7c5c     
kdb> 

I will be glad to submit som more (debug) information?!

I really hope someone can help me :)

Best regards,
Martin Clausen

-- 
                       There's no place like ~

[-- Attachment #2: nfcrash.c --]
[-- Type: text/plain, Size: 3698 bytes --]

/* Compile: gcc -o nfcrash nfcrash.c -lipq
 */	

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>

#include <netinet/in.h>
#include <netinet/in_systm.h>
#include <netinet/ip.h>
#include <arpa/inet.h>

#include <linux/netfilter.h>
#include <linux/netfilter_ipv4.h>
#include <linux/netfilter_ipv4/ip_queue.h>

#include <linux/libipq.h>

#include <sys/types.h>
#include <sys/socket.h>

/* For CRASH_ME <= 32 everything seems to work (2.4.2-ac21).
 * For CRASH_ME > 32 the kernel oops'es in case of TCP traffic (2.4.2-ac21)?!
 * 
 * For CRASH_ME <= 16 everything seems to work (2.4.3).
 * For CRASH_ME > 16 the kernel oops'es in case of TCP traffic (2.4.3)?!
 * 
 */
const int CRASH_ME = 17;

const int ESP_OFF = 0;

unsigned short in_cksum(unsigned short *addr, int len)
{
  int nleft = len;
  int sum 	= 0;
  unsigned short *w = addr;
  unsigned short answer = 0;
  
  while (nleft > 1) {
    sum += *w++;
    nleft -= 2;
  }

  if (nleft == 1) {
    *(unsigned char *)(&answer) = *(unsigned char *) w;
    sum += answer;
  }
  
  sum = (sum >> 16) + (sum & 0xffff); 
  sum += (sum >> 16);                
  answer = ~sum;                     
  
  return answer;
}

int main(int argc, char **argv)
{
  struct ipq_handle *qh;
  struct ipq_packet_msg *qpkt;
  struct iphdr *iph;
  unsigned char	buff[8192];
  unsigned char *esppkt;
  int esppkt_len;
  int type; 
  int len;
      
  /* Init first (flags are not implemented; yet) */
  if ( (qh = ipq_create_handle(0)) == NULL) {
    /* Run away... */
    ipq_perror("create_handle()");
    exit(1);
  }
	
  /* We would like to receive not only metadata... */
  if (ipq_set_mode(qh, IPQ_COPY_PACKET, sizeof(buff)) < 0) {
    ipq_perror("set_mode()");
    goto cleanup;
  }
  
  /* Now real fun begins... Just stay in loop, read packets,
   * accept them if they are IP. */
  while (1) {
    len = ipq_read(qh, buff, sizeof buff, 0);
    
    if (len == 0) {
      fprintf(stderr, "read(): Zero length packet; ingnoring\n");
      continue;
    }
    
    if (len < 0) {
      /* Error, quit the program */
      ipq_perror("read()");
      goto cleanup;
    }    
        
    /* If packet type from kernel is not known to us,
     * just ignore it (but notify user) */
    if ( (type = ipq_message_type(buff)) != IPQM_PACKET) {
      fprintf(stderr, "read(): Unknown message type %u\n", type);
      continue;
    }
    
    /* Now we know that it is IPv4 packet */
    qpkt = ipq_get_packet(buff);
    
    /*esppkt_len = sizeof(struct iphdr) + ESP_OFF + qpkt->data_len + CRASH_ME;*/
    esppkt_len = qpkt->data_len + CRASH_ME;
    esppkt = (unsigned char *) malloc(esppkt_len);
    memset((char *)esppkt, '\0', esppkt_len);

    printf("sizeof(esppkt) = %d\n", esppkt_len);
    
    /* Build new IP header */
    iph = (struct iphdr *) esppkt;
    iph->ihl = 5;
    iph->version = 4; /* IPv4 */
    iph->tot_len = htons(esppkt_len);
    /*iph->id = htons(getpid());*/
    iph->ttl = 64; /* Default TTL */
    iph->protocol = IPPROTO_ESP;
    iph->saddr = inet_addr("130.225.76.37");
    iph->daddr = inet_addr("130.225.76.35");
    iph->check = (unsigned short)in_cksum((unsigned short *) iph, sizeof(struct iphdr));

    /*memcpy(esppkt + sizeof(struct iphdr) + ESP_OFF, qpkt->payload, qpkt->data_len);*/

    /* Return the verdict and the encapsulated (ESP) packet */
    /*if (ipq_set_verdict(qh, qpkt->packet_id, NF_ACCEPT, qpkt->data_len, (unsigned char *) qpkt->payload) < 0) { */
    if (ipq_set_verdict(qh, qpkt->packet_id, NF_ACCEPT, esppkt_len, (unsigned char *) esppkt) < 0) {
      ipq_perror("set_verdict(): Oops!");
      goto cleanup;
    }    
  }
  
 cleanup:
  ipq_destroy_handle(qh);
  
  exit(0);
}

Re: Kernel Oops when using the Netfilter QUEUE target

[James Morris]

On Wed, 25 Apr 2001, Martin Clausen wrote:

> Hi there!
>
> I have encountered a problem (perhaps a bug)! The attached code makes my kernel oops
> in some cases when injecting new packets through Netfilter's QUEUE target. The problem
> only appears when the original packet is a TCP packet; i have tried with ICMP and UDP packets
> also but this does not trigger any oops. I have tried to code on several computers and they
> all oops. The following description regards the case when submitting new packets instead
> of TCP packets.

Please try the patch below.

- James
-- 
James Morris
<jmorris@intercode.com.au>

diff -urN linux-2.4.3/net/ipv4/netfilter/ip_queue.c linux-2.4.3-fix/net/ipv4/netfilter/ip_queue.c
--- linux-2.4.3/net/ipv4/netfilter/ip_queue.c	Tue Dec 12 07:37:04 2000
+++ linux-2.4.3-fix/net/ipv4/netfilter/ip_queue.c	Wed Apr 25 16:10:24 2001
@@ -24,8 +24,10 @@
 #include <linux/sysctl.h>
 #include <linux/proc_fs.h>
 #include <net/sock.h>
+#include <net/route.h>

 #include <linux/netfilter_ipv4/ip_queue.h>
+#include <linux/netfilter_ipv4/ip_tables.h>

 #define IPQ_QMAX_DEFAULT 1024
 #define IPQ_PROC_FS_NAME "ip_queue"
@@ -198,6 +200,32 @@
 	kfree(q);
 }

+/* With a chainsaw... */
+static int route_me_harder(struct sk_buff *skb)
+{
+	struct iphdr *iph = skb->nh.iph;
+	struct rtable *rt;
+
+	struct rt_key key = {
+				dst:iph->daddr, src:iph->saddr,
+				oif:skb->sk ? skb->sk->bound_dev_if : 0,
+				tos:RT_TOS(iph->tos)|RTO_CONN,
+#ifdef CONFIG_IP_ROUTE_FWMARK
+				fwmark:skb->nfmark
+#endif
+			};
+
+	if (ip_route_output_key(&rt, &key) != 0) {
+		printk("route_me_harder: No more route.\n");
+		return -EINVAL;
+	}
+
+	/* Drop old route. */
+	dst_release(skb->dst);
+	skb->dst = &rt->u.dst;
+	return 0;
+}
+
 static int ipq_mangle_ipv4(ipq_verdict_msg_t *v, ipq_queue_element_t *e)
 {
 	int diff;
@@ -223,6 +251,8 @@
 				      "in mangle, dropping packet\n");
 				return -ENOMEM;
 			}
+			if (e->skb->sk)
+				skb_set_owner_w(newskb, e->skb->sk);
 			kfree_skb(e->skb);
 			e->skb = newskb;
 		}
@@ -230,7 +260,13 @@
 	}
 	memcpy(e->skb->data, v->payload, v->data_len);
 	e->skb->nfcache |= NFC_ALTERED;
-	return 0;
+
+	/*
+	 * Extra routing needed on local out, as the QUEUE target never
+	 * returns control to the table.
+	 */
+	return ((e->info->hook == NF_IP_LOCAL_OUT) ?
+			route_me_harder(e->skb) : 0);
 }

 static inline int id_cmp(ipq_queue_element_t *e, unsigned long id)
diff -urN linux-2.4.3/net/ipv4/netfilter/iptable_mangle.c linux-2.4.3-fix/net/ipv4/netfilter/iptable_mangle.c
--- linux-2.4.3/net/ipv4/netfilter/iptable_mangle.c	Tue Jan 30 03:07:30 2001
+++ linux-2.4.3-fix/net/ipv4/netfilter/iptable_mangle.c	Wed Apr 25 16:09:02 2001
@@ -148,7 +148,7 @@

 	ret = ipt_do_table(pskb, hook, in, out, &packet_mangler, NULL);
 	/* Reroute for ANY change. */
-	if (ret != NF_DROP && ret != NF_STOLEN
+	if (ret != NF_DROP && ret != NF_STOLEN && ret != NF_QUEUE
 	    && ((*pskb)->nh.iph->saddr != saddr
 		|| (*pskb)->nh.iph->daddr != daddr
 		|| (*pskb)->nfmark != nfmark

Re: Kernel Oops when using the Netfilter QUEUE target

[Keith Owens]

On Tue, 24 Apr 2001 19:25:47 +0200, 
Martin Clausen <martin@ostenfeld.dk> wrote:
>I have encountered a problem (perhaps a bug)! The attached code makes my kernel oops
>in some cases when injecting new packets through Netfilter's QUEUE target. The problem 
>Entering kdb (current=0xc68f6000, pid 884) Oops: Oops                         
>due to oops @ 0xc01e7456                                                      
>eax = 0x000005dc ebx = 0xc7acf224 ecx = 0x0000000e edx = 0xc72f8440           
>esi = 0xc7cee740 edi = 0x00000000 esp = 0xc68f7c90 eip = 0xc01e7456           
>ebp = 0xc68f7cb0 xss = 0x00000018 xcs = 0x00000010 eflags = 0x00010287        
>xds = 0x00000018 xes = 0x00000018 origeax = 0xffffffff &regs = 0xc68f7c5c     
>kdb> 
>
>I will be glad to submit som more (debug) information?!

At the very least, you need to run the kdb commands 'bt' (backtrace)
and 'id %eip-0x10' (disassemble around failing instruction).  Registers
and eip on their own are almost meaningless.

ps. Don't copy me on the reply, I only maintain kdb, not the failing
    netfilter code.

Re: Kernel Oops when using the Netfilter QUEUE target

[Martin Clausen]

On Wed, Apr 25, 2001 at 04:24:46PM +1000, James Morris wrote:
> > I have encountered a problem (perhaps a bug)! The attached code makes my kernel oops
> > in some cases when injecting new packets through Netfilter's QUEUE target. The problem
> > only appears when the original packet is a TCP packet; i have tried with ICMP and UDP packets
> > also but this does not trigger any oops. I have tried to code on several computers and they
> > all oops. The following description regards the case when submitting new packets instead
> > of TCP packets.
> 
> Please try the patch below.

So i did and it seems to work just fine (= no more oops') under 2.4.3/2.4.2-ac21! The packets 
being sent also seems to be correct; James you're the man :-)

BTW could you describe the problem? And why it caused an oops?

Best regards,
Martin

-- 
                       There's no place like ~

Re: Kernel Oops when using the Netfilter QUEUE target

[Rusty Russell]

In message <20010427004801.A3464@ostenfeld.dk> you write:
> On Wed, Apr 25, 2001 at 04:24:46PM +1000, James Morris wrote:
> > Please try the patch below.
> 
> So i did and it seems to work just fine (= no more oops') under 2.4.3/2.4.2-a

James,  I only glanced at the patch, but IIRC it just did
route_me_harder() on everything.  This is, unfortunately, no longer
"Best Practice": the prevailing trend is to reroute only when
something has actually been changed, to avoid overriding the socket
binding, etc.

See mangle for an example (store old values, see if they changed).  I
think this would be more complex, but still possible in your case, no?

Thanks,
Rusty.
--
Premature optmztion is rt of all evl. --DK