WARNING: at net/core/skbuff.c:154 with tcpdump and ipsec

WARNING: at net/core/skbuff.c:154 with tcpdump and ipsec

[Marco Berizzi]

Hi Folks,

I'm getting this error on 2.6.28.4 when I run tcpdump on
the interface where ipsec packets are enc/decrypted.

TIA

> Feb 11 10:53:55 Pleiadi kernel: ------------[ cut here ]------------
> Feb 11 10:53:55 Pleiadi kernel: WARNING: at net/core/skbuff.c:154 skb_truesize_bug+0x2e/0x33()
> Feb 11 10:53:55 Pleiadi kernel: SKB BUG: Invalid truesize (268) len=134, sizeof(sk_buff)=172
> Feb 11 10:53:55 Pleiadi kernel: Modules linked in: twofish_i586 twofish_common serpent blowfish ecb nf_nat_pptp nf_nat_proto_gre nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_ftp nf_conntrack_ftp 3c59x mii
> Feb 11 10:53:55 Pleiadi kernel: Pid: 4302, comm: tcpdump Tainted: G        W  2.6.28.4 #1
> Feb 11 10:53:55 Pleiadi kernel: Call Trace:
> Feb 11 10:53:55 Pleiadi kernel:  [<c01157b4>] warn_slowpath+0x58/0x71
> Feb 11 10:53:55 Pleiadi kernel:  [<c0111909>] set_next_entity+0x14/0x39
> Feb 11 10:53:55 Pleiadi kernel:  [<c02ae273>] __sched_text_start+0x233/0x24f
> Feb 11 10:53:55 Pleiadi kernel:  [<c024a46d>] wait_for_packet+0x109/0x113
> Feb 11 10:53:55 Pleiadi kernel:  [<c0122d56>] autoremove_wake_function+0x0/0x2d
> Feb 11 10:53:55 Pleiadi kernel:  [<c01ea285>] __copy_to_user_ll+0x3e/0x45
> Feb 11 10:53:55 Pleiadi kernel:  [<c01ea3c4>] copy_to_user+0x27/0x2f
> Feb 11 10:53:55 Pleiadi kernel:  [<c0247612>] skb_truesize_bug+0x2e/0x33
> Feb 11 10:53:55 Pleiadi kernel:  [<c0246363>] sock_rfree+0x1e/0x42
> Feb 11 10:53:55 Pleiadi kernel:  [<c0247909>] skb_release_head_state+0x4c/0x7d
> Feb 11 10:53:55 Pleiadi kernel:  [<c0247942>] skb_release_all+0x8/0x10
> Feb 11 10:53:55 Pleiadi kernel:  [<c0247952>] __kfree_skb+0x8/0x10
> Feb 11 10:53:55 Pleiadi kernel:  [<c024a55b>] skb_free_datagram+0xa/0x29
> Feb 11 10:53:55 Pleiadi kernel:  [<c02a5c19>] packet_recvmsg+0x180/0x18c
> Feb 11 10:53:55 Pleiadi kernel:  [<c0111ac2>] wakeup_preempt_entity+0x26/0x3f
> Feb 11 10:53:55 Pleiadi kernel:  [<c0111b50>] check_preempt_wakeup+0x49/0xac
> Feb 11 10:53:55 Pleiadi kernel:  [<c0243d74>] sock_recvmsg+0xbf/0xda
> Feb 11 10:53:55 Pleiadi kernel:  [<c0112c3d>] __wake_up+0x11/0x1a
> Feb 11 10:53:55 Pleiadi kernel:  [<c0220619>] n_tty_receive_buf+0x811/0x82d
> Feb 11 10:53:55 Pleiadi kernel:  [<c0122d56>] autoremove_wake_function+0x0/0x2d
> Feb 11 10:53:55 Pleiadi kernel:  [<c026cf4b>] ip_forward+0x251/0x292
> Feb 11 10:53:55 Pleiadi kernel:  [<c026c03f>] ip_rcv_finish+0x215/0x229
> Feb 11 10:53:55 Pleiadi kernel:  [<c01ea3c4>] copy_to_user+0x27/0x2f
> Feb 11 10:53:55 Pleiadi kernel:  [<c0244ce2>] sys_recvfrom+0xa9/0x100
> Feb 11 10:53:55 Pleiadi kernel:  [<c021f8d8>] opost+0x184/0x18b
> Feb 11 10:53:55 Pleiadi kernel:  [<c02210ce>] n_tty_write+0x188/0x1a1
> Feb 11 10:53:55 Pleiadi kernel:  [<c0112bc4>] default_wake_function+0x0/0xc
> Feb 11 10:53:55 Pleiadi kernel:  [<c0112c3d>] __wake_up+0x11/0x1a
> Feb 11 10:53:55 Pleiadi kernel:  [<c01e7d6c>] __rb_erase_color+0x95/0x13f
> Feb 11 10:53:55 Pleiadi kernel:  [<c02452ec>] sys_socketcall+0x115/0x18a
> Feb 11 10:53:55 Pleiadi kernel:  [<c0102aae>] syscall_call+0x7/0xb
> Feb 11 10:53:55 Pleiadi kernel: ---[ end trace f36cb14ebd5b6b5f ]---

Maybe related thread: http://lkml.indiana.edu/hypermail/linux/kernel/0902.0/02757.html

Re: WARNING: at net/core/skbuff.c:154 with tcpdump and ipsec

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 666 bytes --]



On Wed, 11 Feb 2009, Marco Berizzi wrote:

> Hi Folks,
>
> I'm getting this error on 2.6.28.4 when I run tcpdump on
> the interface where ipsec packets are enc/decrypted.
>
> TIA
>
>> Feb 11 10:53:55 Pleiadi kernel: ------------[ cut here ]------------
>> Feb 11 10:53:55 Pleiadi kernel: WARNING: at net/core/skbuff.c:154 skb_truesize_bug+0x2e/0x33()
>> Feb 11 10:53:55 Pleiadi kernel: SKB BUG: Invalid truesize (268) len=134, sizeof(sk_buff)=172
<CUT>

This annoying problem is quite old (appeared in 2.6.25) and already known:
  http://bugzilla.kernel.org/show_bug.cgi?id=10996

Sadly, no one is interested in fixing it. :(

Best regards,

 				Krzysztof Olędzki

Re: WARNING: at net/core/skbuff.c:154 with tcpdump and ipsec

[Jarek Poplawski]

Krzysztof Oledzki wrote, On 02/11/2009 04:55 PM:

> <CUT>
> 
> This annoying problem is quite old (appeared in 2.6.25) and already known:
>   http://bugzilla.kernel.org/show_bug.cgi?id=10996
> 
> Sadly, no one is interested in fixing it. :(

":(" looks like interested ;)

Jarek P.

Re: WARNING: at net/core/skbuff.c:154 with tcpdump and ipsec

[Jarek Poplawski]

On 11-02-2009 16:55, Krzysztof Oledzki wrote:
> 
> On Wed, 11 Feb 2009, Marco Berizzi wrote:
> 
>> Hi Folks,
>>
>> I'm getting this error on 2.6.28.4 when I run tcpdump on
>> the interface where ipsec packets are enc/decrypted.
>>
>> TIA
>>
>>> Feb 11 10:53:55 Pleiadi kernel: ------------[ cut here ]------------
>>> Feb 11 10:53:55 Pleiadi kernel: WARNING: at net/core/skbuff.c:154 skb_truesize_bug+0x2e/0x33()
>>> Feb 11 10:53:55 Pleiadi kernel: SKB BUG: Invalid truesize (268) len=134, sizeof(sk_buff)=172
> <CUT>
> 
> This annoying problem is quite old (appeared in 2.6.25) and already known:
>   http://bugzilla.kernel.org/show_bug.cgi?id=10996
> 
> Sadly, no one is interested in fixing it. :(

Here is a debugging patch doing these checks a bit earlier, so maybe
we get something new and interesting. ;)

Thanks,
Jarek P.
---

 include/linux/skbuff.h |    8 ++++++--
 net/packet/af_packet.c |    9 +++++++++
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index cf2cb50..20c3182 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -418,12 +418,16 @@ extern void	      skb_under_panic(struct sk_buff *skb, int len,
 				      void *here);
 extern void	      skb_truesize_bug(struct sk_buff *skb);
 
-static inline void skb_truesize_check(struct sk_buff *skb)
+static inline int skb_truesize_check(struct sk_buff *skb)
 {
 	int len = sizeof(struct sk_buff) + skb->len;
 
-	if (unlikely((int)skb->truesize < len))
+	if (unlikely((int)skb->truesize < len)) {
 		skb_truesize_bug(skb);
+		return 1;
+	}
+
+	return 0;
 }
 
 extern int skb_append_datato_frags(struct sock *sk, struct sk_buff *skb,
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 1fc4a78..08200be 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -307,6 +307,9 @@ static int packet_rcv_spkt(struct sk_buff *skb, struct net_device *dev,  struct
 	if (dev_net(dev) != sock_net(sk))
 		goto out;
 
+	if (skb_truesize_check(skb))
+		goto out;
+
 	if ((skb = skb_share_check(skb, GFP_ATOMIC)) == NULL)
 		goto oom;
 
@@ -495,6 +498,9 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev, struct packet
 	if (dev_net(dev) != sock_net(sk))
 		goto drop;
 
+	if (skb_truesize_check(skb))
+		goto drop;
+
 	skb->dev = dev;
 
 	if (dev->header_ops) {
@@ -617,6 +623,9 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, struct packe
 	if (dev_net(dev) != sock_net(sk))
 		goto drop;
 
+	if (skb_truesize_check(skb))
+		goto drop;
+
 	if (dev->header_ops) {
 		if (sk->sk_type != SOCK_DGRAM)
 			skb_push(skb, skb->data - skb_mac_header(skb));

Re: WARNING: at net/core/skbuff.c:154 with tcpdump and ipsec

[Vlad Yasevich]

Jarek Poplawski wrote:
> On 11-02-2009 16:55, Krzysztof Oledzki wrote:
>> On Wed, 11 Feb 2009, Marco Berizzi wrote:
>>
>>> Hi Folks,
>>>
>>> I'm getting this error on 2.6.28.4 when I run tcpdump on
>>> the interface where ipsec packets are enc/decrypted.
>>>
>>> TIA
>>>
>>>> Feb 11 10:53:55 Pleiadi kernel: ------------[ cut here ]------------
>>>> Feb 11 10:53:55 Pleiadi kernel: WARNING: at net/core/skbuff.c:154 skb_truesize_bug+0x2e/0x33()
>>>> Feb 11 10:53:55 Pleiadi kernel: SKB BUG: Invalid truesize (268) len=134, sizeof(sk_buff)=172
>> <CUT>
>>
>> This annoying problem is quite old (appeared in 2.6.25) and already known:
>>   http://bugzilla.kernel.org/show_bug.cgi?id=10996
>>
>> Sadly, no one is interested in fixing it. :(
> 
> Here is a debugging patch doing these checks a bit earlier, so maybe
> we get something new and interesting. ;)
> 
> Thanks,
> Jarek P.
> ---
> 

I did notice that pskb_expand_head() doesn't change the skb->truesize even
though it could grow the skb.  I saw this problem with tcpdump while
experimenting with some SCTP code.

This is not to say that it is the problem in this case, but it's one of
them that I've seen.

-vlad

Re: WARNING: at net/core/skbuff.c:154 with tcpdump and ipsec

[Jarek Poplawski]

On Fri, Feb 13, 2009 at 01:56:01PM -0500, Vlad Yasevich wrote:
...
> I did notice that pskb_expand_head() doesn't change the skb->truesize even
> though it could grow the skb.  I saw this problem with tcpdump while
> experimenting with some SCTP code.
> 
> This is not to say that it is the problem in this case, but it's one of
> them that I've seen.

Yes, I've read Herbert Xu's message pointing especially to
xfrm_state_check_space(). So I would like to make sure if there is no
other reason it triggers in packet_recvmsg() on these several reports.

If af_packet code is OK, I guess we could update truesize for it:
there is no reason to warn here about bugs from other, well known
places.

Jarek P.

Re: WARNING: at net/core/skbuff.c:154 with tcpdump and ipsec

[Vlad Yasevich]

Jarek Poplawski wrote:
> On Fri, Feb 13, 2009 at 01:56:01PM -0500, Vlad Yasevich wrote:
> ...
>> I did notice that pskb_expand_head() doesn't change the skb->truesize even
>> though it could grow the skb.  I saw this problem with tcpdump while
>> experimenting with some SCTP code.
>>
>> This is not to say that it is the problem in this case, but it's one of
>> them that I've seen.
> 
> Yes, I've read Herbert Xu's message pointing especially to
> xfrm_state_check_space(). So I would like to make sure if there is no
> other reason it triggers in packet_recvmsg() on these several reports.
> 
> If af_packet code is OK, I guess we could update truesize for it:
> there is no reason to warn here about bugs from other, well known
> places.
> 

Personally, I think pskb_expand_head should fix the skb->truesize.  This
way any subsequent clones will not trigger this warning.

Another alternative is to audit the pskb_expand_head() usages and adjust
truesize in each case needed, which is just ugly.

-vlad

> Jarek P.
> 

Re: WARNING: at net/core/skbuff.c:154 with tcpdump and ipsec

[Jarek Poplawski]

On Fri, Feb 13, 2009 at 02:42:07PM -0500, Vlad Yasevich wrote:
...
> Personally, I think pskb_expand_head should fix the skb->truesize.  This
> way any subsequent clones will not trigger this warning.

Personally, I think there is no reason to call skb_truesize_bug() in
anything but some #ifdef CONFIG_XX_DEBUG, if we ignore these reports
for so long.

> Another alternative is to audit the pskb_expand_head() usages and adjust
> truesize in each case needed, which is just ugly.

I guess, it's a lot of work to do it right (if it's possible at all).
Yes, I think about something really ugly here. ;-)

Jarek P.

Re: WARNING: at net/core/skbuff.c:154 with tcpdump and ipsec

[David Miller]

From: Jarek Poplawski <jarkao2@gmail.com>
Date: Fri, 13 Feb 2009 21:00:14 +0100

> On Fri, Feb 13, 2009 at 02:42:07PM -0500, Vlad Yasevich wrote:
> ...
> > Personally, I think pskb_expand_head should fix the skb->truesize.  This
> > way any subsequent clones will not trigger this warning.
> 
> Personally, I think there is no reason to call skb_truesize_bug() in
> anything but some #ifdef CONFIG_XX_DEBUG, if we ignore these reports
> for so long.

If skb->sk is non-NULL, fixing up the truesize will corrupt
socket memory accounting.

Anyways, Herbert and I have talked about %100 removing the
warning.

Re: WARNING: at net/core/skbuff.c:154 with tcpdump and ipsec

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Fri, 13 Feb 2009 18:24:26 -0800 (PST)

> Anyways, Herbert and I have talked about %100 removing the
> warning.

I've committed the following to net-2.6 and will queue
this up for -stable as well.

net: Kill skb_truesize_check(), it only catches false-positives.

A long time ago we had bugs, primarily in TCP, where we would modify
skb->truesize (for TSO queue collapsing) in ways which would corrupt
the socket memory accounting.

skb_truesize_check() was added in order to try and catch this error
more systematically.

However this debugging check has morphed into a Frankenstein of sorts
and these days it does nothing other than catch false-positives.

Signed-off-by: David S. Miller <davem@davemloft.net>
---
 include/linux/skbuff.h |    9 ---------
 include/net/sock.h     |    1 -
 net/core/skbuff.c      |    8 --------
 net/core/sock.c        |    1 -
 4 files changed, 0 insertions(+), 19 deletions(-)

diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index cf2cb50..9dcf956 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -416,15 +416,6 @@ extern void	      skb_over_panic(struct sk_buff *skb, int len,
 				     void *here);
 extern void	      skb_under_panic(struct sk_buff *skb, int len,
 				      void *here);
-extern void	      skb_truesize_bug(struct sk_buff *skb);
-
-static inline void skb_truesize_check(struct sk_buff *skb)
-{
-	int len = sizeof(struct sk_buff) + skb->len;
-
-	if (unlikely((int)skb->truesize < len))
-		skb_truesize_bug(skb);
-}
 
 extern int skb_append_datato_frags(struct sock *sk, struct sk_buff *skb,
 			int getfrag(void *from, char *to, int offset,
diff --git a/include/net/sock.h b/include/net/sock.h
index ce3b5b6..eefeeaf 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -860,7 +860,6 @@ static inline void sk_mem_uncharge(struct sock *sk, int size)
 
 static inline void sk_wmem_free_skb(struct sock *sk, struct sk_buff *skb)
 {
-	skb_truesize_check(skb);
 	sock_set_flag(sk, SOCK_QUEUE_SHRUNK);
 	sk->sk_wmem_queued -= skb->truesize;
 	sk_mem_uncharge(sk, skb->truesize);
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index da74b84..c6a6b16 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -143,14 +143,6 @@ void skb_under_panic(struct sk_buff *skb, int sz, void *here)
 	BUG();
 }
 
-void skb_truesize_bug(struct sk_buff *skb)
-{
-	WARN(net_ratelimit(), KERN_ERR "SKB BUG: Invalid truesize (%u) "
-	       "len=%u, sizeof(sk_buff)=%Zd\n",
-	       skb->truesize, skb->len, sizeof(struct sk_buff));
-}
-EXPORT_SYMBOL(skb_truesize_bug);
-
 /* 	Allocate a new skbuff. We do this ourselves so we can fill in a few
  *	'private' fields and also do memory statistics to find all the
  *	[BEEP] leaks.
diff --git a/net/core/sock.c b/net/core/sock.c
index 6f2e133..6e4f14d 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -1137,7 +1137,6 @@ void sock_rfree(struct sk_buff *skb)
 {
 	struct sock *sk = skb->sk;
 
-	skb_truesize_check(skb);
 	atomic_sub(skb->truesize, &sk->sk_rmem_alloc);
 	sk_mem_uncharge(skb->sk, skb->truesize);
 }
-- 
1.6.1.2.253.ga34a