Re: 2.6.25-rc8: FTP transfer errors

Re: 2.6.25-rc8: FTP transfer errors

[David Miller]

From: vincent-perrier <vincent-perrier@club-internet.fr>
Date: Fri, 11 Apr 2008 01:32:14 +0200

[ Please use netdev@vger.kernel.org so that this discussion
  reaches the networking develops. ]

> Even if the patch is not good, the line dst_free(&rt->u.dst); 
> when rt is still in tree leads to a crash, but when you do not
> do the dst_free, when rt is in tree, then it may have hidden 
> other bugs, but at least I can keep working.
> 
> 
> I never said my patch was good, but it does the minimum to avoid my bug:
> 
> 
>                 if (fn->leaf == NULL) {
>                         bug_8895_clownix_provisional_workaround = 1;
>                         fn->leaf = rt;
>                         atomic_inc(&rt->rt6i_ref);
>                 }
> ...
> 
> ip6_fib.c, line 796:
> 
>                 if (!bug_8895_clownix_provisional_workaround)
>                         dst_free(&rt->u.dst);
> 
> That way at least it does not crash.
> 
> I cannot provide more than the line and the reason for the crash, I am
> one of the numerous brainless users.

Now that the discussion has reached the mailing list, it won't die in
bugzilla like most such bugs do, and very likely will get fixed
quickly as a result.

Thank you.

Re: 2.6.25-rc8: FTP transfer errors

[vincent-perrier]

Thanks to you, and also to Jesper for the "git bisect" explanation,
you have powerfull tools, it is all for the best, millions of users
are relying on you!

On Thu, 2008-04-10 at 16:41 -0700, David Miller wrote:
> From: vincent-perrier <vincent-perrier@club-internet.fr>
> Date: Fri, 11 Apr 2008 01:32:14 +0200
> 
> [ Please use netdev@vger.kernel.org so that this discussion
>   reaches the networking develops. ]
> 
> > Even if the patch is not good, the line dst_free(&rt->u.dst); 
> > when rt is still in tree leads to a crash, but when you do not
> > do the dst_free, when rt is in tree, then it may have hidden 
> > other bugs, but at least I can keep working.
> > 
> > 
> > I never said my patch was good, but it does the minimum to avoid my bug:
> > 
> > 
> >                 if (fn->leaf == NULL) {
> >                         bug_8895_clownix_provisional_workaround = 1;
> >                         fn->leaf = rt;
> >                         atomic_inc(&rt->rt6i_ref);
> >                 }
> > ...
> > 
> > ip6_fib.c, line 796:
> > 
> >                 if (!bug_8895_clownix_provisional_workaround)
> >                         dst_free(&rt->u.dst);
> > 
> > That way at least it does not crash.
> > 
> > I cannot provide more than the line and the reason for the crash, I am
> > one of the numerous brainless users.
> 
> Now that the discussion has reached the mailing list, it won't die in
> bugzilla like most such bugs do, and very likely will get fixed
> quickly as a result.
> 
> Thank you.
> &#0;
> 

Re: 2.6.25-rc8: FTP transfer errors

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Thu, 10 Apr 2008 16:41:06 -0700 (PDT)

> From: vincent-perrier <vincent-perrier@club-internet.fr>
> Date: Fri, 11 Apr 2008 01:32:14 +0200
> 
> > Even if the patch is not good, the line dst_free(&rt->u.dst); 
> > when rt is still in tree leads to a crash, but when you do not
> > do the dst_free, when rt is in tree, then it may have hidden 
> > other bugs, but at least I can keep working.
> > 
> > 
> > I never said my patch was good, but it does the minimum to avoid my bug:
> > 
> > 
> >                 if (fn->leaf == NULL) {
> >                         bug_8895_clownix_provisional_workaround = 1;
> >                         fn->leaf = rt;
> >                         atomic_inc(&rt->rt6i_ref);
> >                 }
> > ...
> > 
> > ip6_fib.c, line 796:
> > 
> >                 if (!bug_8895_clownix_provisional_workaround)
> >                         dst_free(&rt->u.dst);
> > 
> > That way at least it does not crash.

I started looking actively at this.

There are a lot of complicated side effects here, especially when
subtrees are enabled as it is in your case.

The main issue is whether we added any references to 'rt' into
the routing tree.  If we get an error, we have to undo any
such added references.

And that's not being done when the "if (fn->leaf == NULL)" code
runs and fib6_add_rt2node() returns an error.

I think this patch will fix it, could you please test it out?

diff --git a/net/ipv6/ip6_fib.c b/net/ipv6/ip6_fib.c
index b3f6e03..50f3f8f 100644
--- a/net/ipv6/ip6_fib.c
+++ b/net/ipv6/ip6_fib.c
@@ -772,6 +772,10 @@ out:
 		 * If fib6_add_1 has cleared the old leaf pointer in the
 		 * super-tree leaf node we have to find a new one for it.
 		 */
+		if (pn != fn && pn->leaf == rt) {
+			pn->leaf = NULL;
+			atomic_dec(&rt->rt6i_ref);
+		}
 		if (pn != fn && !pn->leaf && !(pn->fn_flags & RTN_RTINFO)) {
 			pn->leaf = fib6_find_prefix(info->nl_net, pn);
 #if RT6_DEBUG >= 2

Re: 2.6.25-rc8: FTP transfer errors

[vincent-perrier]

Hello,
I am very sorry, for the moment I cannot reproduce it and I am going
on holidays tomorow (we have a lot of holidays in france) and I have 
no internet where I go.

Nevertheless, I have tried to reproduce the bug, but all my config has 
changed a lot, the user soft as well as the kernel I use now.
I will try again during the week, but I will not be able to send mail
before next week.

Regards


On Fri, 2008-04-18 at 01:32 -0700, David Miller wrote:
> From: David Miller <davem@davemloft.net>
> Date: Thu, 10 Apr 2008 16:41:06 -0700 (PDT)
> 
> > From: vincent-perrier <vincent-perrier@club-internet.fr>
> > Date: Fri, 11 Apr 2008 01:32:14 +0200
> > 
> > > Even if the patch is not good, the line dst_free(&rt->u.dst); 
> > > when rt is still in tree leads to a crash, but when you do not
> > > do the dst_free, when rt is in tree, then it may have hidden 
> > > other bugs, but at least I can keep working.
> > > 
> > > 
> > > I never said my patch was good, but it does the minimum to avoid my bug:
> > > 
> > > 
> > >                 if (fn->leaf == NULL) {
> > >                         bug_8895_clownix_provisional_workaround = 1;
> > >                         fn->leaf = rt;
> > >                         atomic_inc(&rt->rt6i_ref);
> > >                 }
> > > ...
> > > 
> > > ip6_fib.c, line 796:
> > > 
> > >                 if (!bug_8895_clownix_provisional_workaround)
> > >                         dst_free(&rt->u.dst);
> > > 
> > > That way at least it does not crash.
> 
> I started looking actively at this.
> 
> There are a lot of complicated side effects here, especially when
> subtrees are enabled as it is in your case.
> 
> The main issue is whether we added any references to 'rt' into
> the routing tree.  If we get an error, we have to undo any
> such added references.
> 
> And that's not being done when the "if (fn->leaf == NULL)" code
> runs and fib6_add_rt2node() returns an error.
> 
> I think this patch will fix it, could you please test it out?
> 
> diff --git a/net/ipv6/ip6_fib.c b/net/ipv6/ip6_fib.c
> index b3f6e03..50f3f8f 100644
> --- a/net/ipv6/ip6_fib.c
> +++ b/net/ipv6/ip6_fib.c
> @@ -772,6 +772,10 @@ out:
>  		 * If fib6_add_1 has cleared the old leaf pointer in the
>  		 * super-tree leaf node we have to find a new one for it.
>  		 */
> +		if (pn != fn && pn->leaf == rt) {
> +			pn->leaf = NULL;
> +			atomic_dec(&rt->rt6i_ref);
> +		}
>  		if (pn != fn && !pn->leaf && !(pn->fn_flags & RTN_RTINFO)) {
>  			pn->leaf = fib6_find_prefix(info->nl_net, pn);
>  #if RT6_DEBUG >= 2
> 
> 
> &#0;
> 

Re: 2.6.25-rc8: FTP transfer errors

[Mark Lord]

David Miller wrote:
> From: "Jesper Juhl" <jesper.juhl@gmail.com>
> Date: Fri, 11 Apr 2008 00:09:11 +0200
> 
>> You can't expect users to know how to debug a problem or even bisect
>> it.
> 
> [ The person you are replying to was being sarcastic, BTW. ]
> 
> That's not the case we're talking about in this specific instance.  In
> this particular case the user is more than capable of bisecting, he
> just isn't willing to invest the time.
..

Duh.. more like, "If I take 5-8 hours to attempt a bisect (which may not
even work), then that's 5-8 hours I do not get paid for."

Gotta eat, dude.

Anyways, here's five hours of free consulting for you:


git-bisect start
# bad: [7180c4c9e09888db0a188f729c96c6d7bd61fa83] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/selinux-2.6
git-bisect bad 7180c4c9e09888db0a188f729c96c6d7bd61fa83
# good: [49914084e797530d9baaf51df9eda77babc98fa8] Linux 2.6.24
git-bisect good 49914084e797530d9baaf51df9eda77babc98fa8
# bad: [e5dfb815181fcb186d6080ac3a091eadff2d98fe] [NET_SCHED]: Add flow classifier
git-bisect bad e5dfb815181fcb186d6080ac3a091eadff2d98fe
# good: [00e0b8cb74ed7c16b2bc41eb33a16eae5b6e2d5c] b43: reinit on too many PHY TX errors
git-bisect good 00e0b8cb74ed7c16b2bc41eb33a16eae5b6e2d5c
# good: [42d545c9a4c0d3faeab658a40165c3da2dda91b2] x86: remove depends on X86_32 from PARAVIRT & PARAVIRT_GUEST
git-bisect good 42d545c9a4c0d3faeab658a40165c3da2dda91b2
# good: [6232665040f9a23fafd9d94d4ae8d5a2dc850f65] Merge git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86
git-bisect good 6232665040f9a23fafd9d94d4ae8d5a2dc850f65
# good: [e5723b41abe559bafc52591dcf8ee19cc131d3a1] [ALSA] Remove sequencer instrument layer
git-bisect good e5723b41abe559bafc52591dcf8ee19cc131d3a1
# good: [461e2c78b153e38f284d09721c50c0cd3c47e073] [ALSA] hda-codec - Add Conexant 5051 codec support
git-bisect good 461e2c78b153e38f284d09721c50c0cd3c47e073
# good: [1987e7b4855fcb6a866d3279ee9f2890491bc34d] [AX25]: Kill ax25_bind() user triggable printk.
git-bisect good 1987e7b4855fcb6a866d3279ee9f2890491bc34d
# good: [58a3c9bb0c69f8517c2243cd0912b3f87b4f868c] [NETFILTER]: nf_conntrack: use RCU for conntrack helpers
git-bisect good 58a3c9bb0c69f8517c2243cd0912b3f87b4f868c
# good: [32948588ac4ec54300bae1037e839277fd4536e2] [NETFILTER]: nf_conntrack: annotate l3protos with const
git-bisect good 32948588ac4ec54300bae1037e839277fd4536e2
# bad: [e83a2ea850bf0c0c81c675444080970fc07798c6] [VLAN]: set_rx_mode support for unicast address list
git-bisect bad e83a2ea850bf0c0c81c675444080970fc07798c6
# good: [941b1d22cc035ad58b3d9b44a1c74efac2d7e499] [NETNS]: Make bind buckets live in net namespaces.
git-bisect good 941b1d22cc035ad58b3d9b44a1c74efac2d7e499
# bad: [23717795bee15470b96f9b7aa5ecf4efe14c8e32] [IPV6]: Update MSS even if MTU is unchanged.
git-bisect bad 23717795bee15470b96f9b7aa5ecf4efe14c8e32
# bad: [d86e0dac2ce412715181f792aa0749fe3effff11] [NETNS]: Tcp-v6 sockets per-net lookup.
git-bisect bad d86e0dac2ce412715181f792aa0749fe3effff11
# bad: [c67499c0e772064b37ad75eb69b28fc218752636] [NETNS]: Tcp-v4 sockets per-net lookup.
git-bisect bad c67499c0e772064b37ad75eb69b28fc218752636

[c67499c0e772064b37ad75eb69b28fc218752636 is first bad commit
commit c67499c0e772064b37ad75eb69b28fc218752636
Author: Pavel Emelyanov <xemul@openvz.org>
Date:   Thu Jan 31 05:06:40 2008 -0800

    [NETNS]: Tcp-v4 sockets per-net lookup.

    Add a net argument to inet_lookup and propagate it further
    into lookup calls. Plus tune the __inet_check_established.

    The dccp and inet_diag, which use that lookup functions
    pass the init_net into them.

    Signed-off-by: Pavel Emelyanov <xemul@openvz.org>
    Signed-off-by: David S. Miller <davem@davemloft.net>

---

diff --git a/include/net/inet_hashtables.h b/include/net/inet_hashtables.h
index 55532b9..c23c4ed 100644
--- a/include/net/inet_hashtables.h
+++ b/include/net/inet_hashtables.h
@@ -302,15 +302,17 @@ out:
 		wake_up(&hashinfo->lhash_wait);
 }
 
-extern struct sock *__inet_lookup_listener(struct inet_hashinfo *hashinfo,
+extern struct sock *__inet_lookup_listener(struct net *net,
+					   struct inet_hashinfo *hashinfo,
 					   const __be32 daddr,
 					   const unsigned short hnum,
 					   const int dif);
 
-static inline struct sock *inet_lookup_listener(struct inet_hashinfo *hashinfo,
-						__be32 daddr, __be16 dport, int dif)
+static inline struct sock *inet_lookup_listener(struct net *net,
+		struct inet_hashinfo *hashinfo,
+		__be32 daddr, __be16 dport, int dif)
 {
-	return __inet_lookup_listener(hashinfo, daddr, ntohs(dport), dif);
+	return __inet_lookup_listener(net, hashinfo, daddr, ntohs(dport), dif);
 }
 
 /* Socket demux engine toys. */
@@ -344,26 +346,26 @@ typedef __u64 __bitwise __addrpair;
 				   (((__force __u64)(__be32)(__daddr)) << 32) | \
 				   ((__force __u64)(__be32)(__saddr)));
 #endif /* __BIG_ENDIAN */
-#define INET_MATCH(__sk, __hash, __cookie, __saddr, __daddr, __ports, __dif)\
-	(((__sk)->sk_hash == (__hash))				&&	\
+#define INET_MATCH(__sk, __net, __hash, __cookie, __saddr, __daddr, __ports, __dif)\
+	(((__sk)->sk_hash == (__hash)) && ((__sk)->sk_net == (__net))	&&	\
 	 ((*((__addrpair *)&(inet_sk(__sk)->daddr))) == (__cookie))	&&	\
 	 ((*((__portpair *)&(inet_sk(__sk)->dport))) == (__ports))	&&	\
 	 (!((__sk)->sk_bound_dev_if) || ((__sk)->sk_bound_dev_if == (__dif))))
-#define INET_TW_MATCH(__sk, __hash, __cookie, __saddr, __daddr, __ports, __dif)\
-	(((__sk)->sk_hash == (__hash))				&&	\
+#define INET_TW_MATCH(__sk, __net, __hash, __cookie, __saddr, __daddr, __ports, __dif)\
+	(((__sk)->sk_hash == (__hash)) && ((__sk)->sk_net == (__net))	&&	\
 	 ((*((__addrpair *)&(inet_twsk(__sk)->tw_daddr))) == (__cookie)) &&	\
 	 ((*((__portpair *)&(inet_twsk(__sk)->tw_dport))) == (__ports)) &&	\
 	 (!((__sk)->sk_bound_dev_if) || ((__sk)->sk_bound_dev_if == (__dif))))
 #else /* 32-bit arch */
 #define INET_ADDR_COOKIE(__name, __saddr, __daddr)
-#define INET_MATCH(__sk, __hash, __cookie, __saddr, __daddr, __ports, __dif)	\
-	(((__sk)->sk_hash == (__hash))				&&	\
+#define INET_MATCH(__sk, __net, __hash, __cookie, __saddr, __daddr, __ports, __dif)	\
+	(((__sk)->sk_hash == (__hash)) && ((__sk)->sk_net == (__net))	&&	\
 	 (inet_sk(__sk)->daddr		== (__saddr))		&&	\
 	 (inet_sk(__sk)->rcv_saddr	== (__daddr))		&&	\
 	 ((*((__portpair *)&(inet_sk(__sk)->dport))) == (__ports))	&&	\
 	 (!((__sk)->sk_bound_dev_if) || ((__sk)->sk_bound_dev_if == (__dif))))
-#define INET_TW_MATCH(__sk, __hash,__cookie, __saddr, __daddr, __ports, __dif)	\
-	(((__sk)->sk_hash == (__hash))				&&	\
+#define INET_TW_MATCH(__sk, __net, __hash,__cookie, __saddr, __daddr, __ports, __dif)	\
+	(((__sk)->sk_hash == (__hash)) && ((__sk)->sk_net == (__net))	&&	\
 	 (inet_twsk(__sk)->tw_daddr	== (__saddr))		&&	\
 	 (inet_twsk(__sk)->tw_rcv_saddr	== (__daddr))		&&	\
 	 ((*((__portpair *)&(inet_twsk(__sk)->tw_dport))) == (__ports)) &&	\
@@ -376,32 +378,36 @@ typedef __u64 __bitwise __addrpair;
  *
  * Local BH must be disabled here.
  */
-extern struct sock * __inet_lookup_established(struct inet_hashinfo *hashinfo,
+extern struct sock * __inet_lookup_established(struct net *net,
+		struct inet_hashinfo *hashinfo,
 		const __be32 saddr, const __be16 sport,
 		const __be32 daddr, const u16 hnum, const int dif);
 
 static inline struct sock *
-	inet_lookup_established(struct inet_hashinfo *hashinfo,
+	inet_lookup_established(struct net *net, struct inet_hashinfo *hashinfo,
 				const __be32 saddr, const __be16 sport,
 				const __be32 daddr, const __be16 dport,
 				const int dif)
 {
-	return __inet_lookup_established(hashinfo, saddr, sport, daddr,
+	return __inet_lookup_established(net, hashinfo, saddr, sport, daddr,
 					 ntohs(dport), dif);
 }
 
-static inline struct sock *__inet_lookup(struct inet_hashinfo *hashinfo,
+static inline struct sock *__inet_lookup(struct net *net,
+					 struct inet_hashinfo *hashinfo,
 					 const __be32 saddr, const __be16 sport,
 					 const __be32 daddr, const __be16 dport,
 					 const int dif)
 {
 	u16 hnum = ntohs(dport);
-	struct sock *sk = __inet_lookup_established(hashinfo, saddr, sport, daddr,
-						    hnum, dif);
-	return sk ? : __inet_lookup_listener(hashinfo, daddr, hnum, dif);
+	struct sock *sk = __inet_lookup_established(net, hashinfo,
+				saddr, sport, daddr, hnum, dif);
+
+	return sk ? : __inet_lookup_listener(net, hashinfo, daddr, hnum, dif);
 }
 
-static inline struct sock *inet_lookup(struct inet_hashinfo *hashinfo,
+static inline struct sock *inet_lookup(struct net *net,
+				       struct inet_hashinfo *hashinfo,
 				       const __be32 saddr, const __be16 sport,
 				       const __be32 daddr, const __be16 dport,
 				       const int dif)
@@ -409,7 +415,7 @@ static inline struct sock *inet_lookup(struct inet_hashinfo *hashinfo,
 	struct sock *sk;
 
 	local_bh_disable();
-	sk = __inet_lookup(hashinfo, saddr, sport, daddr, dport, dif);
+	sk = __inet_lookup(net, hashinfo, saddr, sport, daddr, dport, dif);
 	local_bh_enable();
 
 	return sk;
diff --git a/net/dccp/ipv4.c b/net/dccp/ipv4.c
index 9e38b0d..c982ad8 100644
--- a/net/dccp/ipv4.c
+++ b/net/dccp/ipv4.c
@@ -218,7 +218,7 @@ static void dccp_v4_err(struct sk_buff *skb, u32 info)
 		return;
 	}
 
-	sk = inet_lookup(&dccp_hashinfo, iph->daddr, dh->dccph_dport,
+	sk = inet_lookup(&init_net, &dccp_hashinfo, iph->daddr, dh->dccph_dport,
 			 iph->saddr, dh->dccph_sport, inet_iif(skb));
 	if (sk == NULL) {
 		ICMP_INC_STATS_BH(ICMP_MIB_INERRORS);
@@ -436,7 +436,7 @@ static struct sock *dccp_v4_hnd_req(struct sock *sk, struct sk_buff *skb)
 	if (req != NULL)
 		return dccp_check_req(sk, skb, req, prev);
 
-	nsk = inet_lookup_established(&dccp_hashinfo,
+	nsk = inet_lookup_established(&init_net, &dccp_hashinfo,
 				      iph->saddr, dh->dccph_sport,
 				      iph->daddr, dh->dccph_dport,
 				      inet_iif(skb));
@@ -817,7 +817,7 @@ static int dccp_v4_rcv(struct sk_buff *skb)
 
 	/* Step 2:
 	 *	Look up flow ID in table and get corresponding socket */
-	sk = __inet_lookup(&dccp_hashinfo,
+	sk = __inet_lookup(&init_net, &dccp_hashinfo,
 			   iph->saddr, dh->dccph_sport,
 			   iph->daddr, dh->dccph_dport, inet_iif(skb));
 	/*
diff --git a/net/ipv4/inet_diag.c b/net/ipv4/inet_diag.c
index 4cfb15c..95c9f14 100644
--- a/net/ipv4/inet_diag.c
+++ b/net/ipv4/inet_diag.c
@@ -268,7 +268,7 @@ static int inet_diag_get_exact(struct sk_buff *in_skb,
 	err = -EINVAL;
 
 	if (req->idiag_family == AF_INET) {
-		sk = inet_lookup(hashinfo, req->id.idiag_dst[0],
+		sk = inet_lookup(&init_net, hashinfo, req->id.idiag_dst[0],
 				 req->id.idiag_dport, req->id.idiag_src[0],
 				 req->id.idiag_sport, req->id.idiag_if);
 	}
diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c
index db1e53a..48d4500 100644
--- a/net/ipv4/inet_hashtables.c
+++ b/net/ipv4/inet_hashtables.c
@@ -127,7 +127,8 @@ EXPORT_SYMBOL(inet_listen_wlock);
  * remote address for the connection. So always assume those are both
  * wildcarded during the search since they can never be otherwise.
  */
-static struct sock *inet_lookup_listener_slow(const struct hlist_head *head,
+static struct sock *inet_lookup_listener_slow(struct net *net,
+					      const struct hlist_head *head,
 					      const __be32 daddr,
 					      const unsigned short hnum,
 					      const int dif)
@@ -139,7 +140,8 @@ static struct sock *inet_lookup_listener_slow(const struct hlist_head *head,
 	sk_for_each(sk, node, head) {
 		const struct inet_sock *inet = inet_sk(sk);
 
-		if (inet->num == hnum && !ipv6_only_sock(sk)) {
+		if (sk->sk_net == net && inet->num == hnum &&
+				!ipv6_only_sock(sk)) {
 			const __be32 rcv_saddr = inet->rcv_saddr;
 			int score = sk->sk_family == PF_INET ? 1 : 0;
 
@@ -165,7 +167,8 @@ static struct sock *inet_lookup_listener_slow(const struct hlist_head *head,
 }
 
 /* Optimize the common listener case. */
-struct sock *__inet_lookup_listener(struct inet_hashinfo *hashinfo,
+struct sock *__inet_lookup_listener(struct net *net,
+				    struct inet_hashinfo *hashinfo,
 				    const __be32 daddr, const unsigned short hnum,
 				    const int dif)
 {
@@ -180,9 +183,9 @@ struct sock *__inet_lookup_listener(struct inet_hashinfo *hashinfo,
 		if (inet->num == hnum && !sk->sk_node.next &&
 		    (!inet->rcv_saddr || inet->rcv_saddr == daddr) &&
 		    (sk->sk_family == PF_INET || !ipv6_only_sock(sk)) &&
-		    !sk->sk_bound_dev_if)
+		    !sk->sk_bound_dev_if && sk->sk_net == net)
 			goto sherry_cache;
-		sk = inet_lookup_listener_slow(head, daddr, hnum, dif);
+		sk = inet_lookup_listener_slow(net, head, daddr, hnum, dif);
 	}
 	if (sk) {
 sherry_cache:
@@ -193,7 +196,8 @@ sherry_cache:
 }
 EXPORT_SYMBOL_GPL(__inet_lookup_listener);
 
-struct sock * __inet_lookup_established(struct inet_hashinfo *hashinfo,
+struct sock * __inet_lookup_established(struct net *net,
+				  struct inet_hashinfo *hashinfo,
 				  const __be32 saddr, const __be16 sport,
 				  const __be32 daddr, const u16 hnum,
 				  const int dif)
@@ -212,13 +216,15 @@ struct sock * __inet_lookup_established(struct inet_hashinfo *hashinfo,
 	prefetch(head->chain.first);
 	read_lock(lock);
 	sk_for_each(sk, node, &head->chain) {
-		if (INET_MATCH(sk, hash, acookie, saddr, daddr, ports, dif))
+		if (INET_MATCH(sk, net, hash, acookie,
+					saddr, daddr, ports, dif))
 			goto hit; /* You sunk my battleship! */
 	}
 
 	/* Must check for a TIME_WAIT'er before going to listener hash. */
 	sk_for_each(sk, node, &head->twchain) {
-		if (INET_TW_MATCH(sk, hash, acookie, saddr, daddr, ports, dif))
+		if (INET_TW_MATCH(sk, net, hash, acookie,
+					saddr, daddr, ports, dif))
 			goto hit;
 	}
 	sk = NULL;
@@ -249,6 +255,7 @@ static int __inet_check_established(struct inet_timewait_death_row *death_row,
 	struct sock *sk2;
 	const struct hlist_node *node;
 	struct inet_timewait_sock *tw;
+	struct net *net = sk->sk_net;
 
 	prefetch(head->chain.first);
 	write_lock(lock);
@@ -257,7 +264,8 @@ static int __inet_check_established(struct inet_timewait_death_row *death_row,
 	sk_for_each(sk2, node, &head->twchain) {
 		tw = inet_twsk(sk2);
 
-		if (INET_TW_MATCH(sk2, hash, acookie, saddr, daddr, ports, dif)) {
+		if (INET_TW_MATCH(sk2, net, hash, acookie,
+					saddr, daddr, ports, dif)) {
 			if (twsk_unique(sk, sk2, twp))
 				goto unique;
 			else
@@ -268,7 +276,8 @@ static int __inet_check_established(struct inet_timewait_death_row *death_row,
 
 	/* And established part... */
 	sk_for_each(sk2, node, &head->chain) {
-		if (INET_MATCH(sk2, hash, acookie, saddr, daddr, ports, dif))
+		if (INET_MATCH(sk2, net, hash, acookie,
+					saddr, daddr, ports, dif))
 			goto not_unique;
 	}
 
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 9aea88b..77c1939 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -369,8 +369,8 @@ void tcp_v4_err(struct sk_buff *skb, u32 info)
 		return;
 	}
 
-	sk = inet_lookup(&tcp_hashinfo, iph->daddr, th->dest, iph->saddr,
-			 th->source, inet_iif(skb));
+	sk = inet_lookup(skb->dev->nd_net, &tcp_hashinfo, iph->daddr, th->dest,
+			iph->saddr, th->source, inet_iif(skb));
 	if (!sk) {
 		ICMP_INC_STATS_BH(ICMP_MIB_INERRORS);
 		return;
@@ -1503,8 +1503,8 @@ static struct sock *tcp_v4_hnd_req(struct sock *sk, struct sk_buff *skb)
 	if (req)
 		return tcp_check_req(sk, skb, req, prev);
 
-	nsk = inet_lookup_established(&tcp_hashinfo, iph->saddr, th->source,
-				      iph->daddr, th->dest, inet_iif(skb));
+	nsk = inet_lookup_established(sk->sk_net, &tcp_hashinfo, iph->saddr,
+			th->source, iph->daddr, th->dest, inet_iif(skb));
 
 	if (nsk) {
 		if (nsk->sk_state != TCP_TIME_WAIT) {
@@ -1661,8 +1661,8 @@ int tcp_v4_rcv(struct sk_buff *skb)
 	TCP_SKB_CB(skb)->flags	 = iph->tos;
 	TCP_SKB_CB(skb)->sacked	 = 0;
 
-	sk = __inet_lookup(&tcp_hashinfo, iph->saddr, th->source,
-			   iph->daddr, th->dest, inet_iif(skb));
+	sk = __inet_lookup(skb->dev->nd_net, &tcp_hashinfo, iph->saddr,
+			th->source, iph->daddr, th->dest, inet_iif(skb));
 	if (!sk)
 		goto no_tcp_socket;
 
@@ -1735,7 +1735,8 @@ do_time_wait:
 	}
 	switch (tcp_timewait_state_process(inet_twsk(sk), skb, th)) {
 	case TCP_TW_SYN: {
-		struct sock *sk2 = inet_lookup_listener(&tcp_hashinfo,
+		struct sock *sk2 = inet_lookup_listener(skb->dev->nd_net,
+							&tcp_hashinfo,
 							iph->daddr, th->dest,
 							inet_iif(skb));
 		if (sk2) {

Re: 2.6.25-rc8: FTP transfer errors

[David Miller]

From: Mark Lord <lkml@rtr.ca>
Date: Thu, 10 Apr 2008 20:16:11 -0400

> Duh.. more like, "If I take 5-8 hours to attempt a bisect (which may not
> even work), then that's 5-8 hours I do not get paid for."

And if I invest my spare time on your bug how does this statement
apply to me?  Or does it only apply to you?

Every single argument you make that supports why you should not be
investing the necessary time into the bug applies equally to the
very developers you are so quickly to quip at and want help from.

Re: 2.6.25-rc8: FTP transfer errors

[Mark Lord]

David Miller wrote:
> From: Mark Lord <lkml@rtr.ca>
> Date: Thu, 10 Apr 2008 20:16:11 -0400
> 
>> Duh.. more like, "If I take 5-8 hours to attempt a bisect (which may not
>> even work), then that's 5-8 hours I do not get paid for."
> 
> And if I invest my spare time on your bug
> how does this statement apply to me?
..

It's not "my bug".  I'm just the first person to notice,
take time to report it, and even hand it to you on a platter (bisect).

It's *your* bug -- you signed off on the commit.

Cheers

Re: 2.6.25-rc8: FTP transfer errors

[David Miller]

From: Mark Lord <lkml@rtr.ca>
Date: Thu, 10 Apr 2008 20:27:14 -0400

> It's *your* bug -- you signed off on the commit.

I sign off on basically every networking commit, does that mean I have
to fix every networking bug and every networking bug is "mine"?

Of course not, that doesn't scale at all.  What does scale is a
combination of good fully formed bug reports from users combined with
the efforts of the global developer pool.

Linus signs off on every patch from Andrew Morton he puts into the
tree, which is a lot, but does Linus work on every bug introduced by
one of those patches and are such bugs "his" bugs?  Of course he
doesn't, and of course not.  They get pushed up to the person
who wrote the patch once identified as such, and the patch is
reverted if the developer is unresponsive and this will have
consequences for patches they submit in the future.

I still think you have a very self-centered attitude about things.
This is about distributing effort, not forcing it upon individuals
or a constrained resource.

If I get hit by a bus, networking bugs would still get fixed if
handled properly.

And it's a win-win situation.  The incentive for a capable user to do
a bisect or whatever else is that if they do it their bug gets fixed
quickly.  That is the free market economy of Linux kernel bug
reporting.

It addresses the issue that in reality we'll never fix all bugs, and
therefore we prioritize.  And therefore if there is a bisected bug
report and also another one from a user who refuses to do that, guess
which bug gets worked on with a higher priority and which bug gets
fixed first?

Re: 2.6.25-rc8: FTP transfer errors

[Mark Lord]

David Miller wrote:
> From: Mark Lord <lkml@rtr.ca>
> Date: Thu, 10 Apr 2008 20:27:14 -0400
> 
>> It's *your* bug -- you signed off on the commit.
> 
> I sign off on basically every networking commit, does that mean I have
> to fix every networking bug and every networking bug is "mine"?
..

Absolutely, though to a varying degree.  That's the responsibility
that goes with the role of a subsystem maintainer.  I once had
such a role, and gave it up when I felt I could no longer keep up.  

You still keep refering to it as "your (my) bug".
It's not.  I had nothing to do with it, other than stumbling over it.

When people stumble over a libata bug, I look hard to see if my code
could possibly cause it.  Jeff looks even harder, because he's the
current subsystem dude for libata.

I never suggest a user search through a mountain of unrelated commits
for something I've screwed up on.  I give more directed help, patches
to collect more relevant information, and patches to try and resolve it.

The last thing I'd ever do, is diss the reporter.

Regards.

Re: 2.6.25-rc8: FTP transfer errors

[Ilpo Järvinen]

On Thu, 10 Apr 2008, Mark Lord wrote:

> David Miller wrote:
> > From: Mark Lord <lkml@rtr.ca>
> > Date: Thu, 10 Apr 2008 20:27:14 -0400
> > 
> > > It's *your* bug -- you signed off on the commit.
> > 
> > I sign off on basically every networking commit, does that mean I have
> > to fix every networking bug and every networking bug is "mine"?
> ..
> 
> Absolutely, though to a varying degree.  That's the responsibility
> that goes with the role of a subsystem maintainer.  I once had
> such a role, and gave it up when I felt I could no longer keep up.  
> You still keep refering to it as "your (my) bug".
> It's not.  I had nothing to do with it, other than stumbling over it.

This bug is perfect example where bisect clearly was useful :-). Nobody 
knew whose bug it actually was until your bisect gave directions.

> When people stumble over a libata bug, I look hard to see if my code
> could possibly cause it.  Jeff looks even harder, because he's the
> current subsystem dude for libata.
>
> I never suggest a user search through a mountain of unrelated commits
> for something I've screwed up on.

But it is ok for you to ask an innocent net developer to do that (even 
with your terms as I hadn't signed off _anything_ related to that one),
hmm?

...You had this pretty demanding tone earlier:

> Or I can ignore it, like the net developers, since I have a workaround.
> And then we'll see what other apps are broken upon 2.6.25 final release.
>
> Really, folks.  Bug reports are intended to *help* the developers,
> not something to be thrown back in their faces.
>
> There do seem to have been a *lot* of changes around the tcp closing/close
> code (as I see from diff'ing 2.6.24 against latest -git).
>
> *Somebody* is responsible for those changes.
> That particular *somebody* ought to volunteer some help here,
> reducing the mountain of commits to a big handful or two.

...and also...

> > Anyways, here's five hours of free consulting for you

...Sure I could use similar words, but you might use the not-mine
bug approach again to deflect... :-( ...No, I don't mind really :-).
I well understand that I occassionally end up chasing things 
which are bugs that other people have caused, that's part of the
game.

> I give more directed help, patches to collect more relevant information, 
> and patches to try and resolve it.

Now that you have, as stated earlier, first looked the diffs (tcp*.c stuff 
mainly I suppose?!?), and the bisected it and found the breaker, and even 
patch is available already... Seriously, knowing all what's now available, 
how could we have solved _this particular case_ without that very useful 
help (bisect) from your side?

Yes, I went through the commit list (maybe you did as well), I'm not sure 
if Dave did as well. In addition, I checked a number of individual diffs 
too but this just isn't something very obvious (I have to admit though 
that I don't really understand all those namespace things, so I didn't 
even know how to look them too carefully).


-- 
 i.

Re: 2.6.25-rc8: FTP transfer errors

[Mark Lord]

Ilpo Järvinen wrote:
> On Thu, 10 Apr 2008, Mark Lord wrote:
> 
>> David Miller wrote:
>>> From: Mark Lord <lkml@rtr.ca>
>>> Date: Thu, 10 Apr 2008 20:27:14 -0400
>>>
>>>> It's *your* bug -- you signed off on the commit.
>>> I sign off on basically every networking commit, does that mean I have
>>> to fix every networking bug and every networking bug is "mine"?
>> ..
>>
>> Absolutely, though to a varying degree.  That's the responsibility
>> that goes with the role of a subsystem maintainer.  I once had
>> such a role, and gave it up when I felt I could no longer keep up.  
>> You still keep refering to it as "your (my) bug".
>> It's not.  I had nothing to do with it, other than stumbling over it.
> 
> This bug is perfect example where bisect clearly was useful :-). Nobody 
> knew whose bug it actually was until your bisect gave directions.
> 
>> When people stumble over a libata bug, I look hard to see if my code
>> could possibly cause it.  Jeff looks even harder, because he's the
>> current subsystem dude for libata.
>>
>> I never suggest a user search through a mountain of unrelated commits
>> for something I've screwed up on.
> 
> But it is ok for you to ask an innocent net developer to do that (even 
> with your terms as I hadn't signed off _anything_ related to that one),
> hmm?
> 
> ...You had this pretty demanding tone earlier:
> 
>> Or I can ignore it, like the net developers, since I have a workaround.
>> And then we'll see what other apps are broken upon 2.6.25 final release.
..

That's not demanding, that's quite relaxed.  I had a good workaround,
and didn't really care any more at that point.  Just though it was rather
odd that none of the developers seemed interested in tracking it down.
I offered tons of help, gave it, and said I didn't have time for a full
bisect at that juncture.

For that, I get repeatedly slammed by the netdev folks.
Even after I put aside *paid* work to submit to your demands.

Next time around, I won't bother reporting bugs to you folks,
that's for damned sure.

Cheers

Re: 2.6.25-rc8: FTP transfer errors

[Evgeniy Polyakov]

On Fri, Apr 11, 2008 at 09:19:17AM -0400, Mark Lord (lkml@rtr.ca) wrote:
> >>Or I can ignore it, like the net developers, since I have a workaround.
> >>And then we'll see what other apps are broken upon 2.6.25 final release.
> 
> That's not demanding, that's quite relaxed.  I had a good workaround,
> and didn't really care any more at that point.  Just though it was rather
> odd that none of the developers seemed interested in tracking it down.
> I offered tons of help, gave it, and said I didn't have time for a full
> bisect at that juncture.
> 
> For that, I get repeatedly slammed by the netdev folks.
> Even after I put aside *paid* work to submit to your demands.
> 
> Next time around, I won't bother reporting bugs to you folks,
> that's for damned sure.

Actually that will be the best decision from evolutional point of view. 

Bugs, which 'are thrown back to your face' like what you did with this
one, are useless. Developers already know, that bugs exist. 

If you do not care about bug, why do you ever bothered filling it?
You expected that anyone will start running to fix it for you.
You were wrong. Developers only fix bugs, which do not require
mind-reading and magnetic quantification of your brain.

If you do not want to help fixing it, do not expect it will be fixed at
all. Sentence, that you will probably understand better: no one get paid
to fix it.

No one get fun fixing something with description you provided
(not sure about David though, probably he has some masochistic
propensities doing that and trying to get some bits of information
from reportes for years).
You were suggested some simple checks, they did not help.
Developers can not remotely control electrons in your wires, so next
sugestion was bisecting, which ended up with some crap from your point.

If you want bug got fixed, provide info and if it is not enough, help by
trying what you are being suggested, if you do not want, stop.

-- 
	Evgeniy Polyakov

Re: 2.6.25-rc8: FTP transfer errors

[Mark Lord]

Evgeniy Polyakov wrote:
> On Fri, Apr 11, 2008 at 09:19:17AM -0400, Mark Lord (lkml@rtr.ca) wrote:
>>>> Or I can ignore it, like the net developers, since I have a workaround.
>>>> And then we'll see what other apps are broken upon 2.6.25 final release.
>> That's not demanding, that's quite relaxed.  I had a good workaround,
>> and didn't really care any more at that point.  Just though it was rather
>> odd that none of the developers seemed interested in tracking it down.
>> I offered tons of help, gave it, and said I didn't have time for a full
>> bisect at that juncture.
>>
>> For that, I get repeatedly slammed by the netdev folks.
>> Even after I put aside *paid* work to submit to your demands.
>>
>> Next time around, I won't bother reporting bugs to you folks,
>> that's for damned sure.
> 
> Actually that will be the best decision from evolutional point of view. 
> 
> Bugs, which 'are thrown back to your face' like what you did with this
> one, are useless. Developers already know, that bugs exist. 
> 
> If you do not care about bug, why do you ever bothered filling it?
..

Because I care, about Linux's reputation and performance.
I care about basic networking operations, and knew that this bug
would probably affect other applications once widely deployed.

I care about Linux.  That's why.

> If you do not want to help fixing it ...

Where the hell did I *ever* say that?
I did nothing but offer help, and respond quickly.

The one thing I did not have time for initially,
was a painstaking blunt instrument binary search of
every commit since v2.6.24.

There are other ways to debug things and find the causes
quickly, with less impact upon the reporters of bugs.

The current generation of kernel "code submitters" here
seems to have never learned those.  Bummer.

Cheers

Re: 2.6.25-rc8: FTP transfer errors

[Evgeniy Polyakov]

On Fri, Apr 11, 2008 at 10:59:57AM -0400, Mark Lord (lkml@rtr.ca) wrote:
> >If you do not care about bug, why do you ever bothered filling it?
> ..
> 
> Because I care, about Linux's reputation and performance.
> I care about basic networking operations, and knew that this bug
> would probably affect other applications once widely deployed.
> 
> I care about Linux.  That's why.

Blah-blah-blah, you care so much, that pissed people off which suggested
you how to really help Linux. And then you returned with besiect
results. You helped, but if you just cared, you had missed first part.

> >If you do not want to help fixing it ...
> 
> Where the hell did I *ever* say that?
> I did nothing but offer help, and respond quickly.

Citations:
> So you will likely need to bisect.

Or I can ignore it, like the net developers, since I have a workaround.
And then we'll see what other apps are broken upon 2.6.25 final release.

*Somebody* is responsible for those changes.
That particular *somebody* ought to volunteer some help here,
reducing the mountain of commits to a big handful or two.

-----

I can proceed - just reread a thread.

> The one thing I did not have time for initially,
> was a painstaking blunt instrument binary search of
> every commit since v2.6.24.

If you do not know math, binary search takes log2(N), so you
would only need to check at most around dozen commits. That's lot
of time to run 'git bisect good/bad', especially for man, who
"care about Linux".

> There are other ways to debug things and find the causes
> quickly, with less impact upon the reporters of bugs.

I know one. It is guessing.

I will start: did you start hearing voices after 2.6.24 upgrade?
Or did you observed meteorites hitting wires between you machines?

> The current generation of kernel "code submitters" here
> seems to have never learned those.  Bummer.

Next time I will ask soothsayer, she really knows how to debug network
bug with following description: "it worked before I changed a kernel
version. you have to return my puppets back".

-- 
	Evgeniy Polyakov

Re: 2.6.25-rc8: FTP transfer errors

[David Miller]

From: Evgeniy Polyakov <johnpol@2ka.mipt.ru>
Date: Fri, 11 Apr 2008 19:18:21 +0400

> On Fri, Apr 11, 2008 at 10:59:57AM -0400, Mark Lord (lkml@rtr.ca) wrote:
> > >If you do not care about bug, why do you ever bothered filling it?
> > ..
> > 
> > Because I care, about Linux's reputation and performance.
> > I care about basic networking operations, and knew that this bug
> > would probably affect other applications once widely deployed.
> > 
> > I care about Linux.  That's why.
> 
> Blah-blah-blah, you care so much, that pissed people off which suggested
> you how to really help Linux. And then you returned with besiect
> results. You helped, but if you just cared, you had missed first part.

Every time I see someone play the "I care about Linux" card, they are
typically being a hypocrit.  It's a knee jerk, defensive gesture, and
usually has absolutely zero substance.

> > The current generation of kernel "code submitters" here
> > seems to have never learned those.  Bummer.
> 
> Next time I will ask soothsayer, she really knows how to debug network
> bug with following description: "it worked before I changed a kernel
> version. you have to return my puppets back".

Thanks for your support Evgeniy, it is truly appreciated.

We had Mark's bug fixed in 15 minutes once the bisect result was
known, even after Ilpo and myself had scanned through the changesets.

This proves the utility of bisect and in fact that trying to intuit
the cause by continuing to study changesets and code would have been a
complete waste of time.

Yes, Mark, we used to do things that way for every bug in the kernel.
And as a result many bugs sat unfixed for weeks if not months.  Many
of us have left the cave, feel free to join us.

Re: 2.6.25-rc8: FTP transfer errors

[Evgeniy Polyakov]

On Fri, Apr 11, 2008 at 11:07:02AM -0700, David Miller (davem@davemloft.net) wrote:
> Thanks for your support Evgeniy, it is truly appreciated.

No problemo :)

-- 
	Evgeniy Polyakov

Re: 2.6.25-rc8: FTP transfer errors

[Willy Tarreau]

Hi guys,

I've read quite a bunch of this thread, and I think there's some
misunderstanding between both parts, as well as inappropriate
expectations in both cases.

On Fri, Apr 11, 2008 at 11:07:02AM -0700, David Miller wrote:
> We had Mark's bug fixed in 15 minutes once the bisect result was
> known, even after Ilpo and myself had scanned through the changesets.
> 
> This proves the utility of bisect and in fact that trying to intuit
> the cause by continuing to study changesets and code would have been a
> complete waste of time.
> 
> Yes, Mark, we used to do things that way for every bug in the kernel.
> And as a result many bugs sat unfixed for weeks if not months.  Many
> of us have left the cave, feel free to join us.

We should be very careful about git-bisect. First, it does not necessarily
point to the bug, but to the commit which exhibits the bug, so simply
reverting the commit might just hide the bug again. I want to ensure that
people do not forget that it does not replace a brain, it enhances your
eyes by pointing to a change related to the problem.

While it is a powerful tool, we must accept that it cannot efficiently
work in some circumstances, such as :

  - the machine cannot be rebooted often. I've been used to work for
    customers who plan changes once a week, and change absolutely
    nothing on their production if unplanned. This means one bisect
    step per week. Often, those people even require that your changes
    pass through a week of non-regression testing on a pre-production
    system (which was my case), with no overlapping between changes,
    so then you can count on one git-bisect iteration every two weeks.

  - the problem only happens in peak traffic hours on production, and
    the loss of service has already gone far beyond the annual quota.
    The only case they will accept an upgrade if you engage your full
    responsibility that it will definitely fix the problem. I've already
    been in such a situation, you say to the guy in front of you that
    you're putting your balls on the table, it will work (and sometimes
    you're only 90% confident). You obviously cannot do this to just
    check if the current bisect exhibits the problem or not.

  - the reporter has very few spare time. I do have friends in this
    situation. Basically, when your schedule is full of customers
    visits one month ahead, it's very hard to find several consecutive
    hours to track the problem down. Sometimes you're happy if you can
    spend two hours on it in a week. BTW, many developers are also in
    the same situation. Also most of the time, this must be done at
    the customer's and some of them do not accept people out of work
    hours. Then the problem may lay for weeks or months.

  - the problem is very reproducible but takes a lot of time before
    triggering (typically memory leaks).

In these situations, either git-bisect will not be usable, or will
take a lot of time to converge (up to several weeks), so will reveal
inefficient. So the reporter will either stay with the last known
working version, or with the new one accompanied with a workaround.

For this reason, we should not "force" reporters to git-bisect. Just
ask them if they can do so, otherwise investigations on their bug
will not progress until someone else reports the same one, with some
time to bisect it. And there is nothing wrong with that IMHO. If the
problem only affects one person and this person has a solution, is
that really much of a problem ? Sure it would be better fixed, but
nobody suffers from it. On the other hand, being aware that there
exists a person somewhere experiencing a specific bug is useful to
the developers, because when they think they might have fixed it,
they can ping him for validation.

Now, from a developer's point of view, the reporters should not
consider that development in free software is a public service and
that developers have a strong obligation to find and fix new bugs. 
Mark said his time is paid for, but most of the people here will
tend to take that as a customer-provider insult since their time
is also paid for, and while the reporter's work may consist in
consulting customers without much schedule freedom, the developer's
work consists in delivering new features in a more or less agreed
schedule. So everyone's time is valuable.

Of course it's better when developers help, and we must keep in mind
that they're the better placed to understand their code (even more
when it's recent). But due to the long chain of contributors, the
ones in direct contact with the reporter are not often the ones who
will be able to debug the code. So they need to know a bit more to
find whom to ping first.

Both Mark and Ilpo said something true here. It's that they feel
concerned when a bug is reported in an area they have worked on.
It is possible that none of the people who have worked on this bug
was responsible of it, and in this case it's important to insist
on the code author about the fact that he's not only a code author
but also has to support his code, and that next time he'll be
welcome to check if his code might have caused the reported problem.

But clearly, for scalability reasons, we cannot expect people in
the middle of the chain to investigate all bugs. Their experience
in the area is much better used at assisting both reporters and
code authors at taking the right direction though.

So if I can conclude, both reporter's and developer's time is
valuable and may not be spent on chasing every bug down. git-bisect
is very good at saving developer's time in exchange of approximately
the same amount of time on the reporter's side, which makes the whole
process scalable. Sometimes for various reasons the reporter cannot
do this (or not efficiently). We should not call him names in this
case, just tell him that we cannot go further on this bug without
much more information, and that he'll be asked for tests when someone
else reports it and debugs it. If the person expected more investigative
support, he should have gone with a commercialy supported distro.

Now speaking for my case, I know that as a developer, I'm faster than
many others to find bugs in *my* code, but am of little help when it
comes to external contributions to my code. As a user, I will not
always be able to git-bisect (or that would be inefficient, see reasons
above). But I know that a report is a report, and even if I have a
workaround, I feel it as a moral obligation to report the bug, and I
want to be able to do it without the fear of being agressed due to my
lack of involvement in the fix. An no Dave, I'm not hypocritic when I
say this. I really hate people who say "oh yes I know about this bug,
I've already encountered it but did not care to report it". I just
want to ensure that people will always report bugs, whatever the level
of help they will be able to provide.  It's important to know if a
problem happens for the first time or is very wide-spread since
version X or Y. And for such a case, I agree that bugzilla would at
least help not losing those reports.

Willy

Re: 2.6.25-rc8: FTP transfer errors

[David Miller]

From: Willy Tarreau <w@1wt.eu>
Date: Sat, 12 Apr 2008 10:44:00 +0200

> While it is a powerful tool, we must accept that it cannot efficiently
> work in some circumstances, such as :

Everyone is well aware of all of this, that's why I specifically asked
for a bisect, because I knew it would be crucial to pinpointing this
particular bug.

And lo' and behold in 15 minutes after the bisect results were
available it got fixed.

Yes it takes judgment, and nobody ever suggested that a revert
is the way to go.  Git bisect results must be parsed by a human
brain.  Nothing else was ever implied in any way shape or form.

Re: 2.6.25-rc8: FTP transfer errors

[Rafael J. Wysocki]

On Saturday, 12 of April 2008, David Miller wrote:
> From: Willy Tarreau <w@1wt.eu>
> Date: Sat, 12 Apr 2008 10:44:00 +0200
> 
> > While it is a powerful tool, we must accept that it cannot efficiently
> > work in some circumstances, such as :
> 
> Everyone is well aware of all of this, that's why I specifically asked
> for a bisect, because I knew it would be crucial to pinpointing this
> particular bug.
> 
> And lo' and behold in 15 minutes after the bisect results were
> available it got fixed.

I can't find the fix, btw.  Can you please point me to it?

Thanks,
Rafael

Re: 2.6.25-rc8: FTP transfer errors

[Sergio Luis]

On Sun, Apr 13, 2008 at 3:15 PM, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> On Saturday, 12 of April 2008, David Miller wrote:
>  > From: Willy Tarreau <w@1wt.eu>
>  > Date: Sat, 12 Apr 2008 10:44:00 +0200
>  >
>  > > While it is a powerful tool, we must accept that it cannot efficiently
>  > > work in some circumstances, such as :
>  >
>  > Everyone is well aware of all of this, that's why I specifically asked
>  > for a bisect, because I knew it would be crucial to pinpointing this
>  > particular bug.
>  >
>  > And lo' and behold in 15 minutes after the bisect results were
>  > available it got fixed.
>
>  I can't find the fix, btw.  Can you please point me to it?

there you go
http://lkml.org/lkml/2008/4/10/409
http://lkml.org/lkml/2008/4/10/410

>
>  Thanks,
>  Rafael

  -sergio

Re: 2.6.25-rc8: FTP transfer errors

[Rafael J. Wysocki]

On Sunday, 13 of April 2008, Sergio Luis wrote:
> On Sun, Apr 13, 2008 at 3:15 PM, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> > On Saturday, 12 of April 2008, David Miller wrote:
> >  > From: Willy Tarreau <w@1wt.eu>
> >  > Date: Sat, 12 Apr 2008 10:44:00 +0200
> >  >
> >  > > While it is a powerful tool, we must accept that it cannot efficiently
> >  > > work in some circumstances, such as :
> >  >
> >  > Everyone is well aware of all of this, that's why I specifically asked
> >  > for a bisect, because I knew it would be crucial to pinpointing this
> >  > particular bug.
> >  >
> >  > And lo' and behold in 15 minutes after the bisect results were
> >  > available it got fixed.
> >
> >  I can't find the fix, btw.  Can you please point me to it?
> 
> there you go
> http://lkml.org/lkml/2008/4/10/409
> http://lkml.org/lkml/2008/4/10/410

Thanks a lot,
Rafael

Re: 2.6.25-rc8: FTP transfer errors

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 401 bytes --]

On Fri, 11 Apr 2008 10:59:57 EDT, Mark Lord said:

> The one thing I did not have time for initially,
> was a painstaking blunt instrument binary search of
> every commit since v2.6.24.

The nice thing about binary search is that it's by definition an
O(log2(N)) operation, which isn't bad at all as far as algorithms go.

The truly blunt instrument here would be a linear search of every commit...



[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: 2.6.25-rc8: FTP transfer errors

[Tilman Schmidt]

[-- Attachment #1: Type: text/plain, Size: 725 bytes --]

On Fri, 11 Apr 2008 18:35:37 +0400, Evgeniy Polyakov wrote:
> On Fri, Apr 11, 2008 at 09:19:17AM -0400, Mark Lord (lkml@rtr.ca) wrote:

>> I offered tons of help, gave it, and said I didn't have time for a full
>> bisect at that juncture.
>>
>> For that, I get repeatedly slammed by the netdev folks.
>> Even after I put aside *paid* work to submit to your demands.
>>
>> Next time around, I won't bother reporting bugs to you folks,
>> that's for damned sure.
> 
> Actually that will be the best decision from evolutional point of view. 

So I was right after all? Bug reports from people who (for whatever
reason, including having to earn their living) cannot do a bisect are
not welcome?

:-(
T.




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

Re: 2.6.25-rc8: FTP transfer errors

[Evgeniy Polyakov]

On Sat, Apr 12, 2008 at 12:16:28AM +0200, Tilman Schmidt (tilman@imap.cc) wrote:
> > Actually that will be the best decision from evolutional point of view. 
> 
> So I was right after all? Bug reports from people who (for whatever
> reason, including having to earn their living) cannot do a bisect are
> not welcome?

You got it wrong.

If bug is subtle and developers can not reproduce it, there are only two
ways out of the problem: to help developers or not to help.

In the latter case bug report is useless (except that to show that it
exists, since practically no one can fix it until some new details
added).

In the former case there is a discussion between developers and
reporters, so things have progress. In this particular case there were
no healthy discussion, that is why all this is about.

Bisection was just an example of the help, reporter can provide. In this
case there were no other suggestions remotely useful or they were
already tried. If you can not proceed with what was suggested, then do
not piss anyone off because you were told to do something to help.

If you go to the doctor because of aching throat and he asks you to
open a mouth, you will not blame him for asking you to do that.

-- 
	Evgeniy Polyakov

Re: 2.6.25-rc8: FTP transfer errors

[David Miller]

From: Evgeniy Polyakov <johnpol@2ka.mipt.ru>
Date: Sat, 12 Apr 2008 02:25:36 +0400

> If you go to the doctor because of aching throat and he asks you to
> open a mouth, you will not blame him for asking you to do that.

ROFL

Re: 2.6.25-rc8: FTP transfer errors

[Tilman Schmidt]

[-- Attachment #1: Type: text/plain, Size: 3358 bytes --]

On Sat, 12 Apr 2008 02:25:36 +0400, Evgeniy Polyakov wrote:
> On Sat, Apr 12, 2008 at 12:16:28AM +0200, Tilman Schmidt (tilman@imap.cc) wrote:
>> So I was right after all? Bug reports from people who (for whatever
>> reason, including having to earn their living) cannot do a bisect are
>> not welcome?
> 
> You got it wrong.

Did I really? Let's see ...

> If bug is subtle and developers can not reproduce it, there are only two
> ways out of the problem: to help developers or not to help.
> 
> In the latter case bug report is useless (except that to show that it
> exists, since practically no one can fix it until some new details
> added).

Looks like you're saying I was right after all. Useless bug reports
shouldn't be submitted.

So please answer this simple question: If I know beforehand that I won't
have the time to do a bisect (or other similarly time-consuming task the
maintainers might ask from me), should I report the bug, or should I
keep my knowledge to myself?

This question is not theoretical. It's a situation I find myself in
quite regularly, because I allow myself the luxury of building most rc
kernels and even the odd mm kernel just for fun even though I have a
daytime job and a family to feed. It would be quite easy to look the
other way if I encounter a problem in one of those, hoping someone else
with more time on his or her hands will also come across it and report
it. So far my conscience told me not to do that. But if reporting it
without being able to follow up on it is considered useless then my
conscience was apparently wrong. Just say the word, and I'll stop what
I'm doing. I'll have no problem finding other things to do with my time.

> Bisection was just an example of the help, reporter can provide.

Sure. It's not about bisection specifically, but about the time a
reporter is able to invest in addition to what went into the report
already. But bisection is is a good example, because it's the most
time-consuming of all the tasks routinely asked from bug reporters.

> If you can not proceed with what was suggested, then do
> not piss anyone off because you were told to do something to help.

If a polite "sorry, I don't have the time" already counts as pissing
off, the only choice left is to avoid the situation in which I'd have to
say that. IOW, don't report bugs if I don't have the time to follow
through. Again: if that is what you want, I have no problem with it.

> If you go to the doctor because of aching throat and he asks you to
> open a mouth, you will not blame him for asking you to do that.

That analogy is wrong on so many accounts. It is not my throat that's
aching. A doctor would not insult me for not wanting to open my mouth
but rather ask if there was perhaps a valid reason for that. Not to
mention that opening my mouth takes substantially less time than a Linux
kernel bisection ...

A better analogy would be if I see an object lying on the highway, and I
stop at the next service area to call the police and alert them about
the possible danger. If they'd ask me to drive back to the place where I
saw it in order to describe precisely where it lay and what it looked
like, I think I might indeed become a bit upset.

HTH
T.

PS: I'll shut my big mouth now. The topic has been beaten to death.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

Re: 2.6.25-rc8: FTP transfer errors

[Evgeniy Polyakov]

On Sat, Apr 12, 2008 at 01:23:17AM +0200, Tilman Schmidt (tilman@imap.cc) wrote:
> > If you can not proceed with what was suggested, then do
> > not piss anyone off because you were told to do something to help.
> 
> If a polite "sorry, I don't have the time" already counts as pissing
> off, the only choice left is to avoid the situation in which I'd have to
> say that. IOW, don't report bugs if I don't have the time to follow
> through. Again: if that is what you want, I have no problem with it.

We want bug report, but we definitely do not want, when we ask for
additionl help, to listen crap that reporter does not have to help and
it is our roblem.

> > If you go to the doctor because of aching throat and he asks you to
> > open a mouth, you will not blame him for asking you to do that.
> 
> That analogy is wrong on so many accounts. It is not my throat that's
> aching. A doctor would not insult me for not wanting to open my mouth
> but rather ask if there was perhaps a valid reason for that. Not to
> mention that opening my mouth takes substantially less time than a Linux
> kernel bisection ...

It is your throat, since doctor's one is ok, and no one else came with
the same problems. Doctor will not insult you if you will not piss him
off. Read the thread from the beginning.

> A better analogy would be if I see an object lying on the highway, and I
> stop at the next service area to call the police and alert them about
> the possible danger. If they'd ask me to drive back to the place where I
> saw it in order to describe precisely where it lay and what it looked
> like, I think I might indeed become a bit upset.

Yeah, they first asked how it looked and where it was, then they asked
to move here, and you told them, that it is they who have to do that,
that it is exactly their problem, that you are not paid to do that.

Did I miss something, yeah, probably part, where you then tell, that you
care about highways, so you moved there and did what was asked.
Although, no, you first tell to police, that you spend your paid time
and will teach them how to do thing or similar crap. Only few hours
later, to some other people, you will tell that you care about highways,
global warming, Uganda childs and adron collider danger.

Ugh, just removing that object, when you were there 'takes
substantially less time than a Linux kernel bisection'.
Fortunately flooding developers with tons of urine during the whole day
is much more comfortable and ego-boosting, since it allows to close eyes
and do not see, that this took much more time than bisection.

Hope you got it right: we want bug reports and help. If you do not want
to provide some help, do not expect bug will be fixed, although bug
eistence is significant sign. So, be cool, and everything will be ok.

-- 
	Evgeniy Polyakov

Re: 2.6.25-rc8: FTP transfer errors

[Ilpo Järvinen]

On Sat, 12 Apr 2008, Tilman Schmidt wrote:

> On Sat, 12 Apr 2008 02:25:36 +0400, Evgeniy Polyakov wrote:
> > On Sat, Apr 12, 2008 at 12:16:28AM +0200, Tilman Schmidt (tilman@imap.cc) wrote:
> >> So I was right after all? Bug reports from people who (for whatever
> >> reason, including having to earn their living) cannot do a bisect are
> >> not welcome?
> > 
> > You got it wrong.
> 
> Did I really? Let's see ...
> 
> > If bug is subtle and developers can not reproduce it, there are only two
> > ways out of the problem: to help developers or not to help.
> > 
> > In the latter case bug report is useless (except that to show that it
> > exists, since practically no one can fix it until some new details
> > added).
> 
> Looks like you're saying I was right after all. Useless bug reports
> shouldn't be submitted.

...No, useless bug reports don't lead to a solution, ie., that particular 
bug won't get fixed as a result of the report! That's what these people 
are trying to say. Sure the point of bug reports is to get the bugs fixed,
don't you think? :-/ ...Or do you thing it's only secondary to get them 
fixed.

> So please answer this simple question: If I know beforehand that I won't
> have the time to do a bisect (or other similarly time-consuming task the
> maintainers might ask from me), should I report the bug, or should I
> keep my knowledge to myself?
> 
>
> > Bisection was just an example of the help, reporter can provide.
> 
> Sure. It's not about bisection specifically, but about the time a
> reporter is able to invest in addition to what went into the report
> already. But bisection is is a good example, because it's the most
> time-consuming of all the tasks routinely asked from bug reporters.

I'm asking the same thing from you as I did from Mark (it still remains 
unanswered)... What's your suggestion, how should we have solved this 
particular case? Do you join those that ask for developers to "invest" 
time to repeatedly go through the commits that are not guilty? ...One 
would never find the solution by that method :-/.

Yes, I'm fine that you don't want to help (or would want but cannot help 
like have been with many of nearly impossible to reproduce bugs with TCP 
lately) but the sole consequence is that the bug remains unsolved, it's 
plain simple. That's until somebody else is affected and reports and we 
get the necessary information. Or alternatively somebody just reads the 
offending code (possibly much later) and begins to wonder why there's this 
particular thing missing there (this is in fact not related to the bug 
reports at all, many bugs are found this way but it's not a thing one
can force to happen in a timely manner :-)).

> > If you can not proceed with what was suggested, then do
> > not piss anyone off because you were told to do something to help.
> 
> If a polite "sorry, I don't have the time" already counts as pissing
> off, the only choice left is to avoid the situation in which I'd have to
> say that. IOW, don't report bugs if I don't have the time to follow
> through. Again: if that is what you want, I have no problem with it.

Please reread the thread, this couldn't be farther from the truth...

...Dave had suggested Mark would have to bisect, I suppose this was after 
founding out that there wasn't anything particular that should cause this 
kind of behavior, or at least he couldn't find anything even suspicious 
looking. Mark, with rather demanding tone, was _also_ asking for that 
"somebody" who did all those TCP fin/closing changes (that would be me) to 
be responsible over them, ie., those parts that Dave had checked and found 
not suspicious (and bisect also proved them innocent later on). Yes, I 
then went through that "mountain of commits" which Mark "was not willing 
to do" himself. I invested the time even after Dave had also come to the 
same conclusion as I again did, that there is nothing wrong with the 
particular parts of the TCP. Funny, since it seems that even Mark himself 
had come to that same conclusion as both Dave and myself then came (though 
I cannot really speak for him). So I was third one to check those parts 
which were then found not guilty, would I be angry about it, I would call 
that plain waste of time :-). Please note too that I would have gone 
through them without his remarks as well to just check the thing I already 
knew.

So what's the problem with that. The thing that Mark wasn't very willing 
to go through that "mountain of commits" and made some accusions that one 
shouldn't ask user to do that, yet he was doing the same thing, asking for 
me to go through that "mountain of commits". I don't find that as polite
as you do (and maybe you don't honestly either), especially as that was 
_already done by Dave_, yet it wasn't enough for him.


[...doctor part snipped...]

> A better analogy would be if I see an object lying on the highway, and I
> stop at the next service area to call the police and alert them about
> the possible danger. If they'd ask me to drive back to the place where I
> saw it in order to describe precisely where it lay and what it looked
> like, I think I might indeed become a bit upset.

But you would feel qualified to tell how the police/doctor must handle 
it? ...That was the main problem.


To conclude I moved this your case down here...

> This question is not theoretical. It's a situation I find myself in
> quite regularly, because I allow myself the luxury of building most rc
> kernels and even the odd mm kernel just for fun even though I have a
> daytime job and a family to feed. It would be quite easy to look the
> other way if I encounter a problem in one of those, hoping someone else
> with more time on his or her hands will also come across it and report
> it. So far my conscience told me not to do that. But if reporting it
> without being able to follow up on it is considered useless then my
> conscience was apparently wrong. Just say the word, and I'll stop what
> I'm doing. I'll have no problem finding other things to do with my time.

It's perfectly fine that you report bugs, even with little time :-). But 
then if you are asked to do something that is necessary to help developers 
and you are not willing to do that, please don't start adding demands 
(it might actually be quite hard to restrain oneself from adding hidden 
"attacks" to wording, at least for me). Also we have to then accept the 
concequences, ie., the bug won't get fixed because of that report 
(unless something comes up later which connects the pieces).

> PS: I'll shut my big mouth now. The topic has been beaten to death.

I thought the same earlier, but I want to try to correct the misunderstand
you are tryng really hard to get here :-).


-- 
 i.

Re: 2.6.25-rc8: FTP transfer errors

[David Miller]

From: Tilman Schmidt <tilman@imap.cc>
Date: Sat, 12 Apr 2008 00:16:28 +0200

> So I was right after all? Bug reports from people who (for whatever
> reason, including having to earn their living) cannot do a bisect are
> not welcome?

You need to qualify this with: when a bisect is asked of them

You seem to be quite eager to harp on this specitic point, to make it
seem as if a bug report is useless if the person cannot or will not
bisect in all cases.  And that simply is not what we are saying here.

Re: 2.6.25-rc8: FTP transfer errors

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 334 bytes --]

On Thu, 10 Apr 2008 21:23:54 EDT, Mark Lord said:

> You still keep refering to it as "your (my) bug".
> It's not.  I had nothing to do with it, other than stumbling over it.

Like it or not, when you're the owner of the only box that can reliably
reproduce an error condition, it's your bug.

Been there, done that, plenty of times.

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: 2.6.25-rc8: FTP transfer errors

[Tilman Schmidt]

[-- Attachment #1: Type: text/plain, Size: 507 bytes --]

On Fri, 11 Apr 2008 15:58:42 -0400, Valdis.Kletnieks@vt.edu wrote:
> On Thu, 10 Apr 2008 21:23:54 EDT, Mark Lord said:
> 
>> You still keep refering to it as "your (my) bug".
>> It's not.  I had nothing to do with it, other than stumbling over it.
> 
> Like it or not, when you're the owner of the only box that can reliably
> reproduce an error condition, it's your bug.

Thanks for the advice. I'll keep it in mind next time I have to decide
whether to report a bug I'm stumbling over.

T.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

Re: 2.6.25-rc8: FTP transfer errors

[Tilman Schmidt]

[-- Attachment #1: Type: text/plain, Size: 1409 bytes --]

David Miller schrieb:
> From: Mark Lord <lkml@rtr.ca>
> Date: Thu, 10 Apr 2008 20:16:11 -0400
> 
>> Duh.. more like, "If I take 5-8 hours to attempt a bisect (which may not
>> even work), then that's 5-8 hours I do not get paid for."
> 
> And if I invest my spare time on your bug how does this statement
> apply to me?  Or does it only apply to you?
> 
> Every single argument you make that supports why you should not be
> investing the necessary time into the bug applies equally to the
> very developers you are so quickly to quip at and want help from.

I think you got it backwards. Mark and other bug reporters (including,
at times, yours truly) are helping you and other developers to make
Linux better. Most of the times I report a bug, I am not asking for help
- I have no personal need to get it fixed, as I can easily avoid it, and
I only report it to give developers like you a chance to fix it before
it really hurts someone - and I gather that Mark has been in a similar
position wrt to the bug in question.

So what would you have us do? Not report the bugs we find so that you
don't have to invest your spare time on "our" bugs? Report them and
accept a rebuke for our "unwillingness" to do even more benevolent work
than we already did? Report only those for which we really need a fix,
and are consequently willing to invest additional time?

Thanks,
Tilman


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

Re: 2.6.25-rc8: FTP transfer errors

[David Miller]

From: Tilman Schmidt <tilman@imap.cc>
Date: Fri, 11 Apr 2008 02:56:49 +0200

> I think you got it backwards. Mark and other bug reporters (including,
> at times, yours truly) are helping you and other developers to make
> Linux better.

I appreciate the bug reports, believe me.

The issue is which of the limited developer resources get put onto
which bugs.

A developer who does this for fun is going to prioritize to things
that are pleasant and interesting to work on, and also a good
effective use of their time.

So people prioritize.

Therefore, my point is, the net result is that user have a direct
influence on which bugs get worked on with the highest priority and
thus get fixed faster.  And those are the ones that have the most
information available, and in particular bisec results when
appropriate.

Re: 2.6.25-rc8: FTP transfer errors

[David Miller]

From: Mark Lord <lkml@rtr.ca>
Date: Thu, 10 Apr 2008 20:16:11 -0400

> [c67499c0e772064b37ad75eb69b28fc218752636 is first bad commit
> commit c67499c0e772064b37ad75eb69b28fc218752636
> Author: Pavel Emelyanov <xemul@openvz.org>
> Date:   Thu Jan 31 05:06:40 2008 -0800
> 
>     [NETNS]: Tcp-v4 sockets per-net lookup.
> 
>     Add a net argument to inet_lookup and propagate it further
>     into lookup calls. Plus tune the __inet_check_established.
> 
>     The dccp and inet_diag, which use that lookup functions
>     pass the init_net into them.
> 
>     Signed-off-by: Pavel Emelyanov <xemul@openvz.org>
>     Signed-off-by: David S. Miller <davem@davemloft.net>

Thanks Mark.

Pavel can you take a look?  I suspect that the namespace
changes or gets NULL'd out somehow and this leads to the
resets because the socket can no longer be found.  Perhaps
it's even a problem with time-wait socket namespace
propagation.

Re: 2.6.25-rc8: FTP transfer errors

[Mark Lord]

David Miller wrote:
> From: Mark Lord <lkml@rtr.ca>
> Date: Thu, 10 Apr 2008 20:16:11 -0400
> 
>> [c67499c0e772064b37ad75eb69b28fc218752636 is first bad commit
>> commit c67499c0e772064b37ad75eb69b28fc218752636
>> Author: Pavel Emelyanov <xemul@openvz.org>
>> Date:   Thu Jan 31 05:06:40 2008 -0800
>>
>>     [NETNS]: Tcp-v4 sockets per-net lookup.
>>
>>     Add a net argument to inet_lookup and propagate it further
>>     into lookup calls. Plus tune the __inet_check_established.
>>
>>     The dccp and inet_diag, which use that lookup functions
>>     pass the init_net into them.
>>
>>     Signed-off-by: Pavel Emelyanov <xemul@openvz.org>
>>     Signed-off-by: David S. Miller <davem@davemloft.net>
> 
> Thanks Mark.
> 
> Pavel can you take a look?  I suspect that the namespace
> changes or gets NULL'd out somehow and this leads to the
> resets because the socket can no longer be found.  Perhaps
> it's even a problem with time-wait socket namespace
> propagation.
..

My system here is now set up for quick/easy retest, if you have any
suggestions or patches to try out.

Thanks guys.

Re: 2.6.25-rc8: FTP transfer errors

[YOSHIFUJI Hideaki / 吉藤英明]

In article <47FEB0E3.8080507@rtr.ca> (at Thu, 10 Apr 2008 20:29:23 -0400), Mark Lord <lkml@rtr.ca> says:

> David Miller wrote:
> > From: Mark Lord <lkml@rtr.ca>
> > Date: Thu, 10 Apr 2008 20:16:11 -0400
> > 
> >> [c67499c0e772064b37ad75eb69b28fc218752636 is first bad commit
> >> commit c67499c0e772064b37ad75eb69b28fc218752636
> >> Author: Pavel Emelyanov <xemul@openvz.org>
> >> Date:   Thu Jan 31 05:06:40 2008 -0800
> >>
> >>     [NETNS]: Tcp-v4 sockets per-net lookup.
> >>
> >>     Add a net argument to inet_lookup and propagate it further
> >>     into lookup calls. Plus tune the __inet_check_established.
> >>
> >>     The dccp and inet_diag, which use that lookup functions
> >>     pass the init_net into them.
> >>
> >>     Signed-off-by: Pavel Emelyanov <xemul@openvz.org>
> >>     Signed-off-by: David S. Miller <davem@davemloft.net>
> > 
> > Thanks Mark.
> > 
> > Pavel can you take a look?  I suspect that the namespace
> > changes or gets NULL'd out somehow and this leads to the
> > resets because the socket can no longer be found.  Perhaps
> > it's even a problem with time-wait socket namespace
> > propagation.
> ..
> 
> My system here is now set up for quick/easy retest, if you have any
> suggestions or patches to try out.

Please try this, from net-2.6.26 tree.

Signed-off-by: YOSHIFUJI Hideaki <yoshfuji@linux-ipv6.org>

----
>From 8d9f1744cab50acb0c6c9553be533621e01f178b Mon Sep 17 00:00:00 2001
From: Daniel Lezcano <dlezcano@fr.ibm.com>
Date: Fri, 21 Mar 2008 04:12:54 -0700
Subject: [PATCH] [NETNS][IPV6] tcp - assign the netns for timewait sockets

Copy the network namespace from the socket to the timewait socket.

Signed-off-by: Daniel Lezcano <dlezcano@fr.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
---
 net/ipv4/inet_timewait_sock.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/net/ipv4/inet_timewait_sock.c b/net/ipv4/inet_timewait_sock.c
index 876169f..717c411 100644
--- a/net/ipv4/inet_timewait_sock.c
+++ b/net/ipv4/inet_timewait_sock.c
@@ -124,6 +124,7 @@ struct inet_timewait_sock *inet_twsk_alloc(const struct sock *sk, const int stat
 		tw->tw_hash	    = sk->sk_hash;
 		tw->tw_ipv6only	    = 0;
 		tw->tw_prot	    = sk->sk_prot_creator;
+		tw->tw_net          = sk->sk_net;
 		atomic_set(&tw->tw_refcnt, 1);
 		inet_twsk_dead_node_init(tw);
 		__module_get(tw->tw_prot->owner);
-- 
1.4.4.4

-- 
YOSHIFUJI Hideaki @ USAGI Project  <yoshfuji@linux-ipv6.org>
GPG-FP  : 9022 65EB 1ECF 3AD1 0BDF  80D8 4807 F894 E062 0EEA

Re: 2.6.25-rc8: FTP transfer errors

[Pavel Emelyanov]

YOSHIFUJI Hideaki / 吉藤英明 wrote:
> In article <47FEB0E3.8080507@rtr.ca> (at Thu, 10 Apr 2008 20:29:23 -0400), Mark Lord <lkml@rtr.ca> says:
> 
>> David Miller wrote:
>>> From: Mark Lord <lkml@rtr.ca>
>>> Date: Thu, 10 Apr 2008 20:16:11 -0400
>>>
>>>> [c67499c0e772064b37ad75eb69b28fc218752636 is first bad commit
>>>> commit c67499c0e772064b37ad75eb69b28fc218752636
>>>> Author: Pavel Emelyanov <xemul@openvz.org>
>>>> Date:   Thu Jan 31 05:06:40 2008 -0800
>>>>
>>>>     [NETNS]: Tcp-v4 sockets per-net lookup.
>>>>
>>>>     Add a net argument to inet_lookup and propagate it further
>>>>     into lookup calls. Plus tune the __inet_check_established.
>>>>
>>>>     The dccp and inet_diag, which use that lookup functions
>>>>     pass the init_net into them.
>>>>
>>>>     Signed-off-by: Pavel Emelyanov <xemul@openvz.org>
>>>>     Signed-off-by: David S. Miller <davem@davemloft.net>
>>> Thanks Mark.
>>>
>>> Pavel can you take a look?  I suspect that the namespace
>>> changes or gets NULL'd out somehow and this leads to the
>>> resets because the socket can no longer be found.  Perhaps
>>> it's even a problem with time-wait socket namespace
>>> propagation.
>> ..
>>
>> My system here is now set up for quick/easy retest, if you have any
>> suggestions or patches to try out.
> 
> Please try this, from net-2.6.26 tree.
> 
> Signed-off-by: YOSHIFUJI Hideaki <yoshfuji@linux-ipv6.org>

Too late, but still
Acked-by: Pavel Emelyanov <xemul@openvz.org>

Sorry, guys, but my timezone does not allow me to react in time
to found bugs :( So, when I wake up in the morning I usually just
find out that someone has caught a BUG made by me and someone 
else has fixed it already...

> ----
>>From 8d9f1744cab50acb0c6c9553be533621e01f178b Mon Sep 17 00:00:00 2001
> From: Daniel Lezcano <dlezcano@fr.ibm.com>
> Date: Fri, 21 Mar 2008 04:12:54 -0700
> Subject: [PATCH] [NETNS][IPV6] tcp - assign the netns for timewait sockets
> 
> Copy the network namespace from the socket to the timewait socket.
> 
> Signed-off-by: Daniel Lezcano <dlezcano@fr.ibm.com>
> Signed-off-by: David S. Miller <davem@davemloft.net>
> ---
>  net/ipv4/inet_timewait_sock.c |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
> 
> diff --git a/net/ipv4/inet_timewait_sock.c b/net/ipv4/inet_timewait_sock.c
> index 876169f..717c411 100644
> --- a/net/ipv4/inet_timewait_sock.c
> +++ b/net/ipv4/inet_timewait_sock.c
> @@ -124,6 +124,7 @@ struct inet_timewait_sock *inet_twsk_alloc(const struct sock *sk, const int stat
>  		tw->tw_hash	    = sk->sk_hash;
>  		tw->tw_ipv6only	    = 0;
>  		tw->tw_prot	    = sk->sk_prot_creator;
> +		tw->tw_net          = sk->sk_net;
>  		atomic_set(&tw->tw_refcnt, 1);
>  		inet_twsk_dead_node_init(tw);
>  		__module_get(tw->tw_prot->owner);

Re: 2.6.25-rc8: FTP transfer errors

[Ilpo Järvinen]

On Wed, 9 Apr 2008, Mark Lord wrote:

> David Miller wrote:
> > From: Mark Lord <lkml@rtr.ca>
> > Date: Wed, 09 Apr 2008 15:05:47 -0400
> > 
> > > But it would be far more useful for whoever has been working on the
> > > stack to suggest some possible/likely commits to look at instead.
> > 
> > Personally all I see is that one side closes the socket before all
> > data packets received have been read into the application, resulting
> > in a (correct) reset going out.
> > 
> > I can't think of any change we've made over the course of this
> > release that would change behvaior in that area.
> > 
> > So you will likely need to bisect.
> ..
> 
> Or I can ignore it, like the net developers, since I have a workaround.
> And then we'll see what other apps are broken upon 2.6.25 final release.
> 
> Really, folks.  Bug reports are intended to *help* the developers,
> not something to be thrown back in their faces.
> 
> There do seem to have been a *lot* of changes around the tcp closing/close
> code (as I see from diff'ing 2.6.24 against latest -git).

Sure, if you count in all whitespace/indentation/code moving changes to 
that as well... :-)

> *Somebody* is responsible for those changes.
> That particular *somebody* ought to volunteer some help here,

I might help if would add netdev on cc list in case you really want to 
reac net developers, otherwise they might just end up "ignoring it"... ;-)

> reducing the mountain of commits to a big handful or two.

Those touching fin/close are mostly whitespace/move things, so I doubt 
that you find these useful but in case you insist, here's the list:

056834d9f6f6eaf4cc7268569e53acab957aac27 [TCP]: cleanup tcp_{in,out}put.c style
058dc3342b71ffb3531c4f9df7c35f943f392b8d [TCP]: reduce tcp_output's indentation levels a bit
490d5046930276aae50dd16942649bfc626056f7 [TCP]: Uninline tcp_set_state

In addition, there's this one (...though I have read it number of times 
through and still cannot catch something that would cause the wrongness 
you're seeing):

e870a8efcddaaa3da7e180b6ae21239fb96aa2bb [TCP]: Perform setting of common 
control fields in one place

There's very little really on interesting side I can think of, mostly 
thinks are congestion control related changes... ...maybe either one of 
these could cause something unpleasant in some corner case:

bd515c3e48ececd774eb3128e81b669dbbd32637 [TCP]: Fix TSO deferring
0e3a4803aa06cd7bc2cfc1d04289df4f6027640a [TCP]: Force TSO splits to MSS boundaries

...e.g., if the latter causes a return with zero limit under some 
conditions, tso_fragment might generate, well, interesting packets and 
never finish if the condition persists but.


-- 
 i.

Re: 2.6.25-rc8: FTP transfer errors

[Mark Lord]

Ilpo Järvinen wrote:
> On Wed, 9 Apr 2008, Mark Lord wrote:
> 
>> David Miller wrote:
>>> From: Mark Lord <lkml@rtr.ca>
>>> Date: Wed, 09 Apr 2008 15:05:47 -0400
>>>
>>>> But it would be far more useful for whoever has been working on the
>>>> stack to suggest some possible/likely commits to look at instead.
>>> Personally all I see is that one side closes the socket before all
>>> data packets received have been read into the application, resulting
>>> in a (correct) reset going out.
>>>
>>> I can't think of any change we've made over the course of this
>>> release that would change behvaior in that area.
>>>
>>> So you will likely need to bisect.
>> ..
>>
>> Or I can ignore it, like the net developers, since I have a workaround.
>> And then we'll see what other apps are broken upon 2.6.25 final release.
>>
>> Really, folks.  Bug reports are intended to *help* the developers,
>> not something to be thrown back in their faces.
>>
>> There do seem to have been a *lot* of changes around the tcp closing/close
>> code (as I see from diff'ing 2.6.24 against latest -git).
..
> I might help if would add netdev on cc list in case you really want to 
> reac net developers, otherwise they might just end up "ignoring it"... ;-)
..

Oh.. I didn't know about that list.  How does that differ from linux-net ?
(Thanks)

> 
>> reducing the mountain of commits to a big handful or two.
> 
> Those touching fin/close are mostly whitespace/move things, so I doubt 
> that you find these useful but in case you insist, here's the list:
>
> 056834d9f6f6eaf4cc7268569e53acab957aac27 [TCP]: cleanup tcp_{in,out}put.c style
> 058dc3342b71ffb3531c4f9df7c35f943f392b8d [TCP]: reduce tcp_output's indentation levels a bit
> 490d5046930276aae50dd16942649bfc626056f7 [TCP]: Uninline tcp_set_state
> 
> In addition, there's this one (...though I have read it number of times 
> through and still cannot catch something that would cause the wrongness 
> you're seeing):
> 
> e870a8efcddaaa3da7e180b6ae21239fb96aa2bb [TCP]: Perform setting of common 
> control fields in one place
> 
> There's very little really on interesting side I can think of, mostly 
> thinks are congestion control related changes... ...maybe either one of 
> these could cause something unpleasant in some corner case:
> 
> bd515c3e48ececd774eb3128e81b669dbbd32637 [TCP]: Fix TSO deferring
> 0e3a4803aa06cd7bc2cfc1d04289df4f6027640a [TCP]: Force TSO splits to MSS boundaries
> 
> ...e.g., if the latter causes a return with zero limit under some 
> conditions, tso_fragment might generate, well, interesting packets and 
> never finish if the condition persists but.
..

That matches my own assessment there, too: lot's of whitespace changes,
and not much real code difference on most paths.  Bummer.  :)

-ml

Re: 2.6.25-rc8: FTP transfer errors

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2915 bytes --]

On Thu, 10 Apr 2008, Mark Lord wrote:

> Ilpo Järvinen wrote:
>
> > I might help if would add netdev on cc list in case you really want to reac
> > net developers, otherwise they might just end up "ignoring it"... ;-)
> ..
> 
> Oh.. I didn't know about that list.  How does that differ from linux-net ?
> (Thanks)

(Somebody please correct me if I'm wrong) if I've understood it correctly, 
linux-net is meant for users discussions while the developers hang at
netdev, some do read lkml but not all. And in fact, I wouldn't have 
noticed this thread for some time except that I was currently trying to 
see if there are some new tcp warn_on reports showing up.

...In case you have a regression, bug etc. related to networking, netdev 
should definately be included.

> > > reducing the mountain of commits to a big handful or two.
> > 
> > Those touching fin/close are mostly whitespace/move things, so I doubt that
> > you find these useful but in case you insist, here's the list:
> >
> > 056834d9f6f6eaf4cc7268569e53acab957aac27 [TCP]: cleanup tcp_{in,out}put.c
> > style
> > 058dc3342b71ffb3531c4f9df7c35f943f392b8d [TCP]: reduce tcp_output's
> > indentation levels a bit
> > 490d5046930276aae50dd16942649bfc626056f7 [TCP]: Uninline tcp_set_state
> > 
> > In addition, there's this one (...though I have read it number of times
> > through and still cannot catch something that would cause the wrongness
> > you're seeing):
> > 
> > e870a8efcddaaa3da7e180b6ae21239fb96aa2bb [TCP]: Perform setting of common
> > control fields in one place
> > 
> > There's very little really on interesting side I can think of, mostly thinks
> > are congestion control related changes... ...maybe either one of these could
> > cause something unpleasant in some corner case:
> > 
> > bd515c3e48ececd774eb3128e81b669dbbd32637 [TCP]: Fix TSO deferring
> > 0e3a4803aa06cd7bc2cfc1d04289df4f6027640a [TCP]: Force TSO splits to MSS
> > boundaries
> > 
> > ...e.g., if the latter causes a return with zero limit under some
> > conditions, tso_fragment might generate, well, interesting packets and never
> > finish if the condition persists but.
> ..
> That matches my own assessment there, too: lot's of whitespace changes,
> and not much real code difference on most paths.  Bummer.  :)

...I just got tired of seeing all those braindamaged line splits and
other "legacy" formatting over and over again... :-) Much of those things 
predate even 2.4, luckily there isn't yet an agency which would prevent 
changing lines of code with that long historic encumbrance :-).

That last TSO change seem the most potential one from the list of
all net/ipv4/tcp*.c include/net/tcp.h touching commits, trying
0e3a4803aa06cd7bc2cfc1d04289df4f6027640a^ might be worthwhile
(^ = a commit before the "quoted one") and you would be able to reuse
its result anyway if there's a need to bisect it because that commit is
around the halfway.


-- 
 i.