Re: Linux v2.6.16-rc6

Re: Linux v2.6.16-rc6

[Michal Piotrowski]

Hi,

On 12/03/06, Linus Torvalds <torvalds@osdl.org> wrote:
>
> Ok, we're getting closer, although the 2.6.16 release certainly seems to
> drag out more than it should have.
>

I have noticed this warnings
TCP: Treason uncloaked! Peer 82.113.55.2:11759/50967 shrinks window
148470938:148470943. Repaired.
TCP: Treason uncloaked! Peer 82.113.55.2:11759/50967 shrinks window
148470938:148470943. Repaired.
TCP: Treason uncloaked! Peer 82.113.55.2:11759/59768 shrinks window
1124211698:1124211703. Repaired.
TCP: Treason uncloaked! Peer 82.113.55.2:11759/59768 shrinks window
1124211698:1124211703. Repaired.

It maybe problem with ktorrent.

Here is config http://www.stardust.webpages.pl/files/linux/2.6.16-rc6/config
Here is dmesg http://www.stardust.webpages.pl/files/linux/2.6.16-rc6/dmesg

Regards,
Michal

--
Michal K. K. Piotrowski
LTG - Linux Testers Group
(http://www.stardust.webpages.pl/ltg/wiki/)

Re: Linux v2.6.16-rc6

[David S. Miller]

From: "Michal Piotrowski" <michal.k.k.piotrowski@gmail.com>
Date: Sun, 12 Mar 2006 02:51:40 +0100

> I have noticed this warnings
> TCP: Treason uncloaked! Peer 82.113.55.2:11759/50967 shrinks window
> 148470938:148470943. Repaired.
> TCP: Treason uncloaked! Peer 82.113.55.2:11759/50967 shrinks window
> 148470938:148470943. Repaired.
> TCP: Treason uncloaked! Peer 82.113.55.2:11759/59768 shrinks window
> 1124211698:1124211703. Repaired.
> TCP: Treason uncloaked! Peer 82.113.55.2:11759/59768 shrinks window
> 1124211698:1124211703. Repaired.
> 
> It maybe problem with ktorrent.

It is a problem with the remote TCP implementation, it is
illegally advertising a smaller window that it previously
did.

Re: Linux v2.6.16-rc6

[Willy Tarreau]

On Sat, Mar 11, 2006 at 06:39:04PM -0800, David S. Miller wrote:
> From: "Michal Piotrowski" <michal.k.k.piotrowski@gmail.com>
> Date: Sun, 12 Mar 2006 02:51:40 +0100
> 
> > I have noticed this warnings
> > TCP: Treason uncloaked! Peer 82.113.55.2:11759/50967 shrinks window
> > 148470938:148470943. Repaired.
> > TCP: Treason uncloaked! Peer 82.113.55.2:11759/50967 shrinks window
> > 148470938:148470943. Repaired.
> > TCP: Treason uncloaked! Peer 82.113.55.2:11759/59768 shrinks window
> > 1124211698:1124211703. Repaired.
> > TCP: Treason uncloaked! Peer 82.113.55.2:11759/59768 shrinks window
> > 1124211698:1124211703. Repaired.
> > 
> > It maybe problem with ktorrent.
> 
> It is a problem with the remote TCP implementation, it is
> illegally advertising a smaller window that it previously
> did.

on 2005/10/27, Herbert Xu provided a patch merged in 2.6.14 to fix some
erroneous occurences of this message (some of them appeared with Linux
on the other side). It would be interesting to know whether the peer
above is Linux or not, because it might be possible that Herbert's fix
needs to be applied to other places ?

Here comes his patch with his interesting analysis for reference, in
case it might give ideas to anybody.

Cheers,
Willy

---
From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Thu, 27 Oct 2005 08:47:46 +0000 (+1000)
Subject: [TCP]: Clear stale pred_flags when snd_wnd changes
X-Git-Tag: v2.6.14
X-Git-Url: http://kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=2ad41065d9fe518759b695fc2640cf9c07261dd2

[TCP]: Clear stale pred_flags when snd_wnd changes

This bug is responsible for causing the infamous "Treason uncloaked"
messages that's been popping up everywhere since the printk was added.
It has usually been blamed on foreign operating systems.  However,
some of those reports implicate Linux as both systems are running
Linux or the TCP connection is going across the loopback interface.

In fact, there really is a bug in the Linux TCP header prediction code
that's been there since at least 2.1.8.  This bug was tracked down with
help from Dale Blount.

The effect of this bug ranges from harmless "Treason uncloaked"
messages to hung/aborted TCP connections.  The details of the bug
and fix is as follows.

When snd_wnd is updated, we only update pred_flags if
tcp_fast_path_check succeeds.  When it fails (for example,
when our rcvbuf is used up), we will leave pred_flags with
an out-of-date snd_wnd value.

When the out-of-date pred_flags happens to match the next incoming
packet we will again hit the fast path and use the current snd_wnd
which will be wrong.

In the case of the treason messages, it just happens that the snd_wnd
cached in pred_flags is zero while tp->snd_wnd is non-zero.  Therefore
when a zero-window packet comes in we incorrectly conclude that the
window is non-zero.

In fact if the peer continues to send us zero-window pure ACKs we
will continue making the same mistake.  It's only when the peer
transmits a zero-window packet with data attached that we get a
chance to snap out of it.  This is what triggers the treason
message at the next retransmit timeout.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Arnaldo Carvalho de Melo <acme@mandriva.com>
---

--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -2239,6 +2239,7 @@ static int tcp_ack_update_window(struct 
 			/* Note, it is the only place, where
 			 * fast path is recovered for sending TCP.
 			 */
+			tp->pred_flags = 0;
 			tcp_fast_path_check(sk, tp);
 
 			if (nwin > tp->max_window) {


----

Re: Linux v2.6.16-rc6

[Michal Piotrowski]

Hi,

On 12/03/06, David S. Miller <davem@davemloft.net> wrote:
> From: "Michal Piotrowski" <michal.k.k.piotrowski@gmail.com>
> Date: Sun, 12 Mar 2006 02:51:40 +0100
>
> > I have noticed this warnings
> > TCP: Treason uncloaked! Peer 82.113.55.2:11759/50967 shrinks window
> > 148470938:148470943. Repaired.
> > TCP: Treason uncloaked! Peer 82.113.55.2:11759/50967 shrinks window
> > 148470938:148470943. Repaired.
> > TCP: Treason uncloaked! Peer 82.113.55.2:11759/59768 shrinks window
> > 1124211698:1124211703. Repaired.
> > TCP: Treason uncloaked! Peer 82.113.55.2:11759/59768 shrinks window
> > 1124211698:1124211703. Repaired.
> >
> > It maybe problem with ktorrent.
>
> It is a problem with the remote TCP implementation, it is
> illegally advertising a smaller window that it previously
> did.
>

Thanks for explanation.

Regards,
Michal

--
Michal K. K. Piotrowski
LTG - Linux Testers Group
(http://www.stardust.webpages.pl/ltg/wiki/)

Re: Linux v2.6.16-rc6

[Andy Furniss]

David S. Miller wrote:
> From: "Michal Piotrowski" <michal.k.k.piotrowski@gmail.com>
> Date: Sun, 12 Mar 2006 02:51:40 +0100
> 
> 
>>I have noticed this warnings
>>TCP: Treason uncloaked! Peer 82.113.55.2:11759/50967 shrinks window
>>148470938:148470943. Repaired.
>>TCP: Treason uncloaked! Peer 82.113.55.2:11759/50967 shrinks window
>>148470938:148470943. Repaired.
>>TCP: Treason uncloaked! Peer 82.113.55.2:11759/59768 shrinks window
>>1124211698:1124211703. Repaired.
>>TCP: Treason uncloaked! Peer 82.113.55.2:11759/59768 shrinks window
>>1124211698:1124211703. Repaired.
>>
>>It maybe problem with ktorrent.
> 
> 
> It is a problem with the remote TCP implementation, it is
> illegally advertising a smaller window that it previously
> did.
> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Packeteer manipulates window for shaping. I probably misread/read wrong 
RFC on this but I thought it didn't break any MUST NOTs.

I assume Linux + SFQ reordering packets during window growth would not 
trigger it.

Andy.

Re: Window shrinking (was Linux v2.6.16-rc6)

[Roberto Nibali]

>>>> Thanks Mark, I guess packeteer closes window down properly, I 
>>>> thought Dave's reply meant that doing that was Treason.
>>>
>>> Packeteer is almost certainly being cavalier about the way it reduces 
>>> windows.  It could be a serious problem, depending on the way it 
>>> treats traffic on the return path.  The "treason" thing is a joke.  
>>> It is like a bank extending you a credit line one day, and revoking 
>>> it the next.
>>
>> I don't use or know of anyone who uses Packeteer - or have you tested?

I had the distinct pleasure of partly get involved with debugging 
network stalls related to Linux clients (2.6.x kernel) and a Packeteer.

>> to mean that it was illegal to close down a window at all - you 
>> cleared that up - ie it is legal if you close it by <= the amount of 
>> data that has just been acked. I assume this won't cause the Treason 
>> messaage so don't really understand why it is cavalier - or do you 
>> just mean the whole idea of window manipulation to shape may be dodgey 
>> but legal?
> 
> I thought that Packeteer was causing the error messages.  If not then no 
> problem.  The "treason" messages will not occur if the window is reduced 
> normally.  The window is there for a reason - namely flow control.  A 
> zero rwnd means "I can't handle any more data right now".  That is 
> perfectly legal as long as previously granted transmit credit is not 
> withdrawn.  Generally speaking the rwnd always drops to zero when the 
> receive buffer is full.

Regarding Packeteer traffic shaper and Linux TCP stacks:

A customer of ours has had significant problems with the packeteer 
traffic shaper in the past. Unfortunately my consulting contract only 
lasted so long as to point out the problem with the shaper in 
conjunction with Linux clients, thus I cannot provide you with more 
detailed information.

The setup at the customer side (ISP) was like follows: they had 
installed a squid-proxy farm for their clients and used the Packeteer to 
do some sort of micro-billing, shaping and general QoS. The problem of 
window shrinking by the shaper affecting the client's performance 
manifested itself most of the time when their customers were accessing a 
site via that proxy-farm, which delivered its site-pictures using 
content caching services like Akamai (a really horrible example for 
testing squid's patience is, among others, http://www.pro-sieben.de). 
This caused quite a burst of quick TCP connection setups and teardowns, 
eventually resulting in a complete stall for 10-15s.

Disabling the Packeteer traffic shaper completely solved this issue and 
customers were not experiencing any stalls anymore when surfing the net. 
Updating the firmware of the shaper did help somewhat, so I suspect 
their TCP window handling is also error-prone to some extend. So I went 
on and blamed the Packeteer traffic shaper device. It was not until I 
tested the whole setup with their pilot squid-farm based on Solaris 
(SunOS 2.5.1) when I had to rethink my blaming, since routing their 
customers over the Solaris squid proxies did not exhibit the problem 
when enabling the traffic shaper for the same troublesome websites. So, 
this again showed an indication towards an issue with Linux clients and 
the Packeteer traffic shaper. The squid-farm is running on a SuSE 9.3 
based kernel. Due to performance constraints we had to go with the Linux 
solution and disable the traffic shaper.

As soon as I get some more consulting days and if the customer desires, 
I'll be debugging this issue in greater detail. I will send a debug 
report to netdev in this case. Chances are slim though, since they only 
have one Packeteer so far and no test network to perform test conducts, 
so erroneous tests would cause major downtime for a lot of this ISP's 
customers.

My 2 cents,
Roberto Nibali, ratz
-- 
echo 
'[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc

Re: Window shrinking (was Linux v2.6.16-rc6)

[Andy Furniss]

Roberto Nibali wrote:

> 
> I had the distinct pleasure of partly get involved with debugging 
> network stalls related to Linux clients (2.6.x kernel) and a Packeteer.

Dare I suggest that it could be something as trivial as it looks like 
window scaling defaults to off on SunOS 2.5.1 and it's on on Linux - 
maybe packeteer can't handle it properly. Would be an easy test to turn 
it off on the Suse boxes.

Andy.