Re: Disabling "TCP Treason uncloaked"

Re: Disabling "TCP Treason uncloaked"

[Andi Kleen]

On Tuesday 02 May 2006 18:19, Just Marc wrote:

> I thought that maybe it's time to either set TCP_DEBUG to 0 or 
> alternatively allow an admin to toggle the printing of this message 
> off/on?  On a few busy web servers running usually latest versions of 
> 2.6 I have this message displaying hundreds (if not more) times a day, 

You're talking to a lot of broken TCP clients then.

> completely clogging the kernel log and making it harder to see if there 
> are more interesting messages being printed.   TCP_DEBUG is being used 
> in one more place that may also not require it anymore and or have it 
> always enabled.

It's better to use run time switches for these kind of things.

> So, is it time to disable this print altogether? Is it time to allow an 
> admin to toggle printing of this thing off?

If it's disabled there should be probably a netstat counter for the condition instead.

It might be a reasonable simple beginner project for someone:

- Convert all NETDEBUGs/TCP_DEBUG to a single netdebug sysctl
- Then perhaps make a CONFIG to change its default: 0 or 1
- Add statistics counters for all of these conditions

-Andi

Disabling "TCP Treason uncloaked"

[Just Marc]

Hi everyone,

Looking at mailing list archives there has been much talk of this print 
in the past, given the fact that this check/print is surrounded by an 
#ifdef TCP_DEBUG

#ifdef TCP_DEBUG
       if (net_ratelimit()) {
           struct inet_sock *inet = inet_sk(sk);
           printk(KERN_DEBUG "TCP: Treason uncloaked! Peer 
%u.%u.%u.%u:%u/%u shrinks window %u:%u. Repaired.\n",
                  NIPQUAD(inet->daddr), htons(inet->dport),
                  inet->num, tp->snd_una, tp->snd_nxt);
       }
#endif

I thought that maybe it's time to either set TCP_DEBUG to 0 or 
alternatively allow an admin to toggle the printing of this message 
off/on?  On a few busy web servers running usually latest versions of 
2.6 I have this message displaying hundreds (if not more) times a day, 
completely clogging the kernel log and making it harder to see if there 
are more interesting messages being printed.   TCP_DEBUG is being used 
in one more place that may also not require it anymore and or have it 
always enabled.

So, is it time to disable this print altogether? Is it time to allow an 
admin to toggle printing of this thing off?

Thanks for your time

Re: Disabling "TCP Treason uncloaked"

[Herbert Xu]

Just Marc <marc@corky.net> wrote:
> 
> Looking at mailing list archives there has been much talk of this print 
> in the past, given the fact that this check/print is surrounded by an 

A good number of these turned out to be a bug in the Linux TCP stack.
If this message wasn't there we would've never found it.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Disabling "TCP Treason uncloaked"

[Auke Kok]

Herbert Xu wrote:
> Just Marc <marc@corky.net> wrote:
>> Looking at mailing list archives there has been much talk of this print 
>> in the past, given the fact that this check/print is surrounded by an 
> 
> A good number of these turned out to be a bug in the Linux TCP stack.
> If this message wasn't there we would've never found it.

I am surprised that this never made it as a sysctl option that defaulted "on" 
in the first place really.

Auke

Re: Disabling "TCP Treason uncloaked"

[Just Marc]

Andi Kleen wrote:
> On Tuesday 02 May 2006 18:19, Just Marc wrote:
>
>   
>> I thought that maybe it's time to either set TCP_DEBUG to 0 or 
>> alternatively allow an admin to toggle the printing of this message 
>> off/on?  On a few busy web servers running usually latest versions of 
>> 2.6 I have this message displaying hundreds (if not more) times a day, 
>>     
>
> You're talking to a lot of broken TCP clients then.
>
>   
I don't think that's a surprise, with hundreds of thousands of visitors, 
thousands are probably broken.
>> completely clogging the kernel log and making it harder to see if there 
>> are more interesting messages being printed.   TCP_DEBUG is being used 
>> in one more place that may also not require it anymore and or have it 
>> always enabled.
>>     
>
> It's better to use run time switches for these kind of things.
>
>   
Agreed.
>> So, is it time to disable this print altogether? Is it time to allow an 
>> admin to toggle printing of this thing off?
>>     
>
> If it's disabled there should be probably a netstat counter for the condition instead.
>
>   
Yes, that would be better.
> It might be a reasonable simple beginner project for someone:
>
> - Convert all NETDEBUGs/TCP_DEBUG to a single netdebug sysctl
> - Then perhaps make a CONFIG to change its default: 0 or 1
> - Add statistics counters for all of these conditions
>   
It's indeed pretty simple, unfortunately I don't have the time to do 
it.  For myself, I'll just comment it out for now.   But clearly 
something needs to be done about this one.

Thanks

Re: Disabling "TCP Treason uncloaked"

[Just Marc]

Herbert Xu wrote:
> Just Marc <marc@corky.net> wrote:
>   
>> Looking at mailing list archives there has been much talk of this print 
>> in the past, given the fact that this check/print is surrounded by an 
>>     
>
> A good number of these turned out to be a bug in the Linux TCP stack.
> If this message wasn't there we would've never found it.
>   
Agreed, though this would be true for many other cases in the kernel 
should it be much more verbose in many other places.

We're now at 2.6.16.12 and it is still showing thousands of times a day 
on a busy web server, have all the bugs been discovered yet?

Marc

Re: Disabling "TCP Treason uncloaked"

[David S. Miller]

From: Andi Kleen <ak@suse.de>
Date: Tue, 2 May 2006 18:02:43 +0200

> On Tuesday 02 May 2006 18:19, Just Marc wrote:
> 
> > I thought that maybe it's time to either set TCP_DEBUG to 0 or 
> > alternatively allow an admin to toggle the printing of this message 
> > off/on?  On a few busy web servers running usually latest versions of 
> > 2.6 I have this message displaying hundreds (if not more) times a day, 
> 
> You're talking to a lot of broken TCP clients then.

Herbert Xu also fixed a bug that would cause that message
to be emitted erroneously, 9 times out of 10 that is why
people are seeing these messages.

I think disabling that message is a non-starter, we want to
see the message because as we've seen it can point to bugs
on our end too.

Re: Disabling "TCP Treason uncloaked"

[Andi Kleen]

On Wednesday 03 May 2006 08:32, David S. Miller wrote:
> From: Andi Kleen <ak@suse.de>
> Date: Tue, 2 May 2006 18:02:43 +0200
> 
> > On Tuesday 02 May 2006 18:19, Just Marc wrote:
> > 
> > > I thought that maybe it's time to either set TCP_DEBUG to 0 or 
> > > alternatively allow an admin to toggle the printing of this message 
> > > off/on?  On a few busy web servers running usually latest versions of 
> > > 2.6 I have this message displaying hundreds (if not more) times a day, 
> > 
> > You're talking to a lot of broken TCP clients then.
> 
> Herbert Xu also fixed a bug that would cause that message
> to be emitted erroneously, 9 times out of 10 that is why
> people are seeing these messages.

Still if Marc has hundreds per day that would be still tens
per day - and someone with even more traffic than Marc
would have hundreds per day again.

> I think disabling that message is a non-starter, we want to
> see the message because as we've seen it can point to bugs
> on our end too.

Just having the counter would help with that already, no?
Also the default for the sysctl could be maybe switched in -rc* and final? 

-Andi

Re: Disabling "TCP Treason uncloaked"

[Herbert Xu]

Just Marc <marc@corky.net> wrote:
> 
> We're now at 2.6.16.12 and it is still showing thousands of times a day 
> on a busy web server, have all the bugs been discovered yet?

There are no known bugs on the RX side in 2.6.16 that would cause this.
The few busy web servers that I have access to do not see this message
at all.

So either your web server is somehow attracting a lot of buggy TCP stacks,
or there could be a dodgy gateway in your path that is mucking with the
TCP windows.

I suggest that you do a tcpdump to see what it looks like.  It might turn
out that we do still have a bug on our RX side.

BTW, this message is already under net_ratelimit so I don't see any
urgency in getting rid of it completely.  If we're going down the
path of disabling it, we probably should go for something more global
rather than a sysctl that controls this one message.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Disabling "TCP Treason uncloaked"

[Just Marc]

> From: Andi Kleen <ak@suse.de>
> Date: Tue, 2 May 2006 18:02:43 +0200
>
>   
>> On Tuesday 02 May 2006 18:19, Just Marc wrote:
>>
>>     
>>> I thought that maybe it's time to either set TCP_DEBUG to 0 or 
>>> alternatively allow an admin to toggle the printing of this message 
>>> off/on?  On a few busy web servers running usually latest versions of 
>>> 2.6 I have this message displaying hundreds (if not more) times a day, 
>>>       
>> You're talking to a lot of broken TCP clients then.
>>     
>
> Herbert Xu also fixed a bug that would cause that message
> to be emitted erroneously, 9 times out of 10 that is why
> people are seeing these messages.
>
> I think disabling that message is a non-starter, we want to
> see the message because as we've seen it can point to bugs
> on our end too.
>
>   
Correct, I noticed this was applied to 2.6.14, however, I am running 
2.6.16 (and now 2.6.16.12) on all of the machines in question, I am 
seeing hundreds of these messages a day...

Thanks

Re: Disabling "TCP Treason uncloaked"

[Ingo Oeser]

Herbert Xu wrote:
> BTW, this message is already under net_ratelimit so I don't see any
> urgency in getting rid of it completely.  If we're going down the
> path of disabling it, we probably should go for something more global
> rather than a sysctl that controls this one message.

Already there: /proc/sys/net/core/{message_cost,message_burst}

Just set burst to 0 and cost to a very big value to basically supppress
all net_ratelimit()ed messages.

Or did you think of sth. else?

Regards

Ingo Oeser

Re: Disabling "TCP Treason uncloaked"

[Herbert Xu]

On Wed, May 03, 2006 at 01:49:05PM +0100, Just Marc wrote:
>
> You're right, actually this box serves http/ftp file transfers only, 
> it's a mirror with a large amount of downloads a day.

That's interesting.  The RX bug that I fixed earlier would usually
manifest itself under exactly these conditions.  Alas you're running
a fixed kernel.

> I gave it another look, many of the prints are repeat prints for the 
> same remote IPs,

Can you take a tcpdump of the TCP sessions involving those IPs and
then show me the sections that occur when those messages are triggered?

> I agree, however, I think it's pretty good to allow an admin to toggle 
> it off completely.  Maybe that's all we need, really.

The question is always what you want to be off and what to leave on.
After all, you can turn off all printk's to the console through /proc.
If you also tell klogd to ignore them then you won't see anything at
all (unless you run dmesg).

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Disabling "TCP Treason uncloaked"

[Herbert Xu]

Just Marc <marc@corky.net> wrote:
>   
> That's good, but it may (and probably will) suppress many other messages 
> which may be of more interest...

That's the crux of the matter.  There is no way we can satisfy everyone
short of putting a knob on each individual printk.

So the only solution is to do the filtering yourself in userspace through
klogd/syslogd.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Disabling "TCP Treason uncloaked"

[Alexey Toptygin]

On Wed, 3 May 2006, Herbert Xu wrote:

> Can you take a tcpdump of the TCP sessions involving those IPs and
> then show me the sections that occur when those messages are triggered?

I'm curious, how would you do this without filling the disk? With a script 
that starts tcpdump to a ring in the background, waits for the offending 
log entry to appear and then kills tcpdump?

 			Alexey

Re: Disabling "TCP Treason uncloaked"

[Herbert Xu]

On Wed, May 03, 2006 at 01:47:31PM +0200, Ingo Oeser wrote:
> 
> Already there: /proc/sys/net/core/{message_cost,message_burst}
> 
> Just set burst to 0 and cost to a very big value to basically supppress
> all net_ratelimit()ed messages.
> 
> Or did you think of sth. else?

No that'll do just fine.

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Disabling "TCP Treason uncloaked"

[Just Marc]

Hi,
> Just Marc <marc@corky.net> wrote:
>   
>> We're now at 2.6.16.12 and it is still showing thousands of times a day 
>> on a busy web server, have all the bugs been discovered yet?
>>     
>
> There are no known bugs on the RX side in 2.6.16 that would cause this.
> The few busy web servers that I have access to do not see this message
> at all.
>
> So either your web server is somehow attracting a lot of buggy TCP stacks,
> or there could be a dodgy gateway in your path that is mucking with the
> TCP windows.
>
>   
You're right, actually this box serves http/ftp file transfers only, 
it's a mirror with a large amount of downloads a day.

On a pure html/image web server that is just as busy, there are only a 
few of these messages, one or two a day.
> I suggest that you do a tcpdump to see what it looks like.  It might turn
> out that we do still have a bug on our RX side.
>   
I gave it another look, many of the prints are repeat prints for the 
same remote IPs,
> BTW, this message is already under net_ratelimit so I don't see any
> urgency in getting rid of it completely.  If we're going down the
> path of disabling it, we probably should go for something more global
> rather than a sysctl that controls this one message.
>   
I agree, however, I think it's pretty good to allow an admin to toggle 
it off completely.  Maybe that's all we need, really.

Thanks

Re: Disabling "TCP Treason uncloaked"

[Just Marc]

Hi
> Herbert Xu wrote:
>   
>> BTW, this message is already under net_ratelimit so I don't see any
>> urgency in getting rid of it completely.  If we're going down the
>> path of disabling it, we probably should go for something more global
>> rather than a sysctl that controls this one message.
>>     
>
> Already there: /proc/sys/net/core/{message_cost,message_burst}
>
> Just set burst to 0 and cost to a very big value to basically supppress
> all net_ratelimit()ed messages.
>
> Or did you think of sth. else?
>   
That's good, but it may (and probably will) suppress many other messages 
which may be of more interest...

Marc

Re: Disabling "TCP Treason uncloaked"

[Just Marc]

Hi
> On Wed, May 03, 2006 at 01:49:05PM +0100, Just Marc wrote:
>   
>> You're right, actually this box serves http/ftp file transfers only, 
>> it's a mirror with a large amount of downloads a day.
>>     
>
> That's interesting.  The RX bug that I fixed earlier would usually
> manifest itself under exactly these conditions.  Alas you're running
> a fixed kernel.
>
>   
>> I gave it another look, many of the prints are repeat prints for the 
>> same remote IPs,
>>     
>
> Can you take a tcpdump of the TCP sessions involving those IPs and
> then show me the sections that occur when those messages are triggered?
>
>   
OK, I'll try that.
>> I agree, however, I think it's pretty good to allow an admin to toggle 
>> it off completely.  Maybe that's all we need, really.
>>     
>
> The question is always what you want to be off and what to leave on.
> After all, you can turn off all printk's to the console through /proc.
> If you also tell klogd to ignore them then you won't see anything at
> all (unless you run dmesg).
>   
The problem is that it's really BAD to suppress all messages, there may 
be critical ones.  In my opinion while the treason uncloaked message can 
help uncover bugs it is mostly redundant on a day to day usage for 
probably 99% of the scenarios.

Marc

Re: Disabling "TCP Treason uncloaked"

[Herbert Xu]

On Wed, May 03, 2006 at 12:15:30PM +0000, Alexey Toptygin wrote:
> 
> I'm curious, how would you do this without filling the disk? With a script 
> that starts tcpdump to a ring in the background, waits for the offending 
> log entry to appear and then kills tcpdump?

Well if you know the set of IPs that's likely to cause this you just
run tcpdump with an expression to filter out all traffic but those
IPs.

Sinec tcpdump only captures 100 bytes of each packet by default, it
should be manageable assuming that this problem occurs relatively
frequently.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Disabling "TCP Treason uncloaked"

[Just Marc]

Hi
> Just Marc <marc@corky.net> wrote:
>   
>>   
>> That's good, but it may (and probably will) suppress many other messages 
>> which may be of more interest...
>>     
>
> That's the crux of the matter.  There is no way we can satisfy everyone
> short of putting a knob on each individual printk.
>
> So the only solution is to do the filtering yourself in userspace through
> klogd/syslogd.
>   
It is the crux of the matter. This print is a purely informational one, 
absolutely harmless (unless I'm mistaken) and clogs kernel  logs 
throughout the universe :)

Seriously, other kernel prints are usually not so harmless and are 
generally more important to see.  Let me  put it another way, except 
this print, I have no other prints whatsoever on any of my machines and 
if there are prints, I usually do want to know about them, I learned to 
ignore this print over the years.   I believe many people are in the 
same boat.

That's my opinion, anyway.

Thanks