Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Willy Tarreau]

Hi Dale,

I've tried your code right here.
The "attacker" was 10.0.3.1, and the victim 10.0.3.2.

I could successfully generate 1 CLOSE_WAIT on the victim with your program.
It was on port 23 and attached to inetd as fd #3. So I killed inetd, the
connection was then freed, and restarted it.

I changed the code slightly to be able to pass IP/ports as arguments.
On the victim, I straced inetd (pid 1013), and captured all TCP traffic
on port 23.

attacker> ./tcpnclose2 10.0.3.2 22 10.0.3.2 23

I stopped it when it was shouting at me :
socket failed.Connecting to 10.0.3.2:22 (FD: -1)...  FAILED: UNKNOWN ERROR.
socket failed.Connecting to 10.0.3.2:23 (FD: -1)...  FAILED: UNKNOWN ERROR.

Then, on the victim :

victim> sudo netstat -atnp|grep -v LISTEN
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp        1      0 10.0.3.2:23             10.0.3.1:34058          CLOSE_WAIT  1013/inetd          

victim> tcpdump -Svnr capture-victim.cap tcp port 34058
reading from file capture-victim.cap, link-type EN10MB (Ethernet)
19:05:10.360728 IP (tos 0x0, ttl  64, id 8168, offset 0, flags [DF], length: 48) 10.0.3.1.34058 > 10.0.3.2.23: S [tcp sum ok] 2882867180:2882867180(0) win 15920 <mss 7960,nop,nop,sackOK>
19:05:10.360764 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], length: 48) 10.0.3.2.23 > 10.0.3.1.34058: S [tcp sum ok] 2614211278:2614211278(0) ack 2882867181 win 5840 <mss 1460,nop,nop,sackOK>
19:05:10.360863 IP (tos 0x0, ttl  64, id 8169, offset 0, flags [DF], length: 40) 10.0.3.1.34058 > 10.0.3.2.23: . [tcp sum ok] ack 2614211279 win 15920
19:06:17.668670 IP (tos 0x0, ttl  64, id 8170, offset 0, flags [DF], length: 40) 10.0.3.1.34058 > 10.0.3.2.23: F [tcp sum ok] 2882867181:2882867181(0) ack 2614211279 win 15920
19:06:17.671102 IP (tos 0x0, ttl  64, id 11127, offset 0, flags [DF], length: 40) 10.0.3.2.23 > 10.0.3.1.34058: . [tcp sum ok] ack 2882867182 win 5840

==> We see that the victim (10.0.3.2) did not send the FIN in return.

Now let's take a closer look at inetd :

victim> cat /proc/net/tcp 
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode                                                     
  16: 0203000A:0017 0103000A:850A 08 00000000:00000001 00:00000000 00000000     0        0 6420 1 d5dac400 1500 20 0 2 -1                            

==> The socket (state 8 = CLOSE_WAIT) is bound to inode 6420.

victim> sudo ls -l /proc/1013/fd/|grep 6420
lrwx------    1 root     root           64 Sep 12 19:28 3 -> socket:[6420]

==> Again, it's FD #3.

I restarted strace on inetd, and noticed that fd#3 was not in the select fd
list anymore (remember one of the two cases I spoke about a few hours ago ?) :
victim> strace -p 1013
select(22, [4 5 6 7 8 9 11 12 13 14 15 16 17 18 19 20 21], NULL, NULL, NULL <unfinished ...>

Then, I took a look at the strace capture (184 MB !), to which I inserted line
numbers for better readability :

1:1013  accept(10, 0, NULL)               = 3
2:1013  fcntl64(10, F_SETFL, O_RDONLY)    = 0
3:1013  rt_sigprocmask(SIG_BLOCK, [HUP ALRM CHLD], NULL, 8) = 0
4:1013  fork()                            = 1108
5:1013  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
6:1013  close(3)                          = 0

This was the last but one connection assigned to fd #3. As you see, it's
finally closed. But a few lines later :

7:1013  select(22, [4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21], NULL, NULL, NULL) = 1 (in [10])
8:1013  fcntl64(10, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
9:1013  accept(10, 0, NULL)               = 3
10:1013  fcntl64(10, F_SETFL, O_RDONLY)    = 0
11:1013  rt_sigprocmask(SIG_BLOCK, [HUP ALRM CHLD], NULL, 8) = 0
12:1013  gettimeofday({1095008773, 685550}, NULL) = 0

The FD gets re-used, but is never scanned anymore, so never closed either :

35:1013  select(22, [4 5 6 7 8 9 11 12 13 14 15 16 17 18 19 20 21], NULL, NULL, NULL <unfinished ...>

Conclusion :
============

The problem is within inetd. In my case it could be because it was a bit
old (1999), but since you have it too, it might indicate an old bug. The
fact that it affects mysql too does not prove that the problem is in the
kernel, and I suspect that for whatever reason, there are some race
conditions in these two programs if the connection is either reused or
closed very quickly.

To demonstrate this, I've run your program against my reverse-proxy,
haproxy, which I fortunately happen to know better than these other
programs. I could not manage to get even a CLOSE_WAIT session after
several attempts.  All connections are closed normally, and as you'll
see with this extract from strace, the polled file-descriptors are
active once you kill the attacker :

(...)
close(593)                              = 0
select(684, [3 5 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617
618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636
637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655
656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674
675 676 677 678 679 680 681 682 683], NULL, NULL, {4, 835000}) = 81 (in [603
604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622
623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641
642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660
661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679
680 681 682 683], left {4, 836000})
gettimeofday({1095011266, 506966}, NULL) = 0
recv(603, "", 4096, 0x4000)             = 0
recv(604, "", 4096, 0x4000)             = 0
recv(605, "", 4096, 0x4000)             = 0
(...)
close(605)                              = 0
close(604)                              = 0
close(603)                              = 0
select(6, [3 5], NULL, NULL, NULL <unfinished ...>

So I believe you'll have to dig into some programs because at least you found
a vulnerability in both inetd and mysql :-)

Regards,
Willy

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Alan Cox]

On Sul, 2004-09-12 at 18:59, Willy Tarreau wrote:
> The problem is within inetd. In my case it could be because it was a bit
> old (1999), but since you have it too,

Ancient inetd had several fd leak bugs fixed over time and some other
problems with built in services. Not much of a suprise that a 1999 inetd
has it.

Alan

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Willy Tarreau]

Hi again, Dale,

I forgot to say that you don't need to fear releasing your exploit. I
developped its equivalent 4 years ago to stress-test web servers and
proxies, and if I launch it against victim:23, I get the exact same
result within seconds : a CLOSE_WAIT socket :

attacker> ./connectdata 10.0.3.2 23 200 1
ERROR: connect()=-1, nbconn=134 : Connection refused
ERROR: connect()=-1, nbconn=135 : Connection refused
ERROR: connect()=-1, nbconn=136 : Connection refused
ERROR: connect()=-1, nbconn=137 : Connection refused

The program connects 200 sockets to the same IP:port, and sends the begining
of an HTTP request.

victim> sudo netstat -atnp|grep -v LISTEN
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp       17      0 10.0.3.2:23             10.0.3.1:38214          CLOSE_WAIT  1333/inetd          

It's even not necessary to send data, then even faster to block my very old
inetd :

attacker> ./connectdata-nb 10.0.3.2 23 200
200 connections established.
Press any key so exit.

This time, it sends 200 non-blocking connect() calls without any data. It
takes a fraction of a second with the same result. Hopefully, it'll will
help Peter and you reproduce the problem faster on mysql.

Both programs have been freely available here for two years ; I didn't think
they would be useful again !

   http://w.ods.org/tools/connect/

Regards,
Willy

RE: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified)Denial of Service Attack

[Wolfpaw - Dale Corse]

Hi Alan,

 My inetd is not the old one, it is in Slackware 10.0..
apparently 1.79s, still works here. (the s added by
The slackware group according to the readme:

/*      $Slackware: inetd.c 1.79s 2001/02/06 13:18:00 volkerdi Exp $    */
/*      $OpenBSD: inetd.c,v 1.79 2001/01/30 08:30:57 deraadt Exp $      */ 
/*      $NetBSD: inetd.c,v 1.11 1996/02/22 11:14:41 mycroft Exp $       */
/* 

It still dies pretty fast :( Willy is probably right, in that
the bug is application level in this case. Can you explain
though, how it is appropriate to have no timeout on CLOSE_WAIT.
Lets assume (such as the case with the mysql bug), that it is
created, and for some reason never addressed.. The end case being
mysql reporting "Too many Connections". That would make me assume
these connections are still "alive" (perhaps I am assuming wrong?)
and could still cause issues with other services, if the volume of
them was to get too high.

It still seems prudent to me to have some kind of timeout (2 hours,
or something?) to have it expuldge the connection, because obviously,
it isn't going to voluntarily close.

This bug also exists with Apache, the default config of SSH,
and anything controlled by inetd. This is the vast majority of
popular services on a regular internet server.. That is bad, no?

D.

> -----Original Message-----
> From: Alan Cox [mailto:alan@lxorguk.ukuu.org.uk] 
> Sent: Sunday, September 12, 2004 11:17 AM
> To: Willy Tarreau
> Cc: Wolfpaw - Dale Corse; peter@mysql.com; Linux Kernel 
> Mailing List; netdev@oss.sgi.com
> Subject: Re: Linux 2.4.27 SECURITY BUG - TCP Local and 
> REMOTE(verified)Denial of Service Attack
> 
> 
> On Sul, 2004-09-12 at 18:59, Willy Tarreau wrote:
> > The problem is within inetd. In my case it could be because it was a
> > bit old (1999), but since you have it too,
> 
> Ancient inetd had several fd leak bugs fixed over time and 
> some other problems with built in services. Not much of a 
> suprise that a 1999 inetd has it.
> 
> Alan
> 
> 
> --------------------------------------------------------------
> --------------
> -
> This message has been scanned for Spam and Viruses by ClamAV 
> and SpamAssassin
> --------------------------------------------------------------
> --------------
> -
> 
> 
> 
> 
> 

RE: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified)Denial of Service Attack

[Alan Cox]

On Sul, 2004-09-12 at 19:40, Wolfpaw - Dale Corse wrote:
> This bug also exists with Apache, the default config of SSH,
> and anything controlled by inetd. This is the vast majority of
> popular services on a regular internet server.. That is bad, no?

I'm unable to duplicate any such problems with xinetd, or with thttpd,
or with apache. Apache will wait a short time then timeout connections
if you've configured it right. If you can continue making millions of
connections a second you can DoS the server the other end, not exactly
new news. The alternative is that you have an infinite number of running
services and you run out of memory instead.

Thats a high level property of any protocol which allows commitment of
resource without being able to do the security authentication first. Its
very hard to create ones that don't however, thus most devices in life
(eg your telephone) have this form or DoS attack.

My sshd also doesn't show this problem and the manual page indicates it
has a 120 second grace timeout for authentication.

The sshd manual page says:

     Gives the grace time for clients to authenticate themselves
             (default 120 seconds).

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified)Denial of Service Attack

[Willy Tarreau]

On Sun, Sep 12, 2004 at 12:40:56PM -0600, Wolfpaw - Dale Corse wrote:
> It still seems prudent to me to have some kind of timeout (2 hours,
> or something?) to have it expuldge the connection, because obviously,
> it isn't going to voluntarily close.

Dale,

forgive me for insisting, but it would be criminal to do this. You think
that it's sort of closed while it is not. Consider CLOSE_WAIT as ESTABLISHED.
It is not because one end sends a FIN (shutdown(WRITE)) that the connection
is about to end. The culprit really is the application. It is the application
which trusts the other end too much. The system lets the application work as
it expects it, and should never try to cut the grass below the application's
feet.

You must understand how it is supposed to work, basically like this :

        A                                      B

  shutdown(WRITE)
  A enters FIN_WAIT1
          ------- ACK + FIN  ---------->  select(in[fd],out[fd],) returns in[fd]
                                          B enters CLOSE_WAIT
          <------ ACK + data -----------  read(fd) returns 0
                                          shutdown(SHUT_RD)
                                          Then remove fd from read list.
          <---- ACK + data -------------  select(NULL, out[fd],...)
                  (...)                   write(fd, data)...
          <---- ACK + data -------------  select(NULL, out[fd],...)
          <---- ACK + FIN --------------  shutdown(SHUT_WRITE) or close(fd)
 close(fd)
          ------- ACK ----------------->

If the system closed anything, on B side, since the select() does not check
read events anymore, it would only be woken up for a write even, which could
crash the application in a SIG_PIPE if the writer does not check the error
condition on the socket before writing (like most applications do).

So, let me insist, your proposal is not the right solution to this. The
right solution is to carefully check every application and fix them, the
same way you would fix them to handle time-outs on ESTABLISHED connections.

> This bug also exists with Apache, the default config of SSH,
> and anything controlled by inetd. This is the vast majority of
> popular services on a regular internet server.. That is bad, no?

You didn't wait long enough for each of them. I bet that if you wait up
to 2 minutes, SSH will close, and if you wait 5 or 10 minutes, apache
will close too. As to mysql, I have no idea, and inetd, we obviously both
have a buggy version.

I hope this clarifies a bit the situation,
Willy