Reported regression against commit a05d2ad

Reported regression against commit a05d2ad

[Herton Ronaldo Krzesinski]

Hi,

after update to one of the latest 2.6.32.x stable kernels for Ubuntu, we
got a regression report about timeout in tcp connections
(https://launchpad.net/bugs/791512).

We tried help reporter with a bisect process, but it was taking some
time, so we reverted some suspect commits, until we isolated it to
commit "af_unix: Only allow recv on connected seqpacket sockets."

With only commit a05d2ad reverted, testing results so far indicate the
issue doesn't happen.

I'm unfamiliar with unix sockets code, so can't see at first why this
commit in particular is causing problems, for now I can only say may be
something at application level using unix sockets regressed with it (?).
I'm just reporting it right now, and we plan to revert it for that kernel
until more info is found about it.

I'm adding reporter to CC (Lamont), in case more details are necessary
etc.

-- 
[]'s
Herton

Re: Reported regression against commit a05d2ad

[Tim Gardner]

On 06/21/2011 02:15 PM, Herton Ronaldo Krzesinski wrote:
> Hi,
>
> after update to one of the latest 2.6.32.x stable kernels for Ubuntu, we
> got a regression report about timeout in tcp connections
> (https://launchpad.net/bugs/791512).
>
> We tried help reporter with a bisect process, but it was taking some
> time, so we reverted some suspect commits, until we isolated it to
> commit "af_unix: Only allow recv on connected seqpacket sockets."
>
> With only commit a05d2ad reverted, testing results so far indicate the
> issue doesn't happen.
>
> I'm unfamiliar with unix sockets code, so can't see at first why this
> commit in particular is causing problems, for now I can only say may be
> something at application level using unix sockets regressed with it (?).
> I'm just reporting it right now, and we plan to revert it for that kernel
> until more info is found about it.
>
> I'm adding reporter to CC (Lamont), in case more details are necessary
> etc.
>

I believe we're also homing in on the same regression in 2.6.38.6 
('af_unix: Only allow recv on connected seqpacket sockets.'). The 
functional part of the patch is:

+
+       if (sk->sk_state != TCP_ESTABLISHED)
+               return -ENOTCONN;
+
+       return unix_dgram_recvmsg(iocb, sock, msg, size, flags);

What happens with out of order receives? Would fragmentation have an impact?

rtg
-- 
Tim Gardner tim.gardner@canonical.com

Re: Reported regression against commit a05d2ad

[Eric W. Biederman]

Tim Gardner <tim.gardner@canonical.com> writes:

> On 06/21/2011 02:15 PM, Herton Ronaldo Krzesinski wrote:
>> Hi,
>>
>> after update to one of the latest 2.6.32.x stable kernels for Ubuntu, we
>> got a regression report about timeout in tcp connections
>> (https://launchpad.net/bugs/791512).
>>
>> We tried help reporter with a bisect process, but it was taking some
>> time, so we reverted some suspect commits, until we isolated it to
>> commit "af_unix: Only allow recv on connected seqpacket sockets."
>>
>> With only commit a05d2ad reverted, testing results so far indicate the
>> issue doesn't happen.
>>
>> I'm unfamiliar with unix sockets code, so can't see at first why this
>> commit in particular is causing problems, for now I can only say may be
>> something at application level using unix sockets regressed with it (?).
>> I'm just reporting it right now, and we plan to revert it for that kernel
>> until more info is found about it.
>>
>> I'm adding reporter to CC (Lamont), in case more details are necessary
>> etc.
>>
>
> I believe we're also homing in on the same regression in 2.6.38.6 ('af_unix:
> Only allow recv on connected seqpacket sockets.'). The functional part of the
> patch is:
>
> +
> +       if (sk->sk_state != TCP_ESTABLISHED)
> +               return -ENOTCONN;
> +
> +       return unix_dgram_recvmsg(iocb, sock, msg, size, flags);
>
> What happens with out of order receives? Would fragmentation have an
> impact?

That code path has absolutely nothing to do with packets that come in
over the wire.  Zilch, zero, nada.  Fragments don't matter because
fragments can not possibly hit that code path.  IP packets can not
possibly hit that code path.

Eric

Re: Reported regression against commit a05d2ad

[Eric W. Biederman]

Herton Ronaldo Krzesinski <herton.krzesinski@canonical.com> writes:

> Hi,
>
> after update to one of the latest 2.6.32.x stable kernels for Ubuntu, we
> got a regression report about timeout in tcp connections
> (https://launchpad.net/bugs/791512).
>
> We tried help reporter with a bisect process, but it was taking some
> time, so we reverted some suspect commits, until we isolated it to
> commit "af_unix: Only allow recv on connected seqpacket sockets."
>
> With only commit a05d2ad reverted, testing results so far indicate the
> issue doesn't happen.
>
> I'm unfamiliar with unix sockets code, so can't see at first why this
> commit in particular is causing problems, for now I can only say may be
> something at application level using unix sockets regressed with it (?).
> I'm just reporting it right now, and we plan to revert it for that kernel
> until more info is found about it.

The only thing commit a05d2ad will prevent is a non-sense use of a
af_unix socket, and on recent enough kernels a NULL pointer deference.

I respectfully suggest that the bug is elsewhere perhaps a broken user
space application out there that needs to be fixed, or you have a kernel
memory stomp that removing patch a05d2ad happens to shift the memory
layout to be harmful in a different way.

af_unix sockets have nothing to do with tcp and only happen to use
the TCP_ESTABLISHED flag to indicated connected or non-connected
sockets.

Eric

> I'm adding reporter to CC (Lamont), in case more details are necessary
> etc.

Re: Reported regression against commit a05d2ad

[Tim Gardner]

On 06/21/2011 02:49 PM, Eric W. Biederman wrote:
<snip>
> I respectfully suggest that the bug is elsewhere perhaps a broken user
> space application out there that needs to be fixed, or you have a kernel
> memory stomp that removing patch a05d2ad happens to shift the memory
> layout to be harmful in a different way.
>

OK, I'm remembering how PF_UNIX Unix domain sockets are used, so I think 
your theory about a misbehaving user space application is more likely. 
However, I am a bit confused about how an application can attempt to 
receive before the socket is fully opened. Some kind of race condition 
with socketpair() ?

rtg
-- 
Tim Gardner tim.gardner@canonical.com

Re: Reported regression against commit a05d2ad

[Eric W. Biederman]

Tim Gardner <tim.gardner@canonical.com> writes:

> On 06/21/2011 02:49 PM, Eric W. Biederman wrote:
> <snip>
>> I respectfully suggest that the bug is elsewhere perhaps a broken user
>> space application out there that needs to be fixed, or you have a kernel
>> memory stomp that removing patch a05d2ad happens to shift the memory
>> layout to be harmful in a different way.
>>
>
> OK, I'm remembering how PF_UNIX Unix domain sockets are used, so I think your
> theory about a misbehaving user space application is more likely. However, I am
> a bit confused about how an application can attempt to receive before the socket
> is fully opened. Some kind of race condition with socketpair() ?

The case that is relevant is a listening SOCK_SEQPACKET socket.

The case that is affected is when you call receive on a listening
socket.

It isn't that the socket isn't fully opened.  It is that accept is the
only legitimate operation at that point.

It took a mistake while someone was developing an application for this
kernel bug to be found.

Eric