Re: send(), sendmsg(), sendto() not thread-safe

Re: send(), sendmsg(), sendto() not thread-safe

[David S. Miller]

From: Mark A Smith <mark1smi@us.ibm.com>
Date: Mon, 15 May 2006 14:39:06 -0700

> I discovered that in some cases, send(), sendmsg(), and sendto() are not
> thread-safe. Although the man page for these functions does not specify
> whether these functions are supposed to be thread-safe, my reading of the
> POSIX/SUSv3 specification tells me that they should be. I traced the
> problem to tcp_sendmsg(). I was very curious about this issue, so I wrote
> up a small page to describe in more detail my findings. You can find it at:
> http://www.almaden.ibm.com/cs/people/marksmith/sendmsg.html .

I don't understand why the desire is so high to ensure that
individual threads get "atomic" writes, you can't even ensure
that in the general case.

Only sloppy programs that don't do their own internal locking hit into
issues in this area.

>From your findings, the vast majority of systems you investigated do
not provide "atomic" thread safe write semantics over TCP sockets.
And frankly, BSD defines BSD socket semantics here not some wording in
the POSIX standards.

Finally, this discussion belongs on the networking development mailing
list, netdev@vger.kernel.org, not linux-kernel.

Re: send(), sendmsg(), sendto() not thread-safe

[Rick Jones]

David S. Miller wrote:
> From: Mark A Smith <mark1smi@us.ibm.com>
> Date: Mon, 15 May 2006 14:39:06 -0700
> 
> 
>>I discovered that in some cases, send(), sendmsg(), and sendto() are not
>>thread-safe. Although the man page for these functions does not specify
>>whether these functions are supposed to be thread-safe, my reading of the
>>POSIX/SUSv3 specification tells me that they should be. I traced the
>>problem to tcp_sendmsg(). I was very curious about this issue, so I wrote
>>up a small page to describe in more detail my findings. You can find it at:
>>http://www.almaden.ibm.com/cs/people/marksmith/sendmsg.html .


# ./sendmsgclient localhost
ERROR! We should have all 0! We don't!
buff[16384]=1
buff[16385]=1
buff[16386]=1
buff[16387]=1
buff[16388]=1
buff[16389]=1
buff[16390]=1
buff[16391]=1
buff[16392]=1
buff[16393]=1
That's 10/32768 bad bytes
# uname -a
HP-UX tarry B.11.23 U ia64 2397028692 unlimited-user license

Given that the URL above asserts that HP-UX claims atomicity, either 
there is a bug in the UX stack, or perhaps the test?  I took a quick 
look at the HP-UX 11iv2 (aka 11.23) manpage for sendmsg and didn't see 
anything about atomicity there - on which manpage(s) or docs was the 
assertion of HP-UX atomicity made?

I presume this is only for "blocking" sockets?  I cannot at least off 
the top of my head see how a stack could offer it on non-blocking sockets.

> And frankly, BSD defines BSD socket semantics here not some wording in
> the POSIX standards.

Have BSD socket semantics ever been updated/clarified any any 
quasi-official manner since the popular presence of threads?  Or 
are/were Posix/Xopen filling a gap?

rick jones

Re: send(), sendmsg(), sendto() not thread-safe

[Stephen Hemminger]

On Mon, 15 May 2006 16:17:48 -0700
Rick Jones <rick.jones2@hp.com> wrote:

> David S. Miller wrote:
> > From: Mark A Smith <mark1smi@us.ibm.com>
> > Date: Mon, 15 May 2006 14:39:06 -0700
> > 
> > 
> >>I discovered that in some cases, send(), sendmsg(), and sendto() are not
> >>thread-safe. Although the man page for these functions does not specify
> >>whether these functions are supposed to be thread-safe, my reading of the
> >>POSIX/SUSv3 specification tells me that they should be. I traced the
> >>problem to tcp_sendmsg(). I was very curious about this issue, so I wrote
> >>up a small page to describe in more detail my findings. You can find it at:
> >>http://www.almaden.ibm.com/cs/people/marksmith/sendmsg.html .
> 
> 
> # ./sendmsgclient localhost
> ERROR! We should have all 0! We don't!
> buff[16384]=1
> buff[16385]=1
> buff[16386]=1
> buff[16387]=1
> buff[16388]=1
> buff[16389]=1
> buff[16390]=1
> buff[16391]=1
> buff[16392]=1
> buff[16393]=1
> That's 10/32768 bad bytes
> # uname -a
> HP-UX tarry B.11.23 U ia64 2397028692 unlimited-user license
> 
> Given that the URL above asserts that HP-UX claims atomicity, either 
> there is a bug in the UX stack, or perhaps the test?  I took a quick 
> look at the HP-UX 11iv2 (aka 11.23) manpage for sendmsg and didn't see 
> anything about atomicity there - on which manpage(s) or docs was the 
> assertion of HP-UX atomicity made?
> 
> I presume this is only for "blocking" sockets?  I cannot at least off 
> the top of my head see how a stack could offer it on non-blocking sockets.

The test seems to be based on sending a big message. In this case,
on non-blocking sockets, the send call will return partial status. The
return from the system call will be less than the number of bytes requested.

> 
> > And frankly, BSD defines BSD socket semantics here not some wording in
> > the POSIX standards.
> 
> Have BSD socket semantics ever been updated/clarified any any 
> quasi-official manner since the popular presence of threads?  Or 
> are/were Posix/Xopen filling a gap?
> 
> rick jones
> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: send(), sendmsg(), sendto() not thread-safe

[Rick Jones]

>>I presume this is only for "blocking" sockets?  I cannot at least off 
>>the top of my head see how a stack could offer it on non-blocking sockets.
> 
> 
> The test seems to be based on sending a big message. In this case,
> on non-blocking sockets, the send call will return partial status. The
> return from the system call will be less than the number of bytes requested.

Right, and at that point it is already too late to do anything about 
keeping some other thread of execution from shoving _its_ bytes into the 
socket before the rest of the first sends' can be put there.

(Perhaps I'm preaching to the choir)

rick jones

Re: send(), sendmsg(), sendto() not thread-safe

[Rick Jones]

Mark A Smith wrote:
> Hi Rick, Stephen,
> 
> The thread-safe claim is at:
> 
> http://devrsrc1.external.hp.com/STKS/cgi-bin/man2html?manpage=/usr/share/man/man2.Z/send.2
> 
> Specifically,
> 
> "
> MULTITHREAD USAGE
>       The send(), sendmsg(), and sendto() system calls are thread-safe.
>       They each have a cancellation point; and they are async-cancel safe,
>       async-signal safe, and fork-safe.
> "

That looks to be the 11iv1 manpage (aka 11.11).    I wonder if perhaps 
there is a distinction made between "thread-safety" and an atomicity 
semantic?

Also, _strictly_ speaking, since your test is calling fork() rather than 
pthread_create(), it isn't really testing "thread safty" but multiple 
process atomicity right?

> I noticed that you were thinking that the problem may be with my test and
> that the send call is returning partial status. 

Either the test, or the stack.

> Actually, that's exactly
> the issue. On the systems I tested on, and I assume HP-UX, send is _not_
> returning partial status, it is returning that the entire buffer has been
> written, and yet is interleaving data from packets in the other thread.

Ostensibly, I should see some ten byte send messages in the output of 
sendmsgserver yes?  I just ran a test where it was all 32768's, no 10's 
but the client still reported an error.  Is that supposed to be possible?

# sendmsgserver > sendmsgserver.log
# wc sendmsgserver.log
165888 663552 3981312 sendmsgserver.log
# grep -v 32768 sendmsgserver.log
#

# ./sendmsgclient localhost
ERROR! We should have all 0! We don't!
buff[16384]=1
buff[16385]=1
buff[16386]=1
buff[16387]=1
buff[16388]=1
buff[16389]=1
buff[16390]=1
buff[16391]=1
buff[16392]=1
buff[16393]=1
That's 10/32768 bad bytes

I've also seen it fail at 12288 rather than 16384.  I wonder if perhaps 
there are unstated limits to the size of the write that can be atomic?

Looking at the 11.11 manpage for write(2) in the discussion of writes to 
a pipe or FIFO it says:

+  Write requests of {PIPE_BUF} bytes or less will not be
    interleaved with data from other processes doing writes on the
    same pipe.  Writes of greater than {PIPE_BUF} bytes may have
    data interleaved, on arbitrary boundaries, with writes by
    other processes, whether or not the O_NONBLOCK flag of the
    file status flags is set.

from limits.h:

#  define _POSIX_PIPE_BUF       512     /* The number of bytes that can
                                            be written atomically when
                                            writing to a pipe. */


later

#    define PIPE_BUF    8192    /* max number bytes that is guaranteed
                                    to be atomic when writing to a pipe */

Under various #ifdef checks and such.  It would not surprise me if there 
was a limit to the size of a buffer in a send/sendto/sendmsg call 
similar to that for write against a pipe.

I wonder if similar limits exist for the other stacks in the "yes" column.

rick jones

Re: send(), sendmsg(), sendto() not thread-safe

[Mark A Smith]

I cannot think of another possible definition for thread-safe in the
context of these functions. Certainly they should not cause a "crash" when
called from multiple threads, but that's a requirement independent of
thread-safety. The POSIX specification doesn't mention atomicity with
respect to these functions. In the context of read(), it defines atomicity
as follows: "Atomic means that all the bytes from a single operation that
started out together end up together, without interleaving from other I/O
operations." This seems to be the semantics I'm expecting from a
"thread-safe" sendmsg(). What else could thread-safe mean for send(),
sendmsg()?

As for fork() vs. pthreads, my original post indicated that this occurs
with or without pthreads, the same problem occurs if you use pthreads. I
tried to indicate that I was using the term "thread" to be independent of
whether the address space was shared or not.

If you do not see any "send() sent 10 bytes" messages in your server
output, it means that the problem occured the very first time the 10-byte
send was attempted. If you watch the data going over the wire, you'll see
those 10 bytes of 1's.

That write() document is excellent, and reads exactly like the POSIX
specification.

As for limits, I modified my test to use very small buffers, below any
reasonable limit. By removing the printf's, which slow the server
considerably, I can still get the problem to produce.

I think the question really boils down to: "What does thread-safe mean with
respect to send()?" It might be more easily answered by asking, "How would
a non-thread-safe send() behave?" I think it would behave the way we're
seeing it behave.

-Mark



That's excellent documentation, and reads exactly to the write()
specification in POSIX.



|---------+---------------------------->
|         |           Rick Jones       |
|         |           <rick.jones2@hp.c|
|         |           om>              |
|         |                            |
|         |           05/15/2006 05:43 |
|         |           PM               |
|---------+---------------------------->
  >-----------------------------------------------------------------------------------------------------------------|
  |                                                                                                                 |
  |       To:       Mark A Smith/Almaden/IBM@IBMUS                                                                  |
  |       cc:       shemminger@osdl.org, Linux Network Development list <netdev@vger.kernel.org>                    |
  |       Subject:  Re: send(), sendmsg(), sendto() not thread-safe                                                 |
  >-----------------------------------------------------------------------------------------------------------------|




Mark A Smith wrote:
> Hi Rick, Stephen,
>
> The thread-safe claim is at:
>
>
http://devrsrc1.external.hp.com/STKS/cgi-bin/man2html?manpage=/usr/share/man/man2.Z/send.2

>
> Specifically,
>
> "
> MULTITHREAD USAGE
>       The send(), sendmsg(), and sendto() system calls are thread-safe.
>       They each have a cancellation point; and they are async-cancel
safe,
>       async-signal safe, and fork-safe.
> "

That looks to be the 11iv1 manpage (aka 11.11).    I wonder if perhaps
there is a distinction made between "thread-safety" and an atomicity
semantic?

Also, _strictly_ speaking, since your test is calling fork() rather than
pthread_create(), it isn't really testing "thread safty" but multiple
process atomicity right?

> I noticed that you were thinking that the problem may be with my test and
> that the send call is returning partial status.

Either the test, or the stack.

> Actually, that's exactly
> the issue. On the systems I tested on, and I assume HP-UX, send is _not_
> returning partial status, it is returning that the entire buffer has been
> written, and yet is interleaving data from packets in the other thread.

Ostensibly, I should see some ten byte send messages in the output of
sendmsgserver yes?  I just ran a test where it was all 32768's, no 10's
but the client still reported an error.  Is that supposed to be possible?

# sendmsgserver > sendmsgserver.log
# wc sendmsgserver.log
165888 663552 3981312 sendmsgserver.log
# grep -v 32768 sendmsgserver.log
#

# ./sendmsgclient localhost
ERROR! We should have all 0! We don't!
buff[16384]=1
buff[16385]=1
buff[16386]=1
buff[16387]=1
buff[16388]=1
buff[16389]=1
buff[16390]=1
buff[16391]=1
buff[16392]=1
buff[16393]=1
That's 10/32768 bad bytes

I've also seen it fail at 12288 rather than 16384.  I wonder if perhaps
there are unstated limits to the size of the write that can be atomic?

Looking at the 11.11 manpage for write(2) in the discussion of writes to
a pipe or FIFO it says:

+  Write requests of {PIPE_BUF} bytes or less will not be
    interleaved with data from other processes doing writes on the
    same pipe.  Writes of greater than {PIPE_BUF} bytes may have
    data interleaved, on arbitrary boundaries, with writes by
    other processes, whether or not the O_NONBLOCK flag of the
    file status flags is set.

from limits.h:

#  define _POSIX_PIPE_BUF       512     /* The number of bytes that can
                                            be written atomically when
                                            writing to a pipe. */


later

#    define PIPE_BUF    8192    /* max number bytes that is guaranteed
                                    to be atomic when writing to a pipe */

Under various #ifdef checks and such.  It would not surprise me if there
was a limit to the size of a buffer in a send/sendto/sendmsg call
similar to that for write against a pipe.

I wonder if similar limits exist for the other stacks in the "yes" column.

rick jones

Re: send(), sendmsg(), sendto() not thread-safe

[Mike Stroyan]

On Mon, May 15, 2006 at 06:50:36PM -0700, Mark A Smith wrote:
> 
> I cannot think of another possible definition for thread-safe in the
> context of these functions. Certainly they should not cause a "crash" when
> called from multiple threads, but that's a requirement independent of
> thread-safety.
...
> I think the question really boils down to: "What does thread-safe mean with
> respect to send()?" It might be more easily answered by asking, "How would
> a non-thread-safe send() behave?" I think it would behave the way we're
> seeing it behave.

  A non-thread-safe send() could use a global buffer to hold copied
data or results.  When called from multiple threads it could lose data
or mix data from send()s to different sockets.  Or it could return a
result in one thread that reflects the result of an overlapping send()
in another thread.  Or it could sometimes core dump.  The behavior of
a non-thread-safe function called from multiple threads is undefined.
There is really no requirement "independent of thread-safety" on the
behavior of a function called from multiple unsyncronized threads.

  As Rick noted, this is not really a threading test but a multiprocess
test.  A requirement to be thread-safe shouldn't imply anything about
the behavior of functions in single-threaded processes.  The posix
standard has a limit of PIPE_BUF bytes for atomic writes to a pipe.
That might extend to expecting sends of under PIPE_BUF bytes to be atomic
on a socket.  But I don't actually find any guarantee of atomic writes
for sockets, even for sizes below PIPE_BUF.

-- 
Mike Stroyan, mike.stroyan@hp.com

Re: send(), sendmsg(), sendto() not thread-safe

[Benjamin Reed]

David et al, I think you may be missing the point. 

David S. Miller wrote:
> I don't understand why the desire is so high to
> ensure that individual threads get "atomic" writes,
> you can't even ensure that in the general case.

I think Mark's point isn't about "atomic" writes;
instead, he was pointing out that when a programmer
reads that a call is thread-safe he usually
understands that to mean that it does not need to be
protected by a lock when used by multiple threads.

With the current implementation of sendmsg it takes
some imagination to come up with scenario where it is
okay for concurrent sendmsg calls to mix their data.

> Only sloppy programs that don't do their own
internal
> locking hit into issues in this area.

You are looking at it backwards. If sendmsg were
indeed
thread-safe (two concurrent calls to sendmsg would not
mix their data), then it would be stupid for a
programmer to put a lock around the sendmsg call. It
would also be stupid for a programmer to protect a
call to socket() with a lock.

This thread-safe issue is an even bigger deal when you
are sharing the socket amoung many independent
processes since locking in that context is a bit
harder and more inefficient than in a pthread context.

So, if it isn't going to be "fixed". It would be nice
to at least document the issue. A cursory examination
of the sendmsg kernel code leads you to believe that
it is thread-safe in the sense that Mark is talking
about.

ben

Re: send(), sendmsg(), sendto() not thread-safe

[Christopher Friesen]

Benjamin Reed wrote:

> I think Mark's point isn't about "atomic" writes;
> instead, he was pointing out that when a programmer
> reads that a call is thread-safe he usually
> understands that to mean that it does not need to be
> protected by a lock when used by multiple threads.

<snip>

> This thread-safe issue is an even bigger deal when you
> are sharing the socket amoung many independent
> processes since locking in that context is a bit
> harder and more inefficient than in a pthread context.

There have always been possible issues with concurrent access to 
sockets/fds.

Consider what happens if you share a socket between processes/threads, 
you call select() in both of them, then do a blocking recv() in both to 
receive the message.  One of them gets the message, and the other blocks 
unexpectedly.

Or consider calling lseek() from one task, while doing successive read() 
calls in the other.

Or imagine one thread doing recvmsg() with MSG_PEEK, then calling 
recvmsg() to get the data, but meanwhile another thread has already read 
the message.

The calls themselves are thread-safe, but the side-effects of them may 
lead to unexpected behaviour unless the designer enforces appropriate 
synchronization.

Chris

Re: send(), sendmsg(), sendto() not thread-safe

[Benjamin Reed]

You are using the wrong examples, which may be why you
don't understand the problem Mark identified.

--- Christopher Friesen <cfriesen@nortel.com> wrote:
> There have always been possible issues with
> concurrent access to 
> sockets/fds.
> <snip>
> Or consider calling lseek() from one task, while
> doing successive read() 
> calls in the other.

I think all but the most clueless programmer would
know that lseek() and read() need to be in a critical
section. That is not the point. We are not talking
about thread-safe across calls. We are talking about
thread-safety for a given call.

In the case of lseek() and read(), you can use
pread(). pread() is cool because you don't need to
protect it with a critical section. Two threads can do
a pread() to different offsets on the same file
handle. If you protect the pread() with a critical
section, you are just wasting code, which is sloppy.
When reading the doc for sendmsg, it appears you can
avoid the critical section for sendmsg as well, but as
Mark points out, you need the critical section unless
you are running on Windows.

ben 

Re: send(), sendmsg(), sendto() not thread-safe

[Rick Jones]

Benjamin Reed wrote:
> In the case of lseek() and read(), you can use
> pread(). 

What is the meaning of the offset parameter of pread() for TCP or UDP etc?

rick jones

Re: send(), sendmsg(), sendto() not thread-safe

[Benjamin Reed]

These discussions degrade quickly.

I'm not proposing using pread() on a socket. I was
pointing out that you can do two concurrent pread()
invocations on the same file descriptor (for a file)
and get consistent results. It would be nice if you
can do two concurrent sendmsg() invocations on the
same socket descriptor and get consistent results.

pread() does not need to be in a critical section to
work properly. sendmsg() does in general. Usually, the
fact that it needs to be in a critical section would
suggest that it is not thread safe.

ben

--- Rick Jones <rick.jones2@hp.com> wrote:

> Benjamin Reed wrote:
> > In the case of lseek() and read(), you can use
> > pread(). 
> 
> What is the meaning of the offset parameter of
> pread() for TCP or UDP etc?
> 
> rick jones
>