TCP/IPv4 sending using MSG_ZEROCOPY and closing the socket

TCP/IPv4 sending using MSG_ZEROCOPY and closing the socket

[mathias_koehrer]

Hi all,

I have one question on the behavior of TCP/IPv4 sending using the MSG_ZEROCOPY flag,
the kernel version is 4.19.18.

What happens if I close the sending socket immediately after performing a socket 
send() or sendmsg() call (called with the MSG_ZEROCOPY flag)?
I.e. in this situation not all messages have been sent yet, however - as the 
socket is closed - it is no longer possible to retrieve the completion 
notification via the error channel.

Is it fine for the user space program to free all outstanding messages after the
socket close() has returned?
Or is there anything else that has to be considered?

Thanks for any help

Best regards

Mathias

Re: TCP/IPv4 sending using MSG_ZEROCOPY and closing the socket

[Willem de Bruijn]

On Mon, Jan 28, 2019 at 2:57 AM <mathias_koehrer@arcor.de> wrote:
>
> Hi all,
>
> I have one question on the behavior of TCP/IPv4 sending using the MSG_ZEROCOPY flag,
> the kernel version is 4.19.18.
>
> What happens if I close the sending socket immediately after performing a socket
> send() or sendmsg() call (called with the MSG_ZEROCOPY flag)?
> I.e. in this situation not all messages have been sent yet, however - as the
> socket is closed - it is no longer possible to retrieve the completion
> notification via the error channel.
>
> Is it fine for the user space program to free all outstanding messages after the
> socket close() has returned?
> Or is there anything else that has to be considered?

If closing the socket while user memory is still in transmission, it
will not be possible to safely reuse the memory, as the process has no
way of discovering when the kernel has finished transmission.

Depending on type of memory, there may be workarounds to avoid
unbound virtual memory growth, such as unmapping the virtual
address range in the case of mmap()ed data.

But in general, the right approach is to wait for all completions
before closing a socket.

If this takes a long time, say due to the TCP stack hold on to data for
retransmission in the case a peer does not properly close its side,
disconnect (connect() AF_UNSPEC) can be used to purge the
queues and trigger notifications. Again, this is a last resort and
usually not needed.

Re: TCP/IPv4 sending using MSG_ZEROCOPY and closing the socket

[mathias_koehrer]

>> Hi all,
>> 
>> I have one question on the behavior of TCP/IPv4 sending using the MSG_ZEROCOPY flag,
>> the kernel version is 4.19.18.
>> 
>> What happens if I close the sending socket immediately after performing a socket
>> send() or sendmsg() call (called with the MSG_ZEROCOPY flag)?
>> I.e. in this situation not all messages have been sent yet, however - as the
>> socket is closed - it is no longer possible to retrieve the completion
>> notification via the error channel.
>> 
>> Is it fine for the user space program to free all outstanding messages after the
>> socket close() has returned?
>> Or is there anything else that has to be considered?
> 
> If closing the socket while user memory is still in transmission, it
> will not be possible to safely reuse the memory, as the process has no
> way of discovering when the kernel has finished transmission.
> 
> Depending on type of memory, there may be workarounds to avoid
> unbound virtual memory growth, such as unmapping the virtual
> address range in the case of mmap()ed data.
> 
> But in general, the right approach is to wait for all completions
> before closing a socket.
> 
> If this takes a long time, say due to the TCP stack hold on to data for
> retransmission in the case a peer does not properly close its side,
> disconnect (connect() AF_UNSPEC) can be used to purge the
> queues and trigger notifications. Again, this is a last resort and
> usually not needed.
Hi Willem!
Thank you for the clarification.
Do you think you could add this to the documentation of MSG_ZEROCOPY?
I think this could be really helpful for all users of the zerocopy feature.

Thanks a lot

Mathias