dccp send

dccp send

[Tomasz Grobelny]

When I use dccp does sendmsg function block (until it sends the packet)? If 
so, should it? In either case, how to make it just queue the packet and 
return?
-- 
Regards,
Tomasz Grobelny

Re: dccp send

[Arnaldo Carvalho de Melo]

Em Wed, Jan 02, 2008 at 01:41:16AM +0100, Tomasz Grobelny escreveu:
> When I use dccp does sendmsg function block (until it sends the packet)? If 
> so, should it? In either case, how to make it just queue the packet and 
> return?

The interface is the same as for other AF_INET transports, use
O_NONBLOCK (open, fcntl) if you want it to be non blocking.

It queues it in the write routine and tries to send it right away, but
doesn't waits for actually sending the packet, i.e. it only checks if
there is write space available, if you set O_NONBLOCK and there is no
space it returns ENOBUFS, if O_NONBLOCK is not set it will sleep waiting
for write space to be made available, when the process will be awaken.

Use setsockopt(SO_SNDTIMEO) to change the default send timeout, etc.

- Arnaldo

Re: dccp send

[Tomasz Grobelny]

Dnia Wednesday 02 of January 2008, Arnaldo Carvalho de Melo napisa³:
> Em Wed, Jan 02, 2008 at 01:41:16AM +0100, Tomasz Grobelny escreveu:
> > When I use dccp does sendmsg function block (until it sends the packet)?
> > If so, should it? In either case, how to make it just queue the packet
> > and return?
>
> The interface is the same as for other AF_INET transports, use
> O_NONBLOCK (open, fcntl) if you want it to be non blocking.
>
But what are the semantics? In TCP if sendmsg blocks the application has no 
choice but to wait. The time for which sendmsg blocks it tightly connected 
with speed at which packets can be transmitted over the network. Is it the 
case for dccp as well? If so, does it mean that provided stable network 
conditions over longer period of time and using blocking version of sendmsg 
in a loop very few packets should be lost? (By very few I mean < 10%)

> It queues it in the write routine and tries to send it right away, but
> doesn't waits for actually sending the packet, i.e. it only checks if
> there is write space available, if you set O_NONBLOCK and there is no
> space it returns ENOBUFS, if O_NONBLOCK is not set it will sleep waiting
> for write space to be made available, when the process will be awaken.
>
dccp_sendmsg seems to block on call to sock_alloc_send_skb. Which at first 
look does not seems to send anything anywhere. So why does it block at all? 
Can't it just allocate the data needed and return?
-- 
Regards,
Tomasz Grobelny

Re: dccp send

[Arnaldo Carvalho de Melo]

Em Wed, Jan 02, 2008 at 11:34:25PM +0100, Tomasz Grobelny escreveu:
> Dnia Wednesday 02 of January 2008, Arnaldo Carvalho de Melo napisał:
> > Em Wed, Jan 02, 2008 at 01:41:16AM +0100, Tomasz Grobelny escreveu:
> > > When I use dccp does sendmsg function block (until it sends the packet)?
> > > If so, should it? In either case, how to make it just queue the packet
> > > and return?
> >
> > The interface is the same as for other AF_INET transports, use
> > O_NONBLOCK (open, fcntl) if you want it to be non blocking.
> >
> But what are the semantics? In TCP if sendmsg blocks the application has no 
> choice but to wait. The time for which sendmsg blocks it tightly connected 

In DCCP its exactly the same semantic, sendmsg may block waiting for
send buffer space. If you don't want that use O_NONBLOCK and it will
return right away with EAGAIN.

> with speed at which packets can be transmitted over the network. Is it the 
> case for dccp as well? If so, does it mean that provided stable network 
> conditions over longer period of time and using blocking version of sendmsg 
> in a loop very few packets should be lost? (By very few I mean < 10%)

If the application sends many packets in rapid succession before the
DCCP core gets permission from the CCID in use to send those packets to
the network what will happen is that sk_write_queue will have packets
waiting, using the memory allowed to be allocated for the socket send
buffer, which will make sock_alloc_send_skb wait for the packets in
sk_write_queue to be sent, freeing up send buffer space, when it will be
possible for sock_alloc_send_skb to actually allocate memory for the
packet and return to dccp_sendmsg, that in turn will return to the
application after putting the newly allocated packet in sk_write_queue,
doing the send buffer accounting and trying to send something in
dccp_write_xmit, where it will ask the CCID if it can send some more
packets, got it?

Look at these sequences:

      sock_alloc_send_skb -> skb_set_owner_w

      This will set the skb destructor to be sock_wfree

Now look at what happens when the NIC driver finally frees the sk_buff
(the network packet):

      kfree_skb -> __kfree_skb -> skb_release_all -> skb->destructor (sock_wfree)

sock_wfree, in turn, will call sk->sk_write_space, that for DCCP is
dccp_write_space (see inet_create -> sk->sk_prot->init
(dccp_v4_init_sock) -> dccp_init_sock), that will wake up any tasks that
are sleeping in sk_sleep, that is... the process doing the dccp_sendmsg
-> sock_alloc_send_skb.

All this behaviour can be controlled thru O_NONBLOCK _and_
setsockopt(SO_SNDTIMEO), just like TCP.
 
> > It queues it in the write routine and tries to send it right away, but
> > doesn't waits for actually sending the packet, i.e. it only checks if
> > there is write space available, if you set O_NONBLOCK and there is no
> > space it returns ENOBUFS, if O_NONBLOCK is not set it will sleep waiting
> > for write space to be made available, when the process will be awaken.

> dccp_sendmsg seems to block on call to sock_alloc_send_skb. Which at first 
> look does not seems to send anything anywhere. So why does it block at all? 
> Can't it just allocate the data needed and return?

See above.

- Arnaldo

Re: dccp send

[Tomasz Grobelny]

Dnia Thursday 03 January 2008, Arnaldo Carvalho de Melo napisał:
> Em Wed, Jan 02, 2008 at 11:34:25PM +0100, Tomasz Grobelny escreveu:
> > with speed at which packets can be transmitted over the network. Is it
> > the case for dccp as well? If so, does it mean that provided stable
> > network conditions over longer period of time and using blocking version
> > of sendmsg in a loop very few packets should be lost? (By very few I mean
> > < 10%)
>
> If the application sends many packets in rapid succession before the
> DCCP core gets permission from the CCID in use to send those packets to
> the network what will happen is that sk_write_queue will have packets
> waiting, using the memory allowed to be allocated for the socket send
> buffer, which will make sock_alloc_send_skb wait for the packets in
> sk_write_queue to be sent, freeing up send buffer space, when it will be
> possible for sock_alloc_send_skb to actually allocate memory for the
> packet and return to dccp_sendmsg, that in turn will return to the
> application after putting the newly allocated packet in sk_write_queue,
> doing the send buffer accounting and trying to send something in
> dccp_write_xmit, where it will ask the CCID if it can send some more
> packets, got it?
>
I think I almost got it. Thanks a lot for the detailed explanation. But I've 
got two more questions:
1. How can I control the amount of "memory allowed to be allocated for the 
socket send buffer"? It it somehow connected with tx_qlen or is it not?
2. If I decide that for whatever reason a given skb already in sk_write_queue 
should not be transmitted I should remove it from queue and call kfree_skb on 
it, it that right?
-- 
Regards,
Tomasz Grobelny

Re: dccp send

[Ian McDonald]

On Jan 4, 2008 11:19 AM, Tomasz Grobelny <tomasz@grobelny.oswiecenia.net> wrote:
> I think I almost got it. Thanks a lot for the detailed explanation. But I've
> got two more questions:
> 1. How can I control the amount of "memory allowed to be allocated for the
> socket send buffer"? It it somehow connected with tx_qlen or is it not?

I wrote the code here so let me comment - there is no fixed amount of
memory allocated - it is determined by the tx_qlen. I was looking at
implementing memory limits but it got too hard and as DCCP doesn't
combine packets I thought it made it much simpler just to put a limit
by packets.

> 2. If I decide that for whatever reason a given skb already in sk_write_queue
> should not be transmitted I should remove it from queue and call kfree_skb on
> it, it that right?

Correct. Remember of course all your locking mechanisms. If you want a
good example look at the below towards the end:
http://wand.net.nz/~iam4/dccp/patches20/30-best_packet_next.diff

(NB this patch applies against 2.6.20 after previous patches in series).

Regards,

Ian
-- 
Web1: http://wand.net.nz/~iam4/
Web2: http://www.jandi.co.nz

Re: dccp send

[Arnaldo Carvalho de Melo]

Em Thu, Jan 03, 2008 at 11:19:32PM +0100, Tomasz Grobelny escreveu:
> Dnia Thursday 03 January 2008, Arnaldo Carvalho de Melo napisał:
> > Em Wed, Jan 02, 2008 at 11:34:25PM +0100, Tomasz Grobelny escreveu:
> > > with speed at which packets can be transmitted over the network. Is it
> > > the case for dccp as well? If so, does it mean that provided stable
> > > network conditions over longer period of time and using blocking version
> > > of sendmsg in a loop very few packets should be lost? (By very few I mean
> > > < 10%)
> >
> > If the application sends many packets in rapid succession before the
> > DCCP core gets permission from the CCID in use to send those packets to
> > the network what will happen is that sk_write_queue will have packets
> > waiting, using the memory allowed to be allocated for the socket send
> > buffer, which will make sock_alloc_send_skb wait for the packets in
> > sk_write_queue to be sent, freeing up send buffer space, when it will be
> > possible for sock_alloc_send_skb to actually allocate memory for the
> > packet and return to dccp_sendmsg, that in turn will return to the
> > application after putting the newly allocated packet in sk_write_queue,
> > doing the send buffer accounting and trying to send something in
> > dccp_write_xmit, where it will ask the CCID if it can send some more
> > packets, got it?
> >
> I think I almost got it. Thanks a lot for the detailed explanation. But I've 
> got two more questions:
> 1. How can I control the amount of "memory allowed to be allocated for the 
> socket send buffer"? It it somehow connected with tx_qlen or is it not?

setsockopt(socket_descriptor, SOL_SOCKET, SO_SNDBUF,  ...)

execute:

man 7 socket
/SO_SNDBUF

tx_qlen is something experimental, system wide.

> 2. If I decide that for whatever reason a given skb already in sk_write_queue 
> should not be transmitted I should remove it from queue and call kfree_skb on 
> it, it that right?

There is no interface for that right now, once you pass a packet to the
kernel, the application has no control over it anymore. But there has
been talks about things like setting a timeout for packets to stay on
the queue, or even for the app to have an interface for dropping packets
from the head of the queue.

- Arnaldo

Re: dccp send

[Arnaldo Carvalho de Melo]

Em Fri, Jan 04, 2008 at 11:25:29AM +1300, Ian McDonald escreveu:
> On Jan 4, 2008 11:19 AM, Tomasz Grobelny <tomasz@grobelny.oswiecenia.net> wrote:
> > I think I almost got it. Thanks a lot for the detailed explanation. But I've
> > got two more questions:
> > 1. How can I control the amount of "memory allowed to be allocated for the
> > socket send buffer"? It it somehow connected with tx_qlen or is it not?
> 
> I wrote the code here so let me comment - there is no fixed amount of
> memory allocated - it is determined by the tx_qlen. I was looking at

there is no memory allocated in advance nor explicit, DCCP specific
mechanism for limiting the amount of memory a socket can use for its
write queue, but DCCP uses sock_alloc_send_skb and that function does
memory accounting and checks if the maximum memory (SO_SNDBUF) has been
reached, when it makes the app to sleep if O_NONBLOCK is not set.

> implementing memory limits but it got too hard and as DCCP doesn't
> combine packets I thought it made it much simpler just to put a limit
> by packets.
> 
> > 2. If I decide that for whatever reason a given skb already in sk_write_queue
> > should not be transmitted I should remove it from queue and call kfree_skb on
> > it, it that right?
> 
> Correct. Remember of course all your locking mechanisms. If you want a
> good example look at the below towards the end:
> http://wand.net.nz/~iam4/dccp/patches20/30-best_packet_next.diff
> 
> (NB this patch applies against 2.6.20 after previous patches in series).

Correct if he is in-kernel.

- Arnaldo

Re: dccp send

[Tomasz Grobelny]

Quoting Ian McDonald <ian.mcdonald@jandi.co.nz>:
> Correct. Remember of course all your locking mechanisms. If you want a
> good example look at the below towards the end:
> http://wand.net.nz/~iam4/dccp/patches20/30-best_packet_next.diff
>
> (NB this patch applies against 2.6.20 after previous patches in series).
>
Thanks. But while we are at your patch I have one more question: why  
did you choose to implement the algorithm it in ccid3? Shouldn't  
packet priorities and expiration times be processed indepentent of  
ccids? Until now it seemed for me that ccids only control _when_ you  
send data, _what_ you send should be determined by other means. Am I  
missing something here?
-- 
Regards,
Tomasz Grobelny

Re: dccp send

[Ian McDonald]

On Jan 5, 2008 5:28 AM, Tomasz Grobelny <tomasz@grobelny.oswiecenia.net> wrote:
> Thanks. But while we are at your patch I have one more question: why
> did you choose to implement the algorithm it in ccid3? Shouldn't
> packet priorities and expiration times be processed indepentent of
> ccids? Until now it seemed for me that ccids only control _when_ you
> send data, _what_ you send should be determined by other means. Am I
> missing something here?

This is just for my own private research. If I was proposing this to
be included with the kernel I would agree with you totally.

Ian
-- 
Web1: http://wand.net.nz/~iam4/
Web2: http://www.jandi.co.nz