* Re: autotuning of send buffer size of a socket [not found] <4a4634330805121200m739e67bdife66c3ad81af1ceb@mail.gmail.com> @ 2008-05-13 0:47 ` Sridhar Samudrala 2008-05-13 0:51 ` David Miller ` (2 more replies) 0 siblings, 3 replies; 14+ messages in thread From: Sridhar Samudrala @ 2008-05-13 0:47 UTC (permalink / raw) To: Shirish Pargaonkar; +Cc: linux-net, netdev On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote: > Hello, > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try, > I still get the same error and do not see the send buffer size of a socket > changing (increasing) > > The initial buffer sizes are 16384 for send side and 87380 for the receive > side but I see receive side buffer tuning but do not see the same with > send side. > > If tcp does not see a need to increase the send buffer size, wonder why I > get EAGAIN error on this non-blocking socket for kernel_sendmsg! I think the send buffer auto-tuning doesn't happen here because there is already congestion window worth of packets sent that are not yet acknowledged. See tcp_should_expand_sndbuf(). Also, the comments for tcp_new_space() says that sndbuf expansion does not work well with largesends. What is the size of your sends? Adding netdev to the CC list. Thanks Sridhar > > I do subscribe to this mailing list so, please send your responses to this > mail address. > > Regards, > > Shirish > > -------------------------------------------------------------------------------------------------- > uname -r > 2.6.18-91.el5 > > sysctl -a > > net.ipv4.tcp_rmem = 4096 87380 4194304 > net.ipv4.tcp_wmem = 4096 16384 4194304 > net.ipv4.tcp_mem = 98304 131072 196608 > > net.core.rmem_default = 126976 > net.core.wmem_default = 126976 > net.core.rmem_max = 131071 > net.core.wmem_max = 131071 > > net.ipv4.tcp_window_scaling = 1 > net.ipv4.tcp_timestamps = 1 > net.ipv4.tcp_moderate_rcvbuf = 1 > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf > 1 > > > CIFS VFS: sndbuf 16384 rcvbuf 87380 > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080 > stuck for 32 seconds, > error: -11 > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240 > stuck for 32 seconds, > error: -11 > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720 > stuck for 64 seconds, > error: -11 > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720 > stuck for 256 seconds, > error: -11 > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf > is 1) but not > the socket send buffer size. > The send buffer size remains fixed, the auto-tuning for send side is > enabled by default,so I do not see it happening here no matter how > long the c ode tries to > kernel_sendmsg after receiving EAGAIN return code. > -- > To unsubscribe from this list: send the line "unsubscribe linux-net" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket 2008-05-13 0:47 ` autotuning of send buffer size of a socket Sridhar Samudrala @ 2008-05-13 0:51 ` David Miller 2008-05-13 13:54 ` Shirish Pargaonkar 2008-05-15 4:18 ` Shirish Pargaonkar 2 siblings, 0 replies; 14+ messages in thread From: David Miller @ 2008-05-13 0:51 UTC (permalink / raw) To: sri; +Cc: shirishpargaonkar, linux-net, netdev From: Sridhar Samudrala <sri@us.ibm.com> Date: Mon, 12 May 2008 17:47:27 -0700 > Also, the comments for tcp_new_space() says that sndbuf expansion does > not work well with largesends. What is the size of your sends? That's talking about TSO. It's an obsolete comment that is no longer true and thus should be removed. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket 2008-05-13 0:47 ` autotuning of send buffer size of a socket Sridhar Samudrala 2008-05-13 0:51 ` David Miller @ 2008-05-13 13:54 ` Shirish Pargaonkar 2008-05-13 23:48 ` Sridhar Samudrala 2008-05-15 4:18 ` Shirish Pargaonkar 2 siblings, 1 reply; 14+ messages in thread From: Shirish Pargaonkar @ 2008-05-13 13:54 UTC (permalink / raw) To: Sridhar Samudrala; +Cc: linux-net, netdev On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote: > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote: > > Hello, > > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try, > > I still get the same error and do not see the send buffer size of a socket > > changing (increasing) > > > > The initial buffer sizes are 16384 for send side and 87380 for the receive > > side but I see receive side buffer tuning but do not see the same with > > send side. > > > > If tcp does not see a need to increase the send buffer size, wonder why I > > get EAGAIN error on this non-blocking socket for kernel_sendmsg! > > I think the send buffer auto-tuning doesn't happen here because there is > already congestion window worth of packets sent that are not yet acknowledged. > See tcp_should_expand_sndbuf(). > > Also, the comments for tcp_new_space() says that sndbuf expansion does > not work well with largesends. What is the size of your sends? > > Adding netdev to the CC list. > > Thanks > Sridhar > > > > > I do subscribe to this mailing list so, please send your responses to this > > mail address. > > > > Regards, > > > > Shirish > > > > -------------------------------------------------------------------------------------------------- > > uname -r > > 2.6.18-91.el5 > > > > sysctl -a > > > > net.ipv4.tcp_rmem = 4096 87380 4194304 > > net.ipv4.tcp_wmem = 4096 16384 4194304 > > net.ipv4.tcp_mem = 98304 131072 196608 > > > > net.core.rmem_default = 126976 > > net.core.wmem_default = 126976 > > net.core.rmem_max = 131071 > > net.core.wmem_max = 131071 > > > > net.ipv4.tcp_window_scaling = 1 > > net.ipv4.tcp_timestamps = 1 > > net.ipv4.tcp_moderate_rcvbuf = 1 > > > > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf > > 1 > > > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380 > > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080 > > stuck for 32 seconds, > > error: -11 > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240 > > stuck for 32 seconds, > > error: -11 > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720 > > stuck for 64 seconds, > > error: -11 > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720 > > stuck for 256 seconds, > > error: -11 > > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf > > is 1) but not > > the socket send buffer size. > > The send buffer size remains fixed, the auto-tuning for send side is > > enabled by default,so I do not see it happening here no matter how > > long the c ode tries to > > kernel_sendmsg after receiving EAGAIN return code. > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-net" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Sridhar, The size of the sends is 56K. Regards, Shirish ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket 2008-05-13 13:54 ` Shirish Pargaonkar @ 2008-05-13 23:48 ` Sridhar Samudrala 2008-05-14 14:22 ` Shirish Pargaonkar 0 siblings, 1 reply; 14+ messages in thread From: Sridhar Samudrala @ 2008-05-13 23:48 UTC (permalink / raw) To: Shirish Pargaonkar; +Cc: linux-net, netdev On Tue, 2008-05-13 at 08:54 -0500, Shirish Pargaonkar wrote: > On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote: > > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote: > > > Hello, > > > > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try, > > > I still get the same error and do not see the send buffer size of a socket > > > changing (increasing) > > > > > > The initial buffer sizes are 16384 for send side and 87380 for the receive > > > side but I see receive side buffer tuning but do not see the same with > > > send side. > > > > > > If tcp does not see a need to increase the send buffer size, wonder why I > > > get EAGAIN error on this non-blocking socket for kernel_sendmsg! > > > > I think the send buffer auto-tuning doesn't happen here because there is > > already congestion window worth of packets sent that are not yet acknowledged. > > See tcp_should_expand_sndbuf(). > > > > Also, the comments for tcp_new_space() says that sndbuf expansion does > > not work well with largesends. What is the size of your sends? > > > > Adding netdev to the CC list. > > > > Thanks > > Sridhar > > > > > > > > I do subscribe to this mailing list so, please send your responses to this > > > mail address. > > > > > > Regards, > > > > > > Shirish > > > > > > -------------------------------------------------------------------------------------------------- > > > uname -r > > > 2.6.18-91.el5 > > > > > > sysctl -a > > > > > > net.ipv4.tcp_rmem = 4096 87380 4194304 > > > net.ipv4.tcp_wmem = 4096 16384 4194304 > > > net.ipv4.tcp_mem = 98304 131072 196608 > > > > > > net.core.rmem_default = 126976 > > > net.core.wmem_default = 126976 > > > net.core.rmem_max = 131071 > > > net.core.wmem_max = 131071 > > > > > > net.ipv4.tcp_window_scaling = 1 > > > net.ipv4.tcp_timestamps = 1 > > > net.ipv4.tcp_moderate_rcvbuf = 1 > > > > > > > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf > > > 1 > > > > > > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380 > > > > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080 > > > stuck for 32 seconds, > > > error: -11 > > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240 > > > stuck for 32 seconds, > > > error: -11 > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720 > > > stuck for 64 seconds, > > > error: -11 > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720 > > > stuck for 256 seconds, > > > error: -11 > > > > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf > > > is 1) but not > > > the socket send buffer size. > > > The send buffer size remains fixed, the auto-tuning for send side is > > > enabled by default,so I do not see it happening here no matter how > > > long the c ode tries to > > > kernel_sendmsg after receiving EAGAIN return code. > Sridhar, > > The size of the sends is 56K. As David pointed out, the send size may not be an issue. When do you see these stalls? Do they happen frequently or only under stress? It could be that the receiver is not able to drain the receive queue causing the send path to be blocked. You could run netstat -tn on the receiver and take a look at 'Recv-Q' output to see if there is data pending in the receive queue. Thanks Sridhar ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket 2008-05-13 23:48 ` Sridhar Samudrala @ 2008-05-14 14:22 ` Shirish Pargaonkar 2008-05-14 18:11 ` Shirish Pargaonkar 0 siblings, 1 reply; 14+ messages in thread From: Shirish Pargaonkar @ 2008-05-14 14:22 UTC (permalink / raw) To: Sridhar Samudrala; +Cc: linux-net, netdev On 5/13/08, Sridhar Samudrala <sri@us.ibm.com> wrote: > On Tue, 2008-05-13 at 08:54 -0500, Shirish Pargaonkar wrote: > > On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote: > > > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote: > > > > Hello, > > > > > > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try, > > > > I still get the same error and do not see the send buffer size of a socket > > > > changing (increasing) > > > > > > > > The initial buffer sizes are 16384 for send side and 87380 for the receive > > > > side but I see receive side buffer tuning but do not see the same with > > > > send side. > > > > > > > > If tcp does not see a need to increase the send buffer size, wonder why I > > > > get EAGAIN error on this non-blocking socket for kernel_sendmsg! > > > > > > I think the send buffer auto-tuning doesn't happen here because there is > > > already congestion window worth of packets sent that are not yet acknowledged. > > > See tcp_should_expand_sndbuf(). > > > > > > Also, the comments for tcp_new_space() says that sndbuf expansion does > > > not work well with largesends. What is the size of your sends? > > > > > > Adding netdev to the CC list. > > > > > > Thanks > > > Sridhar > > > > > > > > > > > I do subscribe to this mailing list so, please send your responses to this > > > > mail address. > > > > > > > > Regards, > > > > > > > > Shirish > > > > > > > > -------------------------------------------------------------------------------------------------- > > > > uname -r > > > > 2.6.18-91.el5 > > > > > > > > sysctl -a > > > > > > > > net.ipv4.tcp_rmem = 4096 87380 4194304 > > > > net.ipv4.tcp_wmem = 4096 16384 4194304 > > > > net.ipv4.tcp_mem = 98304 131072 196608 > > > > > > > > net.core.rmem_default = 126976 > > > > net.core.wmem_default = 126976 > > > > net.core.rmem_max = 131071 > > > > net.core.wmem_max = 131071 > > > > > > > > net.ipv4.tcp_window_scaling = 1 > > > > net.ipv4.tcp_timestamps = 1 > > > > net.ipv4.tcp_moderate_rcvbuf = 1 > > > > > > > > > > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf > > > > 1 > > > > > > > > > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380 > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080 > > > > stuck for 32 seconds, > > > > error: -11 > > > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240 > > > > stuck for 32 seconds, > > > > error: -11 > > > > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720 > > > > stuck for 64 seconds, > > > > error: -11 > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720 > > > > stuck for 256 seconds, > > > > error: -11 > > > > > > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf > > > > is 1) but not > > > > the socket send buffer size. > > > > The send buffer size remains fixed, the auto-tuning for send side is > > > > enabled by default,so I do not see it happening here no matter how > > > > long the c ode tries to > > > > kernel_sendmsg after receiving EAGAIN return code. > > > Sridhar, > > > > The size of the sends is 56K. > > As David pointed out, the send size may not be an issue. > When do you see these stalls? Do they happen frequently or only under > stress? > > It could be that the receiver is not able to drain the receive queue > causing the send path to be blocked. You could run netstat -tn on > the receiver and take a look at 'Recv-Q' output to see if there is > data pending in the receive queue. > > Thanks > Sridhar > > These errors are logged during stress testing, not otherwise. I am running fsstress on 10 shares mounted on this machine running cifs client which are exported by a samba server on another machine. I was running netstat -tn on the machine running samba server in a while loop in a script untill errors started showing up on the cifs client. Some of the entries captured in the file are listed below, rest of them (34345 out of 34356 ) have Recv-Q as 0. tcp 10080 0 123.456.78.238:445 123.456.78.239:39538 ESTABLISHED tcp 10080 0 123.456.78.238:445 123.456.78.239:39538 ESTABLISHED tcp 10080 51 123.456.78.238:445 123.456.78.239:39538 ESTABLISHED tcp 10983 7200 123.456.78.238:445 123.456.78.239:39538 ESTABLISHED tcp 11884 10080 123.456.78.238:445 123.456.78.239:39538 ESTABLISHED tcp 11925 1440 123.456.78.238:445 123.456.78.239:39538 ESTABLISHED tcp 12116 7200 123.456.78.238:445 123.456.78.239:39538 ESTABLISHED tcp 12406 0 123.456.78.238:445 123.456.78.239:39538 ESTABLISHED tcp 290 0 123.456.78.238:445 123.456.78.239:39538 ESTABLISHED tcp 5028 11627 123.456.78.238:445 123.456.78.239:39538 ESTABLISHED tcp 8640 51 123.456.78.238:445 123.456.78.239:39538 ESTABLISHED It is hard to match exact netstat -tn output on the machine running samba server with the errors on machine running cifs client but as soon as I saw the errors appearing on the client, I ran netstat -tn command on the server and found Recv-Q entry was 0 (may be the Recv-Q entries were processed/cleared by the samba server by then). Regards, Shirish ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket 2008-05-14 14:22 ` Shirish Pargaonkar @ 2008-05-14 18:11 ` Shirish Pargaonkar 2008-05-14 18:26 ` Jeremy Jackson 2008-05-14 20:23 ` Shirish Pargaonkar 0 siblings, 2 replies; 14+ messages in thread From: Shirish Pargaonkar @ 2008-05-14 18:11 UTC (permalink / raw) To: Sridhar Samudrala; +Cc: linux-net, netdev On 5/14/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote: > On 5/13/08, Sridhar Samudrala <sri@us.ibm.com> wrote: > > On Tue, 2008-05-13 at 08:54 -0500, Shirish Pargaonkar wrote: > > > On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote: > > > > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote: > > > > > Hello, > > > > > > > > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try, > > > > > I still get the same error and do not see the send buffer size of a socket > > > > > changing (increasing) > > > > > > > > > > The initial buffer sizes are 16384 for send side and 87380 for the receive > > > > > side but I see receive side buffer tuning but do not see the same with > > > > > send side. > > > > > > > > > > If tcp does not see a need to increase the send buffer size, wonder why I > > > > > get EAGAIN error on this non-blocking socket for kernel_sendmsg! > > > > > > > > I think the send buffer auto-tuning doesn't happen here because there is > > > > already congestion window worth of packets sent that are not yet acknowledged. > > > > See tcp_should_expand_sndbuf(). > > > > > > > > Also, the comments for tcp_new_space() says that sndbuf expansion does > > > > not work well with largesends. What is the size of your sends? > > > > > > > > Adding netdev to the CC list. > > > > > > > > Thanks > > > > Sridhar > > > > > > > > > > > > > > I do subscribe to this mailing list so, please send your responses to this > > > > > mail address. > > > > > > > > > > Regards, > > > > > > > > > > Shirish > > > > > > > > > > -------------------------------------------------------------------------------------------------- > > > > > uname -r > > > > > 2.6.18-91.el5 > > > > > > > > > > sysctl -a > > > > > > > > > > net.ipv4.tcp_rmem = 4096 87380 4194304 > > > > > net.ipv4.tcp_wmem = 4096 16384 4194304 > > > > > net.ipv4.tcp_mem = 98304 131072 196608 > > > > > > > > > > net.core.rmem_default = 126976 > > > > > net.core.wmem_default = 126976 > > > > > net.core.rmem_max = 131071 > > > > > net.core.wmem_max = 131071 > > > > > > > > > > net.ipv4.tcp_window_scaling = 1 > > > > > net.ipv4.tcp_timestamps = 1 > > > > > net.ipv4.tcp_moderate_rcvbuf = 1 > > > > > > > > > > > > > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf > > > > > 1 > > > > > > > > > > > > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380 > > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080 > > > > > stuck for 32 seconds, > > > > > error: -11 > > > > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240 > > > > > stuck for 32 seconds, > > > > > error: -11 > > > > > > > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720 > > > > > stuck for 64 seconds, > > > > > error: -11 > > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720 > > > > > stuck for 256 seconds, > > > > > error: -11 > > > > > > > > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf > > > > > is 1) but not > > > > > the socket send buffer size. > > > > > The send buffer size remains fixed, the auto-tuning for send side is > > > > > enabled by default,so I do not see it happening here no matter how > > > > > long the c ode tries to > > > > > kernel_sendmsg after receiving EAGAIN return code. > > > > > Sridhar, > > > > > > The size of the sends is 56K. > > > > As David pointed out, the send size may not be an issue. > > When do you see these stalls? Do they happen frequently or only under > > stress? > > > > It could be that the receiver is not able to drain the receive queue > > causing the send path to be blocked. You could run netstat -tn on > > the receiver and take a look at 'Recv-Q' output to see if there is > > data pending in the receive queue. > > > > Thanks > > Sridhar > > > > > > These errors are logged during stress testing, not otherwise. > I am running fsstress on 10 shares mounted on this machine running cifs client > which are exported by a samba server on another machine. > > I was running netstat -tn on the machine running samba server in a while loop > in a script untill errors started showing up on the cifs client. > Some of the entries captured in the file are listed below, rest of them > (34345 out of 34356 ) have Recv-Q as 0. > > tcp 10080 0 123.456.78.238:445 123.456.78.239:39538 > ESTABLISHED > tcp 10080 0 123.456.78.238:445 123.456.78.239:39538 > ESTABLISHED > tcp 10080 51 123.456.78.238:445 123.456.78.239:39538 > ESTABLISHED > tcp 10983 7200 123.456.78.238:445 123.456.78.239:39538 > ESTABLISHED > tcp 11884 10080 123.456.78.238:445 123.456.78.239:39538 > ESTABLISHED > tcp 11925 1440 123.456.78.238:445 123.456.78.239:39538 > ESTABLISHED > tcp 12116 7200 123.456.78.238:445 123.456.78.239:39538 > ESTABLISHED > tcp 12406 0 123.456.78.238:445 123.456.78.239:39538 > ESTABLISHED > tcp 290 0 123.456.78.238:445 123.456.78.239:39538 > ESTABLISHED > tcp 5028 11627 123.456.78.238:445 123.456.78.239:39538 > ESTABLISHED > tcp 8640 51 123.456.78.238:445 123.456.78.239:39538 > ESTABLISHED > > It is hard to match exact netstat -tn output on the machine running > samba server > with the errors on machine running cifs client but as soon as I saw the errors > appearing on the client, I ran netstat -tn command on the server and found > Recv-Q entry was 0 (may be the Recv-Q entries were processed/cleared by > the samba server by then). > > Regards, > > Shirish > So I see high count of bytes not copied by the user program but they are read next time netstat -tn is run (within probably less than a second), so it is not as if the samba server is not reading off data for long periods right? Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 164 123.456.78.238:445 123.456.78.239:39538 ESTABLISHED tcp 0 0 ::ffff:123.456.78.238:22 ::ffff:123.456.78.190:34532 ESTABLISHED tcp 0 0 ::ffff:123.456.78.238:22 ::ffff:123.456.78.135:50328 ESTABLISHED tcp 0 0 ::ffff:123.456.78.238:22 ::ffff:123.456.78.135:50333 ESTABLISHED Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 12406 0 123.456.78.238:445 123.456.78.239:39538 ESTABLISHED tcp 0 0 ::ffff:123.456.78.238:22 ::ffff:123.456.78.190:34532 ESTABLISHED tcp 0 0 ::ffff:123.456.78.238:22 ::ffff:123.456.78.135:50328 ESTABLISHED tcp 0 0 ::ffff:123.456.78.238:22 ::ffff:123.456.78.135:50333 ESTABLISHED Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 123.456.78.238:445 123.456.78.239:39538 ESTABLISHED tcp 0 0 ::ffff:123.456.78.238:22 ::ffff:123.456.78.190:34532 ESTABLISHED tcp 0 0 ::ffff:123.456.78.238:22 ::ffff:123.456.78.135:50328 ESTABLISHED tcp 0 0 ::ffff:123.456.78.238:22 ::ffff:123.456.78.135:50333 ESTABLISHED So are these occasional and short-lived spikes in unread data on the server side causing from send buffer from increasing on the client side? Regards, Shirish ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket 2008-05-14 18:11 ` Shirish Pargaonkar @ 2008-05-14 18:26 ` Jeremy Jackson 2008-05-14 18:37 ` Shirish Pargaonkar 2008-05-14 20:23 ` Shirish Pargaonkar 1 sibling, 1 reply; 14+ messages in thread From: Jeremy Jackson @ 2008-05-14 18:26 UTC (permalink / raw) To: Shirish Pargaonkar; +Cc: Sridhar Samudrala, linux-net, netdev Sorry if this has been covered already, but doesn't Samba set the write buffer size, thereby disabling autotuning? -- Jeremy Jackson Coplanar Networks (519)489-4903 http://www.coplanar.net jerj@coplanar.net ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket 2008-05-14 18:26 ` Jeremy Jackson @ 2008-05-14 18:37 ` Shirish Pargaonkar 0 siblings, 0 replies; 14+ messages in thread From: Shirish Pargaonkar @ 2008-05-14 18:37 UTC (permalink / raw) To: Jeremy Jackson; +Cc: Sridhar Samudrala, linux-net, netdev On 5/14/08, Jeremy Jackson <jerj@coplanar.net> wrote: > Sorry if this has been covered already, but doesn't Samba set the write > buffer size, thereby disabling autotuning? > > -- > Jeremy Jackson > Coplanar Networks > (519)489-4903 > http://www.coplanar.net > jerj@coplanar.net > > Jeremy, I would like auto-tuning to happen on the machine running cifs/samba client. That is where the errors are logged. The auto-tune happens on the client side, just not good enough to ward off the delays (and subsequent errors logged by cifs client and reconnects initated by cifs client to samba server running on other machine.) for reasons. Regards, Shirish ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket 2008-05-14 18:11 ` Shirish Pargaonkar 2008-05-14 18:26 ` Jeremy Jackson @ 2008-05-14 20:23 ` Shirish Pargaonkar 1 sibling, 0 replies; 14+ messages in thread From: Shirish Pargaonkar @ 2008-05-14 20:23 UTC (permalink / raw) To: Sridhar Samudrala; +Cc: linux-net, netdev On 5/14/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote: > On 5/14/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote: > > On 5/13/08, Sridhar Samudrala <sri@us.ibm.com> wrote: > > > On Tue, 2008-05-13 at 08:54 -0500, Shirish Pargaonkar wrote: > > > > On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote: > > > > > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote: > > > > > > Hello, > > > > > > > > > > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try, > > > > > > I still get the same error and do not see the send buffer size of a socket > > > > > > changing (increasing) > > > > > > > > > > > > The initial buffer sizes are 16384 for send side and 87380 for the receive > > > > > > side but I see receive side buffer tuning but do not see the same with > > > > > > send side. > > > > > > > > > > > > If tcp does not see a need to increase the send buffer size, wonder why I > > > > > > get EAGAIN error on this non-blocking socket for kernel_sendmsg! > > > > > > > > > > I think the send buffer auto-tuning doesn't happen here because there is > > > > > already congestion window worth of packets sent that are not yet acknowledged. > > > > > See tcp_should_expand_sndbuf(). > > > > > > > > > > Also, the comments for tcp_new_space() says that sndbuf expansion does > > > > > not work well with largesends. What is the size of your sends? > > > > > > > > > > Adding netdev to the CC list. > > > > > > > > > > Thanks > > > > > Sridhar > > > > > > > > > > > > > > > > > I do subscribe to this mailing list so, please send your responses to this > > > > > > mail address. > > > > > > > > > > > > Regards, > > > > > > > > > > > > Shirish > > > > > > > > > > > > -------------------------------------------------------------------------------------------------- > > > > > > uname -r > > > > > > 2.6.18-91.el5 > > > > > > > > > > > > sysctl -a > > > > > > > > > > > > net.ipv4.tcp_rmem = 4096 87380 4194304 > > > > > > net.ipv4.tcp_wmem = 4096 16384 4194304 > > > > > > net.ipv4.tcp_mem = 98304 131072 196608 > > > > > > > > > > > > net.core.rmem_default = 126976 > > > > > > net.core.wmem_default = 126976 > > > > > > net.core.rmem_max = 131071 > > > > > > net.core.wmem_max = 131071 > > > > > > > > > > > > net.ipv4.tcp_window_scaling = 1 > > > > > > net.ipv4.tcp_timestamps = 1 > > > > > > net.ipv4.tcp_moderate_rcvbuf = 1 > > > > > > > > > > > > > > > > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf > > > > > > 1 > > > > > > > > > > > > > > > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380 > > > > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080 > > > > > > stuck for 32 seconds, > > > > > > error: -11 > > > > > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240 > > > > > > stuck for 32 seconds, > > > > > > error: -11 > > > > > > > > > > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720 > > > > > > stuck for 64 seconds, > > > > > > error: -11 > > > > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720 > > > > > > stuck for 256 seconds, > > > > > > error: -11 > > > > > > > > > > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf > > > > > > is 1) but not > > > > > > the socket send buffer size. > > > > > > The send buffer size remains fixed, the auto-tuning for send side is > > > > > > enabled by default,so I do not see it happening here no matter how > > > > > > long the c ode tries to > > > > > > kernel_sendmsg after receiving EAGAIN return code. > > > > > > > Sridhar, > > > > > > > > The size of the sends is 56K. > > > > > > As David pointed out, the send size may not be an issue. > > > When do you see these stalls? Do they happen frequently or only under > > > stress? > > > > > > It could be that the receiver is not able to drain the receive queue > > > causing the send path to be blocked. You could run netstat -tn on > > > the receiver and take a look at 'Recv-Q' output to see if there is > > > data pending in the receive queue. > > > > > > Thanks > > > Sridhar > > > > > > > > > > These errors are logged during stress testing, not otherwise. > > I am running fsstress on 10 shares mounted on this machine running cifs client > > which are exported by a samba server on another machine. > > > > I was running netstat -tn on the machine running samba server in a while loop > > in a script untill errors started showing up on the cifs client. > > Some of the entries captured in the file are listed below, rest of them > > (34345 out of 34356 ) have Recv-Q as 0. > > > > tcp 10080 0 123.456.78.238:445 123.456.78.239:39538 > > ESTABLISHED > > tcp 10080 0 123.456.78.238:445 123.456.78.239:39538 > > ESTABLISHED > > tcp 10080 51 123.456.78.238:445 123.456.78.239:39538 > > ESTABLISHED > > tcp 10983 7200 123.456.78.238:445 123.456.78.239:39538 > > ESTABLISHED > > tcp 11884 10080 123.456.78.238:445 123.456.78.239:39538 > > ESTABLISHED > > tcp 11925 1440 123.456.78.238:445 123.456.78.239:39538 > > ESTABLISHED > > tcp 12116 7200 123.456.78.238:445 123.456.78.239:39538 > > ESTABLISHED > > tcp 12406 0 123.456.78.238:445 123.456.78.239:39538 > > ESTABLISHED > > tcp 290 0 123.456.78.238:445 123.456.78.239:39538 > > ESTABLISHED > > tcp 5028 11627 123.456.78.238:445 123.456.78.239:39538 > > ESTABLISHED > > tcp 8640 51 123.456.78.238:445 123.456.78.239:39538 > > ESTABLISHED > > > > It is hard to match exact netstat -tn output on the machine running > > samba server > > with the errors on machine running cifs client but as soon as I saw the errors > > appearing on the client, I ran netstat -tn command on the server and found > > Recv-Q entry was 0 (may be the Recv-Q entries were processed/cleared by > > the samba server by then). > > > > Regards, > > > > Shirish > > > > So I see high count of bytes not copied by the user program but they are > read next time netstat -tn is run (within probably less than a > second), so it is not > as if the samba server is not reading off data for long periods right? > > Active Internet connections (w/o servers) > Proto Recv-Q Send-Q Local Address Foreign Address > State > tcp 0 164 123.456.78.238:445 123.456.78.239:39538 > ESTABLISHED > tcp 0 0 ::ffff:123.456.78.238:22 > ::ffff:123.456.78.190:34532 ESTABLISHED > tcp 0 0 ::ffff:123.456.78.238:22 > ::ffff:123.456.78.135:50328 ESTABLISHED > tcp 0 0 ::ffff:123.456.78.238:22 > ::ffff:123.456.78.135:50333 ESTABLISHED > Active Internet connections (w/o servers) > Proto Recv-Q Send-Q Local Address Foreign Address > State > tcp 12406 0 123.456.78.238:445 123.456.78.239:39538 > ESTABLISHED > tcp 0 0 ::ffff:123.456.78.238:22 > ::ffff:123.456.78.190:34532 ESTABLISHED > tcp 0 0 ::ffff:123.456.78.238:22 > ::ffff:123.456.78.135:50328 ESTABLISHED > tcp 0 0 ::ffff:123.456.78.238:22 > ::ffff:123.456.78.135:50333 ESTABLISHED > Active Internet connections (w/o servers) > Proto Recv-Q Send-Q Local Address Foreign Address > State > tcp 0 0 123.456.78.238:445 123.456.78.239:39538 > ESTABLISHED > tcp 0 0 ::ffff:123.456.78.238:22 > ::ffff:123.456.78.190:34532 ESTABLISHED > tcp 0 0 ::ffff:123.456.78.238:22 > ::ffff:123.456.78.135:50328 ESTABLISHED > tcp 0 0 ::ffff:123.456.78.238:22 > ::ffff:123.456.78.135:50333 ESTABLISHED > > So are these occasional and short-lived spikes in unread data on the > server side > causing from send buffer from increasing on the client side? > > Regards, > > Shirish > I called kernel_getsockopt after receiving EAGAIN for 15 seconds for kernel_sendmsg. Not sure whether it would be useful. lds is last data sent and ldr is last data received. CIFS VFS: sends on sock 000000000eea1400, sndbuf 34776, rcvbuf 203520 of length 57408 stuck for 15 seconds, error: -11 CIFS VFS: smb_send2 lds 0, ldr 20, cwnd 9, send ssthresh 100, rcv ssthresh 153258 rtt 18750 rttvar 7500 unacked 7 sacked 0 lost 0, retrans 0, fackets 0 ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket 2008-05-13 0:47 ` autotuning of send buffer size of a socket Sridhar Samudrala 2008-05-13 0:51 ` David Miller 2008-05-13 13:54 ` Shirish Pargaonkar @ 2008-05-15 4:18 ` Shirish Pargaonkar 2008-05-15 15:00 ` Shirish Pargaonkar 2 siblings, 1 reply; 14+ messages in thread From: Shirish Pargaonkar @ 2008-05-15 4:18 UTC (permalink / raw) To: Sridhar Samudrala; +Cc: linux-net, netdev On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote: > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote: > > Hello, > > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try, > > I still get the same error and do not see the send buffer size of a socket > > changing (increasing) > > > > The initial buffer sizes are 16384 for send side and 87380 for the receive > > side but I see receive side buffer tuning but do not see the same with > > send side. > > > > If tcp does not see a need to increase the send buffer size, wonder why I > > get EAGAIN error on this non-blocking socket for kernel_sendmsg! > > I think the send buffer auto-tuning doesn't happen here because there is > already congestion window worth of packets sent that are not yet acknowledged. > See tcp_should_expand_sndbuf(). Sridhar, The unacked (packets_out) is 7 and snd_cwnd is 9, so that should not be the case for tcp_should_expand_sndbuf to return 0 right? > > Also, the comments for tcp_new_space() says that sndbuf expansion does > not work well with largesends. What is the size of your sends? > > Adding netdev to the CC list. > > Thanks > Sridhar > > > > > I do subscribe to this mailing list so, please send your responses to this > > mail address. > > > > Regards, > > > > Shirish > > > > -------------------------------------------------------------------------------------------------- > > uname -r > > 2.6.18-91.el5 > > > > sysctl -a > > > > net.ipv4.tcp_rmem = 4096 87380 4194304 > > net.ipv4.tcp_wmem = 4096 16384 4194304 > > net.ipv4.tcp_mem = 98304 131072 196608 > > > > net.core.rmem_default = 126976 > > net.core.wmem_default = 126976 > > net.core.rmem_max = 131071 > > net.core.wmem_max = 131071 > > > > net.ipv4.tcp_window_scaling = 1 > > net.ipv4.tcp_timestamps = 1 > > net.ipv4.tcp_moderate_rcvbuf = 1 > > > > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf > > 1 > > > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380 > > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080 > > stuck for 32 seconds, > > error: -11 > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240 > > stuck for 32 seconds, > > error: -11 > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720 > > stuck for 64 seconds, > > error: -11 > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720 > > stuck for 256 seconds, > > error: -11 > > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf > > is 1) but not > > the socket send buffer size. > > The send buffer size remains fixed, the auto-tuning for send side is > > enabled by default,so I do not see it happening here no matter how > > long the c ode tries to > > kernel_sendmsg after receiving EAGAIN return code. > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-net" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket 2008-05-15 4:18 ` Shirish Pargaonkar @ 2008-05-15 15:00 ` Shirish Pargaonkar 2008-05-15 17:40 ` Sridhar Samudrala 0 siblings, 1 reply; 14+ messages in thread From: Shirish Pargaonkar @ 2008-05-15 15:00 UTC (permalink / raw) To: Sridhar Samudrala; +Cc: linux-net, netdev On 5/14/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote: > On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote: > > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote: > > > Hello, > > > > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try, > > > I still get the same error and do not see the send buffer size of a socket > > > changing (increasing) > > > > > > The initial buffer sizes are 16384 for send side and 87380 for the receive > > > side but I see receive side buffer tuning but do not see the same with > > > send side. > > > > > > If tcp does not see a need to increase the send buffer size, wonder why I > > > get EAGAIN error on this non-blocking socket for kernel_sendmsg! > > > > I think the send buffer auto-tuning doesn't happen here because there is > > already congestion window worth of packets sent that are not yet acknowledged. > > See tcp_should_expand_sndbuf(). > > Sridhar, > > The unacked (packets_out) is 7 and snd_cwnd is 9, so that should not be > the case for tcp_should_expand_sndbuf to return 0 right? > > > > > Also, the comments for tcp_new_space() says that sndbuf expansion does > > not work well with largesends. What is the size of your sends? > > > > Adding netdev to the CC list. > > > > Thanks > > Sridhar > > > > > > > > I do subscribe to this mailing list so, please send your responses to this > > > mail address. > > > > > > Regards, > > > > > > Shirish > > > > > > -------------------------------------------------------------------------------------------------- > > > uname -r > > > 2.6.18-91.el5 > > > > > > sysctl -a > > > > > > net.ipv4.tcp_rmem = 4096 87380 4194304 > > > net.ipv4.tcp_wmem = 4096 16384 4194304 > > > net.ipv4.tcp_mem = 98304 131072 196608 > > > > > > net.core.rmem_default = 126976 > > > net.core.wmem_default = 126976 > > > net.core.rmem_max = 131071 > > > net.core.wmem_max = 131071 > > > > > > net.ipv4.tcp_window_scaling = 1 > > > net.ipv4.tcp_timestamps = 1 > > > net.ipv4.tcp_moderate_rcvbuf = 1 > > > > > > > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf > > > 1 > > > > > > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380 > > > > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080 > > > stuck for 32 seconds, > > > error: -11 > > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240 > > > stuck for 32 seconds, > > > error: -11 > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720 > > > stuck for 64 seconds, > > > error: -11 > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720 > > > stuck for 256 seconds, > > > error: -11 > > > > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf > > > is 1) but not > > > the socket send buffer size. > > > The send buffer size remains fixed, the auto-tuning for send side is > > > enabled by default,so I do not see it happening here no matter how > > > long the c ode tries to > > > kernel_sendmsg after receiving EAGAIN return code. > > > -- > > > To unsubscribe from this list: send the line "unsubscribe linux-net" in > > > the body of a message to majordomo@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > I put some printk in tcp.c (in function tcp_sendmsg) sndbuf grows from 16384 to 34776 but never beyond it. CIFS VFS: sndbuf 16384 rcvbuf 87380 rcvtimeo 0x7fffffffffffffff !sk_stream_memory_free queued 18288, sndbuf 16384 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 28448, sndbuf 27048 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 28448, sndbuf 27048 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 32512, sndbuf 30912 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 32512, sndbuf 30912 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 32512, sndbuf 30912 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 32512, sndbuf 30912 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 32512, sndbuf 30912 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 32512, sndbuf 30912 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 32512, sndbuf 30912 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 32512, sndbuf 30912 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 32512, sndbuf 30912 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 32512, sndbuf 30912 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 32512, sndbuf 30912 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 32512, sndbuf 30912 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 32512, sndbuf 30912 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 32512, sndbuf 30912 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 36576, sndbuf 34776 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 36576, sndbuf 34776 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 36576, sndbuf 34776 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 36576, sndbuf 34776 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 36576, sndbuf 34776 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 36576, sndbuf 34776 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 36576, sndbuf 34776 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 36576, sndbuf 34776 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 36576, sndbuf 34776 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 36576, sndbuf 34776 sk_stream_wait_memory with 0 returned -11 !sk_stream_memory_free queued 36576, sndbuf 34776 sk_stream_wait_memory with 0 returned -11 and so on and the sndbuf does not grow beyond 34776 ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket 2008-05-15 15:00 ` Shirish Pargaonkar @ 2008-05-15 17:40 ` Sridhar Samudrala 2008-05-15 18:03 ` Shirish Pargaonkar 0 siblings, 1 reply; 14+ messages in thread From: Sridhar Samudrala @ 2008-05-15 17:40 UTC (permalink / raw) To: Shirish Pargaonkar; +Cc: linux-net, netdev On Thu, 2008-05-15 at 10:00 -0500, Shirish Pargaonkar wrote: > On 5/14/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote: > > On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote: > > > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote: > > > > Hello, > > > > > > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try, > > > > I still get the same error and do not see the send buffer size of a socket > > > > changing (increasing) > > > > > > > > The initial buffer sizes are 16384 for send side and 87380 for the receive > > > > side but I see receive side buffer tuning but do not see the same with > > > > send side. > > > > > > > > If tcp does not see a need to increase the send buffer size, wonder why I > > > > get EAGAIN error on this non-blocking socket for kernel_sendmsg! > > > > > > I think the send buffer auto-tuning doesn't happen here because there is > > > already congestion window worth of packets sent that are not yet acknowledged. > > > See tcp_should_expand_sndbuf(). > > > > Sridhar, > > > > The unacked (packets_out) is 7 and snd_cwnd is 9, so that should not be > > the case for tcp_should_expand_sndbuf to return 0 right? It looks like sndbuf expansion via tcp_should_expand_sndbuf() happens only in response to acks/data from the receiver. tcp_rcv_established/tcp_rcv_state_process tcp_data_snd_check tcp_check_space tcp_new_space tcp_should_expand_sndbuf auto-tuning doesn't increase sndbuf when trying to send more data. > > > > > > > > Also, the comments for tcp_new_space() says that sndbuf expansion does > > > not work well with largesends. What is the size of your sends? > > > > > > Adding netdev to the CC list. > > > > > > Thanks > > > Sridhar > > > > > > > > > > > I do subscribe to this mailing list so, please send your responses to this > > > > mail address. > > > > > > > > Regards, > > > > > > > > Shirish > > > > > > > > -------------------------------------------------------------------------------------------------- > > > > uname -r > > > > 2.6.18-91.el5 > > > > > > > > sysctl -a > > > > > > > > net.ipv4.tcp_rmem = 4096 87380 4194304 > > > > net.ipv4.tcp_wmem = 4096 16384 4194304 > > > > net.ipv4.tcp_mem = 98304 131072 196608 > > > > > > > > net.core.rmem_default = 126976 > > > > net.core.wmem_default = 126976 > > > > net.core.rmem_max = 131071 > > > > net.core.wmem_max = 131071 > > > > > > > > net.ipv4.tcp_window_scaling = 1 > > > > net.ipv4.tcp_timestamps = 1 > > > > net.ipv4.tcp_moderate_rcvbuf = 1 > > > > > > > > > > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf > > > > 1 > > > > > > > > > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380 > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080 > > > > stuck for 32 seconds, > > > > error: -11 > > > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240 > > > > stuck for 32 seconds, > > > > error: -11 > > > > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720 > > > > stuck for 64 seconds, > > > > error: -11 > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720 > > > > stuck for 256 seconds, > > > > error: -11 > > > > > > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf > > > > is 1) but not > > > > the socket send buffer size. > > > > The send buffer size remains fixed, the auto-tuning for send side is > > > > enabled by default,so I do not see it happening here no matter how > > > > long the c ode tries to > > > > kernel_sendmsg after receiving EAGAIN return code. > > > > -- > > > > To unsubscribe from this list: send the line "unsubscribe linux-net" in > > > > the body of a message to majordomo@vger.kernel.org > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > > > > > I put some printk in tcp.c (in function tcp_sendmsg) > sndbuf grows from 16384 to 34776 but never beyond it. > > > CIFS VFS: sndbuf 16384 rcvbuf 87380 rcvtimeo 0x7fffffffffffffff > > !sk_stream_memory_free queued 18288, sndbuf 16384 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 28448, sndbuf 27048 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 28448, sndbuf 27048 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 32512, sndbuf 30912 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 32512, sndbuf 30912 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 32512, sndbuf 30912 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 32512, sndbuf 30912 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 32512, sndbuf 30912 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 32512, sndbuf 30912 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 32512, sndbuf 30912 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 32512, sndbuf 30912 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 32512, sndbuf 30912 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 32512, sndbuf 30912 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 32512, sndbuf 30912 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 32512, sndbuf 30912 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 32512, sndbuf 30912 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 32512, sndbuf 30912 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 36576, sndbuf 34776 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 36576, sndbuf 34776 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 36576, sndbuf 34776 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 36576, sndbuf 34776 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 36576, sndbuf 34776 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 36576, sndbuf 34776 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 36576, sndbuf 34776 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 36576, sndbuf 34776 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 36576, sndbuf 34776 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 36576, sndbuf 34776 > sk_stream_wait_memory with 0 returned -11 > !sk_stream_memory_free queued 36576, sndbuf 34776 > sk_stream_wait_memory with 0 returned -11 > > and so on and the sndbuf does not grow beyond 34776 So there is outstanding data(sk_wmem_queued) that is not getting acked. If you set the sndbuf manually to a higher value, does it solve the problem or only delay the stalls? Thanks Sridhar ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket 2008-05-15 17:40 ` Sridhar Samudrala @ 2008-05-15 18:03 ` Shirish Pargaonkar 2008-05-15 18:49 ` Shirish Pargaonkar 0 siblings, 1 reply; 14+ messages in thread From: Shirish Pargaonkar @ 2008-05-15 18:03 UTC (permalink / raw) To: Sridhar Samudrala; +Cc: linux-net, netdev On 5/15/08, Sridhar Samudrala <sri@us.ibm.com> wrote: > On Thu, 2008-05-15 at 10:00 -0500, Shirish Pargaonkar wrote: > > On 5/14/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote: > > > On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote: > > > > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote: > > > > > Hello, > > > > > > > > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try, > > > > > I still get the same error and do not see the send buffer size of a socket > > > > > changing (increasing) > > > > > > > > > > The initial buffer sizes are 16384 for send side and 87380 for the receive > > > > > side but I see receive side buffer tuning but do not see the same with > > > > > send side. > > > > > > > > > > If tcp does not see a need to increase the send buffer size, wonder why I > > > > > get EAGAIN error on this non-blocking socket for kernel_sendmsg! > > > > > > > > I think the send buffer auto-tuning doesn't happen here because there is > > > > already congestion window worth of packets sent that are not yet acknowledged. > > > > See tcp_should_expand_sndbuf(). > > > > > > Sridhar, > > > > > > The unacked (packets_out) is 7 and snd_cwnd is 9, so that should not be > > > the case for tcp_should_expand_sndbuf to return 0 right? > > It looks like sndbuf expansion via tcp_should_expand_sndbuf() happens > only in response to acks/data from the receiver. > tcp_rcv_established/tcp_rcv_state_process > tcp_data_snd_check > tcp_check_space > tcp_new_space > tcp_should_expand_sndbuf > auto-tuning doesn't increase sndbuf when trying to send more data. > > > > > > > > > > > > Also, the comments for tcp_new_space() says that sndbuf expansion does > > > > not work well with largesends. What is the size of your sends? > > > > > > > > Adding netdev to the CC list. > > > > > > > > Thanks > > > > Sridhar > > > > > > > > > > > > > > I do subscribe to this mailing list so, please send your responses to this > > > > > mail address. > > > > > > > > > > Regards, > > > > > > > > > > Shirish > > > > > > > > > > -------------------------------------------------------------------------------------------------- > > > > > uname -r > > > > > 2.6.18-91.el5 > > > > > > > > > > sysctl -a > > > > > > > > > > net.ipv4.tcp_rmem = 4096 87380 4194304 > > > > > net.ipv4.tcp_wmem = 4096 16384 4194304 > > > > > net.ipv4.tcp_mem = 98304 131072 196608 > > > > > > > > > > net.core.rmem_default = 126976 > > > > > net.core.wmem_default = 126976 > > > > > net.core.rmem_max = 131071 > > > > > net.core.wmem_max = 131071 > > > > > > > > > > net.ipv4.tcp_window_scaling = 1 > > > > > net.ipv4.tcp_timestamps = 1 > > > > > net.ipv4.tcp_moderate_rcvbuf = 1 > > > > > > > > > > > > > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf > > > > > 1 > > > > > > > > > > > > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380 > > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080 > > > > > stuck for 32 seconds, > > > > > error: -11 > > > > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240 > > > > > stuck for 32 seconds, > > > > > error: -11 > > > > > > > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720 > > > > > stuck for 64 seconds, > > > > > error: -11 > > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720 > > > > > stuck for 256 seconds, > > > > > error: -11 > > > > > > > > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf > > > > > is 1) but not > > > > > the socket send buffer size. > > > > > The send buffer size remains fixed, the auto-tuning for send side is > > > > > enabled by default,so I do not see it happening here no matter how > > > > > long the c ode tries to > > > > > kernel_sendmsg after receiving EAGAIN return code. > > > > > -- > > > > > To unsubscribe from this list: send the line "unsubscribe linux-net" in > > > > > the body of a message to majordomo@vger.kernel.org > > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > > > > > > > > > > I put some printk in tcp.c (in function tcp_sendmsg) > > sndbuf grows from 16384 to 34776 but never beyond it. > > > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380 rcvtimeo 0x7fffffffffffffff > > > > !sk_stream_memory_free queued 18288, sndbuf 16384 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 28448, sndbuf 27048 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 28448, sndbuf 27048 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > sk_stream_wait_memory with 0 returned -11 > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > sk_stream_wait_memory with 0 returned -11 > > > > and so on and the sndbuf does not grow beyond 34776 > > So there is outstanding data(sk_wmem_queued) that is not getting > acked. > If you set the sndbuf manually to a higher value, does it solve > the problem or only delay the stalls? > > Thanks > Sridhar > > Sridhar, snd_cwnd goes from 6 to 7 to 8 to 9 and is capped at that value. That is how sndbuf grows upto but does not grow beyond 34776. If I set the sndbuf manully, it just delays stalls. I have gone as high as 1MB of send size and that prevents from logging these errors longer but eventually it does. Regards, Shirish ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket 2008-05-15 18:03 ` Shirish Pargaonkar @ 2008-05-15 18:49 ` Shirish Pargaonkar 0 siblings, 0 replies; 14+ messages in thread From: Shirish Pargaonkar @ 2008-05-15 18:49 UTC (permalink / raw) To: Sridhar Samudrala; +Cc: linux-net, netdev On 5/15/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote: > On 5/15/08, Sridhar Samudrala <sri@us.ibm.com> wrote: > > On Thu, 2008-05-15 at 10:00 -0500, Shirish Pargaonkar wrote: > > > On 5/14/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote: > > > > On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote: > > > > > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote: > > > > > > Hello, > > > > > > > > > > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try, > > > > > > I still get the same error and do not see the send buffer size of a socket > > > > > > changing (increasing) > > > > > > > > > > > > The initial buffer sizes are 16384 for send side and 87380 for the receive > > > > > > side but I see receive side buffer tuning but do not see the same with > > > > > > send side. > > > > > > > > > > > > If tcp does not see a need to increase the send buffer size, wonder why I > > > > > > get EAGAIN error on this non-blocking socket for kernel_sendmsg! > > > > > > > > > > I think the send buffer auto-tuning doesn't happen here because there is > > > > > already congestion window worth of packets sent that are not yet acknowledged. > > > > > See tcp_should_expand_sndbuf(). > > > > > > > > Sridhar, > > > > > > > > The unacked (packets_out) is 7 and snd_cwnd is 9, so that should not be > > > > the case for tcp_should_expand_sndbuf to return 0 right? > > > > It looks like sndbuf expansion via tcp_should_expand_sndbuf() happens > > only in response to acks/data from the receiver. > > tcp_rcv_established/tcp_rcv_state_process > > tcp_data_snd_check > > tcp_check_space > > tcp_new_space > > tcp_should_expand_sndbuf > > auto-tuning doesn't increase sndbuf when trying to send more data. > > > > > > > > > > > > > > > > Also, the comments for tcp_new_space() says that sndbuf expansion does > > > > > not work well with largesends. What is the size of your sends? > > > > > > > > > > Adding netdev to the CC list. > > > > > > > > > > Thanks > > > > > Sridhar > > > > > > > > > > > > > > > > > I do subscribe to this mailing list so, please send your responses to this > > > > > > mail address. > > > > > > > > > > > > Regards, > > > > > > > > > > > > Shirish > > > > > > > > > > > > -------------------------------------------------------------------------------------------------- > > > > > > uname -r > > > > > > 2.6.18-91.el5 > > > > > > > > > > > > sysctl -a > > > > > > > > > > > > net.ipv4.tcp_rmem = 4096 87380 4194304 > > > > > > net.ipv4.tcp_wmem = 4096 16384 4194304 > > > > > > net.ipv4.tcp_mem = 98304 131072 196608 > > > > > > > > > > > > net.core.rmem_default = 126976 > > > > > > net.core.wmem_default = 126976 > > > > > > net.core.rmem_max = 131071 > > > > > > net.core.wmem_max = 131071 > > > > > > > > > > > > net.ipv4.tcp_window_scaling = 1 > > > > > > net.ipv4.tcp_timestamps = 1 > > > > > > net.ipv4.tcp_moderate_rcvbuf = 1 > > > > > > > > > > > > > > > > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf > > > > > > 1 > > > > > > > > > > > > > > > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380 > > > > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080 > > > > > > stuck for 32 seconds, > > > > > > error: -11 > > > > > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240 > > > > > > stuck for 32 seconds, > > > > > > error: -11 > > > > > > > > > > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720 > > > > > > stuck for 64 seconds, > > > > > > error: -11 > > > > > > > > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720 > > > > > > stuck for 256 seconds, > > > > > > error: -11 > > > > > > > > > > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf > > > > > > is 1) but not > > > > > > the socket send buffer size. > > > > > > The send buffer size remains fixed, the auto-tuning for send side is > > > > > > enabled by default,so I do not see it happening here no matter how > > > > > > long the c ode tries to > > > > > > kernel_sendmsg after receiving EAGAIN return code. > > > > > > -- > > > > > > To unsubscribe from this list: send the line "unsubscribe linux-net" in > > > > > > the body of a message to majordomo@vger.kernel.org > > > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > > > > > > > > > > > > > > > I put some printk in tcp.c (in function tcp_sendmsg) > > > sndbuf grows from 16384 to 34776 but never beyond it. > > > > > > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380 rcvtimeo 0x7fffffffffffffff > > > > > > !sk_stream_memory_free queued 18288, sndbuf 16384 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 28448, sndbuf 27048 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 28448, sndbuf 27048 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 32512, sndbuf 30912 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > > sk_stream_wait_memory with 0 returned -11 > > > !sk_stream_memory_free queued 36576, sndbuf 34776 > > > sk_stream_wait_memory with 0 returned -11 > > > > > > and so on and the sndbuf does not grow beyond 34776 > > > > So there is outstanding data(sk_wmem_queued) that is not getting > > acked. > > If you set the sndbuf manually to a higher value, does it solve > > the problem or only delay the stalls? > > > > Thanks > > Sridhar > > > > > > Sridhar, > > snd_cwnd goes from 6 to 7 to 8 to 9 and is capped at that value. That is how > sndbuf grows upto but does not grow beyond 34776. > > If I set the sndbuf manully, it just delays stalls. I have gone as high as 1MB > of send size and that prevents from logging these errors longer but > eventually it does. > > Regards, > > Shirish > I take it back, sndbuf has gone as high as sk->sk_sndbuf 258888 and snd_cwnd has gone as high as 67. ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2008-05-15 18:49 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <4a4634330805121200m739e67bdife66c3ad81af1ceb@mail.gmail.com>
2008-05-13 0:47 ` autotuning of send buffer size of a socket Sridhar Samudrala
2008-05-13 0:51 ` David Miller
2008-05-13 13:54 ` Shirish Pargaonkar
2008-05-13 23:48 ` Sridhar Samudrala
2008-05-14 14:22 ` Shirish Pargaonkar
2008-05-14 18:11 ` Shirish Pargaonkar
2008-05-14 18:26 ` Jeremy Jackson
2008-05-14 18:37 ` Shirish Pargaonkar
2008-05-14 20:23 ` Shirish Pargaonkar
2008-05-15 4:18 ` Shirish Pargaonkar
2008-05-15 15:00 ` Shirish Pargaonkar
2008-05-15 17:40 ` Sridhar Samudrala
2008-05-15 18:03 ` Shirish Pargaonkar
2008-05-15 18:49 ` Shirish Pargaonkar
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).