* Re: autotuning of send buffer size of a socket
[not found] <4a4634330805121200m739e67bdife66c3ad81af1ceb@mail.gmail.com>
@ 2008-05-13 0:47 ` Sridhar Samudrala
2008-05-13 0:51 ` David Miller
` (2 more replies)
0 siblings, 3 replies; 14+ messages in thread
From: Sridhar Samudrala @ 2008-05-13 0:47 UTC (permalink / raw)
To: Shirish Pargaonkar; +Cc: linux-net, netdev
On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote:
> Hello,
>
> kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try,
> I still get the same error and do not see the send buffer size of a socket
> changing (increasing)
>
> The initial buffer sizes are 16384 for send side and 87380 for the receive
> side but I see receive side buffer tuning but do not see the same with
> send side.
>
> If tcp does not see a need to increase the send buffer size, wonder why I
> get EAGAIN error on this non-blocking socket for kernel_sendmsg!
I think the send buffer auto-tuning doesn't happen here because there is
already congestion window worth of packets sent that are not yet acknowledged.
See tcp_should_expand_sndbuf().
Also, the comments for tcp_new_space() says that sndbuf expansion does
not work well with largesends. What is the size of your sends?
Adding netdev to the CC list.
Thanks
Sridhar
>
> I do subscribe to this mailing list so, please send your responses to this
> mail address.
>
> Regards,
>
> Shirish
>
> --------------------------------------------------------------------------------------------------
> uname -r
> 2.6.18-91.el5
>
> sysctl -a
>
> net.ipv4.tcp_rmem = 4096 87380 4194304
> net.ipv4.tcp_wmem = 4096 16384 4194304
> net.ipv4.tcp_mem = 98304 131072 196608
>
> net.core.rmem_default = 126976
> net.core.wmem_default = 126976
> net.core.rmem_max = 131071
> net.core.wmem_max = 131071
>
> net.ipv4.tcp_window_scaling = 1
> net.ipv4.tcp_timestamps = 1
> net.ipv4.tcp_moderate_rcvbuf = 1
>
>
> cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf
> 1
>
>
> CIFS VFS: sndbuf 16384 rcvbuf 87380
>
> CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080
> stuck for 32 seconds,
> error: -11
> CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240
> stuck for 32 seconds,
> error: -11
>
>
> CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720
> stuck for 64 seconds,
> error: -11
>
> CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720
> stuck for 256 seconds,
> error: -11
>
> I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf
> is 1) but not
> the socket send buffer size.
> The send buffer size remains fixed, the auto-tuning for send side is
> enabled by default,so I do not see it happening here no matter how
> long the c ode tries to
> kernel_sendmsg after receiving EAGAIN return code.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-net" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket
2008-05-13 0:47 ` autotuning of send buffer size of a socket Sridhar Samudrala
@ 2008-05-13 0:51 ` David Miller
2008-05-13 13:54 ` Shirish Pargaonkar
2008-05-15 4:18 ` Shirish Pargaonkar
2 siblings, 0 replies; 14+ messages in thread
From: David Miller @ 2008-05-13 0:51 UTC (permalink / raw)
To: sri; +Cc: shirishpargaonkar, linux-net, netdev
From: Sridhar Samudrala <sri@us.ibm.com>
Date: Mon, 12 May 2008 17:47:27 -0700
> Also, the comments for tcp_new_space() says that sndbuf expansion does
> not work well with largesends. What is the size of your sends?
That's talking about TSO. It's an obsolete comment that is no longer
true and thus should be removed.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket
2008-05-13 0:47 ` autotuning of send buffer size of a socket Sridhar Samudrala
2008-05-13 0:51 ` David Miller
@ 2008-05-13 13:54 ` Shirish Pargaonkar
2008-05-13 23:48 ` Sridhar Samudrala
2008-05-15 4:18 ` Shirish Pargaonkar
2 siblings, 1 reply; 14+ messages in thread
From: Shirish Pargaonkar @ 2008-05-13 13:54 UTC (permalink / raw)
To: Sridhar Samudrala; +Cc: linux-net, netdev
On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote:
> On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote:
> > Hello,
> >
> > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try,
> > I still get the same error and do not see the send buffer size of a socket
> > changing (increasing)
> >
> > The initial buffer sizes are 16384 for send side and 87380 for the receive
> > side but I see receive side buffer tuning but do not see the same with
> > send side.
> >
> > If tcp does not see a need to increase the send buffer size, wonder why I
> > get EAGAIN error on this non-blocking socket for kernel_sendmsg!
>
> I think the send buffer auto-tuning doesn't happen here because there is
> already congestion window worth of packets sent that are not yet acknowledged.
> See tcp_should_expand_sndbuf().
>
> Also, the comments for tcp_new_space() says that sndbuf expansion does
> not work well with largesends. What is the size of your sends?
>
> Adding netdev to the CC list.
>
> Thanks
> Sridhar
>
> >
> > I do subscribe to this mailing list so, please send your responses to this
> > mail address.
> >
> > Regards,
> >
> > Shirish
> >
> > --------------------------------------------------------------------------------------------------
> > uname -r
> > 2.6.18-91.el5
> >
> > sysctl -a
> >
> > net.ipv4.tcp_rmem = 4096 87380 4194304
> > net.ipv4.tcp_wmem = 4096 16384 4194304
> > net.ipv4.tcp_mem = 98304 131072 196608
> >
> > net.core.rmem_default = 126976
> > net.core.wmem_default = 126976
> > net.core.rmem_max = 131071
> > net.core.wmem_max = 131071
> >
> > net.ipv4.tcp_window_scaling = 1
> > net.ipv4.tcp_timestamps = 1
> > net.ipv4.tcp_moderate_rcvbuf = 1
> >
> >
> > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf
> > 1
> >
> >
> > CIFS VFS: sndbuf 16384 rcvbuf 87380
> >
> > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080
> > stuck for 32 seconds,
> > error: -11
> > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240
> > stuck for 32 seconds,
> > error: -11
> >
> >
> > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720
> > stuck for 64 seconds,
> > error: -11
> >
> > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720
> > stuck for 256 seconds,
> > error: -11
> >
> > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf
> > is 1) but not
> > the socket send buffer size.
> > The send buffer size remains fixed, the auto-tuning for send side is
> > enabled by default,so I do not see it happening here no matter how
> > long the c ode tries to
> > kernel_sendmsg after receiving EAGAIN return code.
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-net" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
Sridhar,
The size of the sends is 56K.
Regards,
Shirish
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket
2008-05-13 13:54 ` Shirish Pargaonkar
@ 2008-05-13 23:48 ` Sridhar Samudrala
2008-05-14 14:22 ` Shirish Pargaonkar
0 siblings, 1 reply; 14+ messages in thread
From: Sridhar Samudrala @ 2008-05-13 23:48 UTC (permalink / raw)
To: Shirish Pargaonkar; +Cc: linux-net, netdev
On Tue, 2008-05-13 at 08:54 -0500, Shirish Pargaonkar wrote:
> On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote:
> > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote:
> > > Hello,
> > >
> > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try,
> > > I still get the same error and do not see the send buffer size of a socket
> > > changing (increasing)
> > >
> > > The initial buffer sizes are 16384 for send side and 87380 for the receive
> > > side but I see receive side buffer tuning but do not see the same with
> > > send side.
> > >
> > > If tcp does not see a need to increase the send buffer size, wonder why I
> > > get EAGAIN error on this non-blocking socket for kernel_sendmsg!
> >
> > I think the send buffer auto-tuning doesn't happen here because there is
> > already congestion window worth of packets sent that are not yet acknowledged.
> > See tcp_should_expand_sndbuf().
> >
> > Also, the comments for tcp_new_space() says that sndbuf expansion does
> > not work well with largesends. What is the size of your sends?
> >
> > Adding netdev to the CC list.
> >
> > Thanks
> > Sridhar
> >
> > >
> > > I do subscribe to this mailing list so, please send your responses to this
> > > mail address.
> > >
> > > Regards,
> > >
> > > Shirish
> > >
> > > --------------------------------------------------------------------------------------------------
> > > uname -r
> > > 2.6.18-91.el5
> > >
> > > sysctl -a
> > >
> > > net.ipv4.tcp_rmem = 4096 87380 4194304
> > > net.ipv4.tcp_wmem = 4096 16384 4194304
> > > net.ipv4.tcp_mem = 98304 131072 196608
> > >
> > > net.core.rmem_default = 126976
> > > net.core.wmem_default = 126976
> > > net.core.rmem_max = 131071
> > > net.core.wmem_max = 131071
> > >
> > > net.ipv4.tcp_window_scaling = 1
> > > net.ipv4.tcp_timestamps = 1
> > > net.ipv4.tcp_moderate_rcvbuf = 1
> > >
> > >
> > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf
> > > 1
> > >
> > >
> > > CIFS VFS: sndbuf 16384 rcvbuf 87380
> > >
> > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080
> > > stuck for 32 seconds,
> > > error: -11
> > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240
> > > stuck for 32 seconds,
> > > error: -11
> > >
> > >
> > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720
> > > stuck for 64 seconds,
> > > error: -11
> > >
> > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720
> > > stuck for 256 seconds,
> > > error: -11
> > >
> > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf
> > > is 1) but not
> > > the socket send buffer size.
> > > The send buffer size remains fixed, the auto-tuning for send side is
> > > enabled by default,so I do not see it happening here no matter how
> > > long the c ode tries to
> > > kernel_sendmsg after receiving EAGAIN return code.
> Sridhar,
>
> The size of the sends is 56K.
As David pointed out, the send size may not be an issue.
When do you see these stalls? Do they happen frequently or only under
stress?
It could be that the receiver is not able to drain the receive queue
causing the send path to be blocked. You could run netstat -tn on
the receiver and take a look at 'Recv-Q' output to see if there is
data pending in the receive queue.
Thanks
Sridhar
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket
2008-05-13 23:48 ` Sridhar Samudrala
@ 2008-05-14 14:22 ` Shirish Pargaonkar
2008-05-14 18:11 ` Shirish Pargaonkar
0 siblings, 1 reply; 14+ messages in thread
From: Shirish Pargaonkar @ 2008-05-14 14:22 UTC (permalink / raw)
To: Sridhar Samudrala; +Cc: linux-net, netdev
On 5/13/08, Sridhar Samudrala <sri@us.ibm.com> wrote:
> On Tue, 2008-05-13 at 08:54 -0500, Shirish Pargaonkar wrote:
> > On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote:
> > > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote:
> > > > Hello,
> > > >
> > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try,
> > > > I still get the same error and do not see the send buffer size of a socket
> > > > changing (increasing)
> > > >
> > > > The initial buffer sizes are 16384 for send side and 87380 for the receive
> > > > side but I see receive side buffer tuning but do not see the same with
> > > > send side.
> > > >
> > > > If tcp does not see a need to increase the send buffer size, wonder why I
> > > > get EAGAIN error on this non-blocking socket for kernel_sendmsg!
> > >
> > > I think the send buffer auto-tuning doesn't happen here because there is
> > > already congestion window worth of packets sent that are not yet acknowledged.
> > > See tcp_should_expand_sndbuf().
> > >
> > > Also, the comments for tcp_new_space() says that sndbuf expansion does
> > > not work well with largesends. What is the size of your sends?
> > >
> > > Adding netdev to the CC list.
> > >
> > > Thanks
> > > Sridhar
> > >
> > > >
> > > > I do subscribe to this mailing list so, please send your responses to this
> > > > mail address.
> > > >
> > > > Regards,
> > > >
> > > > Shirish
> > > >
> > > > --------------------------------------------------------------------------------------------------
> > > > uname -r
> > > > 2.6.18-91.el5
> > > >
> > > > sysctl -a
> > > >
> > > > net.ipv4.tcp_rmem = 4096 87380 4194304
> > > > net.ipv4.tcp_wmem = 4096 16384 4194304
> > > > net.ipv4.tcp_mem = 98304 131072 196608
> > > >
> > > > net.core.rmem_default = 126976
> > > > net.core.wmem_default = 126976
> > > > net.core.rmem_max = 131071
> > > > net.core.wmem_max = 131071
> > > >
> > > > net.ipv4.tcp_window_scaling = 1
> > > > net.ipv4.tcp_timestamps = 1
> > > > net.ipv4.tcp_moderate_rcvbuf = 1
> > > >
> > > >
> > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf
> > > > 1
> > > >
> > > >
> > > > CIFS VFS: sndbuf 16384 rcvbuf 87380
> > > >
> > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080
> > > > stuck for 32 seconds,
> > > > error: -11
> > > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240
> > > > stuck for 32 seconds,
> > > > error: -11
> > > >
> > > >
> > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720
> > > > stuck for 64 seconds,
> > > > error: -11
> > > >
> > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720
> > > > stuck for 256 seconds,
> > > > error: -11
> > > >
> > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf
> > > > is 1) but not
> > > > the socket send buffer size.
> > > > The send buffer size remains fixed, the auto-tuning for send side is
> > > > enabled by default,so I do not see it happening here no matter how
> > > > long the c ode tries to
> > > > kernel_sendmsg after receiving EAGAIN return code.
>
> > Sridhar,
> >
> > The size of the sends is 56K.
>
> As David pointed out, the send size may not be an issue.
> When do you see these stalls? Do they happen frequently or only under
> stress?
>
> It could be that the receiver is not able to drain the receive queue
> causing the send path to be blocked. You could run netstat -tn on
> the receiver and take a look at 'Recv-Q' output to see if there is
> data pending in the receive queue.
>
> Thanks
> Sridhar
>
>
These errors are logged during stress testing, not otherwise.
I am running fsstress on 10 shares mounted on this machine running cifs client
which are exported by a samba server on another machine.
I was running netstat -tn on the machine running samba server in a while loop
in a script untill errors started showing up on the cifs client.
Some of the entries captured in the file are listed below, rest of them
(34345 out of 34356 ) have Recv-Q as 0.
tcp 10080 0 123.456.78.238:445 123.456.78.239:39538
ESTABLISHED
tcp 10080 0 123.456.78.238:445 123.456.78.239:39538
ESTABLISHED
tcp 10080 51 123.456.78.238:445 123.456.78.239:39538
ESTABLISHED
tcp 10983 7200 123.456.78.238:445 123.456.78.239:39538
ESTABLISHED
tcp 11884 10080 123.456.78.238:445 123.456.78.239:39538
ESTABLISHED
tcp 11925 1440 123.456.78.238:445 123.456.78.239:39538
ESTABLISHED
tcp 12116 7200 123.456.78.238:445 123.456.78.239:39538
ESTABLISHED
tcp 12406 0 123.456.78.238:445 123.456.78.239:39538
ESTABLISHED
tcp 290 0 123.456.78.238:445 123.456.78.239:39538
ESTABLISHED
tcp 5028 11627 123.456.78.238:445 123.456.78.239:39538
ESTABLISHED
tcp 8640 51 123.456.78.238:445 123.456.78.239:39538
ESTABLISHED
It is hard to match exact netstat -tn output on the machine running
samba server
with the errors on machine running cifs client but as soon as I saw the errors
appearing on the client, I ran netstat -tn command on the server and found
Recv-Q entry was 0 (may be the Recv-Q entries were processed/cleared by
the samba server by then).
Regards,
Shirish
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket
2008-05-14 14:22 ` Shirish Pargaonkar
@ 2008-05-14 18:11 ` Shirish Pargaonkar
2008-05-14 18:26 ` Jeremy Jackson
2008-05-14 20:23 ` Shirish Pargaonkar
0 siblings, 2 replies; 14+ messages in thread
From: Shirish Pargaonkar @ 2008-05-14 18:11 UTC (permalink / raw)
To: Sridhar Samudrala; +Cc: linux-net, netdev
On 5/14/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote:
> On 5/13/08, Sridhar Samudrala <sri@us.ibm.com> wrote:
> > On Tue, 2008-05-13 at 08:54 -0500, Shirish Pargaonkar wrote:
> > > On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote:
> > > > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote:
> > > > > Hello,
> > > > >
> > > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try,
> > > > > I still get the same error and do not see the send buffer size of a socket
> > > > > changing (increasing)
> > > > >
> > > > > The initial buffer sizes are 16384 for send side and 87380 for the receive
> > > > > side but I see receive side buffer tuning but do not see the same with
> > > > > send side.
> > > > >
> > > > > If tcp does not see a need to increase the send buffer size, wonder why I
> > > > > get EAGAIN error on this non-blocking socket for kernel_sendmsg!
> > > >
> > > > I think the send buffer auto-tuning doesn't happen here because there is
> > > > already congestion window worth of packets sent that are not yet acknowledged.
> > > > See tcp_should_expand_sndbuf().
> > > >
> > > > Also, the comments for tcp_new_space() says that sndbuf expansion does
> > > > not work well with largesends. What is the size of your sends?
> > > >
> > > > Adding netdev to the CC list.
> > > >
> > > > Thanks
> > > > Sridhar
> > > >
> > > > >
> > > > > I do subscribe to this mailing list so, please send your responses to this
> > > > > mail address.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Shirish
> > > > >
> > > > > --------------------------------------------------------------------------------------------------
> > > > > uname -r
> > > > > 2.6.18-91.el5
> > > > >
> > > > > sysctl -a
> > > > >
> > > > > net.ipv4.tcp_rmem = 4096 87380 4194304
> > > > > net.ipv4.tcp_wmem = 4096 16384 4194304
> > > > > net.ipv4.tcp_mem = 98304 131072 196608
> > > > >
> > > > > net.core.rmem_default = 126976
> > > > > net.core.wmem_default = 126976
> > > > > net.core.rmem_max = 131071
> > > > > net.core.wmem_max = 131071
> > > > >
> > > > > net.ipv4.tcp_window_scaling = 1
> > > > > net.ipv4.tcp_timestamps = 1
> > > > > net.ipv4.tcp_moderate_rcvbuf = 1
> > > > >
> > > > >
> > > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf
> > > > > 1
> > > > >
> > > > >
> > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380
> > > > >
> > > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080
> > > > > stuck for 32 seconds,
> > > > > error: -11
> > > > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240
> > > > > stuck for 32 seconds,
> > > > > error: -11
> > > > >
> > > > >
> > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720
> > > > > stuck for 64 seconds,
> > > > > error: -11
> > > > >
> > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720
> > > > > stuck for 256 seconds,
> > > > > error: -11
> > > > >
> > > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf
> > > > > is 1) but not
> > > > > the socket send buffer size.
> > > > > The send buffer size remains fixed, the auto-tuning for send side is
> > > > > enabled by default,so I do not see it happening here no matter how
> > > > > long the c ode tries to
> > > > > kernel_sendmsg after receiving EAGAIN return code.
> >
> > > Sridhar,
> > >
> > > The size of the sends is 56K.
> >
> > As David pointed out, the send size may not be an issue.
> > When do you see these stalls? Do they happen frequently or only under
> > stress?
> >
> > It could be that the receiver is not able to drain the receive queue
> > causing the send path to be blocked. You could run netstat -tn on
> > the receiver and take a look at 'Recv-Q' output to see if there is
> > data pending in the receive queue.
> >
> > Thanks
> > Sridhar
> >
> >
>
> These errors are logged during stress testing, not otherwise.
> I am running fsstress on 10 shares mounted on this machine running cifs client
> which are exported by a samba server on another machine.
>
> I was running netstat -tn on the machine running samba server in a while loop
> in a script untill errors started showing up on the cifs client.
> Some of the entries captured in the file are listed below, rest of them
> (34345 out of 34356 ) have Recv-Q as 0.
>
> tcp 10080 0 123.456.78.238:445 123.456.78.239:39538
> ESTABLISHED
> tcp 10080 0 123.456.78.238:445 123.456.78.239:39538
> ESTABLISHED
> tcp 10080 51 123.456.78.238:445 123.456.78.239:39538
> ESTABLISHED
> tcp 10983 7200 123.456.78.238:445 123.456.78.239:39538
> ESTABLISHED
> tcp 11884 10080 123.456.78.238:445 123.456.78.239:39538
> ESTABLISHED
> tcp 11925 1440 123.456.78.238:445 123.456.78.239:39538
> ESTABLISHED
> tcp 12116 7200 123.456.78.238:445 123.456.78.239:39538
> ESTABLISHED
> tcp 12406 0 123.456.78.238:445 123.456.78.239:39538
> ESTABLISHED
> tcp 290 0 123.456.78.238:445 123.456.78.239:39538
> ESTABLISHED
> tcp 5028 11627 123.456.78.238:445 123.456.78.239:39538
> ESTABLISHED
> tcp 8640 51 123.456.78.238:445 123.456.78.239:39538
> ESTABLISHED
>
> It is hard to match exact netstat -tn output on the machine running
> samba server
> with the errors on machine running cifs client but as soon as I saw the errors
> appearing on the client, I ran netstat -tn command on the server and found
> Recv-Q entry was 0 (may be the Recv-Q entries were processed/cleared by
> the samba server by then).
>
> Regards,
>
> Shirish
>
So I see high count of bytes not copied by the user program but they are
read next time netstat -tn is run (within probably less than a
second), so it is not
as if the samba server is not reading off data for long periods right?
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State
tcp 0 164 123.456.78.238:445 123.456.78.239:39538
ESTABLISHED
tcp 0 0 ::ffff:123.456.78.238:22
::ffff:123.456.78.190:34532 ESTABLISHED
tcp 0 0 ::ffff:123.456.78.238:22
::ffff:123.456.78.135:50328 ESTABLISHED
tcp 0 0 ::ffff:123.456.78.238:22
::ffff:123.456.78.135:50333 ESTABLISHED
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State
tcp 12406 0 123.456.78.238:445 123.456.78.239:39538
ESTABLISHED
tcp 0 0 ::ffff:123.456.78.238:22
::ffff:123.456.78.190:34532 ESTABLISHED
tcp 0 0 ::ffff:123.456.78.238:22
::ffff:123.456.78.135:50328 ESTABLISHED
tcp 0 0 ::ffff:123.456.78.238:22
::ffff:123.456.78.135:50333 ESTABLISHED
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State
tcp 0 0 123.456.78.238:445 123.456.78.239:39538
ESTABLISHED
tcp 0 0 ::ffff:123.456.78.238:22
::ffff:123.456.78.190:34532 ESTABLISHED
tcp 0 0 ::ffff:123.456.78.238:22
::ffff:123.456.78.135:50328 ESTABLISHED
tcp 0 0 ::ffff:123.456.78.238:22
::ffff:123.456.78.135:50333 ESTABLISHED
So are these occasional and short-lived spikes in unread data on the
server side
causing from send buffer from increasing on the client side?
Regards,
Shirish
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket
2008-05-14 18:11 ` Shirish Pargaonkar
@ 2008-05-14 18:26 ` Jeremy Jackson
2008-05-14 18:37 ` Shirish Pargaonkar
2008-05-14 20:23 ` Shirish Pargaonkar
1 sibling, 1 reply; 14+ messages in thread
From: Jeremy Jackson @ 2008-05-14 18:26 UTC (permalink / raw)
To: Shirish Pargaonkar; +Cc: Sridhar Samudrala, linux-net, netdev
Sorry if this has been covered already, but doesn't Samba set the write
buffer size, thereby disabling autotuning?
--
Jeremy Jackson
Coplanar Networks
(519)489-4903
http://www.coplanar.net
jerj@coplanar.net
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket
2008-05-14 18:26 ` Jeremy Jackson
@ 2008-05-14 18:37 ` Shirish Pargaonkar
0 siblings, 0 replies; 14+ messages in thread
From: Shirish Pargaonkar @ 2008-05-14 18:37 UTC (permalink / raw)
To: Jeremy Jackson; +Cc: Sridhar Samudrala, linux-net, netdev
On 5/14/08, Jeremy Jackson <jerj@coplanar.net> wrote:
> Sorry if this has been covered already, but doesn't Samba set the write
> buffer size, thereby disabling autotuning?
>
> --
> Jeremy Jackson
> Coplanar Networks
> (519)489-4903
> http://www.coplanar.net
> jerj@coplanar.net
>
>
Jeremy,
I would like auto-tuning to happen on the machine running cifs/samba client.
That is where the errors are logged. The auto-tune happens on the client side,
just not good enough to ward off the delays (and subsequent errors logged
by cifs client and reconnects initated by cifs client to samba server
running on other
machine.) for reasons.
Regards,
Shirish
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket
2008-05-14 18:11 ` Shirish Pargaonkar
2008-05-14 18:26 ` Jeremy Jackson
@ 2008-05-14 20:23 ` Shirish Pargaonkar
1 sibling, 0 replies; 14+ messages in thread
From: Shirish Pargaonkar @ 2008-05-14 20:23 UTC (permalink / raw)
To: Sridhar Samudrala; +Cc: linux-net, netdev
On 5/14/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote:
> On 5/14/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote:
> > On 5/13/08, Sridhar Samudrala <sri@us.ibm.com> wrote:
> > > On Tue, 2008-05-13 at 08:54 -0500, Shirish Pargaonkar wrote:
> > > > On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote:
> > > > > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote:
> > > > > > Hello,
> > > > > >
> > > > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try,
> > > > > > I still get the same error and do not see the send buffer size of a socket
> > > > > > changing (increasing)
> > > > > >
> > > > > > The initial buffer sizes are 16384 for send side and 87380 for the receive
> > > > > > side but I see receive side buffer tuning but do not see the same with
> > > > > > send side.
> > > > > >
> > > > > > If tcp does not see a need to increase the send buffer size, wonder why I
> > > > > > get EAGAIN error on this non-blocking socket for kernel_sendmsg!
> > > > >
> > > > > I think the send buffer auto-tuning doesn't happen here because there is
> > > > > already congestion window worth of packets sent that are not yet acknowledged.
> > > > > See tcp_should_expand_sndbuf().
> > > > >
> > > > > Also, the comments for tcp_new_space() says that sndbuf expansion does
> > > > > not work well with largesends. What is the size of your sends?
> > > > >
> > > > > Adding netdev to the CC list.
> > > > >
> > > > > Thanks
> > > > > Sridhar
> > > > >
> > > > > >
> > > > > > I do subscribe to this mailing list so, please send your responses to this
> > > > > > mail address.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Shirish
> > > > > >
> > > > > > --------------------------------------------------------------------------------------------------
> > > > > > uname -r
> > > > > > 2.6.18-91.el5
> > > > > >
> > > > > > sysctl -a
> > > > > >
> > > > > > net.ipv4.tcp_rmem = 4096 87380 4194304
> > > > > > net.ipv4.tcp_wmem = 4096 16384 4194304
> > > > > > net.ipv4.tcp_mem = 98304 131072 196608
> > > > > >
> > > > > > net.core.rmem_default = 126976
> > > > > > net.core.wmem_default = 126976
> > > > > > net.core.rmem_max = 131071
> > > > > > net.core.wmem_max = 131071
> > > > > >
> > > > > > net.ipv4.tcp_window_scaling = 1
> > > > > > net.ipv4.tcp_timestamps = 1
> > > > > > net.ipv4.tcp_moderate_rcvbuf = 1
> > > > > >
> > > > > >
> > > > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf
> > > > > > 1
> > > > > >
> > > > > >
> > > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380
> > > > > >
> > > > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080
> > > > > > stuck for 32 seconds,
> > > > > > error: -11
> > > > > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240
> > > > > > stuck for 32 seconds,
> > > > > > error: -11
> > > > > >
> > > > > >
> > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720
> > > > > > stuck for 64 seconds,
> > > > > > error: -11
> > > > > >
> > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720
> > > > > > stuck for 256 seconds,
> > > > > > error: -11
> > > > > >
> > > > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf
> > > > > > is 1) but not
> > > > > > the socket send buffer size.
> > > > > > The send buffer size remains fixed, the auto-tuning for send side is
> > > > > > enabled by default,so I do not see it happening here no matter how
> > > > > > long the c ode tries to
> > > > > > kernel_sendmsg after receiving EAGAIN return code.
> > >
> > > > Sridhar,
> > > >
> > > > The size of the sends is 56K.
> > >
> > > As David pointed out, the send size may not be an issue.
> > > When do you see these stalls? Do they happen frequently or only under
> > > stress?
> > >
> > > It could be that the receiver is not able to drain the receive queue
> > > causing the send path to be blocked. You could run netstat -tn on
> > > the receiver and take a look at 'Recv-Q' output to see if there is
> > > data pending in the receive queue.
> > >
> > > Thanks
> > > Sridhar
> > >
> > >
> >
> > These errors are logged during stress testing, not otherwise.
> > I am running fsstress on 10 shares mounted on this machine running cifs client
> > which are exported by a samba server on another machine.
> >
> > I was running netstat -tn on the machine running samba server in a while loop
> > in a script untill errors started showing up on the cifs client.
> > Some of the entries captured in the file are listed below, rest of them
> > (34345 out of 34356 ) have Recv-Q as 0.
> >
> > tcp 10080 0 123.456.78.238:445 123.456.78.239:39538
> > ESTABLISHED
> > tcp 10080 0 123.456.78.238:445 123.456.78.239:39538
> > ESTABLISHED
> > tcp 10080 51 123.456.78.238:445 123.456.78.239:39538
> > ESTABLISHED
> > tcp 10983 7200 123.456.78.238:445 123.456.78.239:39538
> > ESTABLISHED
> > tcp 11884 10080 123.456.78.238:445 123.456.78.239:39538
> > ESTABLISHED
> > tcp 11925 1440 123.456.78.238:445 123.456.78.239:39538
> > ESTABLISHED
> > tcp 12116 7200 123.456.78.238:445 123.456.78.239:39538
> > ESTABLISHED
> > tcp 12406 0 123.456.78.238:445 123.456.78.239:39538
> > ESTABLISHED
> > tcp 290 0 123.456.78.238:445 123.456.78.239:39538
> > ESTABLISHED
> > tcp 5028 11627 123.456.78.238:445 123.456.78.239:39538
> > ESTABLISHED
> > tcp 8640 51 123.456.78.238:445 123.456.78.239:39538
> > ESTABLISHED
> >
> > It is hard to match exact netstat -tn output on the machine running
> > samba server
> > with the errors on machine running cifs client but as soon as I saw the errors
> > appearing on the client, I ran netstat -tn command on the server and found
> > Recv-Q entry was 0 (may be the Recv-Q entries were processed/cleared by
> > the samba server by then).
> >
> > Regards,
> >
> > Shirish
> >
>
> So I see high count of bytes not copied by the user program but they are
> read next time netstat -tn is run (within probably less than a
> second), so it is not
> as if the samba server is not reading off data for long periods right?
>
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address
> State
> tcp 0 164 123.456.78.238:445 123.456.78.239:39538
> ESTABLISHED
> tcp 0 0 ::ffff:123.456.78.238:22
> ::ffff:123.456.78.190:34532 ESTABLISHED
> tcp 0 0 ::ffff:123.456.78.238:22
> ::ffff:123.456.78.135:50328 ESTABLISHED
> tcp 0 0 ::ffff:123.456.78.238:22
> ::ffff:123.456.78.135:50333 ESTABLISHED
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address
> State
> tcp 12406 0 123.456.78.238:445 123.456.78.239:39538
> ESTABLISHED
> tcp 0 0 ::ffff:123.456.78.238:22
> ::ffff:123.456.78.190:34532 ESTABLISHED
> tcp 0 0 ::ffff:123.456.78.238:22
> ::ffff:123.456.78.135:50328 ESTABLISHED
> tcp 0 0 ::ffff:123.456.78.238:22
> ::ffff:123.456.78.135:50333 ESTABLISHED
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address
> State
> tcp 0 0 123.456.78.238:445 123.456.78.239:39538
> ESTABLISHED
> tcp 0 0 ::ffff:123.456.78.238:22
> ::ffff:123.456.78.190:34532 ESTABLISHED
> tcp 0 0 ::ffff:123.456.78.238:22
> ::ffff:123.456.78.135:50328 ESTABLISHED
> tcp 0 0 ::ffff:123.456.78.238:22
> ::ffff:123.456.78.135:50333 ESTABLISHED
>
> So are these occasional and short-lived spikes in unread data on the
> server side
> causing from send buffer from increasing on the client side?
>
> Regards,
>
> Shirish
>
I called kernel_getsockopt after receiving EAGAIN for 15 seconds for
kernel_sendmsg.
Not sure whether it would be useful. lds is last data sent and ldr is
last data received.
CIFS VFS: sends on sock 000000000eea1400, sndbuf 34776, rcvbuf 203520
of length 57408 stuck for 15 seconds, error: -11
CIFS VFS: smb_send2 lds 0, ldr 20, cwnd 9, send ssthresh 100, rcv
ssthresh 153258
rtt 18750 rttvar 7500 unacked 7 sacked 0 lost 0, retrans 0, fackets 0
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket
2008-05-13 0:47 ` autotuning of send buffer size of a socket Sridhar Samudrala
2008-05-13 0:51 ` David Miller
2008-05-13 13:54 ` Shirish Pargaonkar
@ 2008-05-15 4:18 ` Shirish Pargaonkar
2008-05-15 15:00 ` Shirish Pargaonkar
2 siblings, 1 reply; 14+ messages in thread
From: Shirish Pargaonkar @ 2008-05-15 4:18 UTC (permalink / raw)
To: Sridhar Samudrala; +Cc: linux-net, netdev
On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote:
> On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote:
> > Hello,
> >
> > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try,
> > I still get the same error and do not see the send buffer size of a socket
> > changing (increasing)
> >
> > The initial buffer sizes are 16384 for send side and 87380 for the receive
> > side but I see receive side buffer tuning but do not see the same with
> > send side.
> >
> > If tcp does not see a need to increase the send buffer size, wonder why I
> > get EAGAIN error on this non-blocking socket for kernel_sendmsg!
>
> I think the send buffer auto-tuning doesn't happen here because there is
> already congestion window worth of packets sent that are not yet acknowledged.
> See tcp_should_expand_sndbuf().
Sridhar,
The unacked (packets_out) is 7 and snd_cwnd is 9, so that should not be
the case for tcp_should_expand_sndbuf to return 0 right?
>
> Also, the comments for tcp_new_space() says that sndbuf expansion does
> not work well with largesends. What is the size of your sends?
>
> Adding netdev to the CC list.
>
> Thanks
> Sridhar
>
> >
> > I do subscribe to this mailing list so, please send your responses to this
> > mail address.
> >
> > Regards,
> >
> > Shirish
> >
> > --------------------------------------------------------------------------------------------------
> > uname -r
> > 2.6.18-91.el5
> >
> > sysctl -a
> >
> > net.ipv4.tcp_rmem = 4096 87380 4194304
> > net.ipv4.tcp_wmem = 4096 16384 4194304
> > net.ipv4.tcp_mem = 98304 131072 196608
> >
> > net.core.rmem_default = 126976
> > net.core.wmem_default = 126976
> > net.core.rmem_max = 131071
> > net.core.wmem_max = 131071
> >
> > net.ipv4.tcp_window_scaling = 1
> > net.ipv4.tcp_timestamps = 1
> > net.ipv4.tcp_moderate_rcvbuf = 1
> >
> >
> > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf
> > 1
> >
> >
> > CIFS VFS: sndbuf 16384 rcvbuf 87380
> >
> > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080
> > stuck for 32 seconds,
> > error: -11
> > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240
> > stuck for 32 seconds,
> > error: -11
> >
> >
> > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720
> > stuck for 64 seconds,
> > error: -11
> >
> > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720
> > stuck for 256 seconds,
> > error: -11
> >
> > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf
> > is 1) but not
> > the socket send buffer size.
> > The send buffer size remains fixed, the auto-tuning for send side is
> > enabled by default,so I do not see it happening here no matter how
> > long the c ode tries to
> > kernel_sendmsg after receiving EAGAIN return code.
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-net" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket
2008-05-15 4:18 ` Shirish Pargaonkar
@ 2008-05-15 15:00 ` Shirish Pargaonkar
2008-05-15 17:40 ` Sridhar Samudrala
0 siblings, 1 reply; 14+ messages in thread
From: Shirish Pargaonkar @ 2008-05-15 15:00 UTC (permalink / raw)
To: Sridhar Samudrala; +Cc: linux-net, netdev
On 5/14/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote:
> On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote:
> > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote:
> > > Hello,
> > >
> > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try,
> > > I still get the same error and do not see the send buffer size of a socket
> > > changing (increasing)
> > >
> > > The initial buffer sizes are 16384 for send side and 87380 for the receive
> > > side but I see receive side buffer tuning but do not see the same with
> > > send side.
> > >
> > > If tcp does not see a need to increase the send buffer size, wonder why I
> > > get EAGAIN error on this non-blocking socket for kernel_sendmsg!
> >
> > I think the send buffer auto-tuning doesn't happen here because there is
> > already congestion window worth of packets sent that are not yet acknowledged.
> > See tcp_should_expand_sndbuf().
>
> Sridhar,
>
> The unacked (packets_out) is 7 and snd_cwnd is 9, so that should not be
> the case for tcp_should_expand_sndbuf to return 0 right?
>
> >
> > Also, the comments for tcp_new_space() says that sndbuf expansion does
> > not work well with largesends. What is the size of your sends?
> >
> > Adding netdev to the CC list.
> >
> > Thanks
> > Sridhar
> >
> > >
> > > I do subscribe to this mailing list so, please send your responses to this
> > > mail address.
> > >
> > > Regards,
> > >
> > > Shirish
> > >
> > > --------------------------------------------------------------------------------------------------
> > > uname -r
> > > 2.6.18-91.el5
> > >
> > > sysctl -a
> > >
> > > net.ipv4.tcp_rmem = 4096 87380 4194304
> > > net.ipv4.tcp_wmem = 4096 16384 4194304
> > > net.ipv4.tcp_mem = 98304 131072 196608
> > >
> > > net.core.rmem_default = 126976
> > > net.core.wmem_default = 126976
> > > net.core.rmem_max = 131071
> > > net.core.wmem_max = 131071
> > >
> > > net.ipv4.tcp_window_scaling = 1
> > > net.ipv4.tcp_timestamps = 1
> > > net.ipv4.tcp_moderate_rcvbuf = 1
> > >
> > >
> > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf
> > > 1
> > >
> > >
> > > CIFS VFS: sndbuf 16384 rcvbuf 87380
> > >
> > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080
> > > stuck for 32 seconds,
> > > error: -11
> > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240
> > > stuck for 32 seconds,
> > > error: -11
> > >
> > >
> > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720
> > > stuck for 64 seconds,
> > > error: -11
> > >
> > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720
> > > stuck for 256 seconds,
> > > error: -11
> > >
> > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf
> > > is 1) but not
> > > the socket send buffer size.
> > > The send buffer size remains fixed, the auto-tuning for send side is
> > > enabled by default,so I do not see it happening here no matter how
> > > long the c ode tries to
> > > kernel_sendmsg after receiving EAGAIN return code.
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-net" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> >
>
I put some printk in tcp.c (in function tcp_sendmsg)
sndbuf grows from 16384 to 34776 but never beyond it.
CIFS VFS: sndbuf 16384 rcvbuf 87380 rcvtimeo 0x7fffffffffffffff
!sk_stream_memory_free queued 18288, sndbuf 16384
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 28448, sndbuf 27048
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 28448, sndbuf 27048
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 32512, sndbuf 30912
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 32512, sndbuf 30912
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 32512, sndbuf 30912
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 32512, sndbuf 30912
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 32512, sndbuf 30912
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 32512, sndbuf 30912
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 32512, sndbuf 30912
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 32512, sndbuf 30912
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 32512, sndbuf 30912
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 32512, sndbuf 30912
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 32512, sndbuf 30912
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 32512, sndbuf 30912
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 32512, sndbuf 30912
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 32512, sndbuf 30912
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 36576, sndbuf 34776
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 36576, sndbuf 34776
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 36576, sndbuf 34776
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 36576, sndbuf 34776
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 36576, sndbuf 34776
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 36576, sndbuf 34776
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 36576, sndbuf 34776
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 36576, sndbuf 34776
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 36576, sndbuf 34776
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 36576, sndbuf 34776
sk_stream_wait_memory with 0 returned -11
!sk_stream_memory_free queued 36576, sndbuf 34776
sk_stream_wait_memory with 0 returned -11
and so on and the sndbuf does not grow beyond 34776
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket
2008-05-15 15:00 ` Shirish Pargaonkar
@ 2008-05-15 17:40 ` Sridhar Samudrala
2008-05-15 18:03 ` Shirish Pargaonkar
0 siblings, 1 reply; 14+ messages in thread
From: Sridhar Samudrala @ 2008-05-15 17:40 UTC (permalink / raw)
To: Shirish Pargaonkar; +Cc: linux-net, netdev
On Thu, 2008-05-15 at 10:00 -0500, Shirish Pargaonkar wrote:
> On 5/14/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote:
> > On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote:
> > > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote:
> > > > Hello,
> > > >
> > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try,
> > > > I still get the same error and do not see the send buffer size of a socket
> > > > changing (increasing)
> > > >
> > > > The initial buffer sizes are 16384 for send side and 87380 for the receive
> > > > side but I see receive side buffer tuning but do not see the same with
> > > > send side.
> > > >
> > > > If tcp does not see a need to increase the send buffer size, wonder why I
> > > > get EAGAIN error on this non-blocking socket for kernel_sendmsg!
> > >
> > > I think the send buffer auto-tuning doesn't happen here because there is
> > > already congestion window worth of packets sent that are not yet acknowledged.
> > > See tcp_should_expand_sndbuf().
> >
> > Sridhar,
> >
> > The unacked (packets_out) is 7 and snd_cwnd is 9, so that should not be
> > the case for tcp_should_expand_sndbuf to return 0 right?
It looks like sndbuf expansion via tcp_should_expand_sndbuf() happens
only in response to acks/data from the receiver.
tcp_rcv_established/tcp_rcv_state_process
tcp_data_snd_check
tcp_check_space
tcp_new_space
tcp_should_expand_sndbuf
auto-tuning doesn't increase sndbuf when trying to send more data.
> >
> > >
> > > Also, the comments for tcp_new_space() says that sndbuf expansion does
> > > not work well with largesends. What is the size of your sends?
> > >
> > > Adding netdev to the CC list.
> > >
> > > Thanks
> > > Sridhar
> > >
> > > >
> > > > I do subscribe to this mailing list so, please send your responses to this
> > > > mail address.
> > > >
> > > > Regards,
> > > >
> > > > Shirish
> > > >
> > > > --------------------------------------------------------------------------------------------------
> > > > uname -r
> > > > 2.6.18-91.el5
> > > >
> > > > sysctl -a
> > > >
> > > > net.ipv4.tcp_rmem = 4096 87380 4194304
> > > > net.ipv4.tcp_wmem = 4096 16384 4194304
> > > > net.ipv4.tcp_mem = 98304 131072 196608
> > > >
> > > > net.core.rmem_default = 126976
> > > > net.core.wmem_default = 126976
> > > > net.core.rmem_max = 131071
> > > > net.core.wmem_max = 131071
> > > >
> > > > net.ipv4.tcp_window_scaling = 1
> > > > net.ipv4.tcp_timestamps = 1
> > > > net.ipv4.tcp_moderate_rcvbuf = 1
> > > >
> > > >
> > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf
> > > > 1
> > > >
> > > >
> > > > CIFS VFS: sndbuf 16384 rcvbuf 87380
> > > >
> > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080
> > > > stuck for 32 seconds,
> > > > error: -11
> > > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240
> > > > stuck for 32 seconds,
> > > > error: -11
> > > >
> > > >
> > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720
> > > > stuck for 64 seconds,
> > > > error: -11
> > > >
> > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720
> > > > stuck for 256 seconds,
> > > > error: -11
> > > >
> > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf
> > > > is 1) but not
> > > > the socket send buffer size.
> > > > The send buffer size remains fixed, the auto-tuning for send side is
> > > > enabled by default,so I do not see it happening here no matter how
> > > > long the c ode tries to
> > > > kernel_sendmsg after receiving EAGAIN return code.
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-net" in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > >
> > >
> >
>
> I put some printk in tcp.c (in function tcp_sendmsg)
> sndbuf grows from 16384 to 34776 but never beyond it.
>
>
> CIFS VFS: sndbuf 16384 rcvbuf 87380 rcvtimeo 0x7fffffffffffffff
>
> !sk_stream_memory_free queued 18288, sndbuf 16384
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 28448, sndbuf 27048
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 28448, sndbuf 27048
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 32512, sndbuf 30912
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 32512, sndbuf 30912
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 32512, sndbuf 30912
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 32512, sndbuf 30912
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 32512, sndbuf 30912
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 32512, sndbuf 30912
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 32512, sndbuf 30912
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 32512, sndbuf 30912
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 32512, sndbuf 30912
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 32512, sndbuf 30912
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 32512, sndbuf 30912
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 32512, sndbuf 30912
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 32512, sndbuf 30912
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 32512, sndbuf 30912
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 36576, sndbuf 34776
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 36576, sndbuf 34776
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 36576, sndbuf 34776
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 36576, sndbuf 34776
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 36576, sndbuf 34776
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 36576, sndbuf 34776
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 36576, sndbuf 34776
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 36576, sndbuf 34776
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 36576, sndbuf 34776
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 36576, sndbuf 34776
> sk_stream_wait_memory with 0 returned -11
> !sk_stream_memory_free queued 36576, sndbuf 34776
> sk_stream_wait_memory with 0 returned -11
>
> and so on and the sndbuf does not grow beyond 34776
So there is outstanding data(sk_wmem_queued) that is not getting
acked.
If you set the sndbuf manually to a higher value, does it solve
the problem or only delay the stalls?
Thanks
Sridhar
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket
2008-05-15 17:40 ` Sridhar Samudrala
@ 2008-05-15 18:03 ` Shirish Pargaonkar
2008-05-15 18:49 ` Shirish Pargaonkar
0 siblings, 1 reply; 14+ messages in thread
From: Shirish Pargaonkar @ 2008-05-15 18:03 UTC (permalink / raw)
To: Sridhar Samudrala; +Cc: linux-net, netdev
On 5/15/08, Sridhar Samudrala <sri@us.ibm.com> wrote:
> On Thu, 2008-05-15 at 10:00 -0500, Shirish Pargaonkar wrote:
> > On 5/14/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote:
> > > On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote:
> > > > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote:
> > > > > Hello,
> > > > >
> > > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try,
> > > > > I still get the same error and do not see the send buffer size of a socket
> > > > > changing (increasing)
> > > > >
> > > > > The initial buffer sizes are 16384 for send side and 87380 for the receive
> > > > > side but I see receive side buffer tuning but do not see the same with
> > > > > send side.
> > > > >
> > > > > If tcp does not see a need to increase the send buffer size, wonder why I
> > > > > get EAGAIN error on this non-blocking socket for kernel_sendmsg!
> > > >
> > > > I think the send buffer auto-tuning doesn't happen here because there is
> > > > already congestion window worth of packets sent that are not yet acknowledged.
> > > > See tcp_should_expand_sndbuf().
> > >
> > > Sridhar,
> > >
> > > The unacked (packets_out) is 7 and snd_cwnd is 9, so that should not be
> > > the case for tcp_should_expand_sndbuf to return 0 right?
>
> It looks like sndbuf expansion via tcp_should_expand_sndbuf() happens
> only in response to acks/data from the receiver.
> tcp_rcv_established/tcp_rcv_state_process
> tcp_data_snd_check
> tcp_check_space
> tcp_new_space
> tcp_should_expand_sndbuf
> auto-tuning doesn't increase sndbuf when trying to send more data.
>
> > >
> > > >
> > > > Also, the comments for tcp_new_space() says that sndbuf expansion does
> > > > not work well with largesends. What is the size of your sends?
> > > >
> > > > Adding netdev to the CC list.
> > > >
> > > > Thanks
> > > > Sridhar
> > > >
> > > > >
> > > > > I do subscribe to this mailing list so, please send your responses to this
> > > > > mail address.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Shirish
> > > > >
> > > > > --------------------------------------------------------------------------------------------------
> > > > > uname -r
> > > > > 2.6.18-91.el5
> > > > >
> > > > > sysctl -a
> > > > >
> > > > > net.ipv4.tcp_rmem = 4096 87380 4194304
> > > > > net.ipv4.tcp_wmem = 4096 16384 4194304
> > > > > net.ipv4.tcp_mem = 98304 131072 196608
> > > > >
> > > > > net.core.rmem_default = 126976
> > > > > net.core.wmem_default = 126976
> > > > > net.core.rmem_max = 131071
> > > > > net.core.wmem_max = 131071
> > > > >
> > > > > net.ipv4.tcp_window_scaling = 1
> > > > > net.ipv4.tcp_timestamps = 1
> > > > > net.ipv4.tcp_moderate_rcvbuf = 1
> > > > >
> > > > >
> > > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf
> > > > > 1
> > > > >
> > > > >
> > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380
> > > > >
> > > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080
> > > > > stuck for 32 seconds,
> > > > > error: -11
> > > > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240
> > > > > stuck for 32 seconds,
> > > > > error: -11
> > > > >
> > > > >
> > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720
> > > > > stuck for 64 seconds,
> > > > > error: -11
> > > > >
> > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720
> > > > > stuck for 256 seconds,
> > > > > error: -11
> > > > >
> > > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf
> > > > > is 1) but not
> > > > > the socket send buffer size.
> > > > > The send buffer size remains fixed, the auto-tuning for send side is
> > > > > enabled by default,so I do not see it happening here no matter how
> > > > > long the c ode tries to
> > > > > kernel_sendmsg after receiving EAGAIN return code.
> > > > > --
> > > > > To unsubscribe from this list: send the line "unsubscribe linux-net" in
> > > > > the body of a message to majordomo@vger.kernel.org
> > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > > >
> > > >
> > >
> >
> > I put some printk in tcp.c (in function tcp_sendmsg)
> > sndbuf grows from 16384 to 34776 but never beyond it.
> >
> >
> > CIFS VFS: sndbuf 16384 rcvbuf 87380 rcvtimeo 0x7fffffffffffffff
> >
> > !sk_stream_memory_free queued 18288, sndbuf 16384
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 28448, sndbuf 27048
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 28448, sndbuf 27048
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 32512, sndbuf 30912
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 32512, sndbuf 30912
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 32512, sndbuf 30912
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 32512, sndbuf 30912
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 32512, sndbuf 30912
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 32512, sndbuf 30912
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 32512, sndbuf 30912
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 32512, sndbuf 30912
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 32512, sndbuf 30912
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 32512, sndbuf 30912
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 32512, sndbuf 30912
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 32512, sndbuf 30912
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 32512, sndbuf 30912
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 32512, sndbuf 30912
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 36576, sndbuf 34776
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 36576, sndbuf 34776
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 36576, sndbuf 34776
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 36576, sndbuf 34776
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 36576, sndbuf 34776
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 36576, sndbuf 34776
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 36576, sndbuf 34776
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 36576, sndbuf 34776
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 36576, sndbuf 34776
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 36576, sndbuf 34776
> > sk_stream_wait_memory with 0 returned -11
> > !sk_stream_memory_free queued 36576, sndbuf 34776
> > sk_stream_wait_memory with 0 returned -11
> >
> > and so on and the sndbuf does not grow beyond 34776
>
> So there is outstanding data(sk_wmem_queued) that is not getting
> acked.
> If you set the sndbuf manually to a higher value, does it solve
> the problem or only delay the stalls?
>
> Thanks
> Sridhar
>
>
Sridhar,
snd_cwnd goes from 6 to 7 to 8 to 9 and is capped at that value. That is how
sndbuf grows upto but does not grow beyond 34776.
If I set the sndbuf manully, it just delays stalls. I have gone as high as 1MB
of send size and that prevents from logging these errors longer but
eventually it does.
Regards,
Shirish
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: autotuning of send buffer size of a socket
2008-05-15 18:03 ` Shirish Pargaonkar
@ 2008-05-15 18:49 ` Shirish Pargaonkar
0 siblings, 0 replies; 14+ messages in thread
From: Shirish Pargaonkar @ 2008-05-15 18:49 UTC (permalink / raw)
To: Sridhar Samudrala; +Cc: linux-net, netdev
On 5/15/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote:
> On 5/15/08, Sridhar Samudrala <sri@us.ibm.com> wrote:
> > On Thu, 2008-05-15 at 10:00 -0500, Shirish Pargaonkar wrote:
> > > On 5/14/08, Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote:
> > > > On 5/12/08, Sridhar Samudrala <sri@us.ibm.com> wrote:
> > > > > On Mon, 2008-05-12 at 14:00 -0500, Shirish Pargaonkar wrote:
> > > > > > Hello,
> > > > > >
> > > > > > kernel_sendmsg fails with error EAGAIN, yet I no matter how long I try,
> > > > > > I still get the same error and do not see the send buffer size of a socket
> > > > > > changing (increasing)
> > > > > >
> > > > > > The initial buffer sizes are 16384 for send side and 87380 for the receive
> > > > > > side but I see receive side buffer tuning but do not see the same with
> > > > > > send side.
> > > > > >
> > > > > > If tcp does not see a need to increase the send buffer size, wonder why I
> > > > > > get EAGAIN error on this non-blocking socket for kernel_sendmsg!
> > > > >
> > > > > I think the send buffer auto-tuning doesn't happen here because there is
> > > > > already congestion window worth of packets sent that are not yet acknowledged.
> > > > > See tcp_should_expand_sndbuf().
> > > >
> > > > Sridhar,
> > > >
> > > > The unacked (packets_out) is 7 and snd_cwnd is 9, so that should not be
> > > > the case for tcp_should_expand_sndbuf to return 0 right?
> >
> > It looks like sndbuf expansion via tcp_should_expand_sndbuf() happens
> > only in response to acks/data from the receiver.
> > tcp_rcv_established/tcp_rcv_state_process
> > tcp_data_snd_check
> > tcp_check_space
> > tcp_new_space
> > tcp_should_expand_sndbuf
> > auto-tuning doesn't increase sndbuf when trying to send more data.
> >
> > > >
> > > > >
> > > > > Also, the comments for tcp_new_space() says that sndbuf expansion does
> > > > > not work well with largesends. What is the size of your sends?
> > > > >
> > > > > Adding netdev to the CC list.
> > > > >
> > > > > Thanks
> > > > > Sridhar
> > > > >
> > > > > >
> > > > > > I do subscribe to this mailing list so, please send your responses to this
> > > > > > mail address.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Shirish
> > > > > >
> > > > > > --------------------------------------------------------------------------------------------------
> > > > > > uname -r
> > > > > > 2.6.18-91.el5
> > > > > >
> > > > > > sysctl -a
> > > > > >
> > > > > > net.ipv4.tcp_rmem = 4096 87380 4194304
> > > > > > net.ipv4.tcp_wmem = 4096 16384 4194304
> > > > > > net.ipv4.tcp_mem = 98304 131072 196608
> > > > > >
> > > > > > net.core.rmem_default = 126976
> > > > > > net.core.wmem_default = 126976
> > > > > > net.core.rmem_max = 131071
> > > > > > net.core.wmem_max = 131071
> > > > > >
> > > > > > net.ipv4.tcp_window_scaling = 1
> > > > > > net.ipv4.tcp_timestamps = 1
> > > > > > net.ipv4.tcp_moderate_rcvbuf = 1
> > > > > >
> > > > > >
> > > > > > cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf
> > > > > > 1
> > > > > >
> > > > > >
> > > > > > CIFS VFS: sndbuf 16384 rcvbuf 87380
> > > > > >
> > > > > > CIFS VFS: sends on sock 0000000009903100, sendbuf 34776, rcvbuf 190080
> > > > > > stuck for 32 seconds,
> > > > > > error: -11
> > > > > > CIFS VFS: sends on sock 0000000009903a00, sndbuf 34776, rcvbuf 138240
> > > > > > stuck for 32 seconds,
> > > > > > error: -11
> > > > > >
> > > > > >
> > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 126720
> > > > > > stuck for 64 seconds,
> > > > > > error: -11
> > > > > >
> > > > > > CIFS VFS: sends on sock 0000000009903100, sndbuf 34776, rcvbuf 222720
> > > > > > stuck for 256 seconds,
> > > > > > error: -11
> > > > > >
> > > > > > I see the socket receive buffer size fluctuating (tcp_moderate_rcvbuf
> > > > > > is 1) but not
> > > > > > the socket send buffer size.
> > > > > > The send buffer size remains fixed, the auto-tuning for send side is
> > > > > > enabled by default,so I do not see it happening here no matter how
> > > > > > long the c ode tries to
> > > > > > kernel_sendmsg after receiving EAGAIN return code.
> > > > > > --
> > > > > > To unsubscribe from this list: send the line "unsubscribe linux-net" in
> > > > > > the body of a message to majordomo@vger.kernel.org
> > > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > > > >
> > > > >
> > > >
> > >
> > > I put some printk in tcp.c (in function tcp_sendmsg)
> > > sndbuf grows from 16384 to 34776 but never beyond it.
> > >
> > >
> > > CIFS VFS: sndbuf 16384 rcvbuf 87380 rcvtimeo 0x7fffffffffffffff
> > >
> > > !sk_stream_memory_free queued 18288, sndbuf 16384
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 28448, sndbuf 27048
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 28448, sndbuf 27048
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 32512, sndbuf 30912
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 32512, sndbuf 30912
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 32512, sndbuf 30912
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 32512, sndbuf 30912
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 32512, sndbuf 30912
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 32512, sndbuf 30912
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 32512, sndbuf 30912
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 32512, sndbuf 30912
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 32512, sndbuf 30912
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 32512, sndbuf 30912
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 32512, sndbuf 30912
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 32512, sndbuf 30912
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 32512, sndbuf 30912
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 32512, sndbuf 30912
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 36576, sndbuf 34776
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 36576, sndbuf 34776
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 36576, sndbuf 34776
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 36576, sndbuf 34776
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 36576, sndbuf 34776
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 36576, sndbuf 34776
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 36576, sndbuf 34776
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 36576, sndbuf 34776
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 36576, sndbuf 34776
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 36576, sndbuf 34776
> > > sk_stream_wait_memory with 0 returned -11
> > > !sk_stream_memory_free queued 36576, sndbuf 34776
> > > sk_stream_wait_memory with 0 returned -11
> > >
> > > and so on and the sndbuf does not grow beyond 34776
> >
> > So there is outstanding data(sk_wmem_queued) that is not getting
> > acked.
> > If you set the sndbuf manually to a higher value, does it solve
> > the problem or only delay the stalls?
> >
> > Thanks
> > Sridhar
> >
> >
>
> Sridhar,
>
> snd_cwnd goes from 6 to 7 to 8 to 9 and is capped at that value. That is how
> sndbuf grows upto but does not grow beyond 34776.
>
> If I set the sndbuf manully, it just delays stalls. I have gone as high as 1MB
> of send size and that prevents from logging these errors longer but
> eventually it does.
>
> Regards,
>
> Shirish
>
I take it back, sndbuf has gone as high as
sk->sk_sndbuf 258888
and snd_cwnd has gone as high as 67.
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2008-05-15 18:49 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <4a4634330805121200m739e67bdife66c3ad81af1ceb@mail.gmail.com>
2008-05-13 0:47 ` autotuning of send buffer size of a socket Sridhar Samudrala
2008-05-13 0:51 ` David Miller
2008-05-13 13:54 ` Shirish Pargaonkar
2008-05-13 23:48 ` Sridhar Samudrala
2008-05-14 14:22 ` Shirish Pargaonkar
2008-05-14 18:11 ` Shirish Pargaonkar
2008-05-14 18:26 ` Jeremy Jackson
2008-05-14 18:37 ` Shirish Pargaonkar
2008-05-14 20:23 ` Shirish Pargaonkar
2008-05-15 4:18 ` Shirish Pargaonkar
2008-05-15 15:00 ` Shirish Pargaonkar
2008-05-15 17:40 ` Sridhar Samudrala
2008-05-15 18:03 ` Shirish Pargaonkar
2008-05-15 18:49 ` Shirish Pargaonkar
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).