[PATCHv 2] tcp: properly initialize tcp memory limits part 2 (fix nfs regression)

[PATCHv 2] tcp: properly initialize tcp memory limits part 2 (fix nfs regression)

[Sergei Trofimovich]

From: Sergei Trofimovich <slyfox@gentoo.org>

The commit c43b874d5d7 introduced NFS file transfer hangup (proved by bisection).
> Commit 4acb4190 tries to fix the using uninitialized value
> introduced by commit 3dc43e3,  but it would make the
> per-socket memory limits too small.
>
> This patch fixes this and also remove the redundant codes
> introduced in 4acb4190.

The change looks like a typo (division flipped to multiplication):
> limit = nr_free_buffer_pages() / 8;
> limit = nr_free_buffer_pages() << (PAGE_SHIFT - 10);

Patch restores division.

CC: Jason Wang <jasowang@redhat.com>
CC: Glauber Costa <glommer@parallels.com>
CC: David S. Miller <davem@davemloft.net>
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
Tested-by: Sergei Trofimovich <slyfox@gentoo.org>
---

Change since v1: added forgotten commit ID

 net/ipv4/tcp.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 22ef5f9..b11667b 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -3299,7 +3299,7 @@ void __init tcp_init(void)
 
 	tcp_init_mem(&init_net);
 	/* Set per-socket limits to no more than 1/128 the pressure threshold */
-	limit = nr_free_buffer_pages() << (PAGE_SHIFT - 10);
+	limit = nr_free_buffer_pages() >> (PAGE_SHIFT - 10);
 	limit = max(limit, 128UL);
 	max_share = min(4UL*1024*1024, limit);
 
-- 
1.7.8.5

Re: [PATCHv 2] tcp: properly initialize tcp memory limits part 2 (fix nfs regression)

[Jason Wang]

On 03/02/2012 03:59 PM, Sergei Trofimovich wrote:
> From: Sergei Trofimovich<slyfox@gentoo.org>
>
> The commit c43b874d5d7 introduced NFS file transfer hangup (proved by bisection).
>> Commit 4acb4190 tries to fix the using uninitialized value
>> introduced by commit 3dc43e3,  but it would make the
>> per-socket memory limits too small.
>>
>> This patch fixes this and also remove the redundant codes
>> introduced in 4acb4190.
> The change looks like a typo (division flipped to multiplication):
>> limit = nr_free_buffer_pages() / 8;
>> limit = nr_free_buffer_pages()<<  (PAGE_SHIFT - 10);

Hi, thanks for the reporting. It's not a typo. It was previously: 
sysctl_tcp_mem[1] << (PAGE_SHIFT -  7). Looks like we need to do the 
limit check before shift the value. Please try the following patch, thanks.

diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 22ef5f9..4035aab 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -3299,8 +3299,8 @@ void __init tcp_init(void)

         tcp_init_mem(&init_net);
         /* Set per-socket limits to no more than 1/128 the pressure 
threshold */
-       limit = nr_free_buffer_pages() << (PAGE_SHIFT - 10);
-       limit = max(limit, 128UL);
+       limit = nr_free_buffer_pages() / 8;
+       limit = max(limit, 128UL) << (PAGE_SHIFT - 7);
         max_share = min(4UL*1024*1024, limit);

         sysctl_tcp_wmem[0] = SK_MEM_QUANTUM;

Re: [PATCHv 2] tcp: properly initialize tcp memory limits part 2 (fix nfs regression)

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 2236 bytes --]

> > The change looks like a typo (division flipped to multiplication):
> >> limit = nr_free_buffer_pages() / 8;
> >> limit = nr_free_buffer_pages()<<  (PAGE_SHIFT - 10);
> 
> Hi, thanks for the reporting. It's not a typo. It was previously: 
> sysctl_tcp_mem[1] << (PAGE_SHIFT -  7). Looks like we need to do the 
> limit check before shift the value. Please try the following patch, thanks.

Still does not help. I test it by checking sha1sum of a large file over NFS
(small files seem to work simetimes):

    $ strace sha1sum /gentoo/distfiles/gcc-4.6.2.tar.bz2 
    ...
    open("/gentoo/distfiles/gcc-4.6.2.tar.bz2", O_RDONLY
    <HUNG>
After a certain timeout dmesg gets odd spam:
[  314.848094] nfs: server vmhost not responding, still trying
[  314.848134] nfs: server vmhost not responding, still trying
[  314.848145] nfs: server vmhost not responding, still trying
[  314.957047] nfs: server vmhost not responding, still trying
[  314.957066] nfs: server vmhost not responding, still trying
[  314.957075] nfs: server vmhost not responding, still trying
[  314.957085] nfs: server vmhost not responding, still trying
[  314.957100] nfs: server vmhost not responding, still trying
[  314.958023] nfs: server vmhost not responding, still trying
[  314.958035] nfs: server vmhost not responding, still trying
[  314.958044] nfs: server vmhost not responding, still trying
[  314.958054] nfs: server vmhost not responding, still trying

looks like bogus messages. Might be relevant to mishandled timings
somewhere else or a bug in nfs code.

> diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
> index 22ef5f9..4035aab 100644
> --- a/net/ipv4/tcp.c
> +++ b/net/ipv4/tcp.c
> @@ -3299,8 +3299,8 @@ void __init tcp_init(void)
> 
>          tcp_init_mem(&init_net);
>          /* Set per-socket limits to no more than 1/128 the pressure 
> threshold */
> -       limit = nr_free_buffer_pages() << (PAGE_SHIFT - 10);
> -       limit = max(limit, 128UL);
> +       limit = nr_free_buffer_pages() / 8;
> +       limit = max(limit, 128UL) << (PAGE_SHIFT - 7);
>          max_share = min(4UL*1024*1024, limit);
> 
>          sysctl_tcp_wmem[0] = SK_MEM_QUANTUM;
> 
> 
> 


-- 

  Sergei

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [PATCHv 2] tcp: properly initialize tcp memory limits part 2 (fix nfs regression)

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 3817 bytes --]

> > > The change looks like a typo (division flipped to multiplication):
> > >> limit = nr_free_buffer_pages() / 8;
> > >> limit = nr_free_buffer_pages()<<  (PAGE_SHIFT - 10);
> > 
> > Hi, thanks for the reporting. It's not a typo. It was previously: 
> > sysctl_tcp_mem[1] << (PAGE_SHIFT -  7). Looks like we need to do the 
> > limit check before shift the value. Please try the following patch, thanks.
> 
> Still does not help. I test it by checking sha1sum of a large file over NFS
> (small files seem to work simetimes):
> 
>     $ strace sha1sum /gentoo/distfiles/gcc-4.6.2.tar.bz2 
>     ...
>     open("/gentoo/distfiles/gcc-4.6.2.tar.bz2", O_RDONLY
>     <HUNG>
> After a certain timeout dmesg gets odd spam:
> [  314.848094] nfs: server vmhost not responding, still trying
> [  314.848134] nfs: server vmhost not responding, still trying
> [  314.848145] nfs: server vmhost not responding, still trying
> [  314.957047] nfs: server vmhost not responding, still trying
> [  314.957066] nfs: server vmhost not responding, still trying
> [  314.957075] nfs: server vmhost not responding, still trying
> [  314.957085] nfs: server vmhost not responding, still trying
> [  314.957100] nfs: server vmhost not responding, still trying
> [  314.958023] nfs: server vmhost not responding, still trying
> [  314.958035] nfs: server vmhost not responding, still trying
> [  314.958044] nfs: server vmhost not responding, still trying
> [  314.958054] nfs: server vmhost not responding, still trying
> 
> looks like bogus messages. Might be relevant to mishandled timings
> somewhere else or a bug in nfs code.

And after 120 seconds hung tasks shows it might be an OOM issue
Likely caused by patch, as it's a 2GB RAM +4GB swap amd64 box
not running anything heavy:

[  720.798052] INFO: task sha1sum:3811 blocked for more than 120 seconds.
[  720.798056] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  720.798059] sha1sum         D ffff88007bd11d40     0  3811      1 0x00000005
[  720.798065]  ffff880073de9c08 0000000000000082 ffff880073de9af8 ffff880073de9fd8
[  720.798070]  ffff880070db1620 ffff880073de9fd8 ffff880073de8000 0000000000004000
[  720.798075]  ffff880073de8000 ffff880073de9fd8 ffff8800790e0000 ffff880070db1620
[  720.798079] Call Trace:
[  720.798089]  [<ffffffff810fdd53>] ? kfree+0x123/0x150
[  720.798094]  [<ffffffff8123227d>] ? nfs_access_free_entry+0x1d/0x30
[  720.798097]  [<ffffffff810fdd53>] ? kfree+0x123/0x150
[  720.798101]  [<ffffffff8123227d>] ? nfs_access_free_entry+0x1d/0x30
[  720.798104]  [<ffffffff81233cb8>] ? nfs_do_access+0x3a8/0x3d0
[  720.798109]  [<ffffffff8166525a>] schedule+0x3a/0x50
[  720.798112]  [<ffffffff8166390e>] __mutex_lock_slowpath+0xee/0x190
[  720.798117]  [<ffffffff81639228>] ? put_rpccred+0x48/0x130
[  720.798120]  [<ffffffff8166374e>] mutex_lock+0x1e/0x40
[  720.798125]  [<ffffffff81114927>] do_lookup+0x277/0x3a0
[  720.798128]  [<ffffffff811162b8>] do_last.clone.39+0x148/0x7e0
[  720.798132]  [<ffffffff81116a61>] path_openat+0xd1/0x3e0
[  720.798136]  [<ffffffff810604d1>] ? get_parent_ip+0x11/0x50
[  720.798140]  [<ffffffff81060675>] ? add_preempt_count+0x95/0xd0
[  720.798144]  [<ffffffff81666677>] ? _raw_spin_lock_irq+0x17/0x40
[  720.798147]  [<ffffffff81116e84>] do_filp_open+0x44/0xa0
[  720.798151]  [<ffffffff810605a5>] ? sub_preempt_count+0x95/0xd0
[  720.798154]  [<ffffffff81666371>] ? _raw_spin_unlock+0x11/0x40
[  720.798158]  [<ffffffff81123014>] ? alloc_fd+0xe4/0x130
[  720.798163]  [<ffffffff81106f7d>] do_sys_open+0xfd/0x1e0
[  720.798169]  [<ffffffff8100f290>] ? syscall_trace_enter+0xf0/0x1a0
[  720.798172]  [<ffffffff8110707c>] sys_open+0x1c/0x20
[  720.798176]  [<ffffffff81667219>] tracesys+0xd0/0xd5

-- 

  Sergei

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [PATCHv 2] tcp: properly initialize tcp memory limits part 2 (fix nfs regression)

[Glauber Costa]

On 03/02/2012 02:50 PM, Sergei Trofimovich wrote:
>>>> The change looks like a typo (division flipped to multiplication):
>>>>> limit = nr_free_buffer_pages() / 8;
>>>>> limit = nr_free_buffer_pages()<<   (PAGE_SHIFT - 10);
>>>
>>> Hi, thanks for the reporting. It's not a typo. It was previously:
>>> sysctl_tcp_mem[1]<<  (PAGE_SHIFT -  7). Looks like we need to do the
>>> limit check before shift the value. Please try the following patch, thanks.
>>
>> Still does not help. I test it by checking sha1sum of a large file over NFS
>> (small files seem to work simetimes):
>>
>>      $ strace sha1sum /gentoo/distfiles/gcc-4.6.2.tar.bz2
>>      ...
>>      open("/gentoo/distfiles/gcc-4.6.2.tar.bz2", O_RDONLY
>>      <HUNG>
>> After a certain timeout dmesg gets odd spam:
>> [  314.848094] nfs: server vmhost not responding, still trying
>> [  314.848134] nfs: server vmhost not responding, still trying
>> [  314.848145] nfs: server vmhost not responding, still trying
>> [  314.957047] nfs: server vmhost not responding, still trying
>> [  314.957066] nfs: server vmhost not responding, still trying
>> [  314.957075] nfs: server vmhost not responding, still trying
>> [  314.957085] nfs: server vmhost not responding, still trying
>> [  314.957100] nfs: server vmhost not responding, still trying
>> [  314.958023] nfs: server vmhost not responding, still trying
>> [  314.958035] nfs: server vmhost not responding, still trying
>> [  314.958044] nfs: server vmhost not responding, still trying
>> [  314.958054] nfs: server vmhost not responding, still trying
>>
>> looks like bogus messages. Might be relevant to mishandled timings
>> somewhere else or a bug in nfs code.
>
> And after 120 seconds hung tasks shows it might be an OOM issue
> Likely caused by patch, as it's a 2GB RAM +4GB swap amd64 box
> not running anything heavy:

That is a bit weird.

First because with Jason's patch, we should end up with the very same 
calculation, at the same exact order, as it was in older kernels.
Second, because by shifting << 10, you should be ending up with very 
small numbers, effectively having tcp_rmem[1] == tcp_rmem[2], and the 
same for wmem.

Can you share which numbers you end up with at 
/proc/sys/net/ipv4/tcp_{r,w}mem ?

Re: [PATCHv 2] tcp: properly initialize tcp memory limits part 2 (fix nfs regression)

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 2711 bytes --]

On Sat, 3 Mar 2012 11:16:41 -0300
Glauber Costa <glommer@parallels.com> wrote:

> On 03/02/2012 02:50 PM, Sergei Trofimovich wrote:
> >>>> The change looks like a typo (division flipped to multiplication):
> >>>>> limit = nr_free_buffer_pages() / 8;
> >>>>> limit = nr_free_buffer_pages()<<   (PAGE_SHIFT - 10);
> >>>
> >>> Hi, thanks for the reporting. It's not a typo. It was previously:
> >>> sysctl_tcp_mem[1]<<  (PAGE_SHIFT -  7). Looks like we need to do the
> >>> limit check before shift the value. Please try the following patch, thanks.
> >>
> >> Still does not help. I test it by checking sha1sum of a large file over NFS
> >> (small files seem to work simetimes):
> >>
> >>      $ strace sha1sum /gentoo/distfiles/gcc-4.6.2.tar.bz2
> >>      ...
> >>      open("/gentoo/distfiles/gcc-4.6.2.tar.bz2", O_RDONLY
> >>      <HUNG>
> >> After a certain timeout dmesg gets odd spam:
> >> [  314.848094] nfs: server vmhost not responding, still trying
> >> [  314.848134] nfs: server vmhost not responding, still trying
> >> [  314.848145] nfs: server vmhost not responding, still trying
> >> [  314.957047] nfs: server vmhost not responding, still trying
> >> [  314.957066] nfs: server vmhost not responding, still trying
> >> [  314.957075] nfs: server vmhost not responding, still trying
> >> [  314.957085] nfs: server vmhost not responding, still trying
> >> [  314.957100] nfs: server vmhost not responding, still trying
> >> [  314.958023] nfs: server vmhost not responding, still trying
> >> [  314.958035] nfs: server vmhost not responding, still trying
> >> [  314.958044] nfs: server vmhost not responding, still trying
> >> [  314.958054] nfs: server vmhost not responding, still trying
> >>
> >> looks like bogus messages. Might be relevant to mishandled timings
> >> somewhere else or a bug in nfs code.
> >
> > And after 120 seconds hung tasks shows it might be an OOM issue
> > Likely caused by patch, as it's a 2GB RAM +4GB swap amd64 box
> > not running anything heavy:
> 
> That is a bit weird.
> 
> First because with Jason's patch, we should end up with the very same 
> calculation, at the same exact order, as it was in older kernels.
> Second, because by shifting << 10, you should be ending up with very 
> small numbers, effectively having tcp_rmem[1] == tcp_rmem[2], and the 
> same for wmem.
> 
> Can you share which numbers you end up with at 
> /proc/sys/net/ipv4/tcp_{r,w}mem ?
> 

Sure:

    $ cat /proc/sys/net/ipv4/tcp_{r,w}mem
    4096    87380   1999072
    4096    16384   1999072

Nothing special with NFS nere, so I guess it uses UDP.
TCP works fine on machine (I do everything via SSH).

-- 

  Sergei

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [PATCHv 2] tcp: properly initialize tcp memory limits part 2 (fix nfs regression)

[Glauber Costa]

On 03/03/2012 11:43 AM, Sergei Trofimovich wrote:
> On Sat, 3 Mar 2012 11:16:41 -0300
> Glauber Costa<glommer@parallels.com>  wrote:
>
>> On 03/02/2012 02:50 PM, Sergei Trofimovich wrote:
>>>>>> The change looks like a typo (division flipped to multiplication):
>>>>>>> limit = nr_free_buffer_pages() / 8;
>>>>>>> limit = nr_free_buffer_pages()<<    (PAGE_SHIFT - 10);
>>>>>
>>>>> Hi, thanks for the reporting. It's not a typo. It was previously:
>>>>> sysctl_tcp_mem[1]<<   (PAGE_SHIFT -  7). Looks like we need to do the
>>>>> limit check before shift the value. Please try the following patch, thanks.
>>>>
>>>> Still does not help. I test it by checking sha1sum of a large file over NFS
>>>> (small files seem to work simetimes):
>>>>
>>>>       $ strace sha1sum /gentoo/distfiles/gcc-4.6.2.tar.bz2
>>>>       ...
>>>>       open("/gentoo/distfiles/gcc-4.6.2.tar.bz2", O_RDONLY
>>>>       <HUNG>
>>>> After a certain timeout dmesg gets odd spam:
>>>> [  314.848094] nfs: server vmhost not responding, still trying
>>>> [  314.848134] nfs: server vmhost not responding, still trying
>>>> [  314.848145] nfs: server vmhost not responding, still trying
>>>> [  314.957047] nfs: server vmhost not responding, still trying
>>>> [  314.957066] nfs: server vmhost not responding, still trying
>>>> [  314.957075] nfs: server vmhost not responding, still trying
>>>> [  314.957085] nfs: server vmhost not responding, still trying
>>>> [  314.957100] nfs: server vmhost not responding, still trying
>>>> [  314.958023] nfs: server vmhost not responding, still trying
>>>> [  314.958035] nfs: server vmhost not responding, still trying
>>>> [  314.958044] nfs: server vmhost not responding, still trying
>>>> [  314.958054] nfs: server vmhost not responding, still trying
>>>>
>>>> looks like bogus messages. Might be relevant to mishandled timings
>>>> somewhere else or a bug in nfs code.
>>>
>>> And after 120 seconds hung tasks shows it might be an OOM issue
>>> Likely caused by patch, as it's a 2GB RAM +4GB swap amd64 box
>>> not running anything heavy:
>>
>> That is a bit weird.
>>
>> First because with Jason's patch, we should end up with the very same
>> calculation, at the same exact order, as it was in older kernels.
>> Second, because by shifting<<  10, you should be ending up with very
>> small numbers, effectively having tcp_rmem[1] == tcp_rmem[2], and the
>> same for wmem.
>>
>> Can you share which numbers you end up with at
>> /proc/sys/net/ipv4/tcp_{r,w}mem ?
>>
>
> Sure:
>
>      $ cat /proc/sys/net/ipv4/tcp_{r,w}mem
>      4096    87380   1999072
>      4096    16384   1999072
>
Sergei,

Sorry for not being clearer. I was expecting you'd post those values
both in the scenario in which you see the bug, and in the scenario you
don't.

> Nothing special with NFS nere, so I guess it uses UDP.
> TCP works fine on machine (I do everything via SSH).

Can you confirm that? If you're using nfs through udp, it makes
even less sense that the default values of tcp sock mem will harm
you. So it might be a bug somewhere else...

Re: [PATCHv 2] tcp: properly initialize tcp memory limits part 2 (fix nfs regression)

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 4294 bytes --]

On Sat, 3 Mar 2012 20:27:17 -0300
Glauber Costa <glommer@parallels.com> wrote:

> On 03/03/2012 11:43 AM, Sergei Trofimovich wrote:
> > On Sat, 3 Mar 2012 11:16:41 -0300
> > Glauber Costa<glommer@parallels.com>  wrote:
> >
> >> On 03/02/2012 02:50 PM, Sergei Trofimovich wrote:
> >>>>>> The change looks like a typo (division flipped to multiplication):
> >>>>>>> limit = nr_free_buffer_pages() / 8;
> >>>>>>> limit = nr_free_buffer_pages()<<    (PAGE_SHIFT - 10);
> >>>>>
> >>>>> Hi, thanks for the reporting. It's not a typo. It was previously:
> >>>>> sysctl_tcp_mem[1]<<   (PAGE_SHIFT -  7). Looks like we need to do the
> >>>>> limit check before shift the value. Please try the following patch, thanks.
> >>>>
> >>>> Still does not help. I test it by checking sha1sum of a large file over NFS
> >>>> (small files seem to work simetimes):
> >>>>
> >>>>       $ strace sha1sum /gentoo/distfiles/gcc-4.6.2.tar.bz2
> >>>>       ...
> >>>>       open("/gentoo/distfiles/gcc-4.6.2.tar.bz2", O_RDONLY
> >>>>       <HUNG>
> >>>> After a certain timeout dmesg gets odd spam:
> >>>> [  314.848094] nfs: server vmhost not responding, still trying
> >>>> [  314.848134] nfs: server vmhost not responding, still trying
> >>>> [  314.848145] nfs: server vmhost not responding, still trying
> >>>> [  314.957047] nfs: server vmhost not responding, still trying
> >>>> [  314.957066] nfs: server vmhost not responding, still trying
> >>>> [  314.957075] nfs: server vmhost not responding, still trying
> >>>> [  314.957085] nfs: server vmhost not responding, still trying
> >>>> [  314.957100] nfs: server vmhost not responding, still trying
> >>>> [  314.958023] nfs: server vmhost not responding, still trying
> >>>> [  314.958035] nfs: server vmhost not responding, still trying
> >>>> [  314.958044] nfs: server vmhost not responding, still trying
> >>>> [  314.958054] nfs: server vmhost not responding, still trying
> >>>>
> >>>> looks like bogus messages. Might be relevant to mishandled timings
> >>>> somewhere else or a bug in nfs code.
> >>>
> >>> And after 120 seconds hung tasks shows it might be an OOM issue
> >>> Likely caused by patch, as it's a 2GB RAM +4GB swap amd64 box
> >>> not running anything heavy:
> >>
> >> That is a bit weird.
> >>
> >> First because with Jason's patch, we should end up with the very same
> >> calculation, at the same exact order, as it was in older kernels.
> >> Second, because by shifting<<  10, you should be ending up with very
> >> small numbers, effectively having tcp_rmem[1] == tcp_rmem[2], and the
> >> same for wmem.
> >>
> >> Can you share which numbers you end up with at
> >> /proc/sys/net/ipv4/tcp_{r,w}mem ?
> >>
> >
> > Sure:
> >
> >      $ cat /proc/sys/net/ipv4/tcp_{r,w}mem
> >      4096    87380   1999072
> >      4096    16384   1999072
> >
> Sergei,
> 
> Sorry for not being clearer. I was expecting you'd post those values
> both in the scenario in which you see the bug, and in the scenario you
> don't.

Ah, I see.  Sorry. Patches are on top of v3.3-rc5-166-g1f033c1. Buggy one:
> -       limit = nr_free_buffer_pages() << (PAGE_SHIFT - 10);
> -       limit = max(limit, 128UL);
> +       limit = nr_free_buffer_pages() / 8;
> +       limit = max(limit, 128UL) << (PAGE_SHIFT - 7);
>         max_share = min(4UL*1024*1024, limit);
> +       printk(KERN_INFO "TCP: max_share=%u\n", max_share);
    $ cat /proc/sys/net/ipv4/tcp_{r,w}mem
    4096    87380   1999072
    4096    16384   1999072

Working one:
> -       limit = nr_free_buffer_pages() << (PAGE_SHIFT - 10);
> +       limit = nr_free_buffer_pages() >> (PAGE_SHIFT - 10);
>         limit = max(limit, 128UL);
>         max_share = min(4UL*1024*1024, limit);
> +       printk(KERN_INFO "TCP: max_share=%u\n", max_share);
    $ cat /proc/sys/net/ipv4/tcp_{r,w}mem
    4096    87380   124942
    4096    16384   124942

> > Nothing special with NFS nere, so I guess it uses UDP.
> > TCP works fine on machine (I do everything via SSH).
> 
> Can you confirm that? If you're using nfs through udp, it makes
> even less sense that the default values of tcp sock mem will harm
> you. So it might be a bug somewhere else...

Rechecked with tcpdump. It uses TCP.

-- 

  Sergei

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [PATCHv 2] tcp: properly initialize tcp memory limits part 2 (fix nfs regression)

[Jason Wang]

On 03/04/2012 05:14 PM, Sergei Trofimovich wrote:
> On Sat, 3 Mar 2012 20:27:17 -0300
> Glauber Costa<glommer@parallels.com>  wrote:
>
>> On 03/03/2012 11:43 AM, Sergei Trofimovich wrote:
>>> On Sat, 3 Mar 2012 11:16:41 -0300
>>> Glauber Costa<glommer@parallels.com>   wrote:
>>>
>>>> On 03/02/2012 02:50 PM, Sergei Trofimovich wrote:
>>>>>>>> The change looks like a typo (division flipped to multiplication):
>>>>>>>>> limit = nr_free_buffer_pages() / 8;
>>>>>>>>> limit = nr_free_buffer_pages()<<     (PAGE_SHIFT - 10);
>>>>>>> Hi, thanks for the reporting. It's not a typo. It was previously:
>>>>>>> sysctl_tcp_mem[1]<<    (PAGE_SHIFT -  7). Looks like we need to do the
>>>>>>> limit check before shift the value. Please try the following patch, thanks.
>>>>>> Still does not help. I test it by checking sha1sum of a large file over NFS
>>>>>> (small files seem to work simetimes):
>>>>>>
>>>>>>        $ strace sha1sum /gentoo/distfiles/gcc-4.6.2.tar.bz2
>>>>>>        ...
>>>>>>        open("/gentoo/distfiles/gcc-4.6.2.tar.bz2", O_RDONLY
>>>>>>        <HUNG>
Hi Sergei:

Looks like the client does not even start to read the file.
>>>>>> After a certain timeout dmesg gets odd spam:
>>>>>> [  314.848094] nfs: server vmhost not responding, still trying
>>>>>> [  314.848134] nfs: server vmhost not responding, still trying
>>>>>> [  314.848145] nfs: server vmhost not responding, still trying
>>>>>> [  314.957047] nfs: server vmhost not responding, still trying
>>>>>> [  314.957066] nfs: server vmhost not responding, still trying
>>>>>> [  314.957075] nfs: server vmhost not responding, still trying
>>>>>> [  314.957085] nfs: server vmhost not responding, still trying
>>>>>> [  314.957100] nfs: server vmhost not responding, still trying
>>>>>> [  314.958023] nfs: server vmhost not responding, still trying
>>>>>> [  314.958035] nfs: server vmhost not responding, still trying
>>>>>> [  314.958044] nfs: server vmhost not responding, still trying
>>>>>> [  314.958054] nfs: server vmhost not responding, still trying
>>>>>>
>>>>>> looks like bogus messages. Might be relevant to mishandled timings
>>>>>> somewhere else or a bug in nfs code.

  Did you use a virtual machine as your NFS server? Have you tried to 
bisect the server side code?
>>>>> And after 120 seconds hung tasks shows it might be an OOM issue
>>>>> Likely caused by patch, as it's a 2GB RAM +4GB swap amd64 box
>>>>> not running anything heavy:
>>>> That is a bit weird.
>>>>
>>>> First because with Jason's patch, we should end up with the very same
>>>> calculation, at the same exact order, as it was in older kernels.
>>>> Second, because by shifting<<   10, you should be ending up with very
>>>> small numbers, effectively having tcp_rmem[1] == tcp_rmem[2], and the
>>>> same for wmem.
>>>>
>>>> Can you share which numbers you end up with at
>>>> /proc/sys/net/ipv4/tcp_{r,w}mem ?
>>>>
>>> Sure:
>>>
>>>       $ cat /proc/sys/net/ipv4/tcp_{r,w}mem
>>>       4096    87380   1999072
>>>       4096    16384   1999072
>>>
>> Sergei,
>>
>> Sorry for not being clearer. I was expecting you'd post those values
>> both in the scenario in which you see the bug, and in the scenario you
>> don't.
> Ah, I see.  Sorry. Patches are on top of v3.3-rc5-166-g1f033c1. Buggy one:
>> -       limit = nr_free_buffer_pages()<<  (PAGE_SHIFT - 10);
>> -       limit = max(limit, 128UL);
>> +       limit = nr_free_buffer_pages() / 8;
>> +       limit = max(limit, 128UL)<<  (PAGE_SHIFT - 7);
>>          max_share = min(4UL*1024*1024, limit);
>> +       printk(KERN_INFO "TCP: max_share=%u\n", max_share);
>      $ cat /proc/sys/net/ipv4/tcp_{r,w}mem
>      4096    87380   1999072
>      4096    16384   1999072

Nothing strange to me.
> Working one:
>> -       limit = nr_free_buffer_pages()<<  (PAGE_SHIFT - 10);
>> +       limit = nr_free_buffer_pages()>>  (PAGE_SHIFT - 10);
>>          limit = max(limit, 128UL);
>>          max_share = min(4UL*1024*1024, limit);
>> +       printk(KERN_INFO "TCP: max_share=%u\n", max_share);
>      $ cat /proc/sys/net/ipv4/tcp_{r,w}mem
>      4096    87380   124942
>      4096    16384   124942

This one looks small to me, as the tcp_{r,w}mem were count by bytes and 
limit were count by number of pages, so we need to shift PAGE_SHIFT.

As I can't reproduce this locally, in order to narrow down the problem, 
could you please help to check whether the issue were 
introduced/eliminated by commit  4acb4190 or 3dc43e3?

Thanks
>>> Nothing special with NFS nere, so I guess it uses UDP.
>>> TCP works fine on machine (I do everything via SSH).
>> Can you confirm that? If you're using nfs through udp, it makes
>> even less sense that the default values of tcp sock mem will harm
>> you. So it might be a bug somewhere else...
> Rechecked with tcpdump. It uses TCP.
>

Re: [PATCHv 2] tcp: properly initialize tcp memory limits part 2 (fix nfs regression)

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 4903 bytes --]

> >>>>>>>> The change looks like a typo (division flipped to multiplication):
> >>>>>>>>> limit = nr_free_buffer_pages() / 8;
> >>>>>>>>> limit = nr_free_buffer_pages()<<     (PAGE_SHIFT - 10);
> >>>>>>> Hi, thanks for the reporting. It's not a typo. It was previously:
> >>>>>>> sysctl_tcp_mem[1]<<    (PAGE_SHIFT -  7). Looks like we need to do the
> >>>>>>> limit check before shift the value. Please try the following patch, thanks.
> >>>>>> Still does not help. I test it by checking sha1sum of a large file over NFS
> >>>>>> (small files seem to work simetimes):
> >>>>>>
> >>>>>>        $ strace sha1sum /gentoo/distfiles/gcc-4.6.2.tar.bz2
> >>>>>>        ...
> >>>>>>        open("/gentoo/distfiles/gcc-4.6.2.tar.bz2", O_RDONLY
> >>>>>>        <HUNG>
> Hi Sergei:
> 
> Looks like the client does not even start to read the file.
> >>>>>> After a certain timeout dmesg gets odd spam:
> >>>>>> [  314.848094] nfs: server vmhost not responding, still trying
> >>>>>> [  314.848134] nfs: server vmhost not responding, still trying
> >>>>>> [  314.848145] nfs: server vmhost not responding, still trying
> >>>>>> [  314.957047] nfs: server vmhost not responding, still trying
> >>>>>> [  314.957066] nfs: server vmhost not responding, still trying
> >>>>>> [  314.957075] nfs: server vmhost not responding, still trying
> >>>>>> [  314.957085] nfs: server vmhost not responding, still trying
> >>>>>> [  314.957100] nfs: server vmhost not responding, still trying
> >>>>>> [  314.958023] nfs: server vmhost not responding, still trying
> >>>>>> [  314.958035] nfs: server vmhost not responding, still trying
> >>>>>> [  314.958044] nfs: server vmhost not responding, still trying
> >>>>>> [  314.958054] nfs: server vmhost not responding, still trying
> >>>>>>
> >>>>>> looks like bogus messages. Might be relevant to mishandled timings
> >>>>>> somewhere else or a bug in nfs code.
> 
>   Did you use a virtual machine as your NFS server? Have you tried to 
> bisect the server side code?
> >>>>> And after 120 seconds hung tasks shows it might be an OOM issue
> >>>>> Likely caused by patch, as it's a 2GB RAM +4GB swap amd64 box
> >>>>> not running anything heavy:
> >>>> That is a bit weird.
> >>>>
> >>>> First because with Jason's patch, we should end up with the very same
> >>>> calculation, at the same exact order, as it was in older kernels.
> >>>> Second, because by shifting<<   10, you should be ending up with very
> >>>> small numbers, effectively having tcp_rmem[1] == tcp_rmem[2], and the
> >>>> same for wmem.
> >>>>
> >>>> Can you share which numbers you end up with at
> >>>> /proc/sys/net/ipv4/tcp_{r,w}mem ?
> >>>>
> >>> Sure:
> >>>
> >>>       $ cat /proc/sys/net/ipv4/tcp_{r,w}mem
> >>>       4096    87380   1999072
> >>>       4096    16384   1999072
> >>>
> >> Sergei,
> >>
> >> Sorry for not being clearer. I was expecting you'd post those values
> >> both in the scenario in which you see the bug, and in the scenario you
> >> don't.
> > Ah, I see.  Sorry. Patches are on top of v3.3-rc5-166-g1f033c1. Buggy one:
> >> -       limit = nr_free_buffer_pages()<<  (PAGE_SHIFT - 10);
> >> -       limit = max(limit, 128UL);
> >> +       limit = nr_free_buffer_pages() / 8;
> >> +       limit = max(limit, 128UL)<<  (PAGE_SHIFT - 7);
> >>          max_share = min(4UL*1024*1024, limit);
> >> +       printk(KERN_INFO "TCP: max_share=%u\n", max_share);
> >      $ cat /proc/sys/net/ipv4/tcp_{r,w}mem
> >      4096    87380   1999072
> >      4096    16384   1999072
> 
> Nothing strange to me.
> > Working one:
> >> -       limit = nr_free_buffer_pages()<<  (PAGE_SHIFT - 10);
> >> +       limit = nr_free_buffer_pages()>>  (PAGE_SHIFT - 10);
> >>          limit = max(limit, 128UL);
> >>          max_share = min(4UL*1024*1024, limit);
> >> +       printk(KERN_INFO "TCP: max_share=%u\n", max_share);
> >      $ cat /proc/sys/net/ipv4/tcp_{r,w}mem
> >      4096    87380   124942
> >      4096    16384   124942
> 
> This one looks small to me, as the tcp_{r,w}mem were count by bytes and 
> limit were count by number of pages, so we need to shift PAGE_SHIFT.
> 
> As I can't reproduce this locally, in order to narrow down the problem, 
> could you please help to check whether the issue were 
> introduced/eliminated by commit  4acb4190 or 3dc43e3?

I didn't think of server problem. I did run 3.3-rc0 kernel there
from the kvm tree (v3.2-10396-g05ef4c6):
    commit 05ef4c60568ed1740f65bf66a76da30b19060119
    Author: Michael S. Tsirkin <mst@redhat.com>
    Date:   Wed Jan 18 20:07:09 2012 +0200

        kvm: fix error handling for out of range irq

    from git://git.kernel.org/pub/scm/virt/kvm/kvm.git
 
Updating to current vanilla 3.3-rc6 solved the problem.
Are you interested in digging that issue further to find commit
breaking the server?

-- 

  Sergei

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [PATCHv 2] tcp: properly initialize tcp memory limits part 2 (fix nfs regression)

[Jason Wang]

On 03/06/2012 02:22 AM, Sergei Trofimovich wrote:
>> This one looks small to me, as the tcp_{r,w}mem were count by bytes and
>> >  limit were count by number of pages, so we need to shift PAGE_SHIFT.
>> >  
>> >  As I can't reproduce this locally, in order to narrow down the problem,
>> >  could you please help to check whether the issue were
>> >  introduced/eliminated by commit  4acb4190 or 3dc43e3?
> I didn't think of server problem. I did run 3.3-rc0 kernel there
> from the kvm tree (v3.2-10396-g05ef4c6):
>      commit 05ef4c60568ed1740f65bf66a76da30b19060119
>      Author: Michael S. Tsirkin<mst@redhat.com>
>      Date:   Wed Jan 18 20:07:09 2012 +0200
>
>          kvm: fix error handling for out of range irq
>
>      from git://git.kernel.org/pub/scm/virt/kvm/kvm.git
>
> Updating to current vanilla 3.3-rc6 solved the problem.
> Are you interested in digging that issue further to find commit
> breaking the server?
Hi Sergei:

So it's very possible that it's a bug which is blacklisted by early 
commits and uncovered by my commit ( 
c43b874d5d714f271b80d4c3f49e05d0cbf51ed2). I wish I can help in nfs 
debugging, but I'm not familiar with that.

thanks