Re: poor OSD performance using kernel 3.4 => problem found

[Stefan Priebe - Profihost AG <s.priebe@profihost.ag>]

Hi Marc, Hi Stefan,

first thanks for all your help and time.

I found the commit which results in this problem and it is TCP related
but i'm still wondering if the expected behaviour of this commit is
expected?

The commit in question is:
git show c43b874d5d714f271b80d4c3f49e05d0cbf51ed2
commit c43b874d5d714f271b80d4c3f49e05d0cbf51ed2
Author: Jason Wang <jasowang@redhat.com>
Date:   Thu Feb 2 00:07:00 2012 +0000

    tcp: properly initialize tcp memory limits

    Commit 4acb4190 tries to fix the using uninitialized value
    introduced by commit 3dc43e3,  but it would make the
    per-socket memory limits too small.

    This patch fixes this and also remove the redundant codes
    introduced in 4acb4190.

    Signed-off-by: Jason Wang <jasowang@redhat.com>
    Acked-by: Glauber Costa <glommer@parallels.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>

diff --git a/net/ipv4/sysctl_net_ipv4.c b/net/ipv4/sysctl_net_ipv4.c
index 4cb9cd2..7a7724d 100644
--- a/net/ipv4/sysctl_net_ipv4.c
+++ b/net/ipv4/sysctl_net_ipv4.c
@@ -778,7 +778,6 @@ EXPORT_SYMBOL_GPL(net_ipv4_ctl_path);
 static __net_init int ipv4_sysctl_init_net(struct net *net)
 {
        struct ctl_table *table;
-       unsigned long limit;

        table = ipv4_net_table;
        if (!net_eq(net, &init_net)) {
@@ -815,11 +814,6 @@ static __net_init int ipv4_sysctl_init_net(struct
net *net)
        net->ipv4.sysctl_rt_cache_rebuild_count = 4;

        tcp_init_mem(net);
-       limit = nr_free_buffer_pages() / 8;
-       limit = max(limit, 128UL);
-       net->ipv4.sysctl_tcp_mem[0] = limit / 4 * 3;
-       net->ipv4.sysctl_tcp_mem[1] = limit;
-       net->ipv4.sysctl_tcp_mem[2] = net->ipv4.sysctl_tcp_mem[0] * 2;

        net->ipv4.ipv4_hdr = register_net_sysctl_table(net,
                        net_ipv4_ctl_path, table);
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index a34f5cf..37755cc 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -3229,7 +3229,6 @@ __setup("thash_entries=", set_thash_entries);

 void tcp_init_mem(struct net *net)
 {
-       /* Set per-socket limits to no more than 1/128 the pressure
threshold */
        unsigned long limit = nr_free_buffer_pages() / 8;
        limit = max(limit, 128UL);
        net->ipv4.sysctl_tcp_mem[0] = limit / 4 * 3;
@@ -3298,7 +3297,8 @@ void __init tcp_init(void)
        sysctl_max_syn_backlog = max(128, cnt / 256);

        tcp_init_mem(&init_net);
-       limit = nr_free_buffer_pages() / 8;
+       /* Set per-socket limits to no more than 1/128 the pressure
threshold */
+       limit = nr_free_buffer_pages() << (PAGE_SHIFT - 10);
        limit = max(limit, 128UL);
        max_share = min(4UL*1024*1024, limit);

Greets
Stefan