All of lore.kernel.org
 help / color / mirror / Atom feed
From: Roman Gushchin <roman.gushchin-fxUVXftIFDnyG1zEObXtfA@public.gmane.org>
To: Jakub Kicinski <kuba-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
Cc: edumazet-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org,
	netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org,
	pabeni-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
	cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	Shakeel Butt <shakeelb-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>,
	weiwan-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org,
	ncardwell-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org,
	ycheng-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org
Subject: Re: [PATCH net] net-memcg: avoid stalls when under memory pressure
Date: Mon, 24 Oct 2022 09:02:12 -0700	[thread overview]
Message-ID: <Y1a3BHQqllCEymHi@P9FQF9L96D> (raw)
In-Reply-To: <20221021160304.1362511-1-kuba-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>

On Fri, Oct 21, 2022 at 09:03:04AM -0700, Jakub Kicinski wrote:
> As Shakeel explains the commit under Fixes had the unintended
> side-effect of no longer pre-loading the cached memory allowance.
> Even tho we previously dropped the first packet received when
> over memory limit - the consecutive ones would get thru by using
> the cache. The charging was happening in batches of 128kB, so
> we'd let in 128kB (truesize) worth of packets per one drop.
> 
> After the change we no longer force charge, there will be no
> cache filling side effects. This causes significant drops and
> connection stalls for workloads which use a lot of page cache,
> since we can't reclaim page cache under GFP_NOWAIT.
> 
> Some of the latency can be recovered by improving SACK reneg
> handling but nowhere near enough to get back to the pre-5.15
> performance (the application I'm experimenting with still
> sees 5-10x worst latency).
> 
> Apply the suggested workaround of using GFP_ATOMIC. We will now
> be more permissive than previously as we'll drop _no_ packets
> in softirq when under pressure. But I can't think of any good
> and simple way to address that within networking.
> 
> Link: https://lore.kernel.org/all/20221012163300.795e7b86-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org/
> Suggested-by: Shakeel Butt <shakeelb-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
> Fixes: 4b1327be9fe5 ("net-memcg: pass in gfp_t mask to mem_cgroup_charge_skmem()")
> Signed-off-by: Jakub Kicinski <kuba-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>

Acked-by: Roman Gushchin <roman.gushchin-fxUVXftIFDnyG1zEObXtfA@public.gmane.org>

Thanks!

WARNING: multiple messages have this Message-ID (diff)
From: Roman Gushchin <roman.gushchin@linux.dev>
To: Jakub Kicinski <kuba@kernel.org>
Cc: edumazet@google.com, netdev@vger.kernel.org, davem@davemloft.net,
	pabeni@redhat.com, cgroups@vger.kernel.org,
	Shakeel Butt <shakeelb@google.com>,
	weiwan@google.com, ncardwell@google.com, ycheng@google.com
Subject: Re: [PATCH net] net-memcg: avoid stalls when under memory pressure
Date: Mon, 24 Oct 2022 09:02:12 -0700	[thread overview]
Message-ID: <Y1a3BHQqllCEymHi@P9FQF9L96D> (raw)
In-Reply-To: <20221021160304.1362511-1-kuba@kernel.org>

On Fri, Oct 21, 2022 at 09:03:04AM -0700, Jakub Kicinski wrote:
> As Shakeel explains the commit under Fixes had the unintended
> side-effect of no longer pre-loading the cached memory allowance.
> Even tho we previously dropped the first packet received when
> over memory limit - the consecutive ones would get thru by using
> the cache. The charging was happening in batches of 128kB, so
> we'd let in 128kB (truesize) worth of packets per one drop.
> 
> After the change we no longer force charge, there will be no
> cache filling side effects. This causes significant drops and
> connection stalls for workloads which use a lot of page cache,
> since we can't reclaim page cache under GFP_NOWAIT.
> 
> Some of the latency can be recovered by improving SACK reneg
> handling but nowhere near enough to get back to the pre-5.15
> performance (the application I'm experimenting with still
> sees 5-10x worst latency).
> 
> Apply the suggested workaround of using GFP_ATOMIC. We will now
> be more permissive than previously as we'll drop _no_ packets
> in softirq when under pressure. But I can't think of any good
> and simple way to address that within networking.
> 
> Link: https://lore.kernel.org/all/20221012163300.795e7b86@kernel.org/
> Suggested-by: Shakeel Butt <shakeelb@google.com>
> Fixes: 4b1327be9fe5 ("net-memcg: pass in gfp_t mask to mem_cgroup_charge_skmem()")
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>

Acked-by: Roman Gushchin <roman.gushchin@linux.dev>

Thanks!

  parent reply	other threads:[~2022-10-24 16:02 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-10-21 16:03 [PATCH net] net-memcg: avoid stalls when under memory pressure Jakub Kicinski
2022-10-21 16:03 ` Jakub Kicinski
2022-10-21 16:25 ` Shakeel Butt
     [not found]   ` <CALvZod4eezAXpehT4jMiQry4JQ5igJs7Nwi1Q+YhVpDcQ8BMRA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2022-10-21 16:28     ` Eric Dumazet
2022-10-21 16:28       ` Eric Dumazet
     [not found]       ` <CANn89iKTi5TYyFOOpgw3P0eTi1Gqn4k-eX+xRTX78Q4sAunm2Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2022-10-21 16:34         ` Shakeel Butt
2022-10-21 16:34           ` Shakeel Butt
     [not found]           ` <CALvZod5di3saFdDJ1cwFDgvLPmnEJ7XB9P8YBTJ3uzfBKAFi3Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2022-10-21 17:40             ` Jakub Kicinski
2022-10-21 17:40               ` Jakub Kicinski
2022-10-21 18:53               ` Shakeel Butt
     [not found] ` <20221021160304.1362511-1-kuba-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
2022-10-24 16:02   ` Roman Gushchin [this message]
2022-10-24 16:02     ` Roman Gushchin
2022-10-24 18:20   ` patchwork-bot+netdevbpf-DgEjT+Ai2ygdnm+yROfE0A
2022-10-24 18:20     ` patchwork-bot+netdevbpf

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=Y1a3BHQqllCEymHi@P9FQF9L96D \
    --to=roman.gushchin-fxuvxftifdnyg1zeobxtfa@public.gmane.org \
    --cc=cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org \
    --cc=edumazet-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org \
    --cc=kuba-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org \
    --cc=ncardwell-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org \
    --cc=netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=pabeni-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org \
    --cc=shakeelb-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org \
    --cc=weiwan-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org \
    --cc=ycheng-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.