From: Mike Snitzer <snitzer@kernel.org>
To: Chuck Lever <cel@kernel.org>, linux-nfs@vger.kernel.org
Cc: ben.coddington@hammerspace.com, jonathan.flynn@hammerspace.com
Subject: Re: [RFC PATCH 0/2] svcrdma: avoid OOM due to unbounded sc_send_ctxts cache
Date: Tue, 5 May 2026 18:05:34 -0400 [thread overview]
Message-ID: <afpprmApzykJFPci@kernel.org> (raw)
In-Reply-To: <20260505215535.68412-1-snitzer@kernel.org>
On Tue, May 05, 2026 at 05:55:33PM -0400, Mike Snitzer wrote:
> Hi,
>
> I drew the short-straw by having to take a hand-off from Ben on work
> he started with Claude yesterday in response to a really crazy OOM
> situation that hits like a freight train at one large customer's
> install that currently has 121 NFS clients and 9 NFS servers, all
> connected with RDMA networking. Working with Jon Flynn, to bound the
> problem a bit more we later scaled the testing down to 15 clients
> reading from 1 server using 16K O_DIRECT reads.
>
> So I imported Ben's CLAUDE.md that he handed off and carried on, with
> patch 1/2 we're able to avoid OOM killing the NFS servers (each with
> 128GB) -- with the 16K test workload memory use would grow from ~12GB
> to exhaustion (128GB) within ~10 seconds of starting the test.
>
> The 2nd patch in this series provides a diagnostic svcrdma-wq-lag.bt
> bpf script that Claude suggested -- I just dropped it in
> Documentation/filesystems/nfs/ but it isn't intended to go upstream.
Here is svcrdma-wq-lag.bt output before we capped in _get (short run
because otherwise we'd have hit OOM, but shows the flood of
sc_send_ctxts that only become visible once the test was killed at
17:45:36):
# ./svcrdma-wq-lag.bt
Attaching 7 probes...
svcrdma_wq lag diagnostic. Per-5s rates.
Watch for @wc_send_rate >> @release_rate during the test
(workqueue backed up -> ctxts pinned in queue items ->
Ben's cap can't see them).
Ctrl-C to exit.
17:45:06 wc_send=0 release=0 get=0 capped=0
17:45:11 wc_send=1984617 release=1571458 get=1916504 capped=0 queue/release ratio=1
17:45:16 wc_send=2176680 release=1698697 get=2097454 capped=0 queue/release ratio=1
17:45:21 wc_send=2181507 release=1716810 get=2105847 capped=0 queue/release ratio=1
17:45:26 wc_send=2178462 release=1728112 get=2104069 capped=726 queue/release ratio=1
17:45:31 wc_send=2158883 release=1714519 get=2092136 capped=205 queue/release ratio=1
17:45:36 wc_send=1029461 release=1585039 get=996644 capped=757304 queue/release ratio=0
17:45:41 wc_send=0 release=3092605 get=0 capped=1492238 queue/release ratio=0
17:45:46 wc_send=0 release=1000663 get=0 capped=1000764 queue/release ratio=0
17:45:51 wc_send=0 release=0 get=0 capped=0
^C
And here is svcrdma-wq-lag.bt output with patch 1/2 applied (full 5
minute run):
18:13:21 wc_send=1151440 release=1148610 get=1150980 capped=0 queue/release ratio=1
18:13:26 wc_send=2125499 release=2149303 get=2107227 capped=0 queue/release ratio=0
18:13:31 wc_send=1780821 release=1792233 get=1759546 capped=0 queue/release ratio=0
18:13:36 wc_send=3893061 release=2118765 get=2081403 capped=1 queue/release ratio=1
18:13:41 wc_send=1837599 release=1883821 get=1815453 capped=0 queue/release ratio=0
18:13:46 wc_send=1813727 release=1822876 get=1794087 capped=0 queue/release ratio=0
18:13:51 wc_send=1491464 release=1529370 get=1476924 capped=0 queue/release ratio=0
18:13:56 wc_send=1930254 release=1955278 get=1910046 capped=0 queue/release ratio=0
18:14:01 wc_send=1916374 release=1944863 get=1894205 capped=0 queue/release ratio=0
18:14:11 wc_send=4089684 release=4107873 get=4041449 capped=0 queue/release ratio=0
18:14:16 wc_send=1740066 release=1789864 get=5765163 capped=0 queue/release ratio=0
18:14:21 wc_send=3655551 release=1936245 get=1894173 capped=0 queue/release ratio=1
18:14:26 wc_send=1652813 release=3613434 get=1637738 capped=0 queue/release ratio=0
18:14:31 wc_send=1873398 release=1903780 get=1855887 capped=0 queue/release ratio=0
18:14:36 wc_send=1947061 release=1971983 get=1920987 capped=0 queue/release ratio=0
18:14:41 wc_send=1624716 release=1630181 get=1608404 capped=1 queue/release ratio=0
18:14:46 wc_send=1709804 release=1752711 get=1693828 capped=0 queue/release ratio=0
18:14:51 wc_send=1939672 release=1970791 get=3615177 capped=0 queue/release ratio=0
18:14:56 wc_send=1884914 release=3875602 get=1862209 capped=0 queue/release ratio=0
18:15:01 wc_send=2019244 release=2044366 get=1993761 capped=0 queue/release ratio=0
18:15:06 wc_send=2065910 release=2099472 get=2046106 capped=0 queue/release ratio=0
18:15:11 wc_send=1914899 release=1933756 get=1893723 capped=2 queue/release ratio=0
18:15:16 wc_send=1895244 release=1916124 get=1874634 capped=0 queue/release ratio=0
18:15:21 wc_send=2000953 release=2036536 get=3853528 capped=2 queue/release ratio=0
18:15:26 wc_send=3971332 release=1984930 get=5805016 capped=0 queue/release ratio=2
18:15:31 wc_send=1842215 release=1876278 get=1827305 capped=0 queue/release ratio=0
18:15:36 wc_send=1712031 release=1745388 get=1698252 capped=0 queue/release ratio=0
18:15:41 wc_send=1457677 release=1477376 get=1446707 capped=0 queue/release ratio=0
18:15:46 wc_send=1522972 release=1562257 get=1510851 capped=0 queue/release ratio=0
18:15:51 wc_send=2126919 release=2126524 get=2100919 capped=0 queue/release ratio=1
18:15:56 wc_send=1795205 release=1853495 get=1782554 capped=0 queue/release ratio=0
18:16:01 wc_send=1872737 release=1892789 get=1852457 capped=1 queue/release ratio=0
18:16:06 wc_send=1956552 release=1978503 get=1936808 capped=0 queue/release ratio=0
18:16:11 wc_send=1971408 release=2008768 get=1946069 capped=0 queue/release ratio=0
18:16:16 wc_send=1906940 release=1928668 get=1891132 capped=0 queue/release ratio=0
18:16:21 wc_send=2019228 release=2059320 get=1993856 capped=0 queue/release ratio=0
18:16:26 wc_send=2024263 release=2039861 get=2004786 capped=0 queue/release ratio=0
18:16:31 wc_send=1959046 release=1986165 get=1936975 capped=0 queue/release ratio=0
18:16:36 wc_send=3426760 release=3483327 get=1456324 capped=0 queue/release ratio=0
18:16:41 wc_send=2066845 release=2078503 get=2042898 capped=0 queue/release ratio=0
18:16:46 wc_send=1834131 release=1878182 get=1818034 capped=1 queue/release ratio=0
18:16:51 wc_send=1804133 release=1817222 get=1787677 capped=1 queue/release ratio=0
18:16:56 wc_send=1443325 release=1469384 get=1429495 capped=0 queue/release ratio=0
18:17:01 wc_send=1437236 release=1480771 get=1425373 capped=1 queue/release ratio=0
18:17:06 wc_send=3342896 release=1925346 get=1885982 capped=0 queue/release ratio=1
18:17:11 wc_send=2035328 release=2063539 get=2011609 capped=1 queue/release ratio=0
18:17:16 wc_send=1923583 release=1930805 get=1904366 capped=0 queue/release ratio=0
18:17:21 wc_send=1918902 release=1958907 get=1899028 capped=0 queue/release ratio=0
18:17:26 wc_send=1912808 release=1942152 get=1895315 capped=0 queue/release ratio=0
18:17:31 wc_send=2067295 release=2090789 get=2045747 capped=0 queue/release ratio=0
18:17:36 wc_send=1893866 release=1917954 get=3924985 capped=0 queue/release ratio=0
18:17:41 wc_send=1914595 release=1922461 get=1895841 capped=0 queue/release ratio=0
18:17:46 wc_send=1833010 release=1884469 get=1816313 capped=0 queue/release ratio=0
18:17:51 wc_send=1853085 release=3756671 get=3652976 capped=1 queue/release ratio=0
18:17:56 wc_send=1814888 release=1845390 get=1797898 capped=0 queue/release ratio=0
18:18:01 wc_send=1875252 release=3752430 get=1856752 capped=0 queue/release ratio=0
18:18:06 wc_send=1929648 release=1955900 get=1913162 capped=0 queue/release ratio=0
18:18:11 wc_send=1878956 release=1907828 get=1862308 capped=0 queue/release ratio=0
18:18:16 wc_send=1839023 release=1862031 get=1823009 capped=0 queue/release ratio=0
18:18:21 wc_send=77 release=77 get=77 capped=0 queue/release ratio=1
18:18:26 wc_send=0 release=0 get=0 capped=0
18:18:31 wc_send=0 release=0 get=0 capped=0
^C
prev parent reply other threads:[~2026-05-05 22:05 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-05-05 21:55 [RFC PATCH 0/2] svcrdma: avoid OOM due to unbounded sc_send_ctxts cache Mike Snitzer
2026-05-05 21:55 ` [RFC PATCH 1/2] svcrdma: bound per-xprt sc_send_ctxts cache and apply backpressure on _get Mike Snitzer
2026-05-06 6:01 ` Chuck Lever
2026-05-06 11:34 ` Mike Snitzer
2026-05-05 21:55 ` [RFC PATCH 2/2] for diagnostic use only: add svcrdma_wq lag diagnostic Mike Snitzer
2026-05-05 22:05 ` Mike Snitzer [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=afpprmApzykJFPci@kernel.org \
--to=snitzer@kernel.org \
--cc=ben.coddington@hammerspace.com \
--cc=cel@kernel.org \
--cc=jonathan.flynn@hammerspace.com \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.