From: Wang Yugui <wangyugui@e16-tech.com>
To: Dave Chinner <david@fromorbit.com>
Cc: linux-xfs@vger.kernel.org
Subject: Re: performance regression between 6.1.x and 5.15.x
Date: Wed, 10 May 2023 16:50:56 +0800 [thread overview]
Message-ID: <20230510165055.01D5.409509F4@e16-tech.com> (raw)
In-Reply-To: <20230510072706.GX3223426@dread.disaster.area>
[-- Attachment #1: Type: text/plain, Size: 2664 bytes --]
Hi,
> On Wed, May 10, 2023 at 01:46:49PM +0800, Wang Yugui wrote:
> > > Ok, that is further back in time than I expected. In terms of XFS,
> > > there are only two commits between 5.16..5.17 that might impact
> > > performance:
> > >
> > > ebb7fb1557b1 ("xfs, iomap: limit individual ioend chain lengths in writeback")
> > >
> > > and
> > >
> > > 6795801366da ("xfs: Support large folios")
> > >
> > > To test whether ebb7fb1557b1 is the cause, go to
> > > fs/iomap/buffered-io.c and change:
> > >
> > > -#define IOEND_BATCH_SIZE 4096
> > > +#define IOEND_BATCH_SIZE 1048576
> > > This will increase the IO submission chain lengths to at least 4GB
> > > from the 16MB bound that was placed on 5.17 and newer kernels.
> > >
> > > To test whether 6795801366da is the cause, go to fs/xfs/xfs_icache.c
> > > and comment out both calls to mapping_set_large_folios(). This will
> > > ensure the page cache only instantiates single page folios the same
> > > as 5.16 would have.
> >
> > 6.1.x with 'mapping_set_large_folios remove' and 'IOEND_BATCH_SIZE=1048576'
> > fio WRITE: bw=6451MiB/s (6764MB/s)
> >
> > still performance regression when compare to linux 5.16.20
> > fio WRITE: bw=7666MiB/s (8039MB/s),
> >
> > but the performance regression is not too big, then difficult to bisect.
> > We noticed samle level performance regression on btrfs too.
> > so maby some problem of some code that is used by both btrfs and xfs
> > such as iomap and mm/folio.
>
> Yup, that's quite possibly something like the multi-gen LRU changes,
> but that's not the regression we need to find. :/
>
> > 6.1.x with 'mapping_set_large_folios remove' only'
> > fio WRITE: bw=2676MiB/s (2806MB/s)
> >
> > 6.1.x with 'IOEND_BATCH_SIZE=1048576' only'
> > fio WRITE: bw=5092MiB/s (5339MB/s),
> > fio WRITE: bw=6076MiB/s (6371MB/s)
> >
> > maybe we need more fix or ' ebb7fb1557b1 ("xfs, iomap: limit
> > individual ioend chain lengths in writeback")'.
>
> OK, can you re-run the two 6.1.x kernels above (the slow and the
> fast) and record the output of `iostat -dxm 1` whilst the
> fio test is running? I want to see what the overall differences in
> the IO load on the devices are between the two runs. This will tell
> us how the IO sizes and queue depths change between the two kernels,
> etc.
`iostat -dxm 1` result saved in attachment file.
good.txt good performance
bad.txt bad performance
Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2023/05/10
>
> Right now I'm suspecting a contention interaction between write(),
> do_writepages() and folio_end_writeback()...
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
[-- Attachment #2: good.txt --]
[-- Type: application/octet-stream, Size: 17825 bytes --]
[root@T620 ~]# iostat -dxm 1
Linux 6.1.28-0.6.el7.x86_64 (T620) 05/10/2023 _x86_64_ (40 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 203.46 0.17 20.83 0.00 13.92 1357.81 0.21 9.62 0.09 9.70 0.61 1.29
nvme2c2n1 0.00 203.40 0.18 20.87 0.00 13.93 1355.38 0.09 4.37 0.08 4.40 0.27 0.57
nvme1n1 0.00 203.40 0.18 20.89 0.00 13.93 1353.59 0.08 3.53 0.03 3.56 0.29 0.61
nvme3n1 0.00 203.61 0.21 20.87 0.00 13.92 1352.33 0.15 7.20 0.05 7.27 0.38 0.79
sda 0.00 0.02 0.85 0.13 0.03 0.00 60.13 0.00 0.72 0.61 1.42 0.78 0.08
dm-0 0.00 0.00 0.05 42.88 0.00 2.67 127.43 1.04 24.19 0.07 24.22 0.04 0.18
dm-1 0.00 0.00 0.03 42.77 0.00 2.67 127.81 1.37 31.79 0.11 31.81 0.09 0.40
dm-2 0.00 0.00 0.03 42.77 0.00 2.67 127.82 0.89 20.79 0.09 20.80 0.03 0.12
dm-3 0.00 0.00 0.03 42.77 0.00 2.67 127.82 0.70 16.30 0.02 16.31 0.03 0.14
dm-4 0.00 0.00 0.14 171.18 0.00 10.68 127.72 3.99 23.27 0.07 23.29 0.02 0.41
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 25582.00 0.00 1368.00 0.00 1705.69 2553.55 15.84 11.58 0.00 11.58 0.62 85.00
nvme2c2n1 0.00 25583.00 0.00 1347.00 0.00 1680.68 2555.33 5.53 4.10 0.00 4.10 0.45 60.50
nvme1n1 0.00 25590.00 0.00 1348.00 0.00 1680.75 2553.54 3.85 2.86 0.00 2.86 0.45 61.10
nvme3n1 0.00 25585.00 0.00 1371.00 0.00 1709.50 2553.65 14.62 10.66 0.00 10.66 0.65 89.00
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 26940.00 0.00 1682.44 127.90 283.90 10.54 0.00 10.54 0.03 89.20
dm-1 0.00 0.00 0.00 26952.00 0.00 1682.57 127.85 308.07 11.43 0.00 11.43 0.03 85.30
dm-2 0.00 0.00 0.00 26948.00 0.00 1682.62 127.88 102.20 3.79 0.00 3.79 0.02 62.30
dm-3 0.00 0.00 0.00 26951.00 0.00 1682.77 127.87 68.75 2.55 0.00 2.55 0.02 62.80
dm-4 0.00 0.00 0.00 107893.00 0.00 6736.76 127.88 762.46 7.07 0.00 7.07 0.01 91.40
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 26957.00 0.00 1421.00 0.00 1771.55 2553.23 14.10 9.92 0.00 9.92 0.63 89.20
nvme2c2n1 0.00 26955.00 0.00 1421.00 0.00 1771.60 2553.29 5.67 3.99 0.00 3.99 0.44 63.20
nvme1n1 0.00 26953.00 0.00 1421.00 0.00 1771.60 2553.30 3.80 2.67 0.00 2.67 0.44 63.20
nvme3n1 0.00 26955.00 0.00 1421.00 0.00 1771.56 2553.24 15.01 10.56 0.00 10.56 0.64 91.00
sda 0.00 0.00 0.00 5.00 0.00 0.04 18.20 0.01 1.20 0.00 1.20 0.60 0.30
dm-0 0.00 0.00 0.00 28370.00 0.00 1771.02 127.85 292.33 10.30 0.00 10.30 0.03 91.40
dm-1 0.00 0.00 0.00 28357.00 0.00 1770.87 127.90 273.60 9.65 0.00 9.65 0.03 89.30
dm-2 0.00 0.00 0.00 28359.00 0.00 1770.85 127.89 105.64 3.73 0.00 3.73 0.02 64.50
dm-3 0.00 0.00 0.00 28361.00 0.00 1770.73 127.87 68.33 2.41 0.00 2.41 0.02 64.40
dm-4 0.00 0.00 0.00 113345.00 0.00 7077.08 127.87 739.98 6.53 0.00 6.53 0.01 94.40
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 29349.00 0.00 1499.00 0.00 1869.08 2553.62 29.05 19.38 0.00 19.38 0.60 90.60
nvme2c2n1 0.00 29355.00 0.00 1529.00 0.00 1906.56 2553.72 6.26 4.10 0.00 4.10 0.45 68.60
nvme1n1 0.00 29352.00 0.00 1537.00 0.00 1915.27 2552.04 4.21 2.74 0.00 2.74 0.45 68.50
nvme3n1 0.00 29353.00 0.00 1512.00 0.00 1884.06 2551.96 24.56 16.24 0.00 16.24 0.63 94.90
sda 0.00 0.00 0.00 1.00 0.00 0.00 8.00 0.00 0.00 0.00 0.00 1.00 0.10
dm-0 0.00 0.00 0.00 30902.00 0.00 1929.52 127.88 481.12 15.57 0.00 15.57 0.03 95.10
dm-1 0.00 0.00 0.00 30897.00 0.00 1929.46 127.89 571.80 18.51 0.00 18.51 0.03 90.40
dm-2 0.00 0.00 0.00 30903.00 0.00 1929.44 127.87 116.15 3.76 0.00 3.76 0.02 69.30
dm-3 0.00 0.00 0.00 30901.00 0.00 1929.46 127.88 74.80 2.42 0.00 2.42 0.02 69.20
dm-4 0.00 0.00 0.00 123603.00 0.00 7717.87 127.88 1243.94 10.06 0.00 10.06 0.01 96.40
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 25950.00 0.00 1392.00 0.00 1734.20 2551.47 23.06 16.56 0.00 16.56 0.60 83.20
nvme2c2n1 0.00 25954.00 0.00 1387.00 0.00 1729.16 2553.22 5.69 4.10 0.00 4.10 0.44 61.40
nvme1n1 0.00 25947.00 0.00 1380.00 0.00 1720.43 2553.22 4.02 2.91 0.00 2.91 0.45 61.80
nvme3n1 0.00 25954.00 0.00 1378.00 0.00 1717.91 2553.17 18.68 13.55 0.00 13.55 0.63 87.20
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 27322.00 0.00 1705.69 127.85 364.35 13.34 0.00 13.34 0.03 87.40
dm-1 0.00 0.00 0.00 27319.00 0.00 1705.76 127.87 451.39 16.52 0.00 16.52 0.03 83.50
dm-2 0.00 0.00 0.00 27322.00 0.00 1705.72 127.86 104.82 3.84 0.00 3.84 0.02 62.40
dm-3 0.00 0.00 0.00 27315.00 0.00 1705.68 127.89 71.30 2.61 0.00 2.61 0.02 62.50
dm-4 0.00 0.00 0.00 109278.00 0.00 6822.85 127.87 992.64 9.08 0.00 9.08 0.01 89.60
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 26937.00 0.00 1416.00 0.00 1765.16 2552.99 35.40 25.00 0.00 25.00 0.63 89.90
nvme2c2n1 0.00 26944.00 0.00 1420.00 0.00 1770.19 2553.06 5.75 4.05 0.00 4.05 0.45 63.80
nvme1n1 0.00 26938.00 0.00 1420.00 0.00 1770.18 2553.05 3.86 2.72 0.00 2.72 0.45 63.30
nvme3n1 0.00 26936.00 0.00 1448.00 0.00 1803.89 2551.35 6.91 4.77 0.00 4.77 0.52 75.40
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 28357.00 0.00 1770.76 127.89 129.60 4.57 0.00 4.57 0.03 76.20
dm-1 0.00 0.00 0.00 28357.00 0.00 1770.77 127.89 698.41 24.63 0.00 24.63 0.03 89.90
dm-2 0.00 0.00 0.00 28364.00 0.00 1770.81 127.86 106.04 3.74 0.00 3.74 0.02 65.30
dm-3 0.00 0.00 0.00 28358.00 0.00 1770.81 127.89 68.45 2.41 0.00 2.41 0.02 64.80
dm-4 0.00 0.00 0.00 113436.00 0.00 7083.15 127.88 1002.26 8.84 0.00 8.84 0.01 93.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 24627.00 0.00 1318.00 0.00 1645.44 2556.80 11.61 8.81 0.00 8.81 0.63 83.40
nvme2c2n1 0.00 24631.00 0.00 1298.00 0.00 1619.25 2554.87 5.13 3.95 0.00 3.95 0.44 57.60
nvme1n1 0.00 24634.00 0.00 1298.00 0.00 1619.25 2554.88 3.47 2.68 0.00 2.68 0.44 56.90
nvme3n1 0.00 24634.00 0.00 1298.00 0.00 1619.23 2554.84 4.19 3.23 0.00 3.23 0.52 67.70
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 25932.00 0.00 1619.05 127.87 75.80 2.92 0.00 2.92 0.03 69.40
dm-1 0.00 0.00 0.00 25925.00 0.00 1619.02 127.90 224.33 8.65 0.00 8.65 0.03 84.10
dm-2 0.00 0.00 0.00 25929.00 0.00 1619.06 127.88 94.71 3.65 0.00 3.65 0.02 59.30
dm-3 0.00 0.00 0.00 25932.00 0.00 1619.07 127.87 61.35 2.37 0.00 2.37 0.02 58.90
dm-4 0.00 0.00 0.00 103718.00 0.00 6476.21 127.88 455.83 4.39 0.00 4.39 0.01 84.90
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 27534.00 0.00 1462.00 0.00 1821.66 2551.82 25.20 17.24 0.00 17.24 0.60 88.40
nvme2c2n1 0.00 27537.00 0.00 1452.00 0.00 1810.38 2553.48 6.29 4.33 0.00 4.33 0.44 64.00
nvme1n1 0.00 27535.00 0.00 1452.00 0.00 1810.38 2553.48 4.08 2.81 0.00 2.81 0.44 63.90
nvme3n1 0.00 27532.00 0.00 1452.00 0.00 1810.40 2553.51 6.85 4.72 0.00 4.72 0.52 75.00
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 28992.00 0.00 1810.63 127.90 128.65 4.44 0.00 4.44 0.03 75.50
dm-1 0.00 0.00 0.00 28998.00 0.00 1810.60 127.87 494.88 17.07 0.00 17.07 0.03 88.60
dm-2 0.00 0.00 0.00 29001.00 0.00 1810.56 127.86 117.12 4.04 0.00 4.04 0.02 65.00
dm-3 0.00 0.00 0.00 29000.00 0.00 1810.56 127.86 73.24 2.53 0.00 2.53 0.02 64.90
dm-4 0.00 0.00 0.00 116103.00 0.00 7249.36 127.88 813.95 7.01 0.00 7.01 0.01 90.80
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 26478.00 0.00 1391.00 0.00 1735.71 2555.53 22.99 16.53 0.00 16.53 0.63 87.00
nvme2c2n1 0.00 26471.00 0.00 1395.00 0.00 1740.75 2555.60 5.75 4.12 0.00 4.12 0.44 61.60
nvme1n1 0.00 26477.00 0.00 1395.00 0.00 1740.75 2555.60 3.82 2.74 0.00 2.74 0.44 61.20
nvme3n1 0.00 26480.00 0.00 1395.00 0.00 1740.78 2555.64 6.88 4.93 0.00 4.93 0.52 72.60
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 27867.00 0.00 1739.59 127.85 129.73 4.66 0.00 4.66 0.03 73.40
dm-1 0.00 0.00 0.00 27861.00 0.00 1739.65 127.88 452.05 16.23 0.00 16.23 0.03 87.40
dm-2 0.00 0.00 0.00 27854.00 0.00 1739.65 127.91 107.19 3.85 0.00 3.85 0.02 63.00
dm-3 0.00 0.00 0.00 27859.00 0.00 1739.62 127.89 68.79 2.47 0.00 2.47 0.02 62.70
dm-4 0.00 0.00 0.00 111329.00 0.00 6951.52 127.88 757.81 6.81 0.00 6.81 0.01 89.40
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 26664.00 0.00 1399.00 0.00 1741.75 2549.75 34.17 24.42 0.00 24.42 0.62 87.40
nvme2c2n1 0.00 26664.00 0.00 1395.00 0.00 1736.70 2549.64 5.63 4.03 0.00 4.03 0.45 63.10
nvme1n1 0.00 26670.00 0.00 1400.00 0.00 1742.94 2549.67 3.90 2.78 0.00 2.78 0.45 63.40
nvme3n1 0.00 26664.00 0.00 1399.00 0.00 1741.72 2549.71 7.68 5.49 0.00 5.49 0.52 72.40
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 28071.00 0.00 1752.78 127.88 144.22 5.14 0.00 5.14 0.03 73.40
dm-1 0.00 0.00 0.00 28071.00 0.00 1752.81 127.88 673.15 23.98 0.00 23.98 0.03 87.90
dm-2 0.00 0.00 0.00 28071.00 0.00 1752.76 127.88 102.83 3.66 0.00 3.66 0.02 64.10
dm-3 0.00 0.00 0.00 28077.00 0.00 1752.75 127.85 67.94 2.42 0.00 2.42 0.02 64.90
dm-4 0.00 0.00 0.00 112290.00 0.00 7011.10 127.87 988.33 8.80 0.00 8.80 0.01 90.20
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 28862.00 0.00 1524.00 0.00 1900.19 2553.53 38.41 25.20 0.00 25.20 0.61 92.40
nvme2c2n1 0.00 28857.00 0.00 1525.00 0.00 1901.43 2553.53 6.28 4.12 0.00 4.12 0.45 68.50
nvme1n1 0.00 28853.00 0.00 1524.00 0.00 1900.17 2553.51 4.21 2.76 0.00 2.76 0.45 68.40
nvme3n1 0.00 28847.00 0.00 1524.00 0.00 1900.12 2553.45 9.62 6.31 0.00 6.31 0.52 79.70
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 30369.00 0.00 1896.88 127.92 183.75 6.05 0.00 6.05 0.03 80.30
dm-1 0.00 0.00 0.00 30384.00 0.00 1896.86 127.86 758.64 24.97 0.00 24.97 0.03 92.60
dm-2 0.00 0.00 0.00 30379.00 0.00 1896.91 127.88 117.22 3.86 0.00 3.86 0.02 69.80
dm-3 0.00 0.00 0.00 30375.00 0.00 1896.92 127.90 75.70 2.49 0.00 2.49 0.02 69.10
dm-4 0.00 0.00 0.00 121507.00 0.00 7587.55 127.89 1135.30 9.34 0.00 9.34 0.01 94.20
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 26847.00 0.00 1411.00 0.00 1757.56 2551.02 19.05 13.50 0.00 13.50 0.61 86.00
nvme2c2n1 0.00 26836.00 0.00 1425.00 0.00 1775.09 2551.14 5.94 4.17 0.00 4.17 0.44 63.00
nvme1n1 0.00 26847.00 0.00 1421.00 0.00 1770.14 2551.20 3.93 2.77 0.00 2.77 0.44 62.40
nvme3n1 0.00 26843.00 0.00 1422.00 0.00 1771.41 2551.23 5.43 3.82 0.00 3.82 0.52 74.00
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 28259.00 0.00 1764.54 127.88 98.97 3.50 0.00 3.50 0.03 74.60
dm-1 0.00 0.00 0.00 28263.00 0.00 1764.42 127.85 370.40 13.11 0.00 13.11 0.03 86.30
dm-2 0.00 0.00 0.00 28252.00 0.00 1764.46 127.91 108.94 3.86 0.00 3.86 0.02 64.20
dm-3 0.00 0.00 0.00 28263.00 0.00 1764.52 127.86 69.18 2.45 0.00 2.45 0.02 63.60
dm-4 0.00 0.00 0.00 113037.00 0.00 7057.93 127.88 647.52 5.73 0.00 5.73 0.01 88.30
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 24345.00 0.00 1300.00 0.00 1619.89 2551.95 12.49 9.61 0.00 9.61 0.63 82.20
nvme2c2n1 0.00 24347.00 0.00 1285.00 0.00 1601.16 2551.89 5.07 3.94 0.00 3.94 0.45 58.20
nvme1n1 0.00 24349.00 0.00 1285.00 0.00 1601.19 2551.93 3.46 2.69 0.00 2.69 0.46 58.60
nvme3n1 0.00 24350.00 0.00 1285.00 0.00 1601.23 2552.01 4.56 3.55 0.00 3.55 0.53 68.00
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 25635.00 0.00 1600.61 127.87 81.69 3.19 0.00 3.19 0.03 71.80
dm-1 0.00 0.00 0.00 25630.00 0.00 1600.55 127.89 240.19 9.37 0.00 9.37 0.03 85.10
dm-2 0.00 0.00 0.00 25632.00 0.00 1600.54 127.88 91.86 3.58 0.00 3.58 0.02 62.90
dm-3 0.00 0.00 0.00 25634.00 0.00 1600.56 127.88 59.96 2.34 0.00 2.34 0.02 63.40
dm-4 0.00 0.00 0.00 102531.00 0.00 6402.26 127.88 473.77 4.62 0.00 4.62 0.01 86.70
^C
[root@T620 ~]#
[-- Attachment #3: bad.txt --]
[-- Type: application/octet-stream, Size: 8279 bytes --]
[root@T620 ~]# iostat -dxm 1
Linux 6.1.28-0.5.el7.x86_64 (T620) 05/10/2023 _x86_64_ (40 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 779.13 1.45 62.83 0.03 52.31 1667.62 0.36 3.89 0.07 3.97 1.17 7.50
nvme2c2n1 0.00 775.15 1.49 62.80 0.03 52.33 1667.93 0.07 1.06 0.09 1.09 0.47 3.04
nvme1n1 0.00 775.15 1.54 62.86 0.03 52.33 1665.16 0.07 0.92 0.03 0.94 0.52 3.33
nvme3n1 0.00 777.00 1.77 62.83 0.03 52.33 1659.94 0.11 1.53 0.05 1.57 0.60 3.86
sda 0.02 0.10 12.71 0.76 0.43 0.02 67.86 0.00 0.32 0.26 1.45 0.40 0.55
dm-0 0.00 0.00 0.85 839.78 0.01 52.33 127.50 1.41 1.66 0.05 1.66 0.05 3.86
dm-1 0.00 0.00 0.45 841.92 0.01 52.33 127.24 2.74 3.13 0.09 3.13 0.09 7.50
dm-2 0.00 0.00 0.48 837.90 0.00 52.33 127.84 0.94 1.12 0.10 1.12 0.04 3.05
dm-3 0.00 0.00 0.53 837.95 0.00 52.33 127.83 0.76 0.90 0.01 0.90 0.04 3.34
dm-4 0.00 0.00 2.32 3357.55 0.02 209.32 127.60 5.83 1.70 0.06 1.70 0.02 7.72
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 10015.00 0.00 542.00 0.00 663.12 2505.68 1.71 3.15 0.00 3.15 0.96 51.90
nvme2c2n1 0.00 10015.00 0.00 539.00 0.00 659.34 2505.25 1.20 2.22 0.00 2.22 0.76 40.90
nvme1n1 0.00 10014.00 0.00 539.00 0.00 659.31 2505.14 1.11 2.07 0.00 2.07 0.74 39.70
nvme3n1 0.00 10014.00 0.00 539.00 0.00 659.31 2505.14 1.37 2.54 0.00 2.54 0.94 50.70
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 10553.00 0.00 659.56 128.00 18.83 1.78 0.00 1.78 0.05 52.10
dm-1 0.00 0.00 0.00 10554.00 0.00 659.62 128.00 25.53 2.42 0.00 2.42 0.05 53.30
dm-2 0.00 0.00 0.00 10554.00 0.00 659.59 127.99 15.50 1.47 0.00 1.47 0.04 42.80
dm-3 0.00 0.00 0.00 10553.00 0.00 659.56 128.00 13.93 1.32 0.00 1.32 0.04 41.50
dm-4 0.00 0.00 0.00 42214.00 0.00 2638.34 128.00 73.85 1.75 0.00 1.75 0.01 54.70
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 9258.00 0.00 502.00 0.00 610.50 2490.65 1.58 3.15 0.00 3.15 0.96 48.00
nvme2c2n1 0.00 9258.00 0.00 502.00 0.00 610.50 2490.65 1.12 2.24 0.00 2.24 0.75 37.60
nvme1n1 0.00 9258.00 0.00 502.00 0.00 610.50 2490.65 1.05 2.09 0.00 2.09 0.74 37.10
nvme3n1 0.00 9258.00 0.00 503.00 0.00 610.51 2485.73 1.27 2.53 0.00 2.53 0.91 45.60
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 9761.00 0.00 610.01 127.99 16.79 1.72 0.00 1.72 0.05 48.30
dm-1 0.00 0.00 0.00 9760.00 0.00 610.00 128.00 22.74 2.33 0.00 2.33 0.05 50.40
dm-2 0.00 0.00 0.00 9760.00 0.00 610.00 128.00 13.89 1.42 0.00 1.42 0.04 40.40
dm-3 0.00 0.00 0.00 9760.00 0.00 610.00 128.00 12.56 1.29 0.00 1.29 0.04 40.00
dm-4 0.00 0.00 0.00 39041.00 0.00 2440.01 128.00 66.05 1.69 0.00 1.69 0.01 51.30
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 7498.00 0.00 407.00 0.00 494.00 2485.78 1.30 3.19 0.00 3.19 0.94 38.30
nvme2c2n1 0.00 7533.00 0.00 407.00 0.00 493.97 2485.64 0.99 2.43 0.00 2.43 0.75 30.50
nvme1n1 0.00 7497.00 0.00 407.00 0.00 494.00 2485.78 0.92 2.26 0.00 2.26 0.73 29.60
nvme3n1 0.00 7497.00 0.00 407.00 0.00 494.00 2485.78 1.04 2.55 0.00 2.55 0.85 34.50
sda 0.00 0.00 0.00 1.00 0.00 0.00 8.00 0.00 1.00 0.00 1.00 1.00 0.10
dm-0 0.00 0.00 0.00 7904.00 0.00 494.00 128.00 12.50 1.58 0.00 1.58 0.05 37.10
dm-1 0.00 0.00 0.00 7905.00 0.00 494.00 127.98 17.70 2.24 0.00 2.24 0.05 40.60
dm-2 0.00 0.00 0.00 7940.00 0.00 494.00 127.42 11.46 1.44 0.00 1.44 0.04 33.10
dm-3 0.00 0.00 0.00 7904.00 0.00 494.00 128.00 10.23 1.29 0.00 1.29 0.04 32.30
dm-4 0.00 0.00 0.00 31653.00 0.00 1976.00 127.85 51.94 1.64 0.00 1.64 0.01 40.90
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 9697.00 0.00 528.00 0.00 639.25 2479.52 1.65 3.12 0.00 3.12 0.99 52.50
nvme2c2n1 0.00 9725.00 0.00 529.00 0.00 639.28 2474.93 1.11 2.09 0.00 2.09 0.76 40.30
nvme1n1 0.00 9697.00 0.00 528.00 0.00 639.25 2479.52 1.03 1.95 0.00 1.95 0.74 39.30
nvme3n1 0.00 9697.00 0.00 528.00 0.00 639.25 2479.52 1.07 2.02 0.00 2.02 0.77 40.80
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 10224.00 0.00 639.00 128.00 12.52 1.22 0.00 1.22 0.04 42.70
dm-1 0.00 0.00 0.00 10224.00 0.00 639.00 128.00 23.98 2.35 0.00 2.35 0.05 54.20
dm-2 0.00 0.00 0.00 10253.00 0.00 639.00 127.64 13.46 1.31 0.00 1.31 0.04 42.10
dm-3 0.00 0.00 0.00 10224.00 0.00 639.00 128.00 11.66 1.14 0.00 1.14 0.04 41.20
dm-4 0.00 0.00 0.00 40925.00 0.00 2556.00 127.91 61.66 1.51 0.00 1.51 0.01 54.70
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 6690.00 0.00 365.00 0.00 440.75 2473.03 1.25 3.42 0.00 3.42 0.94 34.40
nvme2c2n1 0.00 6737.00 0.00 367.00 0.00 440.72 2459.40 0.95 2.59 0.00 2.59 0.75 27.70
nvme1n1 0.00 6690.00 0.00 365.00 0.00 440.75 2473.03 0.88 2.42 0.00 2.42 0.75 27.20
nvme3n1 0.00 6690.00 0.00 365.00 0.00 440.75 2473.03 0.90 2.46 0.00 2.46 0.75 27.40
sda 0.00 0.00 0.00 1.00 0.00 0.00 8.00 0.00 1.00 0.00 1.00 1.00 0.10
dm-0 0.00 0.00 0.00 7056.00 0.00 441.00 128.00 9.82 1.39 0.00 1.39 0.04 28.80
dm-1 0.00 0.00 0.00 7056.00 0.00 441.00 128.00 16.67 2.36 0.00 2.36 0.05 35.70
dm-2 0.00 0.00 0.00 7105.00 0.00 441.00 127.12 10.70 1.51 0.00 1.51 0.04 29.10
dm-3 0.00 0.00 0.00 7056.00 0.00 441.00 128.00 9.60 1.36 0.00 1.36 0.04 28.50
dm-4 0.00 0.00 0.00 28273.00 0.00 1764.00 127.78 46.83 1.66 0.00 1.66 0.01 36.00
^C
[root@T620 ~]#
next prev parent reply other threads:[~2023-05-10 8:51 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-08 9:24 performance regression between 6.1.x and 5.15.x Wang Yugui
2023-05-08 14:46 ` Wang Yugui
2023-05-08 22:32 ` Dave Chinner
2023-05-08 23:25 ` Wang Yugui
2023-05-09 1:36 ` Dave Chinner
2023-05-09 12:37 ` Wang Yugui
2023-05-09 22:14 ` Dave Chinner
2023-05-10 5:46 ` Wang Yugui
2023-05-10 7:27 ` Dave Chinner
2023-05-10 8:50 ` Wang Yugui [this message]
2023-05-11 1:34 ` Dave Chinner
2023-05-17 13:07 ` Wang Yugui
2023-05-17 22:11 ` Dave Chinner
2023-05-18 18:36 ` Creating large folios in iomap buffered write path Matthew Wilcox
2023-05-18 21:46 ` Matthew Wilcox
2023-05-18 22:03 ` Matthew Wilcox
2023-05-19 2:55 ` Wang Yugui
2023-05-19 15:38 ` Matthew Wilcox
2023-05-20 13:35 ` Wang Yugui
2023-05-20 16:35 ` Matthew Wilcox
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230510165055.01D5.409509F4@e16-tech.com \
--to=wangyugui@e16-tech.com \
--cc=david@fromorbit.com \
--cc=linux-xfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.