From: Zdenek Kabelac <zdenek.kabelac@gmail.com>
To: LVM general discussion and development <linux-lvm@redhat.com>,
Anton Kulshenko <shallriseagain@gmail.com>
Subject: Re: [linux-lvm] ThinPool performance problem with NVMe
Date: Mon, 17 Jul 2023 14:26:22 +0200 [thread overview]
Message-ID: <36ecd456-b80e-7afe-9fc9-a91b160781af@gmail.com> (raw)
In-Reply-To: <CAA3xStJ7bRT5iiUy3fY3fkbiHDChbGw5RjMksSQuhPYUXijBCA@mail.gmail.com>
Dne 24. 06. 23 v 1:22 Anton Kulshenko napsal(a):
> Hello.
>
> Please help me figure out what my problem is. No matter how I configure the
> system, I can't get high performance, especially on writes.
>
> OS: Oracle Linux 8.6, 5.4.17-2136.311.6.el8uek.x86_64
> Platform: Gigabyte R282-Z94 with 2x 7702 64cores AMD EPYC and 2 TB of RAM
> Disks: NVMe Samsung PM1733 7.68 TB
>
> What I do:
> vgcreate vg1 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1
> lvcreate -n thin_pool_1 -L 20T vg1 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1
> /dev/nvme3n1 -i 4 -I 4
>
> -i4 for striping between all disks, -I4 strip size. Also I tried 8, 16, 32...
> In my setup I can't find a big difference.
>
stripe size needs to be aligned with some 'hw' properties.
In the case of 'NVMe' where the write unit for optimal performance is usually
0.5M or more - using 4K block is basically massively destroying your
performance since you generate for each large write huge amount of splits.
> I only get 40k iops, while one drive at the same load easily gives 130k iops.
> I have tried different block sizes, strip sizes, etc. with no result. When I
> look in iostat I see the load on the disk where the metadata is:
> 80 WMB/s, 12500 wrqm/s, 68 %wrqm
>
> I don't understand what I'm missing when configuring the system.
>
As mentioned by Mathew - you likely should start with some 'initial' thin-pool
size - maybe sitting fully on single NVMe - possibly deploy metadata on 2nd.
NVMe for better bus utilization.
For striping - you would need to go with 512K units at least - then it's the
question how it fits your workload...
Anyway now you have way more things to experiment and benchmark and figure out
what is the best on your particular hw.
One more thing - increasing chunksize to 256K or 512K also may significantly
raise the performance - but at the price of reduced sharing in case of taking
a snapshot of a thin volume...
Regards
Zdenek
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
prev parent reply other threads:[~2023-07-17 12:26 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-23 23:22 [linux-lvm] ThinPool performance problem with NVMe Anton Kulshenko
2023-07-14 15:07 ` ComputerAdvancedTechnologySYSTEM
2023-07-17 12:26 ` Zdenek Kabelac [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=36ecd456-b80e-7afe-9fc9-a91b160781af@gmail.com \
--to=zdenek.kabelac@gmail.com \
--cc=linux-lvm@redhat.com \
--cc=shallriseagain@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).