Re: Direct I/O performance problems with 1GB pages

[David Hildenbrand <david@redhat.com>]

On 27.01.25 18:25, Andres Freund wrote:
> Hi,
> 
> On 2025-01-27 15:09:23 +0100, David Hildenbrand wrote:
>> Hmmm ... do we really want to make refcounting more complicated, and more
>> importantly, hugetlb-refcounting more special ?! :)
> 
> I don't know the answer to that - I mainly wanted to report the issue because
> it was pretty nasty to debug and initially surprising (to me).

Thanks for reporting!
> 
> 
>> If the workload doing a lot of single-page try_grab_folio_fast(), could it
>> do so on a larger area (multiple pages at once -> single refcount update)?
> 
> In the original case I hit this I (a VM with 10 PCIe 3x NVMEs JBODed
> together), the IO size averaged something like ~240kB (most 256kB, with some
> smaller ones thrown in). Increasing the IO size further than that starts to
> hurt latency and thus requires even deeper IO queues...

Makes sense.

> 
> Unfortunately for the VMs with those disks I don't have access to hardware
> performance counters :(.
 > >
>> Maybe there is a link to the report you could share, thanks.
> 
> A profile of the "original" case where I hit this, without the patch that
> Willy linked to:
> 
> Note this is a profile *not* using hardware perf counters, thus likely to be
> rather skewed:
> https://gist.github.com/anarazel/304aa6b81d05feb3f4990b467d02dabc
> (this was on Debian Sid's 6.12.6)
> 
> Without the patch I achieved ~18GB/s with 1GB pages and ~35GB/s with 2MB
> pages.

Out of interest, did you ever compare it to 4k?

> 
> After applying the patch to add an unlocked already-dirty check to
> bio_set_pages_dirty() performance improves to ~20GB/s when using 1GB pages.
> 
> A differential profile comparing 2MB and 1GB pages with the patch applied
> (again, without hardware perf counters):
> https://gist.github.com/anarazel/f993c238ea7d2c34f44440336d90ad8f
> 
> 
> Willy then asked me for perf annotate of where in gup_fast_fallback() time is
> spent.  I didn't have access to the VM at that point, and tried to repro the
> problem with local hardware.
> 
> 
> As I don't have quite enough IO throughput available locally, I couldn't repro
> the problem quite as easily. But after lowering the average IO size (which is
> not unrealistic, far from every workload is just a bulk sequential scan), it
> showed up when just using two PCIe 4 NVMe SSDs.
> 
> Here are profiles of the 2MB and 1GB cases, with the bio_set_pages_dirty()
> patch applied:
> https://gist.github.com/anarazel/f0d0a884c55ee18851dc9f15f03f7583
> 
> 2MB pages get ~12.5GB/s, 1GB pages ~7GB/s, with a *lot* of variance.

Thanks!

> 
> This time it's actual hardware perf counters...
> 
> Relevant details about the c2c report, excerpted from IRC:
> 
> andres | willy: Looking at a bit more detail into the c2c report, it looks
>           like the dirtying is due to folio->_pincount and folio->_refcount in
>           about equal measure and folio->flags being modified in
>           gup_fast_fallback(). The modifications then, unsurprisingly, cause a
>           lot of cache misses for reads (like in bio_set_pages_dirty() and
>           bio_check_pages_dirty()).
> 
>   willy | andres: that makes perfect sense, thanks
>   willy | really, the only way to fix that is to split it up
>   willy | and either we can split it per-cpu or per-physical-address-range

As discussed, even better is "not repeatedly pinning/unpinning" at all :)

I'm curious, are multiple processes involved, or is this all within a 
single process?

-- 
Cheers,

David / dhildenb