Re: [LSF/MM/BPF TOPIC] Do not pin pages for various direct-io scheme

[Jerome Glisse <jglisse@redhat.com>]

On Wed, Jan 22, 2020 at 10:04:44AM -0700, Jens Axboe wrote:
> On 1/22/20 9:54 AM, Jerome Glisse wrote:
> > On Wed, Jan 22, 2020 at 08:12:51AM -0700, Jens Axboe wrote:
> >> On 1/22/20 4:59 AM, Michal Hocko wrote:
> >>> On Tue 21-01-20 20:57:23, Jerome Glisse wrote:
> >>>> We can also discuss what kind of knobs we want to expose so that
> >>>> people can decide to choose the tradeof themself (ie from i want low
> >>>> latency io-uring and i don't care wether mm can not do its business; to
> >>>> i want mm to never be impeded in its business and i accept the extra
> >>>> latency burst i might face in io operations).
> >>>
> >>> I do not think it is a good idea to make this configurable. How can
> >>> people sensibly choose between the two without deep understanding of
> >>> internals?
> >>
> >> Fully agree, we can't just punt this to a knob and call it good, that's
> >> a typical fallacy of core changes. And there is only one mode for
> >> io_uring, and that's consistent low latency. If this change introduces
> >> weird reclaim, compaction or migration latencies, then that's a
> >> non-starter as far as I'm concerned.
> >>
> >> And what do those two settings even mean? I don't even know, and a user
> >> sure as hell doesn't either.
> >>
> >> io_uring pins two types of pages - registered buffers, these are used
> >> for actual IO, and the rings themselves. The rings are not used for IO,
> >> just used to communicate between the application and the kernel.
> > 
> > So, do we still want to solve file back pages write back if page in
> > ubuffer are from a file ?
> 
> That's not currently a concern for io_uring, as it disallows file backed
> pages for the IO buffers that are being registered.
> 
> > Also we can introduce a flag when registering buffer that allows to
> > register buffer without pining and thus avoid the RLIMIT_MEMLOCK at
> > the cost of possible latency spike. Then user registering the buffer
> > knows what he gets.
> 
> That may be fine for others users, but I don't think it'll apply
> to io_uring. I can't see anyone selecting that flag, unless you're
> doing something funky where you're registering a substantial amount
> of the system memory for IO buffers. And I don't think that's going
> to be a super valid use case...

Given dataset are getting bigger and bigger i would assume that we
will have people who want to use io-uring with large buffer.

> 
> > Maybe it would be good to test, it might stay in the noise, then it
> > might be a good thing to do. Also they are strategy to avoid latency
> > spike for instance we can block/force skip mm invalidation if buffer
> > has pending/running io in the ring ie only have buffer invalidation
> > happens when there is no pending/running submission entry.
> 
> Would that really work? The buffer could very well be idle right when
> you check, but wanting to do IO the instant you decide you can do
> background work on it. Additionally, that would require accounting
> on when the buffers are inflight, which is exactly the kind of
> overhead we're trying to avoid to begin with.
> 
> > We can also pick what kind of invalidation we allow (compaction,
> > migration, ...) and thus limit the scope and likelyhood of
> > invalidation.
> 
> I think it'd be useful to try and understand the use case first.
> If we're pinning a small percentage of the system memory, do we
> really care at all? Isn't it completely fine to just ignore?

My main motivation is migration in NUMA system, if the process that
did register buffer get migrated to a different node then it might
actualy end up with bad performance because its io buffer are still
on hold node. I am not sure we want to tell application developer to
constantly monitor which node they are on and to re-register buffer
after process migration to allow for memory migration.

Cheers,
Jérôme