Re: [PATCH v7 00/13] fold per-CPU vmstats remotely

Marcelo Tosatti
2023-05-03T13:53:58Z

On Wed, Apr 19, 2023 at 04:15:50PM -0300, Marcelo Tosatti wrote:
> On Wed, Apr 19, 2023 at 06:47:30PM +0200, Vlastimil Babka wrote:
> > On 4/19/23 13:29, Marcelo Tosatti wrote:
> > > On Wed, Apr 19, 2023 at 08:14:09AM -0300, Marcelo Tosatti wrote:
> > >> This was tried before:
> > >> https://lore.kernel.org/lkml/20220127173037.318440631@fedora.localdomain/
> > >>
> > >> My conclusion from that discussion (and work) is that a special system
> > >> call:
> > >>
> > >> 1) Does not allow the benefits to be widely applied (only modified
> > >> applications will benefit). Is not portable across different operating systems.
> > >>
> > >> Removing the vmstat_work interruption is a benefit for HPC workloads,
> > >> for example (in fact, it is a benefit for any kind of application,
> > >> since the interruption causes cache misses).
> > >>
> > >> 2) Increases the system call cost for applications which would use
> > >> the interface.
> > >>
> > >> So avoiding the vmstat_update update interruption, without userspace
> > >> knowledge and modifications, is a better than solution than a modified
> > >> userspace.
> > >
> > > Another important point is this: if an application dirties
> > > its own per-CPU vmstat cache, while performing a system call,
> > > and a vmstat sync event is triggered on a different CPU, you'd have to:
> > >
> > > 1) Wait for that CPU to return to userspace and sync its stats
> > > (unfeasible).
> > >
> > > 2) Queue work to execute on that CPU (undesirable, as that causes
> > > an interruption).
> >
> > So you're saying the application might do a syscall from the isolcpu, so
> > IIUC it cannot expect any latency guarantees at that very moment,
>
> Why not? cyclictest uses nanosleep and its the main tool for measuring
> latency.
>
> > but then
> > it immediately starts expecting them again after returning to userspace,
>
> No, the expectation more generally is this:
>
> For certain types of applications (for example PLC software or
> RAN processing), upon occurrence of an event, it is necessary to
> complete a certain task in a maximum amount of time (deadline).
>
> One way to express this requirement is with a pair of numbers,
> deadline time and execution time, where:
>
> * deadline time: length of time between event and deadline.
> * execution time: length of time it takes for processing of event
> to occur on a particular hardware platform
> (uninterrupted).
>
> The particular values depend on use-case. For the case
> where the realtime application executes in a virtualized
> guest, an interruption which must be serviced in the host will cause
> the following sequence of events:
>
> 1) VM-exit
> 2) execution of IPI (and function call) (or switch to kwork
> thread to execute some work item).
> 3) VM-entry
>
> Which causes an excess of 50us latency as observed by cyclictest
> (this violates the latency requirement of vRAN application with 1ms TTI,
> for example).
>
> > and
> > a single interruption for a one-time flush after the syscall would be too
> > intrusive?
>
> Generally, if you can't complete the task (which involves executing a
> number of instructions) before the deadline, then its a problem.
>
> One-time flush? You mean to switch between:
>
> rt-app -> kworker (to execute vmstat_update flush) -> rt-app
>
> My measurement, which probably had vmstat_update code/data in cache, took 7us.
> It might be the case that the code to execute must be brought in from
> memory, which takes even longer.
>
> > (elsewhere in the thread you described an RT app initialization that may
> > generate vmstats to flush and then entry userspace loop, again, would a
> > single interruption soon after entering the loop be so critical?)
>
> 1) It depends on the application. For the use-case above, where < 50us
> interruption is desired, yes it is critical.
>
> 2) The interruptions can come from different sources.
>
> Time
> 0 rt-app executing instruction 1
> 1 rt-app executing instruction 2
> 2 scheduler switches between rt-app and kworker
> 3 kworker runs vmstat_work
> 4 scheduler switches between kworker and rt-app
> 5 rt-app executing instruction 3
> 6 ipi to handle a KVM request IPI
> 7 fill in your preferred IPI handler
>
> So the argument "a single interruption might not cause your deadline
> to be exceeded" fails (because the time to handle the
> different interruptions might sum).
>
> Does that make sense?

Ping ? (just want to double check the reasoning above makes sense).