Re: [Question] vmalloc latency in RT-Linux

[Zhipeng Shi <zhipeng.shi0@gmail.com>]

On Fri, Jun 24, 2022 at 10:39:43AM +0800, Baoquan He wrote:
> On 06/23/22 at 02:04pm, Waiman Long wrote:
> > On 6/23/22 06:51, Baoquan He wrote:
> > > On 06/21/22 at 08:15pm, Zhipeng Shi wrote:
> > > > I noticed in rt-linux, vmalloc has a large latency. This is because the
> > > > free_vmap_area_lock is held for a long time in the function
> > > > __purge_vmap_area_lazy.
> > > > 
> > > > In non-RT-Linux, because the function spin_is_contended is well
> > > > implemented, so there will be no such problem.
> > > > 
> > > > But in RT-Linux, spin_is_contended simply returns 0. I don't understand
> > > > why this function was implemented like this before, but in order to
> > > > solve this problem, I thought of two ways.
> > > > 
> > > > The first is to modify the spin_is_contended definition in spinlock_rt.h
> > > > as shown below, but I'm not sure if the change has side-effects:
> > > > 
> > > > -#define spin_is_contended(lock)    (((void)(lock), 0))
> > > > +static inline int spin_is_contended(spinlock_t *lock)
> > > > +{
> > > > + unsigned long *p = (unsigned long *) &lock->lock.owner;
> > > > +
> > > > + return (READ_ONCE(*p) & RT_MUTEX_HAS_WAITERS);
> > > > +}
> > > > 
> > > > The second is by reducing the number of lazy_max_pages, but it will lead
> > > > to lower performance of vmalloc.
> > > __purge_vmap_area_lazy() has cond_resched_lock() to reschedule and drop
> > > the lock. From your saying, it's spin_is_contended() which is not
> > > working well to make rescheduling happen during __purge_vmap_area_lazy()
> > > handling. Then the fixing should be done in lock side.
> > 
> > Sebastian had sent out patch last year to fix spin_is_contended().
> > 
> > https://lore.kernel.org/lkml/20210906143004.2259141-3-bigeasy@linutronix.de/
> > 
> > However, there is no follow-up after some discussion and the patch wasn't
> > merged.
> 
> That's great. Thanks, Longman.
> 
> Then this is a good chance to reconsider it, maybe with a test from Zhipeng.

Before that, since I didn't find the patch that Sebastian sent before, 
I sent relevant patch for this problem (now it seems that Sebastian's 
changes are better than mine) and test scripts. please refer to the 
following links:

https://lore.kernel.org/lkml/20220608142457.GA2400218@ubuntu20/

With this patch, max-latency of vmalloc reduce from 10+ msec to
200+ usec, this because spin_lock is released halfway through 
__purge_vmap_area_lazy.

Best regards,
Zhipeng