Re: xen/arm: Domain not fully destroyed when using credit2

[Julien Grall <julien.grall@arm.com>]

Hi Dario,

On 25/01/17 11:10, Dario Faggioli wrote:
> On Tue, 2017-01-24 at 15:06 +0000, Julien Grall wrote:
>> On 24/01/17 14:16, Dario Faggioli wrote:
>>> There, we have tracing (BTW, did that made it to ARM eventually?)
>>> and
>>> there's TRC_PM_IDLE_ENTRY/EXIT which do pretty much the same of
>>> your
>>> printk-s.
>>
>> There is patch on the ML for xentrace support (see [1]) but nothing
>> has
>> been upstreamed yet. Waiting for a new version from the contributor.
>>
> Yep, that was I was remembering, and referring to. Thanks for the
> update.
>
>>> And if I look at it, I do see even totally idle (from the scheduler
>>> point of view) pCPUs, I indeed see them going back and forth from
>>> and
>>> to C3.
>>
>> My knowledge on x86 is limited. When does a CPU decides to leave the
>> idle mode?
>>
> I'm not an expert of that part either. Jan and Andrew for sure know
> best how monitor/mwait works (both in general, and our own
> implementation).
>
> What I know (and can quickly infer from glancing at the code), is that
> timers are certainly involved.
>
> In fact, we wake up when the most imminent timer would expire (see
> mwait_idle_with_hints()), and a timer set by the scheduler fully
> qualifies as being the one (if it's the most imminent).
>
> My point was that, still from scheduling perspective, neither Credit1
> nor Credit2 sets a wakeup timer for idle pCPUs.
>
> Well, in Credit1, the master_ticker timer is never stopped (while,
> e.g., the per-pCPU tick is stopped before entering deep sleep,
> via sched_tick_suspend(), see commit 964fae8ac), but that's only 1
> pCPU.

The function sched_tick_suspend is never called on ARM. The power saving 
in Xen ARM is still very limited and this would need to be updated in 
the future.

So I guess that's why I still see interrupt coming on the idle pCPU when 
credit1 is used. Looking at credit2, the callback tick_suspend is not 
called. Does it mean there is no per-pCPU timer?

Now, from my understanding, if we decide to call sched_tick_suspend on 
ARM before idling. We will likely have the same problem with credit1 
because there is no more interrupt to wake-up the pCPU.

But I don't think this is an issue in the scheduler. IHMO, the problem 
is in the RCU. Indeed a CPU in lower power mode (i.e  wfi on ARM or 
pm_idle on x86 is been executed) will never get out to tell to the RCU : 
"I am quiet, go ahead". So the RCU will never be able to reclaim the 
memory and will result on a memory exhaustion if the pCPU never receive 
an interrupt (this could happen if pCPU has never ran a guest).

The question now, is how to fix it?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel