[Qemu-devel] User space vs kernel space instructions distribution.

[Qemu-devel] User space vs kernel space instructions distribution.

[Shlomo Pongratz]

[-- Attachment #1: Type: text/plain, Size: 615 bytes --]

Hi,

I'm running aarm64 QEMU and I'm counting the number of instructions which
"belong" to user space vs kernel space. My measurements shows that 99
percent of instructions are in kernel space.
I've used both the address of the instructions and the EL just to be sure.
I also added an option to disable block chaining just to make sure that all
the instructions in every TB  is counted.
When examining some kernel's instructions against the objdump of the kernel
I've noticed that most of them are in interrupts/timers area.

Does this make sense?
Did someone also encountered this phenomenon?

Best regards,

S.P.

[-- Attachment #2: Type: text/html, Size: 809 bytes --]

Re: [Qemu-devel] User space vs kernel space instructions distribution.

[Peter Maydell]

On 14 July 2015 at 09:32, Shlomo Pongratz <shlomopongratz@gmail.com> wrote:
> Hi,
>
> I'm running aarm64 QEMU and I'm counting the number of instructions which
> "belong" to user space vs kernel space. My measurements shows that 99
> percent of instructions are in kernel space.
> I've used both the address of the instructions and the EL just to be sure. I
> also added an option to disable block chaining just to make sure that all
> the instructions in every TB  is counted.
> When examining some kernel's instructions against the objdump of the kernel
> I've noticed that most of them are in interrupts/timers area.
>
> Does this make sense?
> Did someone also encountered this phenomenon?

Depends entirely on your workload, obviously. If the system only
boots then most instructions will be in kernel space. If the system
is only sitting idle then it'll just be executing the kernel space
idle loop. If you're measuring solely the section of time where
a userspace program is doing real work with the CPU and you're
still seeing a 99% figure then the obvious conclusion would be that
your measurement approach is wrong...

If your measurement instrumentation is intrusive and is significantly
slowing down QEMU then you'll naturally find that the guest spends
more time in timer interrupt handling, because the timer interrupts
come in in real time, and you've just effectively reduced the speed
of your CPU, so it can get less useful work done between timer
interrupts.

-- PMM

Re: [Qemu-devel] User space vs kernel space instructions distribution.

[Christopher Covington]

On 07/14/2015 04:45 AM, Peter Maydell wrote:
> On 14 July 2015 at 09:32, Shlomo Pongratz <shlomopongratz@gmail.com> wrote:
>> Hi,
>>
>> I'm running aarm64 QEMU and I'm counting the number of instructions which
>> "belong" to user space vs kernel space. My measurements shows that 99
>> percent of instructions are in kernel space.

How are you counting? Instrumenting QEMU?

>> I've used both the address of the instructions and the EL just to be sure. I
>> also added an option to disable block chaining just to make sure that all
>> the instructions in every TB  is counted.
>> When examining some kernel's instructions against the objdump of the kernel
>> I've noticed that most of them are in interrupts/timers area.
>>
>> Does this make sense?
>> Did someone also encountered this phenomenon?
> 
> Depends entirely on your workload, obviously. If the system only
> boots then most instructions will be in kernel space. If the system
> is only sitting idle then it'll just be executing the kernel space
> idle loop. If you're measuring solely the section of time where
> a userspace program is doing real work with the CPU and you're
> still seeing a 99% figure then the obvious conclusion would be that
> your measurement approach is wrong...
> 
> If your measurement instrumentation is intrusive and is significantly
> slowing down QEMU then you'll naturally find that the guest spends
> more time in timer interrupt handling, because the timer interrupts
> come in in real time, and you've just effectively reduced the speed
> of your CPU, so it can get less useful work done between timer
> interrupts.

I find such behavior undesirable. As best I understand, -icount exists to
provide an alternative, although it may have bugs. I've tinkered with -icount
a little, but I have yet to come up with useful tests to verify its correct
behavior. If anyone has suggestions for how to test it, I'd be eager to hear.

Chris

-- 
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [Qemu-devel] User space vs kernel space instructions distribution.

[Peter Maydell]

On 22 July 2015 at 17:45, Christopher Covington <cov@codeaurora.org> wrote:
> On 07/14/2015 04:45 AM, Peter Maydell wrote:
>> If your measurement instrumentation is intrusive and is significantly
>> slowing down QEMU then you'll naturally find that the guest spends
>> more time in timer interrupt handling, because the timer interrupts
>> come in in real time, and you've just effectively reduced the speed
>> of your CPU, so it can get less useful work done between timer
>> interrupts.
>
> I find such behavior undesirable. As best I understand, -icount exists to
> provide an alternative, although it may have bugs.

Well, the default is "go as fast as we can while presenting real
world time and events to the guest". That's mostly what people
want, except that it breaks down when "as fast as we can" turns
out to be "really slowly". (You can see this kind of thing if
you valgrind a TCG QEMU, for instance, I think.)

As you say, icount is supposed to be the other option. I've
never looked closely enough at it to figure out exactly
how it works...

-- PMM

Re: [Qemu-devel] User space vs kernel space instructions distribution.

[Shlomo Pongratz]

[-- Attachment #1: Type: text/plain, Size: 2616 bytes --]

See inline

On Wednesday, July 22, 2015, Christopher Covington <cov@codeaurora.org>
wrote:

> On 07/14/2015 04:45 AM, Peter Maydell wrote:
> > On 14 July 2015 at 09:32, Shlomo Pongratz <shlomopongratz@gmail.com
> <javascript:;>> wrote:
> >> Hi,
> >>
> >> I'm running aarm64 QEMU and I'm counting the number of instructions
> which
> >> "belong" to user space vs kernel space. My measurements shows that 99
> >> percent of instructions are in kernel space.
>
> How are you counting? Instrumenting QEMU?

I have an array of the three address spaces user/unmapped/kernel and array
of 4 els and I'm add the TB's icount to the appropriate entry according to
the env->pc and arm_current_el(env) before the block execution.
As I wrote before I disabled chaining so TB's icount is the number of
executed instructions.

>

>> I've used both the address of the instructions and the EL just to be
> sure. I
> >> also added an option to disable block chaining just to make sure that
> all
> >> the instructions in every TB  is counted.
> >> When examining some kernel's instructions against the objdump of the
> kernel
> >> I've noticed that most of them are in interrupts/timers area.
> >>
> >> Does this make sense?
> >> Did someone also encountered this phenomenon?
> >
> > Depends entirely on your workload, obviously. If the system only
> > boots then most instructions will be in kernel space. If the system
> > is only sitting idle then it'll just be executing the kernel space
> > idle loop. If you're measuring solely the section of time where
> > a userspace program is doing real work with the CPU and you're
> > still seeing a 99% figure then the obvious conclusion would be that
> > your measurement approach is wrong...
> >
> > If your measurement instrumentation is intrusive and is significantly
> > slowing down QEMU then you'll naturally find that the guest spends
> > more time in timer interrupt handling, because the timer interrupts
> > come in in real time, and you've just effectively reduced the speed
> > of your CPU, so it can get less useful work done between timer
> > interrupts.
>
> I find such behavior undesirable. As best I understand, -icount exists to
> provide an alternative, although it may have bugs. I've tinkered with
> -icount
> a little, but I have yet to come up with useful tests to verify its correct
> behavior. If anyone has suggestions for how to test it, I'd be eager to
> hear.
>
I don't use -icount.

>
> Chris
>
> --
> Qualcomm Innovation Center, Inc.
> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> a Linux Foundation Collaborative Project
>

[-- Attachment #2: Type: text/html, Size: 3570 bytes --]

Re: [Qemu-devel] User space vs kernel space instructions distribution.

[Peter Maydell]

On 23 July 2015 at 08:57, Shlomo Pongratz <shlomopongratz@gmail.com> wrote:
> I have an array of the three address spaces user/unmapped/kernel and array
> of 4 els and I'm add the TB's icount to the appropriate entry according to
> the env->pc and arm_current_el(env) before the block execution.
> As I wrote before I disabled chaining so TB's icount is the number of
> executed instructions.

Note that this will overcount unless you also have code to handle the "took
an exception in the middle of the TB" case and reduce the instruction
count accordingly.

thanks
-- PMM