[Qemu-devel] PPC: Timer issues with git master

[Qemu-devel] PPC: Timer issues with git master

[Mark Cave-Ayland]

Hi all,

I've just upgraded my git repository from around the 1.6 release 
timeframe to git master and I'm seeing some strange timing issues 
running through my OpenBIOS test suite which weren't there before my git 
update.

The two most noticeable changes are in FreeBSD and HelenOS; the 
countdown timer in the FreeBSD bootloader used to count down in seconds 
whereas now it whizzes down to zero, and HelenOS now takes a very very 
long time to boot.

In order to reproduce from git master download the appropriate ISOs and 
launch QEMU like this:

./qemu-system-ppc -cdrom 
/home/build/src/qemu/image/ppc/FreeBSD-9.1-RC2-powerpc-release.iso -M 
mac99 -m 512 -boot d

./qemu-system-ppc -cdrom 
/home/build/src/qemu/image/ppc/HelenOS-0.4.2-ppc32.iso -m 512 -boot d

Paulo/Alex B - I know you've been working with timers, so wondered if 
this is related to some of the changes you've been making? I've also 
copied Alex Graf in case this is somehow related to how the timer code 
interacts with the PPC timebase-frequency.


Many thanks,

Mark.

Re: [Qemu-devel] PPC: Timer issues with git master

[Paolo Bonzini]

Il 11/10/2013 00:27, Mark Cave-Ayland ha scritto:
> 
> I've just upgraded my git repository from around the 1.6 release
> timeframe to git master and I'm seeing some strange timing issues
> running through my OpenBIOS test suite which weren't there before my git
> update.
> 
> The two most noticeable changes are in FreeBSD and HelenOS; the
> countdown timer in the FreeBSD bootloader used to count down in seconds
> whereas now it whizzes down to zero, and HelenOS now takes a very very
> long time to boot.
> 
> In order to reproduce from git master download the appropriate ISOs and
> launch QEMU like this:
> 
> ./qemu-system-ppc -cdrom
> /home/build/src/qemu/image/ppc/FreeBSD-9.1-RC2-powerpc-release.iso -M
> mac99 -m 512 -boot d
> 
> ./qemu-system-ppc -cdrom
> /home/build/src/qemu/image/ppc/HelenOS-0.4.2-ppc32.iso -m 512 -boot d
> 
> Paulo/Alex B - I know you've been working with timers, so wondered if
> this is related to some of the changes you've been making? I've also
> copied Alex Graf in case this is somehow related to how the timer code
> interacts with the PPC timebase-frequency.

Can you bisect them?

Paolo

Re: [Qemu-devel] PPC: Timer issues with git master

[Mark Cave-Ayland]

On 11/10/13 09:44, Paolo Bonzini wrote:

>> I've just upgraded my git repository from around the 1.6 release
>> timeframe to git master and I'm seeing some strange timing issues
>> running through my OpenBIOS test suite which weren't there before my git
>> update.
>>
>> The two most noticeable changes are in FreeBSD and HelenOS; the
>> countdown timer in the FreeBSD bootloader used to count down in seconds
>> whereas now it whizzes down to zero, and HelenOS now takes a very very
>> long time to boot.
>>
>> In order to reproduce from git master download the appropriate ISOs and
>> launch QEMU like this:
>>
>> ./qemu-system-ppc -cdrom
>> /home/build/src/qemu/image/ppc/FreeBSD-9.1-RC2-powerpc-release.iso -M
>> mac99 -m 512 -boot d
>>
>> ./qemu-system-ppc -cdrom
>> /home/build/src/qemu/image/ppc/HelenOS-0.4.2-ppc32.iso -m 512 -boot d
>>
>> Paulo/Alex B - I know you've been working with timers, so wondered if
>> this is related to some of the changes you've been making? I've also
>> copied Alex Graf in case this is somehow related to how the timer code
>> interacts with the PPC timebase-frequency.
>
> Can you bisect them?
>
> Paolo

Hi Paolo,

I've just attempted a bisection testing HelenOS, but I'm struggling to 
get a consistent result. Even with the same binary across multiple runs 
then sometimes I see the issue with frequent timer pauses, and sometimes 
I don't which makes tracking this down very difficult.

What I do see post-timer-rework is that when it does occur, the screen 
redraws become really really slow; rectangular blocks (maybe 2 per sec 
or so?) are redrawn very slowly working their way down the screen. I 
have a feeling from memory that HelenOS does tend to hit the timer 
interrupts quite hard on PPC so could it be that the new code somehow 
blocks screen updates under high timer interrupt load?

As for the FreeBSD issue, this seems to be something different from the 
HelenOS issue and I see it even with QEMU 1.6. A quick browse around the 
git logs points me towards this as a possible suspect: 
http://git.qemu.org/?p=qemu.git;a=commit;h=a0f9fdfd98cc0571f9921a7eadd7316532e3e289.


ATB,

Mark.

Re: [Qemu-devel] PPC: Timer issues with git master

[Paolo Bonzini]

Il 17/10/2013 16:40, Mark Cave-Ayland ha scritto:
> 
> Hi Paolo,
> 
> I've just attempted a bisection testing HelenOS, but I'm struggling to
> get a consistent result. Even with the same binary across multiple runs
> then sometimes I see the issue with frequent timer pauses, and sometimes
> I don't which makes tracking this down very difficult.
> 
> What I do see post-timer-rework is that when it does occur, the screen
> redraws become really really slow; rectangular blocks (maybe 2 per sec
> or so?) are redrawn very slowly working their way down the screen. I
> have a feeling from memory that HelenOS does tend to hit the timer
> interrupts quite hard on PPC so could it be that the new code somehow
> blocks screen updates under high timer interrupt load?
> 
> As for the FreeBSD issue, this seems to be something different from the
> HelenOS issue and I see it even with QEMU 1.6. A quick browse around the
> git logs points me towards this as a possible suspect:
> http://git.qemu.org/?p=qemu.git;a=commit;h=a0f9fdfd98cc0571f9921a7eadd7316532e3e289.

Can you try reverting this on top of git master (and test both guests)?

Paolo

Re: [Qemu-devel] PPC: Timer issues with git master

[Mark Cave-Ayland]

On 17/10/13 15:53, Paolo Bonzini wrote:

>> Hi Paolo,
>>
>> I've just attempted a bisection testing HelenOS, but I'm struggling to
>> get a consistent result. Even with the same binary across multiple runs
>> then sometimes I see the issue with frequent timer pauses, and sometimes
>> I don't which makes tracking this down very difficult.
>>
>> What I do see post-timer-rework is that when it does occur, the screen
>> redraws become really really slow; rectangular blocks (maybe 2 per sec
>> or so?) are redrawn very slowly working their way down the screen. I
>> have a feeling from memory that HelenOS does tend to hit the timer
>> interrupts quite hard on PPC so could it be that the new code somehow
>> blocks screen updates under high timer interrupt load?
>>
>> As for the FreeBSD issue, this seems to be something different from the
>> HelenOS issue and I see it even with QEMU 1.6. A quick browse around the
>> git logs points me towards this as a possible suspect:
>> http://git.qemu.org/?p=qemu.git;a=commit;h=a0f9fdfd98cc0571f9921a7eadd7316532e3e289.
>
> Can you try reverting this on top of git master (and test both guests)?

I've tried it with FreeBSD and it doesn't make any difference; the fix 
doesn't matter for the HelenOS regression as that timer is only setup 
for -M mac99.

To summarise what I've found today with HelenOS: I see failures with 
both 1.6 and git master, but git master appears to be a lot worse in 
terms of triggering the problem.

Also with git master, I see time periods where the timer speeds up for a 
couple of seconds at a time (as apparent by the HelenOS circling orbs) 
and then slows back down again. And the keyboard (whilst sluggish under 
1.6) is very unresponsive under git master.

I'm starting to wonder like Alex B if these changes have made an already 
existing bug more apparent.


ATB,

Mark.

Re: [Qemu-devel] PPC: Timer issues with git master

[Alex Bligh]

Mark,

On 17 Oct 2013, at 16:16, Mark Cave-Ayland wrote:

> I've tried it with FreeBSD and it doesn't make any difference; the fix doesn't matter for the HelenOS regression as that timer is only setup for -M mac99.
> 
> To summarise what I've found today with HelenOS: I see failures with both 1.6 and git master, but git master appears to be a lot worse in terms of triggering the problem.
> 
> Also with git master, I see time periods where the timer speeds up for a couple of seconds at a time (as apparent by the HelenOS circling orbs) and then slows back down again. And the keyboard (whilst sluggish under 1.6) is very unresponsive under git master.
> 
> I'm starting to wonder like Alex B if these changes have made an already existing bug more apparent.

Perhaps you might put some debugging in timerlist_run_timers()
in qemu-timer.c before 'cb(opaque)', dumping the address
of the timer (ts) and the expire time (ts->expire_time). If
you see this firing off constantly (note the frequency you
see the message and the increment in expire_time), you will
know some timer is expiring too frequently.

If that's the issue, it will be a matter of tracking it down,
which can probably be achieved with gdb and putting a breakpoint
in timer_mod_ns. Statistically you are likely to hit the "right"
one, and backtrace should give you a clue as to what it is.
Continue a few times and see if it's the right timer, and you
should find the culprit.

If it's an rtc problem on the other hand, you will have a
a different issue.

-- 
Alex Bligh

Re: [Qemu-devel] PPC: Timer issues with git master

[Alex Bligh]

On 10 Oct 2013, at 23:27, Mark Cave-Ayland wrote:

> Paulo/Alex B - I know you've been working with timers, so wondered if this is related to some of the changes you've been making? I've also copied Alex Graf in case this is somehow related to how the timer code interacts with the PPC timebase-frequency.

It sounds more likely to be an RTC problem than a timers problem to be honest, though I prepared to be proved wrong.

-- 
Alex Bligh