Re: [Qemu-devel] Re: [RFT][PATCH 07/15] qemu_irq: Add IRQ handlers with delivery feedback

[Blue Swirl <blauwirbel@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 6788 bytes --]

2010/5/30 Gleb Natapov <gleb@redhat.com>:
> On Sat, May 29, 2010 at 09:21:14PM +0000, Blue Swirl wrote:
>> On Sat, May 29, 2010 at 4:37 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> > On Sat, May 29, 2010 at 04:13:22PM +0000, Blue Swirl wrote:
>> >> On Sat, May 29, 2010 at 2:46 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> >> > On Sat, May 29, 2010 at 09:35:30AM +0000, Blue Swirl wrote:
>> >> >> > I still don't see how the alternative is supposed to simplify our life
>> >> >> > or improve the efficiency of the de-coalescing workaround. It's rather
>> >> >> > problematic like Gleb pointed out: The de-coalescing logic needs to be
>> >> >> > informed about periodicity changes that can only be delivered along
>> >> >> > IRQs. So what to do with the backlog when the timer is stopped?
>> >> >>
>> >> >> What happens with the current design? Gleb only mentioned the
>> >> >> frequency change, I thought that was not so big problem. But I don't
>> >> >> think this case should be allowed happen at all, it can't exist on
>> >> >> real HW.
>> >> >>
>> >> > Hm, why it can't exist on real HW? Do simple exercise. Run WindowsXP
>> >> > inside QEMU, connect with gdb to QEMU process and check what frequency
>> >> > RTC configured with (hint: it will be 64Hz), now run video inside the
>> >> > guest and check frequency again (hint: it will be 1Khz).
>> >>
>> >> You missed the key word 'stopped'. If the timer is really stopped, no
>> >> IRQs should ever come out afterwards, just like on real HW. For the
>> >> emulation, this means loss of ticks which should have been delivered
>> >> before the change.
>> >>
>> > I haven't missed it. I describe to you reality of the situation. You want
>> > to change reality to be more close to what you want it to be by adding
>> > words to my description.
>>
>> Quoting Jan: 'So what to do with the backlog when the timer is
>> stopped?' I didn't add any words to your description, please be more
>> careful with your attributions. Why do you think I want to change the
>> reality?
> Please refer to my words when you answer to my quote. You quoted my
> answer to you statement:
>  Gleb only mentioned the frequency change, I thought that was not so big
>  problem. But I don't think this case should be allowed happen at all,
>  it can't exist on real HW.

With 'this case' I was referring to 'case with timer stopped', not
'case which Gleb mentioned'.

> No 'stopped' was under discussion nowhere.

It's clearly written there in the sentence Jan wrote.

> FWIW 'stopped' is just a case
> of frequency change.

True.

>
>>
>> XP frequency change isn't the same case as timer being stopped.
>>
> And what is the big difference exactly?

Because after the timer is stopped, its extremely unrealistic to send
any IRQs. Whereas if the frequency is changed to some other nonzero
value, we can cheat and inject some more queued IRQs.

Anyway, if this case is not interesting because it doesn't happen in
real life emulation scenarios, we can forget it no matter how buggy
the current QEMU implementation is.

>> > Please just go write code, experiment, debug
>> > and _then_ come here with design.
>>
>> I added some debugging to RTC, PIC and APIC. I also built a small
>> guest in x86 assembly to test the coalescing. However, in the tests
>> with this guest and others I noticed that the coalescing only happens
>> in some obscure conditions.
> So try with real guest and with real load.

Well, I'd like to get the test program also trigger it. Now I'm getting:
apic: write: 00000350 = 00000000
apic: apic_reset_irq_delivered: old coalescing 0
apic: apic_local_deliver: vector 3 delivery mode 0
apic: apic_set_irq: coalescing 1
apic: apic_get_irq_delivered: returning coalescing 1
apic: apic_reset_irq_delivered: old coalescing 1
apic: apic_local_deliver: vector 3 delivery mode 0
apic: apic_set_irq: coalescing 0
apic: apic_get_irq_delivered: returning coalescing 0
apic: apic_reset_irq_delivered: old coalescing 0
apic: apic_local_deliver: vector 3 delivery mode 0
apic: apic_set_irq: coalescing 0

It looks like some other IRQs cause the coalescing, because also
looking at RTC code, it seems it's not possible for RTC to raise the
IRQ (except update IRQ, alarm etc.) without calling
apic_reset_irq_delivered().

I've attached my test program. Compile:
gcc -m32 -o coalescing coalescing.S -ffreestanding -nostdlib -Wl,-T
coalescing.ld -g && objcopy -Obinary coalescing coalescing.bin

Run:
qemu -L . -bios coalescing.bin -no-hpet -rtc-td-hack

>>
>> By default the APIC's delivery method for IRQs is ExtInt and
>> coalescing counting happens only with Fixed. This means that the guest
>> needs to reprogram APIC. It also looks like RTC interrupts need to be
>> triggered. But I didn't see both of these to happen simultaneously in
>> my tests with Linux and Windows guests. Of course, -rtc-td-hack flag
>> must be used and I also disabled HPET to be sure that RTC would be
>> used.
>>
>> With DEBUG_COALESCING enabled, I just get increasing numbers for
>> apic_irq_delivered:
>> apic: apic_set_irq: coalescing 67123
>> apic: apic_set_irq: coalescing 67124
>> apic: apic_set_irq: coalescing 67125
> So have you actually used -rtc-td-hack option? I compiled head of
> qemu.git with DEBUG_COALESCING and run WindowsXP guest with -rtc-td-hack
> and I get:
> apic: apic_reset_irq_delivered: old coalescing 3
> apic: apic_set_irq: coalescing 1
> apic: apic_get_irq_delivered: returning coalescing 1
> apic: apic_set_irq: coalescing 2
> apic: apic_set_irq: coalescing 3
> apic: apic_set_irq: coalescing 4
> apic: apic_set_irq: coalescing 5
> apic: apic_set_irq: coalescing 6
> apic: apic_reset_irq_delivered: old coalescing 6
> apic: apic_set_irq: coalescing 1
> apic: apic_get_irq_delivered: returning coalescing 1
>
>>
>> If the hack were active, the numbers would be close to zero (or at
>> least some point) because apic_reset_irq_delivered would be called,
>> but this does not happen. Could you specify a clear test case with
>> which the coalescing action could be tested? Linux or BSD based,
>> please.
> Linux don't use RTC as time source and I don't know about BSD, so no
> Linux or BSD test case for you, sorry. Run WindowXP standard HAL and put
> heavy load on the host. You can run video inside the gust to trigger
> coalescing more easily.

I don't have Windows XP, sorry.

>
>>
>> >> But what if the guest changed the frequency very often, and between
>> >> changes used zero value, like 64Hz -> 0Hz -> 128Hz -> 0Hz -> 64Hz?
>> > Too bad, the world is not perfect.
>> >
>> > --
>> >                        Gleb.
>> >
>
> --
>                        Gleb.
>

[-- Attachment #2: coalescing.S --]
[-- Type: application/octet-stream, Size: 2397 bytes --]


        .section .text, "ax"
        .code16gcc

#define O(port, val)            \
        movb    $val, %al;      \
        out     %al, $port
#define RTC_O(port, val)        \
        O(0x70, port);          \
        O(0x71, val)
#define RTC_I(port)             \
        O(0x70, port);          \
        in      $0x71, %al

        .org 0x10000
        .globl start
start:
        /* A20 */
        O(0x92, 0x01)

        /* IDT */
        lidtw   %cs:0x2000

        /* GDT */
        lgdtw   %cs:0x2010

        /* Switch to protected mode */
        movl    %cr0, %eax
        orl     $1, %eax
        movl    %eax, %cr0
        ljmpl   $8, $0xf4000
        .org 0x11000

        .code32

        /* INT 70 handler */
        RTC_I(0xc)
        O(0xa0, 0x20)
        O(0x20, 0x20)
        iret

        .org 0x12000

        /* IDTD */
        .short 0x0400
        .long 0x0f3000

        .org 0x12010

        /* GDTD */
        .short 0x0020
        .long 0xf3400

        .org 0x13380

        /* IDT entry for INT 70 */
        .long 0x00081000, 0x000f8e00

        .org 0x13408

        /* GDT entry for 1st descriptor, CS */
        .short 0xffff, 0x0000, 0x9b00, 0x00cf
        /* GDT entry for 2nd descriptor, DS etc. */
        .short 0xffff, 0x0000, 0x9300, 0x00cf

        .org 0x14000
        movl    $0x10, %eax
        movw    %ax, %ds
        movw    %ax, %ss
        movl    $0x1000, %esp

        /* Master PIC */
        /* ICW1 */
        O(0x20, 0x11)
        /* ICW2 */
        O(0x21, 0x08)
        /* ICW3 */
        O(0x21, 0x04)
        /* ICW4 */
        O(0x21, 0x01)
        /* OCW1: only slave IRQs */
        O(0x21, 0xfb)

        /* Slave PIC */
        /* ICW1 */
        O(0xa0, 0x11)
        /* ICW2 */
        O(0xa1, 0x70)
        /* ICW3 */
        O(0xa1, 0x02)
        /* ICW4 */
        O(0xa1, 0x01)
        /* OCW1: only RTC IRQ */
        O(0xa1, 0xfe)

        /* set up APIC LVT */
        xor     %eax, %eax
        mov     %eax, 0xfee00350

        /* RTC: frequency 1kHz */
        RTC_O(0xa, 0x26)
        /* Enable IRQ */
        RTC_O(0xb, 0x40)

        sti
1:      jmp     1b
        hlt

        .section .reset, "ax"
        .globl  entry
        .code16gcc
entry:
        cli
        ljmp    $0xf000, $0x0000
        nop
        nop
        nop
        nop
        nop
        nop
        nop
        nop
        nop
        nop


[-- Attachment #3: coalescing.ld --]
[-- Type: application/octet-stream, Size: 205 bytes --]

OUTPUT_FORMAT(elf32-i386)
OUTPUT_ARCH(i386)

ENTRY(entry)

SECTIONS
{
    . = 0xe0000;

    .text : { *(.text) }

    . = 0xf0000;

    .text : { *(.text) }

    . = 0xffff0;

    .reset : { *(.reset) }
}