[Qemu-devel] Performance regression "ioport: Switch dispatching to memory core layer"

[Qemu-devel] Performance regression "ioport: Switch dispatching to memory core layer"

[Stefan Hajnoczi]

The following qemu.git commit adds a ~2 minute regression in boot time
to a x86_64 live CD:

  commit b40acf99bef69fa8ab0f9092ff162fde945eec12
  Author: Jan Kiszka <jan.kiszka@siemens.com>
  Date:   Mon Jun 24 10:45:09 2013 +0200

      ioport: Switch dispatching to memory core layer

After the ISOLINUX menu, the display goes white and sits there for about
2 minutes on my laptop before the usual Linux boot screen appears.

$ qemu-system-x86_64 -enable-kvm -cpu host -m 1024 -cdrom ~/iso/livecd.iso

The parent commit immediately goes to the usual boot screen after
ISOLINUX.  Unfortunately I don't have a ISO download link for the live
CD.

Any ideas?

Stefan

Re: [Qemu-devel] Performance regression "ioport: Switch dispatching to memory core layer"

[Jan Kiszka]

On 2013-07-15 12:52, Stefan Hajnoczi wrote:
> The following qemu.git commit adds a ~2 minute regression in boot time
> to a x86_64 live CD:
> 
>   commit b40acf99bef69fa8ab0f9092ff162fde945eec12
>   Author: Jan Kiszka <jan.kiszka@siemens.com>
>   Date:   Mon Jun 24 10:45:09 2013 +0200
> 
>       ioport: Switch dispatching to memory core layer
> 
> After the ISOLINUX menu, the display goes white and sits there for about
> 2 minutes on my laptop before the usual Linux boot screen appears.
> 
> $ qemu-system-x86_64 -enable-kvm -cpu host -m 1024 -cdrom ~/iso/livecd.iso
> 
> The parent commit immediately goes to the usual boot screen after
> ISOLINUX.  Unfortunately I don't have a ISO download link for the live
> CD.

Can you share it otherwise (before I spend time on recreating something
that does not trigger it)?

> 
> Any ideas?

What does perf say during that time? What ftrace regarding the port
numbers and access types?

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Performance regression "ioport: Switch dispatching to memory core layer"

[Stefan Hajnoczi]

On Mon, Jul 15, 2013 at 7:01 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> On 2013-07-15 12:52, Stefan Hajnoczi wrote:
>> The following qemu.git commit adds a ~2 minute regression in boot time
>> to a x86_64 live CD:
>>
>>   commit b40acf99bef69fa8ab0f9092ff162fde945eec12
>>   Author: Jan Kiszka <jan.kiszka@siemens.com>
>>   Date:   Mon Jun 24 10:45:09 2013 +0200
>>
>>       ioport: Switch dispatching to memory core layer
>>
>> After the ISOLINUX menu, the display goes white and sits there for about
>> 2 minutes on my laptop before the usual Linux boot screen appears.
>>
>> $ qemu-system-x86_64 -enable-kvm -cpu host -m 1024 -cdrom ~/iso/livecd.iso
>>
>> The parent commit immediately goes to the usual boot screen after
>> ISOLINUX.  Unfortunately I don't have a ISO download link for the live
>> CD.
>
> Can you share it otherwise (before I spend time on recreating something
> that does not trigger it)?

I reproduced this now with an installed RHEL 6.4 guest (no ISO).

>>
>> Any ideas?
>
> What does perf say during that time? What ftrace regarding the port
> numbers and access types?

The guest kernel prints the following IPMI initialization output:

acpiphp: Slot [31] registered
ipmi message handler version 39.2
IPMI System Interface driver.
ipmi_si: Adding default-specified kcs state machine
ipmi_si: Trying default-specified kcs state machine at i/o address
0xca2, slave address 0x0, irq 0
ipmi_si: There appears to be no BMC at this location
ipmi_si: Adding default-specified smic state machine
ipmi_si: Trying default-specified smic state machine at i/o address
0xca9, slave address 0x0, irq 0
ipmi_smic_drv: smic hosed: state = SMIC_OP_OK, status != SMIC_SC_SMS_READY
ipmi_si: There appears to be no BMC at this location
ipmi_si: Adding default-specified bt state machine
ipmi_si: Trying default-specified bt state machine at i/o address
0xe4, slave address 0x0, irq 0
Refined TSC clocksource calibration: 2895.094 MHz.
[...wait...]
IPMI BT: timeout in RD_WAIT [ ] 1 retries left

Before this commit the guest boots immediately and does not get stuck
in IPMI initialization.

kvmstat:
 kvm_exit                                   2567180    3007
 kvm_entry                                  2567180    3007
 kvm_inj_virq                                 16226    1002
 kvm_emulate_insn                           2044389    1001
 kvm_userspace_exit                          538441    1001
 kvm_pio                                     502055    1001
 kvm_msr                                      16182    1001
 kvm_apic                                     15994    1001
 kvm_eoi                                      15876    1001
 kvm_apic_accept_irq                          15876    1001
 kvm_pv_eoi                                   15763    1001
 kvm_mmio                                   2006613       0
 vcpu_match_mmio                            1970359       0
 kvm_cr                                      144706       0
 kvm_cpuid                                    33451       0
 kvm_set_irq                                   1032       0
 kvm_pic_set_irq                               1024  0
 kvm_ioapic_set_irq                            1024       0
 kvm_page_fault                                1020  0
 kvm_ack_irq                                    277  0
 kvm_inj_exception                               33  0
 kvm_fpu                                          6  0
 kvm_track_tsc                                    2  0
 kvm_apic_ipi                                     2  0
 kvm_update_master_clock                          2  0

IPMI seems like the smoking gun.  perf shows a number of mmio/pio/msr
accesses, the longest repeated sequence is:

 qemu-system-x86  7318 [003]  4188.001381: kvm:kvm_pio: pio_read at
0xe4 size 1 count 1
 qemu-system-x86  7318 [003]  4188.002343: kvm:kvm_msr: msr_write 838 = 0xd6ff

Stefan