public inbox for kvm@vger.kernel.org
 help / color / mirror / Atom feed
* qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
@ 2011-01-06  7:48 Nikola Ciprich
  2011-01-06  9:03 ` Stefan Hajnoczi
  2011-01-06  9:08 ` Avi Kivity
  0 siblings, 2 replies; 29+ messages in thread
From: Nikola Ciprich @ 2011-01-06  7:48 UTC (permalink / raw)
  To: KVM list; +Cc: nikola.ciprich

Hi,
I'd like to ask for advice with following problem.
I have windows 2008 terminal server guest running on 2.6.36 x86_64
host (kvm 0.13.0).
guest has 4GB of RAM, 40GB storage on top of LVM volume and two cores. 
So far everything was running fine, but during periodic maintenance 
I wanted to force chkdisk after reboot.
So windows started checking disk integrity, but the problem is, that
it's waaay too slow - after ~12 hours, it's still running and seeems
like it'll take ages to finish.
Both CPU cores seem to be fully loaded.
Is there some way I could check why it's taking so long, and fix
it eventually?
can I use kvm_trace to achieve this task? how?
I'll be very gratefull for any help...
with best regards
nik

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06  7:48 qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow Nikola Ciprich
@ 2011-01-06  9:03 ` Stefan Hajnoczi
  2011-01-06  9:06   ` Nikola Ciprich
  2011-01-06  9:08 ` Avi Kivity
  1 sibling, 1 reply; 29+ messages in thread
From: Stefan Hajnoczi @ 2011-01-06  9:03 UTC (permalink / raw)
  To: Nikola Ciprich; +Cc: KVM list, nikola.ciprich

On Thu, Jan 6, 2011 at 7:48 AM, Nikola Ciprich <extmaillist@linuxbox.cz> wrote:
> So windows started checking disk integrity, but the problem is, that
> it's waaay too slow - after ~12 hours, it's still running and seeems
> like it'll take ages to finish.

Please post your KVM command-line.

Have you run storage benchmarks on the host to check what sort of
maximum I/O performance you can expect?  Do you have a RAID setup
underneath LVM?

Stefan

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06  9:03 ` Stefan Hajnoczi
@ 2011-01-06  9:06   ` Nikola Ciprich
  0 siblings, 0 replies; 29+ messages in thread
From: Nikola Ciprich @ 2011-01-06  9:06 UTC (permalink / raw)
  To: Stefan Hajnoczi; +Cc: KVM list, nikola.ciprich

Hello Stefan!
> Please post your KVM command-line.
/usr/bin/qemu-kvm -S -M pc-0.13 -enable-kvm -m 4096 -smp 2,sockets=2,cores=1,threads=1 -name vmwts02 -uuid 1e501300-dc48-11df-a690-00304834195b -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/vmwts02.monitor,server,nowait -mon chardev=monitor,mode=readline -rtc base=localtime -boot c -drive file=/dev/vgshared/vmwts02-1,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=22,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=00:16:3e:61:01:00,bus=pci.0,addr=0x3 -usb -vnc 0.0.0.0:30801 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4

I see I could disable cache for storage, but I don't want to kill
fsck now just to test if it helps (and I guess it shouldn't make
such a difference).

> Have you run storage benchmarks on the host to check what sort of
> maximum I/O performance you can expect?  Do you have a RAID setup
> underneath LVM?
Not for windows, but in general it is running quite fast, only the chkdsk
seems to be bad. In other VMs (linux), I'm achieving write speeds >40MB/s.
Storage configuration is a bit comples, it's DRBD replicated storage,
on top of it sits clustered LVM and KVMS use logical volumes on top of it.
but as I said, overall performance is OK.

> 
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06  7:48 qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow Nikola Ciprich
  2011-01-06  9:03 ` Stefan Hajnoczi
@ 2011-01-06  9:08 ` Avi Kivity
  2011-01-06  9:20   ` Nikola Ciprich
  1 sibling, 1 reply; 29+ messages in thread
From: Avi Kivity @ 2011-01-06  9:08 UTC (permalink / raw)
  To: Nikola Ciprich; +Cc: KVM list, nikola.ciprich

On 01/06/2011 09:48 AM, Nikola Ciprich wrote:
> Hi,
> I'd like to ask for advice with following problem.
> I have windows 2008 terminal server guest running on 2.6.36 x86_64
> host (kvm 0.13.0).
> guest has 4GB of RAM, 40GB storage on top of LVM volume and two cores.
> So far everything was running fine, but during periodic maintenance
> I wanted to force chkdisk after reboot.
> So windows started checking disk integrity, but the problem is, that
> it's waaay too slow - after ~12 hours, it's still running and seeems
> like it'll take ages to finish.
> Both CPU cores seem to be fully loaded.
> Is there some way I could check why it's taking so long, and fix
> it eventually?
> can I use kvm_trace to achieve this task? how?

Let's start with a few 'kvm_stat -1' snapshots while this is going on.

http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=blob_plain;f=kvm/kvm_stat;hb=HEAD

-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06  9:08 ` Avi Kivity
@ 2011-01-06  9:20   ` Nikola Ciprich
  2011-01-06  9:27     ` Avi Kivity
  0 siblings, 1 reply; 29+ messages in thread
From: Nikola Ciprich @ 2011-01-06  9:20 UTC (permalink / raw)
  To: Avi Kivity; +Cc: KVM list, nikola.ciprich

Hello Avi!
On Thu, Jan 06, 2011 at 11:08:32AM +0200, Avi Kivity wrote:
> Let's start with a few 'kvm_stat -1' snapshots while this is going on.

here it is, but note that there are few more machines running on this host.
but they're almost idle in contrast to this windows one, so I hope it's not problem.

kvm_ack_irq                  299       290
kvm_age_page                  71        69
kvm_apic                    1126      1087
kvm_apic_accept_irq          453       438
kvm_apic_ipi                 236       228
kvm_cpuid                      0         0
kvm_cr                       780       772
kvm_emulate_insn           46181     44982
kvm_entry                  55012     53867
kvm_exit                   55182     53884
kvm_exit(APIC_ACCESS)          3         0
kvm_exit(CPUID)                3         0
kvm_exit(CR_ACCESS)          784       772
kvm_exit(DR_ACCESS)            3         0
kvm_exit(EPT_MISCONFIG)         4         0
kvm_exit(EPT_VIOLATION)         4         0
kvm_exit(EXCEPTION_NMI)     50194     49039
kvm_exit(EXTERNAL_INTERRUPT)      2314      2213
kvm_exit(HLT)                260       248
kvm_exit(INVALID_STATE)         3         0
kvm_exit(INVLPG)             103        98
kvm_exit(IO_INSTRUCTION)      1538      1534
kvm_exit(MCE_DURING_VMENTRY)         5         0
kvm_exit(MONITOR_INSTRUCTION)         3         0
kvm_exit(MSR_READ)             5         0
kvm_exit(MSR_WRITE)            4         0
kvm_exit(MWAIT_INSTRUCTION)         3         0
kvm_exit(NMI_WINDOW)           4         0
kvm_exit(PAUSE_INSTRUCTION)         4         0
kvm_exit(PENDING_INTERRUPT)        12         7
kvm_exit(RDPMC)                4         0
kvm_exit(RDTSC)                4         0
kvm_exit(TASK_SWITCH)          3         0
kvm_exit(TPR_BELOW_THRESHOLD)         6         2
kvm_exit(TRIPLE_FAULT)         5         0
kvm_exit(VMCALL)               4         0
kvm_exit(VMCLEAR)              3         0
kvm_exit(VMLAUNCH)             3         0
kvm_exit(VMOFF)                4         0
kvm_exit(VMON)                 3         0
kvm_exit(VMPTRLD)              4         0
kvm_exit(VMPTRST)              4         0
kvm_exit(VMREAD)               5         0
kvm_exit(VMRESUME)             3         0
kvm_exit(VMWRITE)              4         0
kvm_exit(WBINVD)               3         0
kvm_exit(XSETBV)               4         0
kvm_fpu                      247       243
kvm_hv_hypercall               0         0
kvm_hypercall                  0         0
kvm_inj_exception              0         0
kvm_inj_virq                 514       500
kvm_invlpga                    0         0
kvm_ioapic_set_irq           363       353
kvm_mmio                   67725     65974
kvm_msi_set_irq                0         0
kvm_msr                        0         0
kvm_nested_intercepts          0         0
kvm_nested_intr_vmexit         0         0
kvm_nested_vmexit              0         0
kvm_nested_vmexit_inject         0         0
kvm_nested_vmrun               0         0
kvm_page_fault             50200     48785
kvm_pic_set_irq              363       353
kvm_pio                     1541      1541
kvm_set_irq                  363       353
kvm_skinit                     0         0


>
> http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=blob_plain;f=kvm/kvm_stat;hb=HEAD
>
> -- 
> error compiling committee.c: too many arguments to function
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06  9:20   ` Nikola Ciprich
@ 2011-01-06  9:27     ` Avi Kivity
  2011-01-06  9:30       ` Avi Kivity
  2011-01-06  9:42       ` Nikola Ciprich
  0 siblings, 2 replies; 29+ messages in thread
From: Avi Kivity @ 2011-01-06  9:27 UTC (permalink / raw)
  To: Nikola Ciprich; +Cc: KVM list, nikola.ciprich

On 01/06/2011 11:20 AM, Nikola Ciprich wrote:
> Hello Avi!
> On Thu, Jan 06, 2011 at 11:08:32AM +0200, Avi Kivity wrote:
> >  Let's start with a few 'kvm_stat -1' snapshots while this is going on.
>
> here it is, but note that there are few more machines running on this host.
> but they're almost idle in contrast to this windows one, so I hope it's not problem.

> kvm_cr                       780       772
> kvm_emulate_insn           46181     44982
> kvm_entry                  55012     53867
> kvm_exit                   55182     53884

It's emulating quite a bit.  Let's see why.

- install udis86 and udis86-devel
- build and install 
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
- run trace-cmd record -e kvm -b 100000 -P pid1 -P pid2, ctrl-C after a 
few seconds (pid1/pid2 are thread ids from 'info cpus' of the bad guest, 
plus the pid of the qemu process itself)
- post the resulting trace.dat somewhere


-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06  9:27     ` Avi Kivity
@ 2011-01-06  9:30       ` Avi Kivity
  2011-01-06  9:42       ` Nikola Ciprich
  1 sibling, 0 replies; 29+ messages in thread
From: Avi Kivity @ 2011-01-06  9:30 UTC (permalink / raw)
  To: Nikola Ciprich; +Cc: KVM list, nikola.ciprich

On 01/06/2011 11:27 AM, Avi Kivity wrote:
> On 01/06/2011 11:20 AM, Nikola Ciprich wrote:
>> Hello Avi!
>> On Thu, Jan 06, 2011 at 11:08:32AM +0200, Avi Kivity wrote:
>> >  Let's start with a few 'kvm_stat -1' snapshots while this is going 
>> on.
>>
>> here it is, but note that there are few more machines running on this 
>> host.
>> but they're almost idle in contrast to this windows one, so I hope 
>> it's not problem.
>
>> kvm_cr                       780       772
>> kvm_emulate_insn           46181     44982
>> kvm_entry                  55012     53867
>> kvm_exit                   55182     53884
>
> It's emulating quite a bit.  Let's see why.
>
> - install udis86 and udis86-devel
> - build and install 
> git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
> - run trace-cmd record -e kvm -b 100000 -P pid1 -P pid2, ctrl-C after 
> a few seconds (pid1/pid2 are thread ids from 'info cpus' of the bad 
> guest, plus the pid of the qemu process itself)
> - post the resulting trace.dat somewhere
>

btw, that 100000 is 400MB of nonswappable kernel memory per cpu, so if 
you have lots of cpus and not enough memory, adjust it down.

-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06  9:27     ` Avi Kivity
  2011-01-06  9:30       ` Avi Kivity
@ 2011-01-06  9:42       ` Nikola Ciprich
  2011-01-06 10:19         ` Avi Kivity
  1 sibling, 1 reply; 29+ messages in thread
From: Nikola Ciprich @ 2011-01-06  9:42 UTC (permalink / raw)
  To: Avi Kivity; +Cc: KVM list, nikola.ciprich

> - run trace-cmd record -e kvm -b 100000 -P pid1 -P pid2, ctrl-C after a  
seems like it's not possible to specify multiple pids, so
I've run 4 commands in parallel. Also I can't get monitor information
since vm is started using libvirt, so I've just used all machine's qemu-kvm
pids..
hope it's OK
here's the trace:
http://nelide.cz/downloads/nik/trace.tar.bz2
n.

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06  9:42       ` Nikola Ciprich
@ 2011-01-06 10:19         ` Avi Kivity
  2011-01-06 10:25           ` Nikola Ciprich
                             ` (2 more replies)
  0 siblings, 3 replies; 29+ messages in thread
From: Avi Kivity @ 2011-01-06 10:19 UTC (permalink / raw)
  To: Nikola Ciprich; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

On 01/06/2011 11:42 AM, Nikola Ciprich wrote:
> >  - run trace-cmd record -e kvm -b 100000 -P pid1 -P pid2, ctrl-C after a
> seems like it's not possible to specify multiple pids, so

Did you get 'overrun: something' reports from trace-cmd, where something 
!= 0?

If you're not sure, please run the trace again.  Also try adding '-r 10' 
to the command line.

> I've run 4 commands in parallel. Also I can't get monitor information
> since vm is started using libvirt, so I've just used all machine's qemu-kvm
> pids..

Dan, is there a way to hijack the monitor so we can run some commands on 
it?  Things like 'info registers' and disassembly.

> hope it's OK
> here's the trace:
> http://nelide.cz/downloads/nik/trace.tar.bz2
> n.
>

Looks like vcpu 1 is spinning; perhaps that's normal.  If you get hold 
of the monitor, please disassemble around 0xfffff80001575d59.

vcpu 0 is busy writing to vga (can you confirm)? looks like bank 
switching is hitting synchronize_srcu_expedited(), which is known slow.  
Unfortunately that only gets better in 2.6.38.

You can try applying 
http://git.kernel.org/?p=linux/kernel/git/sfr/linux-next.git;a=commit;h=46fdb0937f26124700fc9fc80da4776330cc00d3 
and see if it helps.

-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06 10:19         ` Avi Kivity
@ 2011-01-06 10:25           ` Nikola Ciprich
  2011-01-06 10:50             ` Avi Kivity
  2011-01-06 11:06           ` Daniel P. Berrange
  2011-01-06 12:18           ` Nikola Ciprich
  2 siblings, 1 reply; 29+ messages in thread
From: Nikola Ciprich @ 2011-01-06 10:25 UTC (permalink / raw)
  To: Avi Kivity; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

> Did you get 'overrun: something' reports from trace-cmd, where something  
> != 0?
nope, all entries were 0.

> Dan, is there a way to hijack the monitor so we can run some commands on  
> it?  Things like 'info registers' and disassembly.
AFAIK that's intentionally not possible :(
pity..

> Looks like vcpu 1 is spinning; perhaps that's normal.  If you get hold  
> of the monitor, please disassemble around 0xfffff80001575d59.
>
> vcpu 0 is busy writing to vga (can you confirm)? looks like bank  
> switching is hitting synchronize_srcu_expedited(), which is known slow.   
> Unfortunately that only gets better in 2.6.38.
I guess it might help if I would just risk killing the machine and run
it without libvirt so we can debug better right?
or even better, I'll try to reproduce on another 2K8 windows on testing
machine and then we can play more with it.
gimme a few minutes please..
>
> You can try applying  
> http://git.kernel.org/?p=linux/kernel/git/sfr/linux-next.git;a=commit;h=46fdb0937f26124700fc9fc80da4776330cc00d3 
> and see if it helps.
I'll also prepare testing kernel including this patch..
BTW Is it just me, or is git.kernel.org pretty slow today?
>
> -- 
> error compiling committee.c: too many arguments to function
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06 10:25           ` Nikola Ciprich
@ 2011-01-06 10:50             ` Avi Kivity
  0 siblings, 0 replies; 29+ messages in thread
From: Avi Kivity @ 2011-01-06 10:50 UTC (permalink / raw)
  To: Nikola Ciprich; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

On 01/06/2011 12:25 PM, Nikola Ciprich wrote:
> >  Did you get 'overrun: something' reports from trace-cmd, where something
> >  != 0?
> nope, all entries were 0.
>
> >  Dan, is there a way to hijack the monitor so we can run some commands on
> >  it?  Things like 'info registers' and disassembly.
> AFAIK that's intentionally not possible :(
> pity..
>
> >  Looks like vcpu 1 is spinning; perhaps that's normal.  If you get hold
> >  of the monitor, please disassemble around 0xfffff80001575d59.
> >
> >  vcpu 0 is busy writing to vga (can you confirm)? looks like bank
> >  switching is hitting synchronize_srcu_expedited(), which is known slow.
> >  Unfortunately that only gets better in 2.6.38.
> I guess it might help if I would just risk killing the machine and run
> it without libvirt so we can debug better right?
> or even better, I'll try to reproduce on another 2K8 windows on testing
> machine and then we can play more with it.
> gimme a few minutes please..


Thanks, a test machine that we can do horrible things to would be best.

> >
> >  You can try applying
> >  http://git.kernel.org/?p=linux/kernel/git/sfr/linux-next.git;a=commit;h=46fdb0937f26124700fc9fc80da4776330cc00d3
> >  and see if it helps.
> I'll also prepare testing kernel including this patch..
> BTW Is it just me, or is git.kernel.org pretty slow today?


2.6.37 is out, perhaps people are all over it.

-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06 10:19         ` Avi Kivity
  2011-01-06 10:25           ` Nikola Ciprich
@ 2011-01-06 11:06           ` Daniel P. Berrange
  2011-01-06 12:19             ` Nikola Ciprich
  2011-01-06 12:18           ` Nikola Ciprich
  2 siblings, 1 reply; 29+ messages in thread
From: Daniel P. Berrange @ 2011-01-06 11:06 UTC (permalink / raw)
  To: Avi Kivity; +Cc: Nikola Ciprich, KVM list, nikola.ciprich

On Thu, Jan 06, 2011 at 12:19:21PM +0200, Avi Kivity wrote:
> On 01/06/2011 11:42 AM, Nikola Ciprich wrote:
> >>  - run trace-cmd record -e kvm -b 100000 -P pid1 -P pid2, ctrl-C after a
> >seems like it's not possible to specify multiple pids, so
> 
> Did you get 'overrun: something' reports from trace-cmd, where
> something != 0?
> 
> If you're not sure, please run the trace again.  Also try adding '-r
> 10' to the command line.
> 
> >I've run 4 commands in parallel. Also I can't get monitor information
> >since vm is started using libvirt, so I've just used all machine's qemu-kvm
> >pids..
> 
> Dan, is there a way to hijack the monitor so we can run some
> commands on it?  Things like 'info registers' and disassembly.

Depends on the libvirt version. For most, you'll need to
look for the monitor path in the QEMU argv:

  -chardev
+socket,id=monitor,path=/var/lib/libvirt/qemu/vmwts02.monitor,server,nowait -mon   chardev=monitor,mode=readline

then, 'service libvirtd stop' and now you can connect to
the monitor at that path & run commands you want, and then
disconnect and start libvirtd again. If you run any commands
that change the VM state, things may well get confused when
you start libvirtd again, but if its just 'info registers'
etc it should be pretty safe.

If you have a new enough libvirt, then you can also send
commands directly using 'virsh qemu-monitor-command' (checking
whether you need JSON or HMP syntax first - in this case you
can see it needs HMP).

Regards,
Daniel

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06 10:19         ` Avi Kivity
  2011-01-06 10:25           ` Nikola Ciprich
  2011-01-06 11:06           ` Daniel P. Berrange
@ 2011-01-06 12:18           ` Nikola Ciprich
  2011-01-06 13:26             ` Avi Kivity
  2 siblings, 1 reply; 29+ messages in thread
From: Nikola Ciprich @ 2011-01-06 12:18 UTC (permalink / raw)
  To: Avi Kivity; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

OK, got test environment running, but it seems to be running much faster
there :(
but as dan suggested, I can type monitor commands using virsh, so I can
(carefully:)) continue debugging on this production machine..
here's info registers:
RAX=0000000000000007 RBX=00000000000000ac RCX=fffff880009d1015 RDX=00000000000003ce
RSI=000000000000018a RDI=fffff8000163f737 RBP=0000000000000007 RSP=fffff88002588b08
R8 =000000000000000f R9 =00000000000000ac R10=0000000000007b20 R11=0000000000000008
R12=00000000000000a8 R13=0000000000000000 R14=000000000012c690 R15=00000000001d52d0
RIP=fffff8000156ae48 RFL=00000202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
CS =0010 0000000000000000 00000000 00209b00 DPL=0 CS64 [-RA]
SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
FS =0053 00000000fffe0000 00003c00 0040f300 DPL=3 DS   [-WA]
GS =002b fffff80001644d00 ffffffff 00c0f300 DPL=3 DS   [-WA]
LDT=0000 0000000000000000 ffffffff 00000000
TR =0040 fffff80002767080 00000067 00008b00 DPL=0 TSS64-busy
GDT=     fffff80002766000 0000007f
IDT=     fffff80002766080 00000fff
CR0=80050031 CR2=000000000047029a CR3=0000000094925000 CR4=000006f8
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
FCW=027f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=0000000000288c300000000000a000a0 XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000
XMM08=00000000000000000000000000000000 XMM09=00000000000000000000000000000000
XMM10=00000000000000000000000000000000 XMM11=00000000000000000000000000000000
XMM12=00000000000000000000000000000000 XMM13=00000000000000000000000000000000
XMM14=00000000000000000000000000000000 XMM15=00000000000000000000000000000000

> Looks like vcpu 1 is spinning; perhaps that's normal.  If you get hold  
> of the monitor, please disassemble around 0xfffff80001575d59.
ouch, can You advice me on how do I do it? :-[

>
> vcpu 0 is busy writing to vga (can you confirm)? looks like bank  
yes, seems like screen refreshing is quite slow, certainly in this rescue mode
or what it is, it's not using any acceleration...

> switching is hitting synchronize_srcu_expedited(), which is known slow.   
> Unfortunately that only gets better in 2.6.38.
>
> You can try applying  
> http://git.kernel.org/?p=linux/kernel/git/sfr/linux-next.git;a=commit;h=46fdb0937f26124700fc9fc80da4776330cc00d3 
I'll be able to test this only on testing machine, or on this production maybe overnight..
I'll prepare the kernel anyways..


> and see if it helps.
>
> -- 
> error compiling committee.c: too many arguments to function
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06 11:06           ` Daniel P. Berrange
@ 2011-01-06 12:19             ` Nikola Ciprich
  0 siblings, 0 replies; 29+ messages in thread
From: Nikola Ciprich @ 2011-01-06 12:19 UTC (permalink / raw)
  To: Daniel P. Berrange; +Cc: Avi Kivity, KVM list, nikola.ciprich

> If you have a new enough libvirt, then you can also send
> commands directly using 'virsh qemu-monitor-command' (checking
> whether you need JSON or HMP syntax first - in this case you
> can see it needs HMP).
Thanks Dan!
didn't know this is possible, works pretty well!
n.

> 
> Regards,
> Daniel
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06 12:18           ` Nikola Ciprich
@ 2011-01-06 13:26             ` Avi Kivity
  2011-01-06 13:55               ` Nikola Ciprich
  0 siblings, 1 reply; 29+ messages in thread
From: Avi Kivity @ 2011-01-06 13:26 UTC (permalink / raw)
  To: Nikola Ciprich; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

On 01/06/2011 02:18 PM, Nikola Ciprich wrote:
> OK, got test environment running, but it seems to be running much faster
> there :(

Same host kernel?

> but as dan suggested, I can type monitor commands using virsh, so I can
> (carefully:)) continue debugging on this production machine..
> here's info registers:
> RAX=0000000000000007 RBX=00000000000000ac RCX=fffff880009d1015 RDX=00000000000003ce
> RSI=000000000000018a RDI=fffff8000163f737 RBP=0000000000000007 RSP=fffff88002588b08
> R8 =000000000000000f R9 =00000000000000ac R10=0000000000007b20 R11=0000000000000008
> R12=00000000000000a8 R13=0000000000000000 R14=000000000012c690 R15=00000000001d52d0
> RIP=fffff8000156ae48 RFL=00000202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>

That's cpu 0, which is busy writing stuff to the screen, not very 
interesting.

> >  Looks like vcpu 1 is spinning; perhaps that's normal.  If you get hold
> >  of the monitor, please disassemble around 0xfffff80001575d59.
> ouch, can You advice me on how do I do it? :-[

(qemu) cpu 1
(qemu) info registers
(qemu) x/100i 0xfffff80001575d59 - 35

> >
> >  vcpu 0 is busy writing to vga (can you confirm)? looks like bank
> yes, seems like screen refreshing is quite slow, certainly in this rescue mode
> or what it is, it's not using any acceleration...

It's actually decelerated by synchronized_srcu_expedited().  It's one 
area which got a large slowdown as the price for the great scalability 
we achieved with srcu.

But wait, this doesn't make sense.  If we see mmio to vga, then bank 
switching is not involved, yet I see huge latencies on writes to vga io 
ports.

Please install the qemu debuginfo package (if you built it yourself, I 
hope it was with debug symbols enabled) and run 'perf top'.

> >  switching is hitting synchronize_srcu_expedited(), which is known slow.
> >  Unfortunately that only gets better in 2.6.38.
> >
> >  You can try applying
> >  http://git.kernel.org/?p=linux/kernel/git/sfr/linux-next.git;a=commit;h=46fdb0937f26124700fc9fc80da4776330cc00d3
> I'll be able to test this only on testing machine, or on this production maybe overnight..
> I'll prepare the kernel anyways..
>

I'm no longer sure this is the problem.

-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06 13:26             ` Avi Kivity
@ 2011-01-06 13:55               ` Nikola Ciprich
  2011-01-06 14:37                 ` Avi Kivity
  0 siblings, 1 reply; 29+ messages in thread
From: Nikola Ciprich @ 2011-01-06 13:55 UTC (permalink / raw)
  To: Avi Kivity; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

> Same host kernel?
yes, I also disabled KSM now, se below.
>
> (qemu) cpu 1
> (qemu) info registers

RAX=0000000000000000 RBX=0000000000000000 RCX=0000000000000002 RDX=000055a900000000
RSI=fffffa8003660450 RDI=0000000000000001 RBP=0000000000000080 RSP=fffff880009f7cc0
R8 =0000000000000000 R9 =0000000000000f44 R10=fffff8000145a000 R11=0000000000000000
R12=0000000000000000 R13=fffff800015cada0 R14=0000000000000000 R15=fffff880009bcec0
RIP=fffff80001575d5b RFL=00000202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
CS =0010 0000000000000000 00000000 00209b00 DPL=0 CS64 [-RA]
SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
FS =0053 00000000fffe0000 00007c00 0040f300 DPL=3 DS   [-WA]
GS =002b fffff880009b8000 ffffffff 00c0f300 DPL=3 DS   [-WA]
LDT=0000 0000000000000000 ffffffff 00000000
TR =0040 fffff880009bcec0 00000067 00008b00 DPL=0 TSS64-busy
GDT=     fffff880009c34c0 0000007f
IDT=     fffff880009c3540 00000fff
CR0=80050031 CR2=fffff8800121ca30 CR3=0000000000187000 CR4=000006f8
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
FCW=027f FSW=3800 [ST=7] FTW=80 MXCSR=00001f80
FPR0=9fc0000000000000 4008 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=99b6438668a3a8ed8cfd2d2540e9389a XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000
XMM08=00000000000000000000000000000000 XMM09=00000000000000000000000000000000
XMM10=00000000000000000000000000000000 XMM11=00000000000000000000000000000000
XMM12=00000000000000000000000000000000 XMM13=00000000000000000000000000000000
XMM14=00000000000000000000000000000000 XMM15=00000000000000000000000000000000



> (qemu) x/100i 0xfffff80001575d59 - 35
virsh # qemu-monitor-command 2 "x/100i 0xfffff80001575d59 - 35"
0xfffff80001575d36:  mov    %ebp,0x18(%rsp)
0xfffff80001575d3a:  push   %rsi
0xfffff80001575d3b:  push   %rdi
0xfffff80001575d3c:  push   %r12
0xfffff80001575d3e:  sub    $0x30,%rsp
0xfffff80001575d42:  mov    %cr8,%rbx
0xfffff80001575d46:  mov    $0x2,%ecx
0xfffff80001575d4b:  cmp    %cl,%bl
0xfffff80001575d4d:  ja     0xfffff80001575d67
0xfffff80001575d4f:  cmpq   $0x0,0x1598d1(%rip)        # 0xfffff800016cf628
0xfffff80001575d57:  je     0xfffff80001575d5d
0xfffff80001575d59:  pause  
0xfffff80001575d5b:  jmp    0xfffff80001575d4f
0xfffff80001575d5d:  mov    %cr8,%rax
0xfffff80001575d61:  mov    %rcx,%cr8
0xfffff80001575d65:  mov    %al,%bl
0xfffff80001575d67:  mov    %gs:0x20,%rdi
0xfffff80001575d70:  xor    %r9d,%r9d
0xfffff80001575d73:  btl    $0x10,0xe38c9(%rip)        # 0xfffff80001659644
0xfffff80001575d7b:  jae    0xfffff80001575d95
0xfffff80001575d7d:  mov    $0x1,%sil
0xfffff80001575d80:  rdtsc  
0xfffff80001575d82:  mov    0x4700(%rdi),%r12d
0xfffff80001575d89:  shl    $0x20,%rdx
0xfffff80001575d8d:  or     %rdx,%rax
0xfffff80001575d90:  mov    %rax,%rbp
0xfffff80001575d93:  jmp    0xfffff80001575da2
0xfffff80001575d95:  mov    0x50(%rsp),%rbp
0xfffff80001575d9a:  mov    0x50(%rsp),%r12d
0xfffff80001575d9f:  xor    %sil,%sil
0xfffff80001575da2:  incl   0x4b00(%rdi)
0xfffff80001575da8:  lock btsq $0x0,0x159876(%rip)        # 0xfffff800016cf628
0xfffff80001575db2:  jae    0xfffff80001575dd1
0xfffff80001575db4:  lea    0x15986d(%rip),%rcx        # 0xfffff800016cf628
0xfffff80001575dbb:  callq  0xfffff8000148c1c0
0xfffff80001575dc0:  incl   0x4b04(%rdi)
0xfffff80001575dc6:  add    %eax,0x4b08(%rdi)
0xfffff80001575dcc:  mov    %eax,%r9d
0xfffff80001575dcf:  jmp    0xfffff80001575dd4
0xfffff80001575dd1:  lfence 
0xfffff80001575dd4:  test   %sil,%sil
0xfffff80001575dd7:  je     0xfffff80001575e01
0xfffff80001575dd9:  rdtsc  
0xfffff80001575ddb:  shl    $0x20,%rdx
0xfffff80001575ddf:  lea    0x159842(%rip),%rcx        # 0xfffff800016cf628
0xfffff80001575de6:  movb   $0x0,0x28(%rsp)
0xfffff80001575deb:  or     %rdx,%rax
0xfffff80001575dee:  mov    %r12d,0x20(%rsp)
0xfffff80001575df3:  mov    %eax,%r8d
0xfffff80001575df6:  mov    %rax,%rdx
0xfffff80001575df9:  sub    %ebp,%r8d
0xfffff80001575dfc:  callq  0xfffff80001560f10
0xfffff80001575e01:  mov    0x60(%rsp),%rbp
0xfffff80001575e06:  mov    %bl,0x159818(%rip)        # 0xfffff800016cf624
0xfffff80001575e0c:  mov    0x58(%rsp),%rbx
0xfffff80001575e11:  add    $0x30,%rsp
0xfffff80001575e15:  pop    %r12
0xfffff80001575e17:  pop    %rdi
0xfffff80001575e18:  pop    %rsi
0xfffff80001575e19:  retq   
0xfffff80001575e1a:  nop    
0xfffff80001575e1b:  nop    
0xfffff80001575e1c:  nop    
0xfffff80001575e1d:  nop    
0xfffff80001575e1e:  nop    
0xfffff80001575e1f:  nop    
0xfffff80001575e20:  sub    $0x58,%rsp
0xfffff80001575e24:  mov    %r8b,0x40(%rsp)
0xfffff80001575e29:  xor    %eax,%eax
0xfffff80001575e2b:  movb   $0x1,0x38(%rsp)
0xfffff80001575e30:  cmp    %al,0x10(%rdx)
0xfffff80001575e33:  mov    %rax,0x30(%rsp)
0xfffff80001575e38:  mov    0x14(%rdx),%eax
0xfffff80001575e3b:  mov    %eax,0x28(%rsp)
0xfffff80001575e3f:  mov    0x20(%rdx),%rax
0xfffff80001575e43:  lea    0x8(%rdx),%r9
0xfffff80001575e47:  mov    %rdx,%r8
0xfffff80001575e4a:  mov    0x18(%rdx),%rdx
0xfffff80001575e4e:  mov    %rax,0x20(%rsp)
0xfffff80001575e53:  je     0xfffff80001575e5c
0xfffff80001575e55:  callq  0xfffff8000147b73c
0xfffff80001575e5a:  jmp    0xfffff80001575e61
0xfffff80001575e5c:  callq  0xfffff8000147b144
0xfffff80001575e61:  add    $0x58,%rsp
0xfffff80001575e65:  retq   
0xfffff80001575e66:  nop    
0xfffff80001575e67:  nop    
0xfffff80001575e68:  nop    
0xfffff80001575e69:  nop    
0xfffff80001575e6a:  nop    
0xfffff80001575e6b:  nop    
0xfffff80001575e6c:  nop    
0xfffff80001575e6d:  nop    
0xfffff80001575e6e:  nop    
0xfffff80001575e6f:  nop    
0xfffff80001575e70:  mov    %rbx,0x10(%rsp)
0xfffff80001575e75:  mov    %rbp,0x18(%rsp)
0xfffff80001575e7a:  mov    %rsi,0x20(%rsp)
0xfffff80001575e7f:  push   %rdi
0xfffff80001575e80:  push   %r12


> It's actually decelerated by synchronized_srcu_expedited().  It's one  
> area which got a large slowdown as the price for the great scalability  
> we achieved with srcu.
>
> But wait, this doesn't make sense.  If we see mmio to vga, then bank  
> switching is not involved, yet I see huge latencies on writes to vga io  
> ports.
>
> Please install the qemu debuginfo package (if you built it yourself, I  
> hope it was with debug symbols enabled) and run 'perf top'.
sure, here it is. it showed a lot of KSM activity:
             2276.00 12.1% memcmp_pages              [kernel.kallsyms]                                        
             1613.00  8.6% ksm_scan_thread           [kernel.kallsyms]                                        
hence I disabled KSM.
here's perf top then, checkdisk is still slow:
              412.00  8.6% do_raw_spin_lock             [kernel.kallsyms]                                        
              235.00  4.9% send_mono_rect               /usr/bin/qemu-kvm                                        
              215.00  4.5% rb_next                      [kernel.kallsyms]                                        
              166.00  3.5% schedule                     [kernel.kallsyms]                                        
              141.00  3.0% add_preempt_count            [kernel.kallsyms]                                        
              137.00  2.9% gen_rotc_rm_T1               /usr/bin/qemu-kvm                                        
              127.00  2.7% vmx_vcpu_run                 /lib/modules/2.6.36lb.03/kernel/arch/x86/kvm/kvm-intel.ko
              120.00  2.5% kvm_mmu_prepare_zap_page     /lib/modules/2.6.36lb.03/kernel/arch/x86/kvm/kvm.ko      
              103.00  2.2% sub_preempt_count            [kernel.kallsyms]                                        
              102.00  2.1% debug_smp_processor_id       [kernel.kallsyms]                                        
               87.00  1.8% delay_tsc                    [kernel.kallsyms]                                        
               62.00  1.3% cpupri_set                   [kernel.kallsyms]                                        
               60.00  1.3% vmcs_readl                   /lib/modules/2.6.36lb.03/kernel/arch/x86/kvm/kvm-intel.ko
               59.00  1.2% cpu_stopper_thread           [kernel.kallsyms]                                        
               57.00  1.2% select_nohz_load_balancer    [kernel.kallsyms]                                        
               56.00  1.2% memset                       /lib64/libc-2.5.so                                       
               52.00  1.1% alloc_vmap_area              [kernel.kallsyms]                                        
               50.00  1.0% copy_user_generic_string     [kernel.kallsyms]                                        
               46.00  1.0% gfn_to_memslot               /lib/modules/2.6.36lb.03/kernel/arch/x86/kvm/kvm.ko      
               38.00  0.8% is_shadow_present_pte        /lib/modules/2.6.36lb.03/kernel/arch/x86/kvm/kvm.ko      
               38.00  0.8% read_tsc                     [kernel.kallsyms]                                        
               35.00  0.7% sched_clock_local            [kernel.kallsyms]                                        
               35.00  0.7% sched_clock                  [kernel.kallsyms]                                        
               35.00  0.7% cirrus_fill_src_or_dst_32    /usr/bin/qemu-kvm                                        
               33.00  0.7% cpu_stop_signal_done         [kernel.kallsyms]                                        
               33.00  0.7% gen_rot_rm_T1                /usr/bin/qemu-kvm                                        
               31.00  0.7% do_raw_spin_unlock           [kernel.kallsyms]                                        
               30.00  0.6% __enqueue_rt_entity          [kernel.kallsyms]                                        
               30.00  0.6% gen_shift_rm_T1              /usr/bin/qemu-kvm                                        
               29.00  0.6% load_balance                 [kernel.kallsyms]                                        
               29.00  0.6% sched_clock_cpu              [kernel.kallsyms]                                        
               29.00  0.6% _raw_spin_unlock_irqrestore  [kernel.kallsyms]                                        
               28.00  0.6% __might_sleep                [kernel.kallsyms]                                        
               28.00  0.6% finish_task_switch           [kernel.kallsyms]                                        
               28.00  0.6% pick_next_task_rt            [kernel.kallsyms]                                        
               28.00  0.6% tick_nohz_restart_sched_tick [kernel.kallsyms]                                        
.
.
.


>
> I'm no longer sure this is the problem.
>
> -- 
> error compiling committee.c: too many arguments to function
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06 13:55               ` Nikola Ciprich
@ 2011-01-06 14:37                 ` Avi Kivity
  2011-01-06 15:38                   ` Nikola Ciprich
  0 siblings, 1 reply; 29+ messages in thread
From: Avi Kivity @ 2011-01-06 14:37 UTC (permalink / raw)
  To: Nikola Ciprich; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

On 01/06/2011 03:55 PM, Nikola Ciprich wrote:
> >  Same host kernel?
> yes, I also disabled KSM now, se below.
> >
> >  (qemu) cpu 1
> >  (qemu) info registers
>
> RAX=0000000000000000 RBX=0000000000000000 RCX=0000000000000002 RDX=000055a900000000
> RSI=fffffa8003660450 RDI=0000000000000001 RBP=0000000000000080 RSP=fffff880009f7cc0
> R8 =0000000000000000 R9 =0000000000000f44 R10=fffff8000145a000 R11=0000000000000000
> R12=0000000000000000 R13=fffff800015cada0 R14=0000000000000000 R15=fffff880009bcec0
> RIP=fffff80001575d5b RFL=00000202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
> CS =0010 0000000000000000 00000000 00209b00 DPL=0 CS64 [-RA]
> SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
> DS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
> FS =0053 00000000fffe0000 00007c00 0040f300 DPL=3 DS   [-WA]
> GS =002b fffff880009b8000 ffffffff 00c0f300 DPL=3 DS   [-WA]
> LDT=0000 0000000000000000 ffffffff 00000000
> TR =0040 fffff880009bcec0 00000067 00008b00 DPL=0 TSS64-busy
> GDT=     fffff880009c34c0 0000007f
> IDT=     fffff880009c3540 00000fff
> CR0=80050031 CR2=fffff8800121ca30 CR3=0000000000187000 CR4=000006f8
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000d01
> FCW=027f FSW=3800 [ST=7] FTW=80 MXCSR=00001f80
> FPR0=9fc0000000000000 4008 FPR1=0000000000000000 0000
> FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
> FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
> FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
> XMM00=99b6438668a3a8ed8cfd2d2540e9389a XMM01=00000000000000000000000000000000
> XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
> XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
> XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000
> XMM08=00000000000000000000000000000000 XMM09=00000000000000000000000000000000
> XMM10=00000000000000000000000000000000 XMM11=00000000000000000000000000000000
> XMM12=00000000000000000000000000000000 XMM13=00000000000000000000000000000000
> XMM14=00000000000000000000000000000000 XMM15=00000000000000000000000000000000
>
>
>
> >  (qemu) x/100i 0xfffff80001575d59 - 35
> virsh # qemu-monitor-command 2 "x/100i 0xfffff80001575d59 - 35"
> 0xfffff80001575d36:  mov    %ebp,0x18(%rsp)
> 0xfffff80001575d3a:  push   %rsi
> 0xfffff80001575d3b:  push   %rdi
> 0xfffff80001575d3c:  push   %r12
> 0xfffff80001575d3e:  sub    $0x30,%rsp
> 0xfffff80001575d42:  mov    %cr8,%rbx
> 0xfffff80001575d46:  mov    $0x2,%ecx
> 0xfffff80001575d4b:  cmp    %cl,%bl
> 0xfffff80001575d4d:  ja     0xfffff80001575d67
> 0xfffff80001575d4f:  cmpq   $0x0,0x1598d1(%rip)        # 0xfffff800016cf628
> 0xfffff80001575d57:  je     0xfffff80001575d5d
> 0xfffff80001575d59:  pause

rip points here, reasonable for a spin loop, but what it's spinning on, 
I can't imagine.

Maybe some bug in kvm caused Windows to spin here endlessly and that's 
what's causing the problem, this is consistent with you not reproducing 
it on the test machine.

> hence I disabled KSM.
> here's perf top then, checkdisk is still slow:
>                412.00  8.6% do_raw_spin_lock             [kernel.kallsyms]

$ perf record -a -f -g
$ perf report -g

will show who calls do_raw_spin_lock.

>                235.00  4.9% send_mono_rect               /usr/bin/qemu-kvm
>                215.00  4.5% rb_next                      [kernel.kallsyms]
>                166.00  3.5% schedule                     [kernel.kallsyms]

What's the context switch rate? 'vmstat 1'

>                141.00  3.0% add_preempt_count            [kernel.kallsyms]
>                137.00  2.9% gen_rotc_rm_T1               /usr/bin/qemu-kvm

Do you have a guest running with kvm disabled?!

>                127.00  2.7% vmx_vcpu_run                 /lib/modules/2.6.36lb.03/kernel/arch/x86/kvm/kvm-intel.ko

That's guest time.  Puny.

>                120.00  2.5% kvm_mmu_prepare_zap_page     /lib/modules/2.6.36lb.03/kernel/arch/x86/kvm/kvm.ko

Could be a couple of things.



-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06 14:37                 ` Avi Kivity
@ 2011-01-06 15:38                   ` Nikola Ciprich
  2011-01-09 12:36                     ` Nikola Ciprich
  0 siblings, 1 reply; 29+ messages in thread
From: Nikola Ciprich @ 2011-01-06 15:38 UTC (permalink / raw)
  To: Avi Kivity; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

>
> $ perf record -a -f -g
> $ perf report -g
here we go:
here we go:
      - 49.72% _raw_spin_lock                                                                                                                          ▒
         - 32.32% kvm_mmu_pte_write                                                                                                                    ▒
            - 98.02% emulator_write_phys                                                                                                               ▒
                 emulator_write_emulated_onepage                                                                                                       ▒
                 emulator_write_emulated                                                                                                               ▒
                 x86_emulate_insn                                                                                                                      ▒
                 emulate_instruction                                                                                                                   ▒
                 kvm_mmu_page_fault                                                                                                                    ▒
                 handle_exception                                                                                                                      ▒
                 vmx_handle_exit                                                                                                                       ▒
                 kvm_arch_vcpu_ioctl_run                                                                                                               ▒
                 kvm_vcpu_ioctl                                                                                                                        ▒
                 vfs_ioctl                                                                                                                             ▒
                 do_vfs_ioctl                                                                                                                          ▒
                 sys_ioctl                                                                                                                             ▒
                 system_call_fastpath                                                                                                                  ▒
                 __GI_ioctl                                                                                                                            ▒
            - 1.98% paging64_invlpg                                                                                                                    ▒
                 kvm_mmu_invlpg                                                                                                                        ▒
                 handle_invlpg                                                                                                                         ▒
                 vmx_handle_exit                                                                                                                       ▒
                 kvm_arch_vcpu_ioctl_run                                                                                                               ▒
                 kvm_vcpu_ioctl                                                                                                                        ▒
                 vfs_ioctl                                                                                                                             ▒
                 do_vfs_ioctl                                                                                                                          ▒
                 sys_ioctl                                                                                                                             ▒
                 system_call_fastpath                                                                                                                  ▒
                 __GI_ioctl                                                                                                                            ▒
         - 23.66% task_rq_lock                                                                                                                         ▒
            - try_to_wake_up                                                                                                                           ▒
               - 94.76% wake_up_process                                                                                                                ▒
                    cpu_stop_queue_work                                                                                                                ▒
                    __stop_cpus                                                                                                                        ▒
                    try_stop_cpus                                                                                                                      ▒
                    synchronize_sched_expedited                                                                                                        ▒
                    __synchronize_srcu                                                                                                                 ▒
                    synchronize_srcu_expedited                                                                                                         ▒
                    __kvm_set_memory_region                                                                                                            ▒
                    kvm_set_memory_region                                                                                                              ▒
                    kvm_vm_ioctl_set_memory_region                                                                                                     ▒
                    kvm_vm_ioctl                                                                                                                       ▒
                    vfs_ioctl                                                                                                                          ▒
                    do_vfs_ioctl                                                                                                                       ▒
                    sys_ioctl                                                                                                                          ▒
                    system_call_fastpath                                                                                                               ▒
                    __GI_ioctl                                                                                                                         ▒
               - 4.44% default_wake_function                                                                                                           ▒
                  - 53.81% autoremove_wake_function                                                                                                    ▒
                       __wake_up_common                                                                                                                ▒
                       __wake_up                                                                                                                       ▒
                       kvm_vcpu_kick                                                                                                                   ▒
                       __apic_accept_irq                                                                                                               ▒
                       kvm_apic_set_irq                                                                                                                ▒
                     - kvm_irq_delivery_to_apic                                                                                                        ▒
                        - 58.41% apic_reg_write                                                                                                        ▒
                             apic_mmio_write                                                                                                           ▒
                             emulator_write_emulated_onepage                                                                                           ▒
                             emulator_write_emulated                                                                                                   ▒
                             x86_emulate_insn                                                                                                          ▒
                             emulate_instruction                                                                                                       ▒
                             kvm_mmu_page_fault                                                                                                        ▒
                             handle_exception                                                                                                          ▒
                             vmx_handle_exit                                                                                                           ▒
                             kvm_arch_vcpu_ioctl_run                                                                                                   ▒
                             kvm_vcpu_ioctl                                                                                                            ▒
                             vfs_ioctl                                                                                                                 ▒
                             do_vfs_ioctl                                                                                                              ▒
                             sys_ioctl                                                                                                                 ▒
                             system_call_fastpath                                                                                                      ▒
                             __GI_ioctl                                                                                                                ▒
                        - 29.60% ioapic_service                                                                                                        ▒
                             kvm_ioapic_set_irq                                                                                                        ▒
                             kvm_set_ioapic_irq                                                                                                        ▒
                             kvm_set_irq                                                                                                               ▒
                             kvm_arch_vm_ioctl                                                                                                         ▒
                             kvm_vm_ioctl                                                                                                              ▒
                             vfs_ioctl                                                                                                                 ▒
                             do_vfs_ioctl                                                                                                              ▒
                             sys_ioctl                                                                                                                 ▒
                             system_call_fastpath                                                                                                      ▒
                             __GI_ioctl                                                                                                                ▒
                        - 11.99% kvm_set_msi                                                                                                           ▒
                             kvm_set_irq                                                                                                               ▒
                             kvm_arch_vm_ioctl                                                                                                         ▒
                             kvm_vm_ioctl                                                                                                              ▒
                             vfs_ioctl                                                                                                                 ▒
                             do_vfs_ioctl                                                                                                              ▒
                             sys_ioctl                                                                                                                 ▒
                             system_call_fastpath                                                                                                      ▒
                             __GI_ioctl                                                                                                                ▒
                  - 46.19% pollwake                                                                                                                    ▒
                       __wake_up_common                                                                                                                ▒
                     - __wake_up                                                                                                                       ▒
                        - 77.30% __send_signal                                                                                                         ▒
                             send_signal                                                                                                               ▒
                           - do_send_sig_info                                                                                                          ▒
                              - 68.83% do_send_specific                                                                                                ▒
                                   do_tkill                                                                                                            ▒
                                   sys_tgkill                                                                                                          ▒
                                   system_call_fastpath                                                                                                ▒
                                   __pthread_kill                                                                                                      ▒
                              - 31.17% group_send_sig_info                                                                                             ▒
                                   kill_pid_info                                                                                                       ▒
                                   sys_kill                                                                                                            ▒
                                   system_call_fastpath                                                                                                ▒
                                   __kill                                                                                                              ▒
                        - 22.70% send_sigqueue                                                                                                         ▒
                             posix_timer_event                                                                                                         ▒
                             posix_timer_fn                                                                                                            ▒
                             __run_hrtimer                                                                                                             ▒
                             hrtimer_interrupt                                                                                                         ▒
                             smp_apic_timer_interrupt                                                                                                  ▒
                             apic_timer_interrupt                                                                                                      ▒
                             kvm_arch_commit_memory_region                                                                                             ▒
                             __kvm_set_memory_region                                                                                                   ▒
                             kvm_set_memory_region                                                                                                     ▒
                             kvm_vm_ioctl_set_memory_region                                                                                            ▒
                             kvm_vm_ioctl                                                                                                              ▒
                             vfs_ioctl                                                                                                                 ▒
                             do_vfs_ioctl                                                                                                              ▒
                             sys_ioctl                                                                                                                 ▒
                             system_call_fastpath                                                                                                      ▒
                             __GI_ioctl                                                                                                                ▒
               - 0.80% wake_up_state                                                                                                                   ▒
                  - wake_futex                                                                                                                         ▒
                     - 68.01% do_futex                                                                                                                 ▒
                          sys_futex                                                                                                                    ▒
                          system_call_fastpath                                                                                                         ▒
                          __pthread_cond_signal                                                                                                        ▒
                     - 31.99% futex_wake                                                                                                               ▒
                          do_futex                                                                                                                     ▒
                          sys_futex                                                                                                                    ▒
                          system_call_fastpath                                                                                                         ▒
                          __lll_unlock_wake                                                                                                            ▒
         - 11.51% mmu_free_roots                                                                                                                       ▒
            - 96.89% kvm_mmu_unload                                                                                                                    ▒
                 kvm_arch_vcpu_ioctl_run                                                                                                               ▒
                 kvm_vcpu_ioctl                                                                                                                        ▒
                 vfs_ioctl                                                                                                                             ▒
                 do_vfs_ioctl                                                                                                                          ▒
                 sys_ioctl                                                                                                                             ▒
                 system_call_fastpath                                                                                                                  ▒
                 __GI_ioctl                                                                                                                            ▒
            - 3.11% paging_new_cr3                                                                                                                     ▒
                 kvm_set_cr3                                                                                                                           ▒
                 handle_cr                                                                                                                             ▒
                 vmx_handle_exit                                                                                                                       ▒
                 kvm_arch_vcpu_ioctl_run                                                                                                               ▒
                 kvm_vcpu_ioctl                                                                                                                        ▒
                 vfs_ioctl                                                                                                                             ▒
                 do_vfs_ioctl                                                                                                                          ▒
                 sys_ioctl                                                                                                                             ▒
                 system_call_fastpath                                                                                                                  ▒
                 __GI_ioctl                                                                                                                            ▒
         - 6.78% paging64_page_fault                                                                                                                   ▒
              kvm_mmu_page_fault                                                                                                                       ▒
              handle_exception                                                                                                                         ▒
              vmx_handle_exit                                                                                                                          ▒
              kvm_arch_vcpu_ioctl_run                                                                                                                  ▒
              kvm_vcpu_ioctl                                                                                                                           ▒
              vfs_ioctl                                                                                                                                ▒
              do_vfs_ioctl                                                                                                                             ▒
              sys_ioctl                                                                                                                                ▒
              system_call_fastpath                                                                                                                     ▒
              __GI_ioctl                                                                                                                               ▒
         - 5.72% make_all_cpus_request                                                                                                                 ▒
            - 62.13% kvm_reload_remote_mmus                                                                                                            ▒
               - kvm_mmu_prepare_zap_page                                                                                                              ▒
                  - 96.56% kvm_mmu_zap_all                                                                                                             ▒
                       kvm_arch_flush_shadow                                                                                                           ▒
                       __kvm_set_memory_region                                                                                                         ▒
                       kvm_set_memory_region                                                                                                           ▒
                       kvm_vm_ioctl_set_memory_region                                                                                                  ▒
                       kvm_vm_ioctl                                                                                                                    ▒
                       vfs_ioctl                                                                                                                       ▒
                       do_vfs_ioctl                                                                                                                    ▒
                       sys_ioctl                                                                                                                       ▒
                       system_call_fastpath                                                                                                            ▒
                       __GI_ioctl                                                                                                                      ▒
                  - 3.44% kvm_mmu_pte_write                                                                                                            ▒
                       emulator_write_phys                                                                                                             ▒
                       emulator_write_emulated_onepage                                                                                                 ▒
                       emulator_write_emulated                                                                                                         ▒
                       x86_emulate_insn                                                                                                                ▒
                       emulate_instruction                                                                                                             ▒
                       kvm_mmu_page_fault                                                                                                              ▒
                       handle_exception                                                                                                                ▒
                       vmx_handle_exit                                                                                                                 ▒
                       kvm_arch_vcpu_ioctl_run                                                                                                         ▒
                       kvm_vcpu_ioctl                                                                                                                  ▒
                       vfs_ioctl                                                                                                                       ▒
                       do_vfs_ioctl                                                                                                                    ▒
                         sys_ioctl                                                                                                                       ▒
                       system_call_fastpath                                                                                                            ▒
                       __GI_ioctl                                                                                                                      ▒
.
.
it's not all, is this enough? or can I simply export whole tree somehow?
>
> will show who calls do_raw_spin_lock.
>
>>                235.00  4.9% send_mono_rect               /usr/bin/qemu-kvm
>>                215.00  4.5% rb_next                      [kernel.kallsyms]
>>                166.00  3.5% schedule                     [kernel.kallsyms]
>
> What's the context switch rate? 'vmstat 1'
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0 168920 5841008 4614064    0    0    28    21   32   28 15 22 63  0  0
 2  0      0 168820 5841008 4614088    0    0     0     0 13489 76739 15 16 69  0  0
 2  0      0 167656 5841008 4614088    0    0     0   104 6089 33390 16 13 71  0  0
 3  0      0 169184 5841008 4614088    0    0     0     0 12489 71263 17 15 69  0  0
 2  0      0 169200 5841012 4614092    0    0     0     8 7034 33908 17 12 72  0  0
 2  0      0 169432 5841024 4614080    0    0     0    16 10924 67008 16 12 72  0  0
 2  0      0 168084 5841028 4614088    0    0     0     4 8955 47767 17 13 71  0  0
 2  0      0 168936 5841032 4614088    0    0     0    80 9528 50119 16 13 71  0  0
.
.


>
>>                141.00  3.0% add_preempt_count            [kernel.kallsyms]
>>                137.00  2.9% gen_rotc_rm_T1               /usr/bin/qemu-kvm
>
> Do you have a guest running with kvm disabled?!
nope, all seem to be using KVM

>
>

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-06 15:38                   ` Nikola Ciprich
@ 2011-01-09 12:36                     ` Nikola Ciprich
  2011-01-09 12:39                       ` Avi Kivity
  0 siblings, 1 reply; 29+ messages in thread
From: Nikola Ciprich @ 2011-01-09 12:36 UTC (permalink / raw)
  To: Avi Kivity; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

Avi,
I was forced to interrupt checkdisk on this machine, I tried it again
and it's reproducible (even on another (testing) W2K8 guests).
I transfered the image to testing machine where chkdsk runs OK and finished
it there.
So in case You'd like to continue debugging, I'm at Your disposal, we can create
more testing guests and play with them on host where I can reproduce.
So far thanks a lot for all the time You've spent with it.
BR
nik


-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-09 12:36                     ` Nikola Ciprich
@ 2011-01-09 12:39                       ` Avi Kivity
  2011-01-09 13:36                         ` Nikola Ciprich
  0 siblings, 1 reply; 29+ messages in thread
From: Avi Kivity @ 2011-01-09 12:39 UTC (permalink / raw)
  To: Nikola Ciprich; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

On 01/09/2011 02:36 PM, Nikola Ciprich wrote:
> Avi,
> I was forced to interrupt checkdisk on this machine, I tried it again
> and it's reproducible (even on another (testing) W2K8 guests).
> I transfered the image to testing machine where chkdsk runs OK and finished
> it there.
> So in case You'd like to continue debugging, I'm at Your disposal, we can create
> more testing guests and play with them on host where I can reproduce.
> So far thanks a lot for all the time You've spent with it.
> BR
> nik
>

What are the differences between the host the completes chkdsk and the 
host that fails?  Hardware-wise and software-wise.

-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-09 12:39                       ` Avi Kivity
@ 2011-01-09 13:36                         ` Nikola Ciprich
  2011-01-09 13:44                           ` Nikola Ciprich
  2011-01-09 14:13                           ` Avi Kivity
  0 siblings, 2 replies; 29+ messages in thread
From: Nikola Ciprich @ 2011-01-09 13:36 UTC (permalink / raw)
  To: Avi Kivity; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

> What are the differences between the host the completes chkdsk and the  
> host that fails?  Hardware-wise and software-wise.
Software-wise, I'm not aware of any, both nodes use:
- kernel 2.6.36.2
- qemu-kvm-0.13.0
- seabios-0.6.0

hardware-wise, problematic machine is older 8core, 16GB RAM,
cpuinfo snippet:

processor      : 7
vendor_id      : GenuineIntel
cpu family     : 6
model	       	 : 15
model name	 : Intel(R) Xeon(R) CPU           E5310  @ 1.60GHz
stepping	 : 7
cpu MHz		   : 1599.873
cache size	   : 4096 KB
physical id	   : 1
siblings : 4
core id	   : 3
cpu cores  : 4
apicid	     : 7
initial apicid : 7
fpu	       : yes
fpu_exception  : yes
cpuid level    : 10
wp    	       : yes
flags	       	 : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx tm2 ssse3 cx16 xtpr pdcm dca lahf_lm dts tpr_shadow
bogomips	 : 3200.27
clflush size	 : 64
cache_alignment	 : 64
address sizes	 : 36 bits physical, 48 bits virtual

nonproblematic machine is 8core 4GB, cpuinfo snippet:
processor      : 7
vendor_id      : GenuineIntel
cpu family     : 6
model	       	 : 30
model name	 : Intel(R) Xeon(R) CPU           X3440  @ 2.53GHz
stepping	 : 5
cpu MHz		   : 2533.260
cache size	   : 8192 KB
physical id	   : 0
siblings : 8
core id	   : 3
cpu cores  : 4
apicid	     : 7
initial apicid : 7
fpu	       : yes
fpu_exception  : yes
cpuid level    : 11
wp    	       : yes
flags	       	 : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm ida dts tpr_shadow vnmi flexpriority ept vpid
bogomips	 : 5066.35
clflush size	 : 64
cache_alignment	 : 64
address sizes	 : 36 bits physical, 48 bits virtual

important to note is, that botch machines are failover clusters, I can reproduce even on second node
of "problematic" cluster, although CPU is a bit different, snippet follows:
processor	 : 7
vendor_id	 : GenuineIntel
cpu family	 : 6
model		   : 23
model name	   : Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping	   : 6
cpu MHz		     : 2500.038
cache size	     : 6144 KB
physical id	     : 1
siblings : 4
core id	   : 3
cpu cores  : 4
apicid	     : 7
initial apicid : 7
fpu	       : yes
fpu_exception  : yes
cpuid level    : 10
wp    	       : yes
flags	       	 : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dts tpr_shadow vnmi flexpriority
bogomips	 : 5000.40
clflush size	 : 64
cache_alignment	 : 64
address sizes	 : 38 bits physical, 48 bits virtual

What else could I check?
n.


>
> -- 
> error compiling committee.c: too many arguments to function
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-09 13:36                         ` Nikola Ciprich
@ 2011-01-09 13:44                           ` Nikola Ciprich
  2011-01-09 14:13                             ` Avi Kivity
  2011-01-09 14:13                           ` Avi Kivity
  1 sibling, 1 reply; 29+ messages in thread
From: Nikola Ciprich @ 2011-01-09 13:44 UTC (permalink / raw)
  To: Avi Kivity; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

On Sun, Jan 09, 2011 at 02:36:55PM +0100, Nikola Ciprich wrote:
> > What are the differences between the host the completes chkdsk and the  
> > host that fails?  Hardware-wise and software-wise.
> Software-wise, I'm not aware of any, both nodes use:
> - kernel 2.6.36.2
> - qemu-kvm-0.13.0
> - seabios-0.6.0
huh, I noticed now, it's not clean 0.13.0, but 0.13.0 + patches up to 420fe74769cc67baec6f3d962dc054e2972ca3ae.
but it's the same on both machines, problematic and nonproblematic.
hope it's not problem, sorry for the mess :(
n.

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-09 13:36                         ` Nikola Ciprich
  2011-01-09 13:44                           ` Nikola Ciprich
@ 2011-01-09 14:13                           ` Avi Kivity
  2011-01-09 14:27                             ` Nikola Ciprich
  1 sibling, 1 reply; 29+ messages in thread
From: Avi Kivity @ 2011-01-09 14:13 UTC (permalink / raw)
  To: Nikola Ciprich; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

On 01/09/2011 03:36 PM, Nikola Ciprich wrote:
> >  What are the differences between the host the completes chkdsk and the
> >  host that fails?  Hardware-wise and software-wise.
> Software-wise, I'm not aware of any, both nodes use:
> - kernel 2.6.36.2
> - qemu-kvm-0.13.0
> - seabios-0.6.0
>
> hardware-wise, problematic machine is older 8core, 16GB RAM,
> cpuinfo snippet:
>
> processor      : 7
> vendor_id      : GenuineIntel
> cpu family     : 6
> model	       	 : 15
> model name	 : Intel(R) Xeon(R) CPU           E5310  @ 1.60GHz

Please try the newer machine, after rmmoding kvm-intel, and reloading it 
with the module parameter ept=0.


-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-09 13:44                           ` Nikola Ciprich
@ 2011-01-09 14:13                             ` Avi Kivity
  0 siblings, 0 replies; 29+ messages in thread
From: Avi Kivity @ 2011-01-09 14:13 UTC (permalink / raw)
  To: Nikola Ciprich; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

On 01/09/2011 03:44 PM, Nikola Ciprich wrote:
> On Sun, Jan 09, 2011 at 02:36:55PM +0100, Nikola Ciprich wrote:
> >  >  What are the differences between the host the completes chkdsk and the
> >  >  host that fails?  Hardware-wise and software-wise.
> >  Software-wise, I'm not aware of any, both nodes use:
> >  - kernel 2.6.36.2
> >  - qemu-kvm-0.13.0
> >  - seabios-0.6.0
> huh, I noticed now, it's not clean 0.13.0, but 0.13.0 + patches up to 420fe74769cc67baec6f3d962dc054e2972ca3ae.
> but it's the same on both machines, problematic and nonproblematic.
> hope it's not problem, sorry for the mess :(

It's not a problem, and in any case unlikely to cause the bug.

-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-09 14:13                           ` Avi Kivity
@ 2011-01-09 14:27                             ` Nikola Ciprich
  2011-01-09 14:36                               ` Avi Kivity
  0 siblings, 1 reply; 29+ messages in thread
From: Nikola Ciprich @ 2011-01-09 14:27 UTC (permalink / raw)
  To: Avi Kivity; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange


> Please try the newer machine, after rmmoding kvm-intel, and reloading it  
> with the module parameter ept=0.
seems to be OK even with parameter..
n.

>
>
> -- 
> error compiling committee.c: too many arguments to function
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-09 14:27                             ` Nikola Ciprich
@ 2011-01-09 14:36                               ` Avi Kivity
  2011-01-09 15:15                                 ` Nikola Ciprich
  0 siblings, 1 reply; 29+ messages in thread
From: Avi Kivity @ 2011-01-09 14:36 UTC (permalink / raw)
  To: Nikola Ciprich; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

On 01/09/2011 04:27 PM, Nikola Ciprich wrote:
> >  Please try the newer machine, after rmmoding kvm-intel, and reloading it
> >  with the module parameter ept=0.
> seems to be OK even with parameter..
> n.

Ok.  Is chkdsk outside the preboot environment working okay on the bad host?

-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-09 14:36                               ` Avi Kivity
@ 2011-01-09 15:15                                 ` Nikola Ciprich
  2011-01-09 20:10                                   ` Nikola Ciprich
  0 siblings, 1 reply; 29+ messages in thread
From: Nikola Ciprich @ 2011-01-09 15:15 UTC (permalink / raw)
  To: Avi Kivity; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

> Ok.  Is chkdsk outside the preboot environment working okay on the bad host?
I'll check, give me some time, I'll have to add another large volume which I can
check while the system is up..

in the meantime, on hosts, I noticed following messages:
[322172.007569] kvm: 17743: cpu1 unhandled wrmsr: 0x198 data 0
it's on all (problematic and nonproblematic machines).
dunno if this could be somehow related..
n.

>
> -- 
> error compiling committee.c: too many arguments to function
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-09 15:15                                 ` Nikola Ciprich
@ 2011-01-09 20:10                                   ` Nikola Ciprich
  2011-01-10  9:23                                     ` Avi Kivity
  0 siblings, 1 reply; 29+ messages in thread
From: Nikola Ciprich @ 2011-01-09 20:10 UTC (permalink / raw)
  To: Avi Kivity; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

Hi Avi,
sorry it took so long.
running chkdsk from windows is OK, runs pretty fast.
so there must be something different in this preboot environment.
n.

On Sun, Jan 09, 2011 at 04:15:19PM +0100, Nikola Ciprich wrote:
> > Ok.  Is chkdsk outside the preboot environment working okay on the bad host?
> I'll check, give me some time, I'll have to add another large volume which I can
> check while the system is up..
> 
> in the meantime, on hosts, I noticed following messages:
> [322172.007569] kvm: 17743: cpu1 unhandled wrmsr: 0x198 data 0
> it's on all (problematic and nonproblematic machines).
> dunno if this could be somehow related..
> n.
> 
> >
> > -- 
> > error compiling committee.c: too many arguments to function
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 
> -- 
> -------------------------------------
> Ing. Nikola CIPRICH
> LinuxBox.cz, s.r.o.
> 28. rijna 168, 709 01 Ostrava
> 
> tel.:   +420 596 603 142
> fax:    +420 596 621 273
> mobil:  +420 777 093 799
> 
> www.linuxbox.cz
> 
> mobil servis: +420 737 238 656
> email servis: servis@linuxbox.cz
> -------------------------------------
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow
  2011-01-09 20:10                                   ` Nikola Ciprich
@ 2011-01-10  9:23                                     ` Avi Kivity
  0 siblings, 0 replies; 29+ messages in thread
From: Avi Kivity @ 2011-01-10  9:23 UTC (permalink / raw)
  To: Nikola Ciprich; +Cc: KVM list, nikola.ciprich, Daniel P. Berrange

On 01/09/2011 10:10 PM, Nikola Ciprich wrote:
> Hi Avi,
> sorry it took so long.
> running chkdsk from windows is OK, runs pretty fast.
> so there must be something different in this preboot environment.

Let's try to see what's the difference between the working machine and 
the bad one.

Please compare several kvm_stat snapshots, see if something stands out.


-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 29+ messages in thread

end of thread, other threads:[~2011-01-10  9:23 UTC | newest]

Thread overview: 29+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-01-06  7:48 qemu-kvm-0.13.0 - winsows 2008 - chkdisk too slow Nikola Ciprich
2011-01-06  9:03 ` Stefan Hajnoczi
2011-01-06  9:06   ` Nikola Ciprich
2011-01-06  9:08 ` Avi Kivity
2011-01-06  9:20   ` Nikola Ciprich
2011-01-06  9:27     ` Avi Kivity
2011-01-06  9:30       ` Avi Kivity
2011-01-06  9:42       ` Nikola Ciprich
2011-01-06 10:19         ` Avi Kivity
2011-01-06 10:25           ` Nikola Ciprich
2011-01-06 10:50             ` Avi Kivity
2011-01-06 11:06           ` Daniel P. Berrange
2011-01-06 12:19             ` Nikola Ciprich
2011-01-06 12:18           ` Nikola Ciprich
2011-01-06 13:26             ` Avi Kivity
2011-01-06 13:55               ` Nikola Ciprich
2011-01-06 14:37                 ` Avi Kivity
2011-01-06 15:38                   ` Nikola Ciprich
2011-01-09 12:36                     ` Nikola Ciprich
2011-01-09 12:39                       ` Avi Kivity
2011-01-09 13:36                         ` Nikola Ciprich
2011-01-09 13:44                           ` Nikola Ciprich
2011-01-09 14:13                             ` Avi Kivity
2011-01-09 14:13                           ` Avi Kivity
2011-01-09 14:27                             ` Nikola Ciprich
2011-01-09 14:36                               ` Avi Kivity
2011-01-09 15:15                                 ` Nikola Ciprich
2011-01-09 20:10                                   ` Nikola Ciprich
2011-01-10  9:23                                     ` Avi Kivity

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox