[Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Peter Lieven]

Hi,

while migrating vServers from 2.1.0 to 2.2.0 I see the following assertion that seems to have
been introduced between 2.1.0 and 2.2.0 trigger on the destination. This happens in a serious
amount of cases. In most cases it works flawlessly on the second attempt to migrate the vServer
(same source and same destination).

Any hints how to debug the root cause?

Thank you,
Peter

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Paolo Bonzini]

On 10/02/2015 10:26, Peter Lieven wrote:
> Hi,
> 
> while migrating vServers from 2.1.0 to 2.2.0 I see the following
> assertion that seems to have
> been introduced between 2.1.0 and 2.2.0 trigger on the destination. This
> happens in a serious
> amount of cases. In most cases it works flawlessly on the second attempt
> to migrate the vServer
> (same source and same destination).
> 
> Any hints how to debug the root cause?

How can anyone help without knowing anything about your configuration? :)

Paolo

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Peter Lieven]

Am 10.02.2015 um 10:49 schrieb Paolo Bonzini:
>
> On 10/02/2015 10:26, Peter Lieven wrote:
>> Hi,
>>
>> while migrating vServers from 2.1.0 to 2.2.0 I see the following
>> assertion that seems to have
>> been introduced between 2.1.0 and 2.2.0 trigger on the destination. This
>> happens in a serious
>> amount of cases. In most cases it works flawlessly on the second attempt
>> to migrate the vServer
>> (same source and same destination).
>>
>> Any hints how to debug the root cause?
> How can anyone help without knowing anything about your configuration? :)

My hope was that anyone has observed this post 2.2.0 already and there
is a fix available ;-)

Can you indicate what info would be helpful debugging this?

Cmdline is:
/usr/bin/qemu-2.2.0  -enable-kvm  -M pc-i440fx-2.1  -nodefaults -netdev type=tap,id=guest19,script=no,downscript=no,ifname=tap19,vnet_hdr -device virtio-net-pci,netdev=guest19,mac=52:54:00:80:00:55 -netdev 
type=tap,id=guest20,script=no,downscript=no,ifname=tap20,vnet_hdr -device virtio-net-pci,netdev=guest20,mac=52:54:00:80:00:6f -netdev type=tap,id=guest21,script=no,downscript=no,ifname=tap21,vnet_hdr -device 
virtio-net-pci,netdev=guest21,mac=52:54:00:80:00:75 -serial null  -parallel null  -m 496 -monitor tcp:0:4011,server,nowait -vnc :11 -qmp tcp:0:3011,server,nowait  -name 'gw-5000123'  -boot order=nc,menu=on  -drive 
index=2,media=cdrom,if=ide,cache=unsafe,aio=native,readonly=on  -k de  -incoming tcp:0:5011  -pidfile /var/run/qemu/vm-115.pid -mem-path /hugepages  -mem-prealloc  -rtc base=utc -usb -usbdevice tablet -no-hpet -vga vmware  -cpu qemu64

Linux guest kernels observed so far:
  3.2.0
  3.13

Thanks,
Peter

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Paolo Bonzini]

On 10/02/2015 11:03, Peter Lieven wrote:
> 
> My hope was that anyone has observed this post 2.2.0 already and there
> is a fix available ;-)
> 
> Can you indicate what info would be helpful debugging this?
> 
> Cmdline is:
> /usr/bin/qemu-2.2.0  -enable-kvm  -M pc-i440fx-2.1  -nodefaults -netdev
> type=tap,id=guest19,script=no,downscript=no,ifname=tap19,vnet_hdr
> -device virtio-net-pci,netdev=guest19,mac=52:54:00:80:00:55 -netdev
> type=tap,id=guest20,script=no,downscript=no,ifname=tap20,vnet_hdr
> -device virtio-net-pci,netdev=guest20,mac=52:54:00:80:00:6f -netdev
> type=tap,id=guest21,script=no,downscript=no,ifname=tap21,vnet_hdr
> -device virtio-net-pci,netdev=guest21,mac=52:54:00:80:00:75 -serial
> null  -parallel null  -m 496 -monitor tcp:0:4011,server,nowait -vnc :11
> -qmp tcp:0:3011,server,nowait  -name 'gw-5000123'  -boot
> order=nc,menu=on  -drive
> index=2,media=cdrom,if=ide,cache=unsafe,aio=native,readonly=on  -k de 
> -incoming tcp:0:5011  -pidfile /var/run/qemu/vm-115.pid -mem-path
> /hugepages  -mem-prealloc  -rtc base=utc -usb -usbdevice tablet -no-hpet
> -vga vmware  -cpu qemu64

First of all (but unrelated to the bug) do not use "-cpu qemu64" with KVM.

Second, what downtime or bandwidth setting?  What is the actual
downtime?  Can you print time.tsc_timestamp and migration_tsc on the
destination?

Thanks,

Paolo

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Peter Lieven]

Am 10.02.2015 um 11:17 schrieb Paolo Bonzini:
>
> On 10/02/2015 11:03, Peter Lieven wrote:
>> My hope was that anyone has observed this post 2.2.0 already and there
>> is a fix available ;-)
>>
>> Can you indicate what info would be helpful debugging this?
>>
>> Cmdline is:
>> /usr/bin/qemu-2.2.0  -enable-kvm  -M pc-i440fx-2.1  -nodefaults -netdev
>> type=tap,id=guest19,script=no,downscript=no,ifname=tap19,vnet_hdr
>> -device virtio-net-pci,netdev=guest19,mac=52:54:00:80:00:55 -netdev
>> type=tap,id=guest20,script=no,downscript=no,ifname=tap20,vnet_hdr
>> -device virtio-net-pci,netdev=guest20,mac=52:54:00:80:00:6f -netdev
>> type=tap,id=guest21,script=no,downscript=no,ifname=tap21,vnet_hdr
>> -device virtio-net-pci,netdev=guest21,mac=52:54:00:80:00:75 -serial
>> null  -parallel null  -m 496 -monitor tcp:0:4011,server,nowait -vnc :11
>> -qmp tcp:0:3011,server,nowait  -name 'gw-5000123'  -boot
>> order=nc,menu=on  -drive
>> index=2,media=cdrom,if=ide,cache=unsafe,aio=native,readonly=on  -k de
>> -incoming tcp:0:5011  -pidfile /var/run/qemu/vm-115.pid -mem-path
>> /hugepages  -mem-prealloc  -rtc base=utc -usb -usbdevice tablet -no-hpet
>> -vga vmware  -cpu qemu64
> First of all (but unrelated to the bug) do not use "-cpu qemu64" with KVM.

I remember there was an issue with kvm64 anytime in the past, but this
is  ages ago I think. I will change that for new vserver starts. What (which flag) is
the exact issue with qemu64 vs. kvm64

>
> Second, what downtime or bandwidth setting?  What is the actual
> downtime?  Can you print time.tsc_timestamp and migration_tsc on the
> destination?

 From my logs:

migrate_set_speed 1200M
migrate_set_capability auto-converge on
migrate_set_capability xbzrle off

The max_downtime was default.

I will add debugging output around that assertion and try to reproduce.

Thank you,
Peter

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Paolo Bonzini]

On 10/02/2015 11:24, Peter Lieven wrote:
>>>
>> First of all (but unrelated to the bug) do not use "-cpu qemu64" with
>> KVM.
> 
> I remember there was an issue with kvm64 anytime in the past, but this
> is  ages ago I think. I will change that for new vserver starts. What
> (which flag) is the exact issue with qemu64 vs. kvm64

QEMU64 does not resemble any actual processor.  You want the lowest
denominator of your cluster, probably Nehalem.

Paolo

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Peter Lieven]

Am 10.02.2015 um 11:25 schrieb Paolo Bonzini:
>
> On 10/02/2015 11:24, Peter Lieven wrote:
>>> First of all (but unrelated to the bug) do not use "-cpu qemu64" with
>>> KVM.
>> I remember there was an issue with kvm64 anytime in the past, but this
>> is  ages ago I think. I will change that for new vserver starts. What
>> (which flag) is the exact issue with qemu64 vs. kvm64
> QEMU64 does not resemble any actual processor.  You want the lowest
> denominator of your cluster, probably Nehalem.

Ah okay. Normally, I compare the available cpu flags of my cluster
and pass those flags individually. Maybe I have to revisit that logic.

Peter

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Paolo Bonzini]

On 10/02/2015 11:30, Peter Lieven wrote:
>>>
>> QEMU64 does not resemble any actual processor.  You want the lowest
>> denominator of your cluster, probably Nehalem.
> 
> Ah okay. Normally, I compare the available cpu flags of my cluster
> and pass those flags individually. Maybe I have to revisit that logic.

Seems to be okay, but that work has been already done for you. :)

If you use "-cpu Nehalem,enforce" it will also fail to start on
pre-Nehalem processors.

Paolo

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Peter Lieven]

Am 10.02.2015 um 11:17 schrieb Paolo Bonzini:
>
> On 10/02/2015 11:03, Peter Lieven wrote:
>> My hope was that anyone has observed this post 2.2.0 already and there
>> is a fix available ;-)
>>
>> Can you indicate what info would be helpful debugging this?
>>
>> Cmdline is:
>> /usr/bin/qemu-2.2.0  -enable-kvm  -M pc-i440fx-2.1  -nodefaults -netdev
>> type=tap,id=guest19,script=no,downscript=no,ifname=tap19,vnet_hdr
>> -device virtio-net-pci,netdev=guest19,mac=52:54:00:80:00:55 -netdev
>> type=tap,id=guest20,script=no,downscript=no,ifname=tap20,vnet_hdr
>> -device virtio-net-pci,netdev=guest20,mac=52:54:00:80:00:6f -netdev
>> type=tap,id=guest21,script=no,downscript=no,ifname=tap21,vnet_hdr
>> -device virtio-net-pci,netdev=guest21,mac=52:54:00:80:00:75 -serial
>> null  -parallel null  -m 496 -monitor tcp:0:4011,server,nowait -vnc :11
>> -qmp tcp:0:3011,server,nowait  -name 'gw-5000123'  -boot
>> order=nc,menu=on  -drive
>> index=2,media=cdrom,if=ide,cache=unsafe,aio=native,readonly=on  -k de
>> -incoming tcp:0:5011  -pidfile /var/run/qemu/vm-115.pid -mem-path
>> /hugepages  -mem-prealloc  -rtc base=utc -usb -usbdevice tablet -no-hpet
>> -vga vmware  -cpu qemu64
> First of all (but unrelated to the bug) do not use "-cpu qemu64" with KVM.
>
> Second, what downtime or bandwidth setting?  What is the actual
> downtime?  Can you print time.tsc_timestamp and migration_tsc on the
> destination?

I also found this 'info migrate' output:

capabilities: xbzrle: off
rdma-pin-all: off
auto-converge: on
zero-blocks: on
Migration status: completed
total time: 604 milliseconds
downtime: 187 milliseconds
setup: 1 milliseconds
transferred ram: 331529 kbytes
throughput: 4498.48 mbps
remaining ram: 0 kbytes
total ram: 525700 kbytes
duplicate: 49083 pages
normal: 82613 pages
normal bytes: 330452 kbytes
dirty sync count: 0

Peter

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Marcelo Tosatti]

On Tue, Feb 10, 2015 at 10:26:33AM +0100, Peter Lieven wrote:
> Hi,
> 
> while migrating vServers from 2.1.0 to 2.2.0 I see the following assertion that seems to have
> been introduced between 2.1.0 and 2.2.0 trigger on the destination. This happens in a serious
> amount of cases. In most cases it works flawlessly on the second attempt to migrate the vServer
> (same source and same destination).
> 
> Any hints how to debug the root cause?
> 
> Thank you,
> Peter

Peter,

Make sure you have the following commit applied to the host:

commit 7f187922ddf6b67f2999a76dcb71663097b75497
KVM: x86: update masterclock values on TSC writes

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Peter Lieven]

Am 10.02.2015 um 11:50 schrieb Marcelo Tosatti:
> On Tue, Feb 10, 2015 at 10:26:33AM +0100, Peter Lieven wrote:
>> Hi,
>>
>> while migrating vServers from 2.1.0 to 2.2.0 I see the following assertion that seems to have
>> been introduced between 2.1.0 and 2.2.0 trigger on the destination. This happens in a serious
>> amount of cases. In most cases it works flawlessly on the second attempt to migrate the vServer
>> (same source and same destination).
>>
>> Any hints how to debug the root cause?
>>
>> Thank you,
>> Peter
> Peter,
>
> Make sure you have the following commit applied to the host:
>
> commit 7f187922ddf6b67f2999a76dcb71663097b75497
> KVM: x86: update masterclock values on TSC writes

I have build my kvm module from

http://git.kiszka.org/?p=kvm-kmod.git

but it seems not to have been updates since October.
The patch is not in there.

Is that repository not up to date anymore?

Peter

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Paolo Bonzini]

On 10/02/2015 13:17, Peter Lieven wrote:
> 
> I have build my kvm module from
> 
> http://git.kiszka.org/?p=kvm-kmod.git
> 
> but it seems not to have been updates since October.
> The patch is not in there.
> 
> Is that repository not up to date anymore?

Ahem, 3.19 was released last Sunday.  Give us some breathing room...

Paolo

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Peter Lieven]

Am 10.02.2015 um 13:35 schrieb Paolo Bonzini:
>
> On 10/02/2015 13:17, Peter Lieven wrote:
>> I have build my kvm module from
>>
>> http://git.kiszka.org/?p=kvm-kmod.git
>>
>> but it seems not to have been updates since October.
>> The patch is not in there.
>>
>> Is that repository not up to date anymore?
> Ahem, 3.19 was released last Sunday.  Give us some breathing room...

I missed the next branch. I checked that out and updated the submodule
to commit bfa76d49576599a4b9f9b7a71f23d73d6dcff735.

Does that make sense?

Peter

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Paolo Bonzini]

On 10/02/2015 13:56, Peter Lieven wrote:
>> Ahem, 3.19 was released last Sunday.  Give us some breathing room...
> 
> I missed the next branch. I checked that out and updated the submodule
> to commit bfa76d49576599a4b9f9b7a71f23d73d6dcff735.
> 
> Does that make sense?

Yes, but it is still not ready.  It only works with 3.18 and 3.16.

You should just use the stable kernels.  I sent out Marcelo's patch and
it should percolate to the stable kernels in the next few days/weeks.

Paolo

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Peter Lieven]

Am 10.02.2015 um 14:15 schrieb Paolo Bonzini:
>
> On 10/02/2015 13:56, Peter Lieven wrote:
>>> Ahem, 3.19 was released last Sunday.  Give us some breathing room...
>> I missed the next branch. I checked that out and updated the submodule
>> to commit bfa76d49576599a4b9f9b7a71f23d73d6dcff735.
>>
>> Does that make sense?
> Yes, but it is still not ready.  It only works with 3.18 and 3.16.

Okay, but for testing I will use 3.19 with the patch for testing if the assertion
disappears?!

Peter

>
> You should just use the stable kernels.  I sent out Marcelo's patch and
> it should percolate to the stable kernels in the next few days/weeks.
>
> Paolo


-- 

Mit freundlichen Grüßen

Peter Lieven

...........................................................

   KAMP Netzwerkdienste GmbH
   Vestische Str. 89-91 | 46117 Oberhausen
   Tel: +49 (0) 208.89 402-50 | Fax: +49 (0) 208.89 402-40
   pl@kamp.de | http://www.kamp.de

   Geschäftsführer: Heiner Lante | Michael Lante
   Amtsgericht Duisburg | HRB Nr. 12154
   USt-Id-Nr.: DE 120607556

...........................................................

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Paolo Bonzini]

On 10/02/2015 14:31, Peter Lieven wrote:
>>>
>> Yes, but it is still not ready.  It only works with 3.18 and 3.16.
> 
> Okay, but for testing I will use 3.19 with the patch for testing if the
> assertion disappears?!

If you use 3.19 in the host, you don't need kvm-kmod.

If you use 3.19 with kvm-kmod, you can use origin/next but only if your
host kernel is 3.16 or 3.18.

Paolo

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Peter Lieven]

Am 10.02.2015 um 14:40 schrieb Paolo Bonzini:
>
> On 10/02/2015 14:31, Peter Lieven wrote:
>>> Yes, but it is still not ready.  It only works with 3.18 and 3.16.
>> Okay, but for testing I will use 3.19 with the patch for testing if the
>> assertion disappears?!
> If you use 3.19 in the host, you don't need kvm-kmod.
>
> If you use 3.19 with kvm-kmod, you can use origin/next but only if your
> host kernel is 3.16 or 3.18.

Ah okay. I currently have to use 3.13 on the host. The kvm-kmod with branch origin/next
compiles just fine with the linux submodule at v3.19.

Is that supposed to work?

Peter

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Paolo Bonzini]

On 10/02/2015 14:58, Peter Lieven wrote:
>>
>>>> Yes, but it is still not ready.  It only works with 3.18 and 3.16.
>>> Okay, but for testing I will use 3.19 with the patch for testing if the
>>> assertion disappears?!
>> If you use 3.19 in the host, you don't need kvm-kmod.
>>
>> If you use 3.19 with kvm-kmod, you can use origin/next but only if your
>> host kernel is 3.16 or 3.18.
> 
> Ah okay. I currently have to use 3.13 on the host. The kvm-kmod with
> branch origin/next
> compiles just fine with the linux submodule at v3.19.

The buildbot (http://buildbot.kiszka.org/kvm-kmod/builders/3-next) tells
me that it shouldn't have compiled :) but maybe you have a more recent
3.13 stable release than the buildbot.  So if it compiles, you can use
it to test.

Paolo

Re: [Qemu-devel] kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc'

[Peter Lieven]

Am 10.02.2015 um 11:50 schrieb Marcelo Tosatti:
> On Tue, Feb 10, 2015 at 10:26:33AM +0100, Peter Lieven wrote:
>> Hi,
>>
>> while migrating vServers from 2.1.0 to 2.2.0 I see the following assertion that seems to have
>> been introduced between 2.1.0 and 2.2.0 trigger on the destination. This happens in a serious
>> amount of cases. In most cases it works flawlessly on the second attempt to migrate the vServer
>> (same source and same destination).
>>
>> Any hints how to debug the root cause?
>>
>> Thank you,
>> Peter
> Peter,
>
> Make sure you have the following commit applied to the host:
>
> commit 7f187922ddf6b67f2999a76dcb71663097b75497
> KVM: x86: update masterclock values on TSC writes

Thanks for the pointer. It seems that has fixed the assertion issue.

Peter