time does not move forward in HVM guests

time does not move forward in HVM guests

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 994 bytes --]

In my testing with sysbench in a HVM domU running a linux-4.4 based
pvops kernel on a xen-4.7 based dom0 the time does not move forward
properly:

There (URL below) is basically code like this:
  clock_gettime(CLOCK_MONOTONIC, a)
  do_work
  clock_gettime(CLOCK_MONOTONIC, b)
  diff_time(a,b)

All 'do_work' does is writing zeros to a block of memory.
clock_getres(CLOCK_MONOTONIC) indicates a resolution of 1ns.
If 'do_work' takes like 100ns or less: a==b. I think this is something
that should not happen. In case of vcpu overcommit this happens also
when 'do_work' takes around 800ns. At some point I have also seen cases
of time going backward. I can not reproduce this anymore, might have
been bugs in my code or the domU.cfg changed.

A workaround is booting the domU kernel with 'clocksource=tsc nohz=off highres=off'.

Why does this happen? Are the expectations too high?


Olaf


 https://github.com/olafhering/sysbench/compare/master...pv
 bash autogen.sh
 make -j
 bash mem.1K.on.sh

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: time does not move forward in HVM guests

[Jan Beulich]

>>> On 04.07.17 at 18:34, <olaf@aepfle.de> wrote:
> In my testing with sysbench in a HVM domU running a linux-4.4 based
> pvops kernel on a xen-4.7 based dom0 the time does not move forward
> properly:
> 
> There (URL below) is basically code like this:
>   clock_gettime(CLOCK_MONOTONIC, a)
>   do_work
>   clock_gettime(CLOCK_MONOTONIC, b)
>   diff_time(a,b)
> 
> All 'do_work' does is writing zeros to a block of memory.
> clock_getres(CLOCK_MONOTONIC) indicates a resolution of 1ns.

But what's the implied meaning of resolution here? See below.

> If 'do_work' takes like 100ns or less: a==b. I think this is something
> that should not happen. In case of vcpu overcommit this happens also
> when 'do_work' takes around 800ns. At some point I have also seen cases
> of time going backward. I can not reproduce this anymore, might have
> been bugs in my code or the domU.cfg changed.

Or did you perhaps test with an older version, where the time
handling backports from master hadn't been there yet?

> A workaround is booting the domU kernel with 'clocksource=tsc nohz=off 
> highres=off'.

What clocksource does the system use by default? HPET?
According to what the hypervisor tells the guest, vHPET
resolution is 16ns. That still wouldn't explain a steady value
over a period of 100ns, but it's at least a hint that what the
kernel tells you may not be what underlying (virtual)
hardware reports.

Additionally - are all three options indeed required to work
around this, i.e. no pair out of the three is enough?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: time does not move forward in HVM guests

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 1331 bytes --]

On Wed, Jul 05, Jan Beulich wrote:

> > clock_getres(CLOCK_MONOTONIC) indicates a resolution of 1ns.
> But what's the implied meaning of resolution here? See below.

I have no ide what the returned value is supposed to promise.

> Or did you perhaps test with an older version, where the time
> handling backports from master hadn't been there yet?

It was weeks ago, and I have not seen it since then. I think it is fixed
in one way or another.

> > A workaround is booting the domU kernel with 'clocksource=tsc nohz=off 
> > highres=off'.
> What clocksource does the system use by default? HPET?

HPET would be really really slow. The default clocksource is "xen".

> According to what the hypervisor tells the guest, vHPET
> resolution is 16ns. That still wouldn't explain a steady value
> over a period of 100ns, but it's at least a hint that what the
> kernel tells you may not be what underlying (virtual)
> hardware reports.

If clocksource=xen relies on the hypervisor, perhaps the kernel should
be aware of it in some way. So far I have not checked where clock_getres
gets its data.


> Additionally - are all three options indeed required to work
> around this, i.e. no pair out of the three is enough?

Yes, otherwise the kernel would complain, forgot the exact error
message.

Olaf

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: time does not move forward in HVM guests

[Jan Beulich]

>>> On 05.07.17 at 09:25, <olaf@aepfle.de> wrote:
> On Wed, Jul 05, Jan Beulich wrote:
> 
>> > clock_getres(CLOCK_MONOTONIC) indicates a resolution of 1ns.
>> But what's the implied meaning of resolution here? See below.
> 
> I have no ide what the returned value is supposed to promise.
> 
>> Or did you perhaps test with an older version, where the time
>> handling backports from master hadn't been there yet?
> 
> It was weeks ago, and I have not seen it since then. I think it is fixed
> in one way or another.
> 
>> > A workaround is booting the domU kernel with 'clocksource=tsc nohz=off 
>> > highres=off'.
>> What clocksource does the system use by default? HPET?
> 
> HPET would be really really slow. The default clocksource is "xen".

Oh, even for HVM. Doesn't that go back to the missing vDSO
support then again, which we had discussed just last week?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: time does not move forward in HVM guests

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 437 bytes --]

Am Wed, 05 Jul 2017 02:14:23 -0600
schrieb "Jan Beulich" <JBeulich@suse.com>:

> Oh, even for HVM. Doesn't that go back to the missing vDSO
> support then again, which we had discussed just last week?

Yes. This is part of it. With clocksource=tsc there is a performance boost because vdso is used. With clocksource=hpet there is a performance drop to 20%, depending on the workload, due to the emulation of it (I guess).

Olaf

[-- Attachment #1.2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel