[Qemu-devel] QEMU postcopy-test failing on ppc64

[Qemu-devel] QEMU postcopy-test failing on ppc64

[Stefan Hajnoczi]

I hit a failure running "make check" on ppc64 for the first time.  Ideas?

Stefan

commit 682df581c65ed2c1b9e77093e332214ecaa1ee93

  GTESTER check-qtest-ppc64
Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
current = 7c hit_edge = 1
Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
current = 1b hit_edge = 1
Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
current = 1a hit_edge = 1
**
ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Dr. David Alan Gilbert]

* Stefan Hajnoczi (stefanha@gmail.com) wrote:
> I hit a failure running "make check" on ppc64 for the first time.  Ideas?

Hmm shouldn't happen; cc'ing in lvivier, I've not checked it since Laurent got
the test working on PPC.

> Stefan
> 
> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
> 
>   GTESTER check-qtest-ppc64
> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
> current = 7c hit_edge = 1
> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
> current = 1b hit_edge = 1

The idea is that one byte in every 4k chunk of memory is being incremented
and the loop increments all those bytes once and then loops back to the
start.  So if you pause it at any point and look at memory you should
see two different values in memory, one before the edge that's getting
incremented and one after the edge - and the difference should be
exactly one.  So in this case we're seeing that the start of memory
has the value '1b' and at 5af3000 we have the value 1a (last_byte)
so somewhere between the two is the edge where it's currently incrementing;
then it should stay at 1a until the end of memory - but instead a '7c'
has somehow got in.

My only thought would be to bisect.

Dave

> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
> current = 1a hit_edge = 1
> **
> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Laurent Vivier]

On 14/11/2016 21:52, Stefan Hajnoczi wrote:
> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
> 
> Stefan
> 
> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
> 
>   GTESTER check-qtest-ppc64
> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
> current = 7c hit_edge = 1
> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
> current = 1b hit_edge = 1
> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
> current = 1a hit_edge = 1
> **
> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
> 

Are you using KVM PR?

it  was working fine with TCG and KVM HV.

Apparently, USERFAULTFD doesn't work with KVM PR.

I've already seen this kind of error with nested KVM on Power:
guest in guest with KVM PR in host.

This problem was reported on IRC by Greg if I remember correctly (CC:)

Laurent

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Greg Kurz]

On Tue, 15 Nov 2016 10:53:35 +0100
Laurent Vivier <lvivier@redhat.com> wrote:

> On 14/11/2016 21:52, Stefan Hajnoczi wrote:
> > I hit a failure running "make check" on ppc64 for the first time.  Ideas?
> > 
> > Stefan
> > 
> > commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
> > 
> >   GTESTER check-qtest-ppc64
> > Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
> > current = 7c hit_edge = 1
> > Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
> > current = 1b hit_edge = 1
> > Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
> > current = 1a hit_edge = 1
> > **
> > ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
> > GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
> >   
> 
> Are you using KVM PR?
> 
> it  was working fine with TCG and KVM HV.
> 
> Apparently, USERFAULTFD doesn't work with KVM PR.
> 
> I've already seen this kind of error with nested KVM on Power:
> guest in guest with KVM PR in host.
> 
> This problem was reported on IRC by Greg if I remember correctly (CC:)
> 

Yeah I hit this when running make check in a PPC64 BE guest which
has kvm_pr loaded. I did not find time to investigate though... I've
switched to run make check on bare metal POWER7 instead.

> Laurent

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Stefan Hajnoczi]

On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:
> On Tue, 15 Nov 2016 10:53:35 +0100
> Laurent Vivier <lvivier@redhat.com> wrote:
>
>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:
>> > I hit a failure running "make check" on ppc64 for the first time.  Ideas?
>> >
>> > Stefan
>> >
>> > commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
>> >
>> >   GTESTER check-qtest-ppc64
>> > Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
>> > current = 7c hit_edge = 1
>> > Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
>> > current = 1b hit_edge = 1
>> > Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
>> > current = 1a hit_edge = 1
>> > **
>> > ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
>> > GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
>> >
>>
>> Are you using KVM PR?
>>
>> it  was working fine with TCG and KVM HV.
>>
>> Apparently, USERFAULTFD doesn't work with KVM PR.
>>
>> I've already seen this kind of error with nested KVM on Power:
>> guest in guest with KVM PR in host.
>>
>> This problem was reported on IRC by Greg if I remember correctly (CC:)
>>
>
> Yeah I hit this when running make check in a PPC64 BE guest which
> has kvm_pr loaded. I did not find time to investigate though... I've
> switched to run make check on bare metal POWER7 instead.

Right, it's POWER7 PPC64 BE with kvm_pr.

If this should be fixed for QEMU 2.8 please add it to
http://qemu-project.org/Planning/2.8 and I'll track it (i.e. won't
release before it's resolved).

If this is a known issue that will be in QEMU 2.8 please add it to
http://qemu-project.org/ChangeLog/2.8#Known_issues.

Stefan

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Dr. David Alan Gilbert]

* Stefan Hajnoczi (stefanha@gmail.com) wrote:
> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:
> > On Tue, 15 Nov 2016 10:53:35 +0100
> > Laurent Vivier <lvivier@redhat.com> wrote:
> >
> >> On 14/11/2016 21:52, Stefan Hajnoczi wrote:
> >> > I hit a failure running "make check" on ppc64 for the first time.  Ideas?
> >> >
> >> > Stefan
> >> >
> >> > commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
> >> >
> >> >   GTESTER check-qtest-ppc64
> >> > Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
> >> > current = 7c hit_edge = 1
> >> > Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
> >> > current = 1b hit_edge = 1
> >> > Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
> >> > current = 1a hit_edge = 1
> >> > **
> >> > ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
> >> > GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
> >> >
> >>
> >> Are you using KVM PR?
> >>
> >> it  was working fine with TCG and KVM HV.
> >>
> >> Apparently, USERFAULTFD doesn't work with KVM PR.
> >>
> >> I've already seen this kind of error with nested KVM on Power:
> >> guest in guest with KVM PR in host.
> >>
> >> This problem was reported on IRC by Greg if I remember correctly (CC:)
> >>
> >
> > Yeah I hit this when running make check in a PPC64 BE guest which
> > has kvm_pr loaded. I did not find time to investigate though... I've
> > switched to run make check on bare metal POWER7 instead.
> 
> Right, it's POWER7 PPC64 BE with kvm_pr.
> 
> If this should be fixed for QEMU 2.8 please add it to
> http://qemu-project.org/Planning/2.8 and I'll track it (i.e. won't
> release before it's resolved).
> 
> If this is a known issue that will be in QEMU 2.8 please add it to
> http://qemu-project.org/ChangeLog/2.8#Known_issues.

I don't think it's new with 2.8 (Laurent?) and I think it's actually
a kernel issue that needs fixing.

Dave

> Stefan
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Laurent Vivier]

On 15/11/2016 13:20, Dr. David Alan Gilbert wrote:
> * Stefan Hajnoczi (stefanha@gmail.com) wrote:
>> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:
>>> On Tue, 15 Nov 2016 10:53:35 +0100
>>> Laurent Vivier <lvivier@redhat.com> wrote:
>>>
>>>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:
>>>>> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
>>>>>
>>>>> Stefan
>>>>>
>>>>> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
>>>>>
>>>>>   GTESTER check-qtest-ppc64
>>>>> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
>>>>> current = 7c hit_edge = 1
>>>>> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
>>>>> current = 1b hit_edge = 1
>>>>> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
>>>>> current = 1a hit_edge = 1
>>>>> **
>>>>> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
>>>>> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
>>>>>
>>>>
>>>> Are you using KVM PR?
>>>>
>>>> it  was working fine with TCG and KVM HV.
>>>>
>>>> Apparently, USERFAULTFD doesn't work with KVM PR.
>>>>
>>>> I've already seen this kind of error with nested KVM on Power:
>>>> guest in guest with KVM PR in host.
>>>>
>>>> This problem was reported on IRC by Greg if I remember correctly (CC:)
>>>>
>>>
>>> Yeah I hit this when running make check in a PPC64 BE guest which
>>> has kvm_pr loaded. I did not find time to investigate though... I've
>>> switched to run make check on bare metal POWER7 instead.
>>
>> Right, it's POWER7 PPC64 BE with kvm_pr.
>>
>> If this should be fixed for QEMU 2.8 please add it to
>> http://qemu-project.org/Planning/2.8 and I'll track it (i.e. won't
>> release before it's resolved).
>>
>> If this is a known issue that will be in QEMU 2.8 please add it to
>> http://qemu-project.org/ChangeLog/2.8#Known_issues.
> 
> I don't think it's new with 2.8 (Laurent?) and I think it's actually
> a kernel issue that needs fixing.

I's not a regression and I think it's a kernel issue.

As it has never worked with KVM PR and can't be fixed at QEMU level, we
can't solve it for 2.8 (no tracking).

I'm going to update the know issues for 2.8.
[and I'm going to try to understand what really happens]

Laurent

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2611 bytes --]

On Tue, Nov 15, 2016 at 01:58:38PM +0100, Laurent Vivier wrote:
> 
> 
> On 15/11/2016 13:20, Dr. David Alan Gilbert wrote:
> > * Stefan Hajnoczi (stefanha@gmail.com) wrote:
> >> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:
> >>> On Tue, 15 Nov 2016 10:53:35 +0100
> >>> Laurent Vivier <lvivier@redhat.com> wrote:
> >>>
> >>>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:
> >>>>> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
> >>>>>
> >>>>> Stefan
> >>>>>
> >>>>> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
> >>>>>
> >>>>>   GTESTER check-qtest-ppc64
> >>>>> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
> >>>>> current = 7c hit_edge = 1
> >>>>> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
> >>>>> current = 1b hit_edge = 1
> >>>>> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
> >>>>> current = 1a hit_edge = 1
> >>>>> **
> >>>>> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
> >>>>> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
> >>>>>
> >>>>
> >>>> Are you using KVM PR?
> >>>>
> >>>> it  was working fine with TCG and KVM HV.
> >>>>
> >>>> Apparently, USERFAULTFD doesn't work with KVM PR.
> >>>>
> >>>> I've already seen this kind of error with nested KVM on Power:
> >>>> guest in guest with KVM PR in host.
> >>>>
> >>>> This problem was reported on IRC by Greg if I remember correctly (CC:)
> >>>>
> >>>
> >>> Yeah I hit this when running make check in a PPC64 BE guest which
> >>> has kvm_pr loaded. I did not find time to investigate though... I've
> >>> switched to run make check on bare metal POWER7 instead.
> >>
> >> Right, it's POWER7 PPC64 BE with kvm_pr.
> >>
> >> If this should be fixed for QEMU 2.8 please add it to
> >> http://qemu-project.org/Planning/2.8 and I'll track it (i.e. won't
> >> release before it's resolved).
> >>
> >> If this is a known issue that will be in QEMU 2.8 please add it to
> >> http://qemu-project.org/ChangeLog/2.8#Known_issues.
> > 
> > I don't think it's new with 2.8 (Laurent?) and I think it's actually
> > a kernel issue that needs fixing.
> 
> I's not a regression and I think it's a kernel issue.
> 
> As it has never worked with KVM PR and can't be fixed at QEMU level, we
> can't solve it for 2.8 (no tracking).
> 
> I'm going to update the know issues for 2.8.
> [and I'm going to try to understand what really happens]

Should the test be skipped on KVM PR?

It's a shame that make check fails.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Greg Kurz]

[-- Attachment #1: Type: text/plain, Size: 3078 bytes --]

On Tue, 15 Nov 2016 14:48:30 +0000
Stefan Hajnoczi <stefanha@gmail.com> wrote:

> On Tue, Nov 15, 2016 at 01:58:38PM +0100, Laurent Vivier wrote:
> > 
> > 
> > On 15/11/2016 13:20, Dr. David Alan Gilbert wrote:  
> > > * Stefan Hajnoczi (stefanha@gmail.com) wrote:  
> > >> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:  
> > >>> On Tue, 15 Nov 2016 10:53:35 +0100
> > >>> Laurent Vivier <lvivier@redhat.com> wrote:
> > >>>  
> > >>>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:  
> > >>>>> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
> > >>>>>
> > >>>>> Stefan
> > >>>>>
> > >>>>> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
> > >>>>>
> > >>>>>   GTESTER check-qtest-ppc64
> > >>>>> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
> > >>>>> current = 7c hit_edge = 1
> > >>>>> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
> > >>>>> current = 1b hit_edge = 1
> > >>>>> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
> > >>>>> current = 1a hit_edge = 1
> > >>>>> **
> > >>>>> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
> > >>>>> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
> > >>>>>  
> > >>>>
> > >>>> Are you using KVM PR?
> > >>>>
> > >>>> it  was working fine with TCG and KVM HV.
> > >>>>
> > >>>> Apparently, USERFAULTFD doesn't work with KVM PR.
> > >>>>
> > >>>> I've already seen this kind of error with nested KVM on Power:
> > >>>> guest in guest with KVM PR in host.
> > >>>>
> > >>>> This problem was reported on IRC by Greg if I remember correctly (CC:)
> > >>>>  
> > >>>
> > >>> Yeah I hit this when running make check in a PPC64 BE guest which
> > >>> has kvm_pr loaded. I did not find time to investigate though... I've
> > >>> switched to run make check on bare metal POWER7 instead.  
> > >>
> > >> Right, it's POWER7 PPC64 BE with kvm_pr.
> > >>
> > >> If this should be fixed for QEMU 2.8 please add it to
> > >> http://qemu-project.org/Planning/2.8 and I'll track it (i.e. won't
> > >> release before it's resolved).
> > >>
> > >> If this is a known issue that will be in QEMU 2.8 please add it to
> > >> http://qemu-project.org/ChangeLog/2.8#Known_issues.  
> > > 
> > > I don't think it's new with 2.8 (Laurent?) and I think it's actually
> > > a kernel issue that needs fixing.  
> > 
> > I's not a regression and I think it's a kernel issue.
> > 
> > As it has never worked with KVM PR and can't be fixed at QEMU level, we
> > can't solve it for 2.8 (no tracking).
> > 
> > I'm going to update the know issues for 2.8.
> > [and I'm going to try to understand what really happens]  
> 
> Should the test be skipped on KVM PR?
> 
> It's a shame that make check fails.
> 

Yeah I agree this is lame but KVM PR isn't very cherished these days...

Since postcopy-test runs QEMU with -accel kvm:tcg, a possible workaround is
to kick KVM out of the environment you're running make check in.

> Stefan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Laurent Vivier]

On 15/11/2016 16:03, Greg Kurz wrote:
> On Tue, 15 Nov 2016 14:48:30 +0000
> Stefan Hajnoczi <stefanha@gmail.com> wrote:
> 
>> On Tue, Nov 15, 2016 at 01:58:38PM +0100, Laurent Vivier wrote:
>>>
>>>
>>> On 15/11/2016 13:20, Dr. David Alan Gilbert wrote:  
>>>> * Stefan Hajnoczi (stefanha@gmail.com) wrote:  
>>>>> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:  
>>>>>> On Tue, 15 Nov 2016 10:53:35 +0100
>>>>>> Laurent Vivier <lvivier@redhat.com> wrote:
>>>>>>  
>>>>>>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:  
>>>>>>>> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>>
>>>>>>>> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
>>>>>>>>
>>>>>>>>   GTESTER check-qtest-ppc64
>>>>>>>> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
>>>>>>>> current = 7c hit_edge = 1
>>>>>>>> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
>>>>>>>> current = 1b hit_edge = 1
>>>>>>>> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
>>>>>>>> current = 1a hit_edge = 1
>>>>>>>> **
>>>>>>>> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
>>>>>>>> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
>>>>>>>>  
>>>>>>>
>>>>>>> Are you using KVM PR?
>>>>>>>
>>>>>>> it  was working fine with TCG and KVM HV.
>>>>>>>
>>>>>>> Apparently, USERFAULTFD doesn't work with KVM PR.
>>>>>>>
>>>>>>> I've already seen this kind of error with nested KVM on Power:
>>>>>>> guest in guest with KVM PR in host.
>>>>>>>
>>>>>>> This problem was reported on IRC by Greg if I remember correctly (CC:)
>>>>>>>  
>>>>>>
>>>>>> Yeah I hit this when running make check in a PPC64 BE guest which
>>>>>> has kvm_pr loaded. I did not find time to investigate though... I've
>>>>>> switched to run make check on bare metal POWER7 instead.  
>>>>>
>>>>> Right, it's POWER7 PPC64 BE with kvm_pr.
>>>>>
>>>>> If this should be fixed for QEMU 2.8 please add it to
>>>>> http://qemu-project.org/Planning/2.8 and I'll track it (i.e. won't
>>>>> release before it's resolved).
>>>>>
>>>>> If this is a known issue that will be in QEMU 2.8 please add it to
>>>>> http://qemu-project.org/ChangeLog/2.8#Known_issues.  
>>>>
>>>> I don't think it's new with 2.8 (Laurent?) and I think it's actually
>>>> a kernel issue that needs fixing.  
>>>
>>> I's not a regression and I think it's a kernel issue.
>>>
>>> As it has never worked with KVM PR and can't be fixed at QEMU level, we
>>> can't solve it for 2.8 (no tracking).
>>>
>>> I'm going to update the know issues for 2.8.
>>> [and I'm going to try to understand what really happens]  
>>
>> Should the test be skipped on KVM PR?
>>
>> It's a shame that make check fails.
>>
> 
> Yeah I agree this is lame but KVM PR isn't very cherished these days...
> 
> Since postcopy-test runs QEMU with -accel kvm:tcg, a possible workaround is
> to kick KVM out of the environment you're running make check in.

Moreover, using "-accel kvm:tcg" induces an error when the test is
running in cross-arch case (ppc on intel, or intel on ppc).

But I think postcopy is typically something we should test with KVM, not
TCG, as there are some kernel dependencies (USERFAULTFD).

It has been suggested a while to use an environment variable, like for
kvm-unit-tests, something like QTEST_ACCEL="kvm".

Laurent

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Dr. David Alan Gilbert]

* Laurent Vivier (lvivier@redhat.com) wrote:
> 
> 
> On 15/11/2016 16:03, Greg Kurz wrote:
> > On Tue, 15 Nov 2016 14:48:30 +0000
> > Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > 
> >> On Tue, Nov 15, 2016 at 01:58:38PM +0100, Laurent Vivier wrote:
> >>>
> >>>
> >>> On 15/11/2016 13:20, Dr. David Alan Gilbert wrote:  
> >>>> * Stefan Hajnoczi (stefanha@gmail.com) wrote:  
> >>>>> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:  
> >>>>>> On Tue, 15 Nov 2016 10:53:35 +0100
> >>>>>> Laurent Vivier <lvivier@redhat.com> wrote:
> >>>>>>  
> >>>>>>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:  
> >>>>>>>> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
> >>>>>>>>
> >>>>>>>> Stefan
> >>>>>>>>
> >>>>>>>> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
> >>>>>>>>
> >>>>>>>>   GTESTER check-qtest-ppc64
> >>>>>>>> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
> >>>>>>>> current = 7c hit_edge = 1
> >>>>>>>> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
> >>>>>>>> current = 1b hit_edge = 1
> >>>>>>>> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
> >>>>>>>> current = 1a hit_edge = 1
> >>>>>>>> **
> >>>>>>>> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
> >>>>>>>> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
> >>>>>>>>  
> >>>>>>>
> >>>>>>> Are you using KVM PR?
> >>>>>>>
> >>>>>>> it  was working fine with TCG and KVM HV.
> >>>>>>>
> >>>>>>> Apparently, USERFAULTFD doesn't work with KVM PR.
> >>>>>>>
> >>>>>>> I've already seen this kind of error with nested KVM on Power:
> >>>>>>> guest in guest with KVM PR in host.
> >>>>>>>
> >>>>>>> This problem was reported on IRC by Greg if I remember correctly (CC:)
> >>>>>>>  
> >>>>>>
> >>>>>> Yeah I hit this when running make check in a PPC64 BE guest which
> >>>>>> has kvm_pr loaded. I did not find time to investigate though... I've
> >>>>>> switched to run make check on bare metal POWER7 instead.  
> >>>>>
> >>>>> Right, it's POWER7 PPC64 BE with kvm_pr.
> >>>>>
> >>>>> If this should be fixed for QEMU 2.8 please add it to
> >>>>> http://qemu-project.org/Planning/2.8 and I'll track it (i.e. won't
> >>>>> release before it's resolved).
> >>>>>
> >>>>> If this is a known issue that will be in QEMU 2.8 please add it to
> >>>>> http://qemu-project.org/ChangeLog/2.8#Known_issues.  
> >>>>
> >>>> I don't think it's new with 2.8 (Laurent?) and I think it's actually
> >>>> a kernel issue that needs fixing.  
> >>>
> >>> I's not a regression and I think it's a kernel issue.
> >>>
> >>> As it has never worked with KVM PR and can't be fixed at QEMU level, we
> >>> can't solve it for 2.8 (no tracking).
> >>>
> >>> I'm going to update the know issues for 2.8.
> >>> [and I'm going to try to understand what really happens]  
> >>
> >> Should the test be skipped on KVM PR?
> >>
> >> It's a shame that make check fails.
> >>
> > 
> > Yeah I agree this is lame but KVM PR isn't very cherished these days...
> > 
> > Since postcopy-test runs QEMU with -accel kvm:tcg, a possible workaround is
> > to kick KVM out of the environment you're running make check in.
> 
> Moreover, using "-accel kvm:tcg" induces an error when the test is
> running in cross-arch case (ppc on intel, or intel on ppc).
> 
> But I think postcopy is typically something we should test with KVM, not
> TCG, as there are some kernel dependencies (USERFAULTFD).
> 
> It has been suggested a while to use an environment variable, like for
> kvm-unit-tests, something like QTEST_ACCEL="kvm".

Can we detect kvm_pr at runtime in the test and just make it skip?

Dave

> Laurent
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Laurent Vivier]

On 15/11/2016 16:08, Dr. David Alan Gilbert wrote:
> * Laurent Vivier (lvivier@redhat.com) wrote:
>>
>>
>> On 15/11/2016 16:03, Greg Kurz wrote:
>>> On Tue, 15 Nov 2016 14:48:30 +0000
>>> Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>>
>>>> On Tue, Nov 15, 2016 at 01:58:38PM +0100, Laurent Vivier wrote:
>>>>>
>>>>>
>>>>> On 15/11/2016 13:20, Dr. David Alan Gilbert wrote:  
>>>>>> * Stefan Hajnoczi (stefanha@gmail.com) wrote:  
>>>>>>> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:  
>>>>>>>> On Tue, 15 Nov 2016 10:53:35 +0100
>>>>>>>> Laurent Vivier <lvivier@redhat.com> wrote:
>>>>>>>>  
>>>>>>>>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:  
>>>>>>>>>> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
>>>>>>>>>>
>>>>>>>>>> Stefan
>>>>>>>>>>
>>>>>>>>>> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
>>>>>>>>>>
>>>>>>>>>>   GTESTER check-qtest-ppc64
>>>>>>>>>> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
>>>>>>>>>> current = 7c hit_edge = 1
>>>>>>>>>> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
>>>>>>>>>> current = 1b hit_edge = 1
>>>>>>>>>> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
>>>>>>>>>> current = 1a hit_edge = 1
>>>>>>>>>> **
>>>>>>>>>> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
>>>>>>>>>> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
>>>>>>>>>>  
>>>>>>>>>
>>>>>>>>> Are you using KVM PR?
>>>>>>>>>
>>>>>>>>> it  was working fine with TCG and KVM HV.
>>>>>>>>>
>>>>>>>>> Apparently, USERFAULTFD doesn't work with KVM PR.
>>>>>>>>>
>>>>>>>>> I've already seen this kind of error with nested KVM on Power:
>>>>>>>>> guest in guest with KVM PR in host.
>>>>>>>>>
>>>>>>>>> This problem was reported on IRC by Greg if I remember correctly (CC:)
>>>>>>>>>  
>>>>>>>>
>>>>>>>> Yeah I hit this when running make check in a PPC64 BE guest which
>>>>>>>> has kvm_pr loaded. I did not find time to investigate though... I've
>>>>>>>> switched to run make check on bare metal POWER7 instead.  
>>>>>>>
>>>>>>> Right, it's POWER7 PPC64 BE with kvm_pr.
>>>>>>>
>>>>>>> If this should be fixed for QEMU 2.8 please add it to
>>>>>>> http://qemu-project.org/Planning/2.8 and I'll track it (i.e. won't
>>>>>>> release before it's resolved).
>>>>>>>
>>>>>>> If this is a known issue that will be in QEMU 2.8 please add it to
>>>>>>> http://qemu-project.org/ChangeLog/2.8#Known_issues.  
>>>>>>
>>>>>> I don't think it's new with 2.8 (Laurent?) and I think it's actually
>>>>>> a kernel issue that needs fixing.  
>>>>>
>>>>> I's not a regression and I think it's a kernel issue.
>>>>>
>>>>> As it has never worked with KVM PR and can't be fixed at QEMU level, we
>>>>> can't solve it for 2.8 (no tracking).
>>>>>
>>>>> I'm going to update the know issues for 2.8.
>>>>> [and I'm going to try to understand what really happens]  
>>>>
>>>> Should the test be skipped on KVM PR?
>>>>
>>>> It's a shame that make check fails.
>>>>
>>>
>>> Yeah I agree this is lame but KVM PR isn't very cherished these days...
>>>
>>> Since postcopy-test runs QEMU with -accel kvm:tcg, a possible workaround is
>>> to kick KVM out of the environment you're running make check in.
>>
>> Moreover, using "-accel kvm:tcg" induces an error when the test is
>> running in cross-arch case (ppc on intel, or intel on ppc).
>>
>> But I think postcopy is typically something we should test with KVM, not
>> TCG, as there are some kernel dependencies (USERFAULTFD).
>>
>> It has been suggested a while to use an environment variable, like for
>> kvm-unit-tests, something like QTEST_ACCEL="kvm".
> 
> Can we detect kvm_pr at runtime in the test and just make it skip?

There was a discussion on the subject: tests must be run in a predictive
way, so always with the same parametes and conditions.
[I think it was said by danpb or Drew +CC]

Laurent

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Greg Kurz]

On Tue, 15 Nov 2016 16:07:40 +0100
Laurent Vivier <lvivier@redhat.com> wrote:

> On 15/11/2016 16:03, Greg Kurz wrote:
> > On Tue, 15 Nov 2016 14:48:30 +0000
> > Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >   
> >> On Tue, Nov 15, 2016 at 01:58:38PM +0100, Laurent Vivier wrote:  
> >>>
> >>>
> >>> On 15/11/2016 13:20, Dr. David Alan Gilbert wrote:    
> >>>> * Stefan Hajnoczi (stefanha@gmail.com) wrote:    
> >>>>> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:    
> >>>>>> On Tue, 15 Nov 2016 10:53:35 +0100
> >>>>>> Laurent Vivier <lvivier@redhat.com> wrote:
> >>>>>>    
> >>>>>>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:    
> >>>>>>>> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
> >>>>>>>>
> >>>>>>>> Stefan
> >>>>>>>>
> >>>>>>>> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
> >>>>>>>>
> >>>>>>>>   GTESTER check-qtest-ppc64
> >>>>>>>> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
> >>>>>>>> current = 7c hit_edge = 1
> >>>>>>>> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
> >>>>>>>> current = 1b hit_edge = 1
> >>>>>>>> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
> >>>>>>>> current = 1a hit_edge = 1
> >>>>>>>> **
> >>>>>>>> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
> >>>>>>>> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
> >>>>>>>>    
> >>>>>>>
> >>>>>>> Are you using KVM PR?
> >>>>>>>
> >>>>>>> it  was working fine with TCG and KVM HV.
> >>>>>>>
> >>>>>>> Apparently, USERFAULTFD doesn't work with KVM PR.
> >>>>>>>
> >>>>>>> I've already seen this kind of error with nested KVM on Power:
> >>>>>>> guest in guest with KVM PR in host.
> >>>>>>>
> >>>>>>> This problem was reported on IRC by Greg if I remember correctly (CC:)
> >>>>>>>    
> >>>>>>
> >>>>>> Yeah I hit this when running make check in a PPC64 BE guest which
> >>>>>> has kvm_pr loaded. I did not find time to investigate though... I've
> >>>>>> switched to run make check on bare metal POWER7 instead.    
> >>>>>
> >>>>> Right, it's POWER7 PPC64 BE with kvm_pr.
> >>>>>
> >>>>> If this should be fixed for QEMU 2.8 please add it to
> >>>>> http://qemu-project.org/Planning/2.8 and I'll track it (i.e. won't
> >>>>> release before it's resolved).
> >>>>>
> >>>>> If this is a known issue that will be in QEMU 2.8 please add it to
> >>>>> http://qemu-project.org/ChangeLog/2.8#Known_issues.    
> >>>>
> >>>> I don't think it's new with 2.8 (Laurent?) and I think it's actually
> >>>> a kernel issue that needs fixing.    
> >>>
> >>> I's not a regression and I think it's a kernel issue.
> >>>
> >>> As it has never worked with KVM PR and can't be fixed at QEMU level, we
> >>> can't solve it for 2.8 (no tracking).
> >>>
> >>> I'm going to update the know issues for 2.8.
> >>> [and I'm going to try to understand what really happens]    
> >>
> >> Should the test be skipped on KVM PR?
> >>
> >> It's a shame that make check fails.
> >>  
> > 
> > Yeah I agree this is lame but KVM PR isn't very cherished these days...
> > 
> > Since postcopy-test runs QEMU with -accel kvm:tcg, a possible workaround is
> > to kick KVM out of the environment you're running make check in.  
> 
> Moreover, using "-accel kvm:tcg" induces an error when the test is
> running in cross-arch case (ppc on intel, or intel on ppc).
> 

You're talking about the '"kvm" accelerator not found' error ?

> But I think postcopy is typically something we should test with KVM, not
> TCG, as there are some kernel dependencies (USERFAULTFD).
> 

I agree it should be tested with KVM when applicable but I don't understand
the 'not TCG' since it is the only way to test when you don't have access
to specific hardware...

> It has been suggested a while to use an environment variable, like for
> kvm-unit-tests, something like QTEST_ACCEL="kvm".
> 

Yeah, I think it would be better than "-accel kvm:tcg"

> Laurent

Cheers.

--
Greg

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Laurent Vivier]

On 15/11/2016 19:01, Greg Kurz wrote:
> On Tue, 15 Nov 2016 16:07:40 +0100
> Laurent Vivier <lvivier@redhat.com> wrote:
> 
>> On 15/11/2016 16:03, Greg Kurz wrote:
>>> On Tue, 15 Nov 2016 14:48:30 +0000
>>> Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>>   
>>>> On Tue, Nov 15, 2016 at 01:58:38PM +0100, Laurent Vivier wrote:  
>>>>>
>>>>>
>>>>> On 15/11/2016 13:20, Dr. David Alan Gilbert wrote:    
>>>>>> * Stefan Hajnoczi (stefanha@gmail.com) wrote:    
>>>>>>> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:    
>>>>>>>> On Tue, 15 Nov 2016 10:53:35 +0100
>>>>>>>> Laurent Vivier <lvivier@redhat.com> wrote:
>>>>>>>>    
>>>>>>>>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:    
>>>>>>>>>> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
>>>>>>>>>>
>>>>>>>>>> Stefan
>>>>>>>>>>
>>>>>>>>>> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
>>>>>>>>>>
>>>>>>>>>>   GTESTER check-qtest-ppc64
>>>>>>>>>> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
>>>>>>>>>> current = 7c hit_edge = 1
>>>>>>>>>> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
>>>>>>>>>> current = 1b hit_edge = 1
>>>>>>>>>> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
>>>>>>>>>> current = 1a hit_edge = 1
>>>>>>>>>> **
>>>>>>>>>> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
>>>>>>>>>> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
>>>>>>>>>>    
>>>>>>>>>
>>>>>>>>> Are you using KVM PR?
>>>>>>>>>
>>>>>>>>> it  was working fine with TCG and KVM HV.
>>>>>>>>>
>>>>>>>>> Apparently, USERFAULTFD doesn't work with KVM PR.
>>>>>>>>>
>>>>>>>>> I've already seen this kind of error with nested KVM on Power:
>>>>>>>>> guest in guest with KVM PR in host.
>>>>>>>>>
>>>>>>>>> This problem was reported on IRC by Greg if I remember correctly (CC:)
>>>>>>>>>    
>>>>>>>>
>>>>>>>> Yeah I hit this when running make check in a PPC64 BE guest which
>>>>>>>> has kvm_pr loaded. I did not find time to investigate though... I've
>>>>>>>> switched to run make check on bare metal POWER7 instead.    
>>>>>>>
>>>>>>> Right, it's POWER7 PPC64 BE with kvm_pr.
>>>>>>>
>>>>>>> If this should be fixed for QEMU 2.8 please add it to
>>>>>>> http://qemu-project.org/Planning/2.8 and I'll track it (i.e. won't
>>>>>>> release before it's resolved).
>>>>>>>
>>>>>>> If this is a known issue that will be in QEMU 2.8 please add it to
>>>>>>> http://qemu-project.org/ChangeLog/2.8#Known_issues.    
>>>>>>
>>>>>> I don't think it's new with 2.8 (Laurent?) and I think it's actually
>>>>>> a kernel issue that needs fixing.    
>>>>>
>>>>> I's not a regression and I think it's a kernel issue.
>>>>>
>>>>> As it has never worked with KVM PR and can't be fixed at QEMU level, we
>>>>> can't solve it for 2.8 (no tracking).
>>>>>
>>>>> I'm going to update the know issues for 2.8.
>>>>> [and I'm going to try to understand what really happens]    
>>>>
>>>> Should the test be skipped on KVM PR?
>>>>
>>>> It's a shame that make check fails.
>>>>  
>>>
>>> Yeah I agree this is lame but KVM PR isn't very cherished these days...
>>>
>>> Since postcopy-test runs QEMU with -accel kvm:tcg, a possible workaround is
>>> to kick KVM out of the environment you're running make check in.  
>>
>> Moreover, using "-accel kvm:tcg" induces an error when the test is
>> running in cross-arch case (ppc on intel, or intel on ppc).
>>
> 
> You're talking about the '"kvm" accelerator not found' error ?

Yes

> 
>> But I think postcopy is typically something we should test with KVM, not
>> TCG, as there are some kernel dependencies (USERFAULTFD).
>>
> 
> I agree it should be tested with KVM when applicable but I don't understand
> the 'not TCG' since it is the only way to test when you don't have access
> to specific hardware...

I agree; What I mean is we must test this preferably with KVM because it
is more likely to have a problem with KVM than with TCG because of the
MMU virtualization.

>> It has been suggested a while to use an environment variable, like for
>> kvm-unit-tests, something like QTEST_ACCEL="kvm".
>>
> 
> Yeah, I think it would be better than "-accel kvm:tcg"
> 
>> Laurent
> 
> Cheers.
> 
> --
> Greg

Thanks,
Laurent

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Greg Kurz]

On Tue, 15 Nov 2016 19:24:24 +0100
Laurent Vivier <lvivier@redhat.com> wrote:

> On 15/11/2016 19:01, Greg Kurz wrote:
> > On Tue, 15 Nov 2016 16:07:40 +0100
> > Laurent Vivier <lvivier@redhat.com> wrote:
> >   
> >> On 15/11/2016 16:03, Greg Kurz wrote:  
> >>> On Tue, 15 Nov 2016 14:48:30 +0000
> >>> Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >>>     
> >>>> On Tue, Nov 15, 2016 at 01:58:38PM +0100, Laurent Vivier wrote:    
> >>>>>
> >>>>>
> >>>>> On 15/11/2016 13:20, Dr. David Alan Gilbert wrote:      
> >>>>>> * Stefan Hajnoczi (stefanha@gmail.com) wrote:      
> >>>>>>> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:      
> >>>>>>>> On Tue, 15 Nov 2016 10:53:35 +0100
> >>>>>>>> Laurent Vivier <lvivier@redhat.com> wrote:
> >>>>>>>>      
> >>>>>>>>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:      
> >>>>>>>>>> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
> >>>>>>>>>>
> >>>>>>>>>> Stefan
> >>>>>>>>>>
> >>>>>>>>>> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
> >>>>>>>>>>
> >>>>>>>>>>   GTESTER check-qtest-ppc64
> >>>>>>>>>> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
> >>>>>>>>>> current = 7c hit_edge = 1
> >>>>>>>>>> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
> >>>>>>>>>> current = 1b hit_edge = 1
> >>>>>>>>>> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
> >>>>>>>>>> current = 1a hit_edge = 1
> >>>>>>>>>> **
> >>>>>>>>>> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
> >>>>>>>>>> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
> >>>>>>>>>>      
> >>>>>>>>>
> >>>>>>>>> Are you using KVM PR?
> >>>>>>>>>
> >>>>>>>>> it  was working fine with TCG and KVM HV.
> >>>>>>>>>
> >>>>>>>>> Apparently, USERFAULTFD doesn't work with KVM PR.
> >>>>>>>>>
> >>>>>>>>> I've already seen this kind of error with nested KVM on Power:
> >>>>>>>>> guest in guest with KVM PR in host.
> >>>>>>>>>
> >>>>>>>>> This problem was reported on IRC by Greg if I remember correctly (CC:)
> >>>>>>>>>      
> >>>>>>>>
> >>>>>>>> Yeah I hit this when running make check in a PPC64 BE guest which
> >>>>>>>> has kvm_pr loaded. I did not find time to investigate though... I've
> >>>>>>>> switched to run make check on bare metal POWER7 instead.      
> >>>>>>>
> >>>>>>> Right, it's POWER7 PPC64 BE with kvm_pr.
> >>>>>>>
> >>>>>>> If this should be fixed for QEMU 2.8 please add it to
> >>>>>>> http://qemu-project.org/Planning/2.8 and I'll track it (i.e. won't
> >>>>>>> release before it's resolved).
> >>>>>>>
> >>>>>>> If this is a known issue that will be in QEMU 2.8 please add it to
> >>>>>>> http://qemu-project.org/ChangeLog/2.8#Known_issues.      
> >>>>>>
> >>>>>> I don't think it's new with 2.8 (Laurent?) and I think it's actually
> >>>>>> a kernel issue that needs fixing.      
> >>>>>
> >>>>> I's not a regression and I think it's a kernel issue.
> >>>>>
> >>>>> As it has never worked with KVM PR and can't be fixed at QEMU level, we
> >>>>> can't solve it for 2.8 (no tracking).
> >>>>>
> >>>>> I'm going to update the know issues for 2.8.
> >>>>> [and I'm going to try to understand what really happens]      
> >>>>
> >>>> Should the test be skipped on KVM PR?
> >>>>
> >>>> It's a shame that make check fails.
> >>>>    
> >>>
> >>> Yeah I agree this is lame but KVM PR isn't very cherished these days...
> >>>
> >>> Since postcopy-test runs QEMU with -accel kvm:tcg, a possible workaround is
> >>> to kick KVM out of the environment you're running make check in.    
> >>
> >> Moreover, using "-accel kvm:tcg" induces an error when the test is
> >> running in cross-arch case (ppc on intel, or intel on ppc).
> >>  
> > 
> > You're talking about the '"kvm" accelerator not found' error ?  
> 
> Yes
> 
> >   
> >> But I think postcopy is typically something we should test with KVM, not
> >> TCG, as there are some kernel dependencies (USERFAULTFD).
> >>  
> > 
> > I agree it should be tested with KVM when applicable but I don't understand
> > the 'not TCG' since it is the only way to test when you don't have access
> > to specific hardware...  
> 
> I agree; What I mean is we must test this preferably with KVM because it
> is more likely to have a problem with KVM than with TCG because of the
> MMU virtualization.
> 

Fair enough but this doesn't address the problem we currently have: a broken
KVM can cause make check to fail... and BTW, until the root cause is found,
I'm not sure if the culprit is KVM PR alone or QEMU+KVM PR...

Cheers.

--
Greg

> >> It has been suggested a while to use an environment variable, like for
> >> kvm-unit-tests, something like QTEST_ACCEL="kvm".
> >>  
> > 
> > Yeah, I think it would be better than "-accel kvm:tcg"
> >   
> >> Laurent  
> > 
> > Cheers.
> > 
> > --
> > Greg  
> 
> Thanks,
> Laurent

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Thomas Huth]

On 15.11.2016 21:02, Greg Kurz wrote:
> On Tue, 15 Nov 2016 19:24:24 +0100
> Laurent Vivier <lvivier@redhat.com> wrote:
> 
>> On 15/11/2016 19:01, Greg Kurz wrote:
>>> On Tue, 15 Nov 2016 16:07:40 +0100
>>> Laurent Vivier <lvivier@redhat.com> wrote:
>>>   
>>>> On 15/11/2016 16:03, Greg Kurz wrote:  
>>>>> On Tue, 15 Nov 2016 14:48:30 +0000
>>>>> Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>>>>     
>>>>>> On Tue, Nov 15, 2016 at 01:58:38PM +0100, Laurent Vivier wrote:    
>>>>>>>
>>>>>>>
>>>>>>> On 15/11/2016 13:20, Dr. David Alan Gilbert wrote:      
>>>>>>>> * Stefan Hajnoczi (stefanha@gmail.com) wrote:      
>>>>>>>>> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:      
>>>>>>>>>> On Tue, 15 Nov 2016 10:53:35 +0100
>>>>>>>>>> Laurent Vivier <lvivier@redhat.com> wrote:
>>>>>>>>>>      
>>>>>>>>>>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:      
>>>>>>>>>>>> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
>>>>>>>>>>>>
>>>>>>>>>>>> Stefan
>>>>>>>>>>>>
>>>>>>>>>>>> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
>>>>>>>>>>>>
>>>>>>>>>>>>   GTESTER check-qtest-ppc64
>>>>>>>>>>>> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
>>>>>>>>>>>> current = 7c hit_edge = 1
>>>>>>>>>>>> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
>>>>>>>>>>>> current = 1b hit_edge = 1
>>>>>>>>>>>> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
>>>>>>>>>>>> current = 1a hit_edge = 1
>>>>>>>>>>>> **
>>>>>>>>>>>> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
>>>>>>>>>>>> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
>>>>>>>>>>>>      
>>>>>>>>>>>
>>>>>>>>>>> Are you using KVM PR?
>>>>>>>>>>>
>>>>>>>>>>> it  was working fine with TCG and KVM HV.
>>>>>>>>>>>
>>>>>>>>>>> Apparently, USERFAULTFD doesn't work with KVM PR.
>>>>>>>>>>>
>>>>>>>>>>> I've already seen this kind of error with nested KVM on Power:
>>>>>>>>>>> guest in guest with KVM PR in host.
>>>>>>>>>>>
>>>>>>>>>>> This problem was reported on IRC by Greg if I remember correctly (CC:)
>>>>>>>>>>>      
>>>>>>>>>>
>>>>>>>>>> Yeah I hit this when running make check in a PPC64 BE guest which
>>>>>>>>>> has kvm_pr loaded. I did not find time to investigate though... I've
>>>>>>>>>> switched to run make check on bare metal POWER7 instead.      
>>>>>>>>>
>>>>>>>>> Right, it's POWER7 PPC64 BE with kvm_pr.
>>>>>>>>>
>>>>>>>>> If this should be fixed for QEMU 2.8 please add it to
>>>>>>>>> http://qemu-project.org/Planning/2.8 and I'll track it (i.e. won't
>>>>>>>>> release before it's resolved).
>>>>>>>>>
>>>>>>>>> If this is a known issue that will be in QEMU 2.8 please add it to
>>>>>>>>> http://qemu-project.org/ChangeLog/2.8#Known_issues.      
>>>>>>>>
>>>>>>>> I don't think it's new with 2.8 (Laurent?) and I think it's actually
>>>>>>>> a kernel issue that needs fixing.      
>>>>>>>
>>>>>>> I's not a regression and I think it's a kernel issue.
>>>>>>>
>>>>>>> As it has never worked with KVM PR and can't be fixed at QEMU level, we
>>>>>>> can't solve it for 2.8 (no tracking).
>>>>>>>
>>>>>>> I'm going to update the know issues for 2.8.
>>>>>>> [and I'm going to try to understand what really happens]      
>>>>>>
>>>>>> Should the test be skipped on KVM PR?
>>>>>>
>>>>>> It's a shame that make check fails.
>>>>>>    
>>>>>
>>>>> Yeah I agree this is lame but KVM PR isn't very cherished these days...
>>>>>
>>>>> Since postcopy-test runs QEMU with -accel kvm:tcg, a possible workaround is
>>>>> to kick KVM out of the environment you're running make check in.    
>>>>
>>>> Moreover, using "-accel kvm:tcg" induces an error when the test is
>>>> running in cross-arch case (ppc on intel, or intel on ppc).
>>>>  
>>>
>>> You're talking about the '"kvm" accelerator not found' error ?  
>>
>> Yes
>>
>>>   
>>>> But I think postcopy is typically something we should test with KVM, not
>>>> TCG, as there are some kernel dependencies (USERFAULTFD).
>>>>  
>>>
>>> I agree it should be tested with KVM when applicable but I don't understand
>>> the 'not TCG' since it is the only way to test when you don't have access
>>> to specific hardware...  
>>
>> I agree; What I mean is we must test this preferably with KVM because it
>> is more likely to have a problem with KVM than with TCG because of the
>> MMU virtualization.
>>
> 
> Fair enough but this doesn't address the problem we currently have: a broken
> KVM can cause make check to fail...

Thinking about this again ... maybe the code to exercise KVM should
rather go into kvm-unit-tests instead of being part of the QEMU tests?

 Thomas

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Greg Kurz]

On Tue, 15 Nov 2016 11:14:57 +0000
Stefan Hajnoczi <stefanha@gmail.com> wrote:

> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:
> > On Tue, 15 Nov 2016 10:53:35 +0100
> > Laurent Vivier <lvivier@redhat.com> wrote:
> >  
> >> On 14/11/2016 21:52, Stefan Hajnoczi wrote:  
> >> > I hit a failure running "make check" on ppc64 for the first time.  Ideas?
> >> >
> >> > Stefan
> >> >
> >> > commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
> >> >
> >> >   GTESTER check-qtest-ppc64
> >> > Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
> >> > current = 7c hit_edge = 1
> >> > Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
> >> > current = 1b hit_edge = 1
> >> > Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
> >> > current = 1a hit_edge = 1
> >> > **
> >> > ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
> >> > GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
> >> >  
> >>
> >> Are you using KVM PR?
> >>
> >> it  was working fine with TCG and KVM HV.
> >>
> >> Apparently, USERFAULTFD doesn't work with KVM PR.
> >>
> >> I've already seen this kind of error with nested KVM on Power:
> >> guest in guest with KVM PR in host.
> >>
> >> This problem was reported on IRC by Greg if I remember correctly (CC:)
> >>  
> >
> > Yeah I hit this when running make check in a PPC64 BE guest which
> > has kvm_pr loaded. I did not find time to investigate though... I've
> > switched to run make check on bare metal POWER7 instead.  
> 
> Right, it's POWER7 PPC64 BE with kvm_pr.
> 

And you cannot install a PPC64 BE distro on this POWER7 host and use
kvm_hv instead ?

> If this should be fixed for QEMU 2.8 please add it to
> http://qemu-project.org/Planning/2.8 and I'll track it (i.e. won't
> release before it's resolved).
> 
> If this is a known issue that will be in QEMU 2.8 please add it to
> http://qemu-project.org/ChangeLog/2.8#Known_issues.
> 
> Stefan

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Stefan Hajnoczi]

On Tue, Nov 15, 2016 at 2:56 PM, Greg Kurz <groug@kaod.org> wrote:
> On Tue, 15 Nov 2016 11:14:57 +0000
> Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
>> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:
>> > On Tue, 15 Nov 2016 10:53:35 +0100
>> > Laurent Vivier <lvivier@redhat.com> wrote:
>> >
>> >> On 14/11/2016 21:52, Stefan Hajnoczi wrote:
>> >> > I hit a failure running "make check" on ppc64 for the first time.  Ideas?
>> >> >
>> >> > Stefan
>> >> >
>> >> > commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
>> >> >
>> >> >   GTESTER check-qtest-ppc64
>> >> > Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
>> >> > current = 7c hit_edge = 1
>> >> > Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
>> >> > current = 1b hit_edge = 1
>> >> > Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
>> >> > current = 1a hit_edge = 1
>> >> > **
>> >> > ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
>> >> > GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
>> >> >
>> >>
>> >> Are you using KVM PR?
>> >>
>> >> it  was working fine with TCG and KVM HV.
>> >>
>> >> Apparently, USERFAULTFD doesn't work with KVM PR.
>> >>
>> >> I've already seen this kind of error with nested KVM on Power:
>> >> guest in guest with KVM PR in host.
>> >>
>> >> This problem was reported on IRC by Greg if I remember correctly (CC:)
>> >>
>> >
>> > Yeah I hit this when running make check in a PPC64 BE guest which
>> > has kvm_pr loaded. I did not find time to investigate though... I've
>> > switched to run make check on bare metal POWER7 instead.
>>
>> Right, it's POWER7 PPC64 BE with kvm_pr.
>>
>
> And you cannot install a PPC64 BE distro on this POWER7 host and use
> kvm_hv instead ?

Sure, I could get a non-kvm_pr environment.  I'm just concerned that
QEMU 2.8-rc0 is being tagged today and there is a POWER environment
where "make check" fails.

Is kvm_pr supported by QEMU?

If it is supported then "make check" ought to pass.

Stefan

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Thomas Huth]

On 15.11.2016 16:18, Stefan Hajnoczi wrote:
> On Tue, Nov 15, 2016 at 2:56 PM, Greg Kurz <groug@kaod.org> wrote:
>> On Tue, 15 Nov 2016 11:14:57 +0000
>> Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>
>>> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:
>>>> On Tue, 15 Nov 2016 10:53:35 +0100
>>>> Laurent Vivier <lvivier@redhat.com> wrote:
>>>>
>>>>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:
>>>>>> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
>>>>>>
>>>>>>   GTESTER check-qtest-ppc64
>>>>>> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
>>>>>> current = 7c hit_edge = 1
>>>>>> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
>>>>>> current = 1b hit_edge = 1
>>>>>> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
>>>>>> current = 1a hit_edge = 1
>>>>>> **
>>>>>> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
>>>>>> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
>>>>>>
>>>>>
>>>>> Are you using KVM PR?
>>>>>
>>>>> it  was working fine with TCG and KVM HV.
>>>>>
>>>>> Apparently, USERFAULTFD doesn't work with KVM PR.
>>>>>
>>>>> I've already seen this kind of error with nested KVM on Power:
>>>>> guest in guest with KVM PR in host.
>>>>>
>>>>> This problem was reported on IRC by Greg if I remember correctly (CC:)
>>>>>
>>>>
>>>> Yeah I hit this when running make check in a PPC64 BE guest which
>>>> has kvm_pr loaded. I did not find time to investigate though... I've
>>>> switched to run make check on bare metal POWER7 instead.
>>>
>>> Right, it's POWER7 PPC64 BE with kvm_pr.
>>>
>>
>> And you cannot install a PPC64 BE distro on this POWER7 host and use
>> kvm_hv instead ?
> 
> Sure, I could get a non-kvm_pr environment.  I'm just concerned that
> QEMU 2.8-rc0 is being tagged today and there is a POWER environment
> where "make check" fails.
> 
> Is kvm_pr supported by QEMU?

Basically yes, it's just like Greg said in another mail, it does not get
much attention these days.

> If it is supported then "make check" ought to pass.

OK, since nobody got a really, really good idea how to fix this so far:
What about changing the QEMU test to use only tcg for now, so we've got
a clean release with QEMU v2.8? And then afterwards, we can either fix
kvm-pr in the kernel, or introduce some other mechanism to select
whether the test should be run with KVM or not (either a detection of
the KVM type, or an environment variable or something similar).

 Thomas

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Dr. David Alan Gilbert]

* Thomas Huth (thuth@redhat.com) wrote:
> On 15.11.2016 16:18, Stefan Hajnoczi wrote:
> > On Tue, Nov 15, 2016 at 2:56 PM, Greg Kurz <groug@kaod.org> wrote:
> >> On Tue, 15 Nov 2016 11:14:57 +0000
> >> Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >>
> >>> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:
> >>>> On Tue, 15 Nov 2016 10:53:35 +0100
> >>>> Laurent Vivier <lvivier@redhat.com> wrote:
> >>>>
> >>>>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:
> >>>>>> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
> >>>>>>
> >>>>>> Stefan
> >>>>>>
> >>>>>> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
> >>>>>>
> >>>>>>   GTESTER check-qtest-ppc64
> >>>>>> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
> >>>>>> current = 7c hit_edge = 1
> >>>>>> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
> >>>>>> current = 1b hit_edge = 1
> >>>>>> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
> >>>>>> current = 1a hit_edge = 1
> >>>>>> **
> >>>>>> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
> >>>>>> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
> >>>>>>
> >>>>>
> >>>>> Are you using KVM PR?
> >>>>>
> >>>>> it  was working fine with TCG and KVM HV.
> >>>>>
> >>>>> Apparently, USERFAULTFD doesn't work with KVM PR.
> >>>>>
> >>>>> I've already seen this kind of error with nested KVM on Power:
> >>>>> guest in guest with KVM PR in host.
> >>>>>
> >>>>> This problem was reported on IRC by Greg if I remember correctly (CC:)
> >>>>>
> >>>>
> >>>> Yeah I hit this when running make check in a PPC64 BE guest which
> >>>> has kvm_pr loaded. I did not find time to investigate though... I've
> >>>> switched to run make check on bare metal POWER7 instead.
> >>>
> >>> Right, it's POWER7 PPC64 BE with kvm_pr.
> >>>
> >>
> >> And you cannot install a PPC64 BE distro on this POWER7 host and use
> >> kvm_hv instead ?
> > 
> > Sure, I could get a non-kvm_pr environment.  I'm just concerned that
> > QEMU 2.8-rc0 is being tagged today and there is a POWER environment
> > where "make check" fails.
> > 
> > Is kvm_pr supported by QEMU?
> 
> Basically yes, it's just like Greg said in another mail, it does not get
> much attention these days.
> 
> > If it is supported then "make check" ought to pass.
> 
> OK, since nobody got a really, really good idea how to fix this so far:
> What about changing the QEMU test to use only tcg for now, so we've got
> a clean release with QEMU v2.8? And then afterwards, we can either fix
> kvm-pr in the kernel, or introduce some other mechanism to select
> whether the test should be run with KVM or not (either a detection of
> the KVM type, or an environment variable or something similar).

I'm OK if you want to do that for Power; I'd prefer it kept preferring KVM on
x86.

Dave

>  Thomas
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Greg Kurz]

On Tue, 15 Nov 2016 17:43:39 +0000
"Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:

> * Thomas Huth (thuth@redhat.com) wrote:
> > On 15.11.2016 16:18, Stefan Hajnoczi wrote:  
> > > On Tue, Nov 15, 2016 at 2:56 PM, Greg Kurz <groug@kaod.org> wrote:  
> > >> On Tue, 15 Nov 2016 11:14:57 +0000
> > >> Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > >>  
> > >>> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:  
> > >>>> On Tue, 15 Nov 2016 10:53:35 +0100
> > >>>> Laurent Vivier <lvivier@redhat.com> wrote:
> > >>>>  
> > >>>>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:  
> > >>>>>> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
> > >>>>>>
> > >>>>>> Stefan
> > >>>>>>
> > >>>>>> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
> > >>>>>>
> > >>>>>>   GTESTER check-qtest-ppc64
> > >>>>>> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
> > >>>>>> current = 7c hit_edge = 1
> > >>>>>> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
> > >>>>>> current = 1b hit_edge = 1
> > >>>>>> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
> > >>>>>> current = 1a hit_edge = 1
> > >>>>>> **
> > >>>>>> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
> > >>>>>> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
> > >>>>>>  
> > >>>>>
> > >>>>> Are you using KVM PR?
> > >>>>>
> > >>>>> it  was working fine with TCG and KVM HV.
> > >>>>>
> > >>>>> Apparently, USERFAULTFD doesn't work with KVM PR.
> > >>>>>
> > >>>>> I've already seen this kind of error with nested KVM on Power:
> > >>>>> guest in guest with KVM PR in host.
> > >>>>>
> > >>>>> This problem was reported on IRC by Greg if I remember correctly (CC:)
> > >>>>>  
> > >>>>
> > >>>> Yeah I hit this when running make check in a PPC64 BE guest which
> > >>>> has kvm_pr loaded. I did not find time to investigate though... I've
> > >>>> switched to run make check on bare metal POWER7 instead.  
> > >>>
> > >>> Right, it's POWER7 PPC64 BE with kvm_pr.
> > >>>  
> > >>
> > >> And you cannot install a PPC64 BE distro on this POWER7 host and use
> > >> kvm_hv instead ?  
> > > 
> > > Sure, I could get a non-kvm_pr environment.  I'm just concerned that
> > > QEMU 2.8-rc0 is being tagged today and there is a POWER environment
> > > where "make check" fails.
> > > 
> > > Is kvm_pr supported by QEMU?  
> > 
> > Basically yes, it's just like Greg said in another mail, it does not get
> > much attention these days.
> >   
> > > If it is supported then "make check" ought to pass.  
> > 
> > OK, since nobody got a really, really good idea how to fix this so far:
> > What about changing the QEMU test to use only tcg for now, so we've got
> > a clean release with QEMU v2.8? And then afterwards, we can either fix
> > kvm-pr in the kernel, or introduce some other mechanism to select
> > whether the test should be run with KVM or not (either a detection of
> > the KVM type, or an environment variable or something similar).  
> 
> I'm OK if you want to do that for Power; I'd prefer it kept preferring KVM on
> x86.
> 

Even for Power, I'd prefer to keep KVM since the problem only happens with
KVM PR which isn't the preferred way to do KVM on bare metal... until this
get fixed, I'd rather suggest people to run make check with KVM HV.

Cheers.

--
Greg

> Dave
> 
> >  Thomas
> >   
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Thomas Huth]

On 15.11.2016 19:13, Greg Kurz wrote:
> On Tue, 15 Nov 2016 17:43:39 +0000
> "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:
> 
>> * Thomas Huth (thuth@redhat.com) wrote:
>>> On 15.11.2016 16:18, Stefan Hajnoczi wrote:  
>>>> On Tue, Nov 15, 2016 at 2:56 PM, Greg Kurz <groug@kaod.org> wrote:  
>>>>> On Tue, 15 Nov 2016 11:14:57 +0000
>>>>> Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>>>>  
>>>>>> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:  
>>>>>>> On Tue, 15 Nov 2016 10:53:35 +0100
>>>>>>> Laurent Vivier <lvivier@redhat.com> wrote:
>>>>>>>  
>>>>>>>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:  
>>>>>>>>> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
>>>>>>>>>
>>>>>>>>> Stefan
>>>>>>>>>
>>>>>>>>> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
>>>>>>>>>
>>>>>>>>>   GTESTER check-qtest-ppc64
>>>>>>>>> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
>>>>>>>>> current = 7c hit_edge = 1
>>>>>>>>> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
>>>>>>>>> current = 1b hit_edge = 1
>>>>>>>>> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
>>>>>>>>> current = 1a hit_edge = 1
>>>>>>>>> **
>>>>>>>>> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
>>>>>>>>> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
>>>>>>>>>  
>>>>>>>>
>>>>>>>> Are you using KVM PR?
>>>>>>>>
>>>>>>>> it  was working fine with TCG and KVM HV.
>>>>>>>>
>>>>>>>> Apparently, USERFAULTFD doesn't work with KVM PR.
>>>>>>>>
>>>>>>>> I've already seen this kind of error with nested KVM on Power:
>>>>>>>> guest in guest with KVM PR in host.
>>>>>>>>
>>>>>>>> This problem was reported on IRC by Greg if I remember correctly (CC:)
>>>>>>>>  
>>>>>>>
>>>>>>> Yeah I hit this when running make check in a PPC64 BE guest which
>>>>>>> has kvm_pr loaded. I did not find time to investigate though... I've
>>>>>>> switched to run make check on bare metal POWER7 instead.  
>>>>>>
>>>>>> Right, it's POWER7 PPC64 BE with kvm_pr.
>>>>>>  
>>>>>
>>>>> And you cannot install a PPC64 BE distro on this POWER7 host and use
>>>>> kvm_hv instead ?  
>>>>
>>>> Sure, I could get a non-kvm_pr environment.  I'm just concerned that
>>>> QEMU 2.8-rc0 is being tagged today and there is a POWER environment
>>>> where "make check" fails.
>>>>
>>>> Is kvm_pr supported by QEMU?  
>>>
>>> Basically yes, it's just like Greg said in another mail, it does not get
>>> much attention these days.
>>>   
>>>> If it is supported then "make check" ought to pass.  
>>>
>>> OK, since nobody got a really, really good idea how to fix this so far:
>>> What about changing the QEMU test to use only tcg for now, so we've got
>>> a clean release with QEMU v2.8? And then afterwards, we can either fix
>>> kvm-pr in the kernel, or introduce some other mechanism to select
>>> whether the test should be run with KVM or not (either a detection of
>>> the KVM type, or an environment variable or something similar).  
>>
>> I'm OK if you want to do that for Power; I'd prefer it kept preferring KVM on
>> x86.
> 
> Even for Power, I'd prefer to keep KVM since the problem only happens with
> KVM PR which isn't the preferred way to do KVM on bare metal... until this
> get fixed, I'd rather suggest people to run make check with KVM HV.

OK ... what do you think about a patch like this:

diff --git a/tests/postcopy-test.c b/tests/postcopy-test.c
--- a/tests/postcopy-test.c
+++ b/tests/postcopy-test.c
@@ -380,17 +380,19 @@ static void test_migrate(void)
                                   " -incoming %s",
                                   tmpfs, bootpath, uri);
     } else if (strcmp(arch, "ppc64") == 0) {
+        const char *accel;
         init_bootfile_ppc(bootpath);
-        cmd_src = g_strdup_printf("-machine accel=kvm:tcg -m 256M"
+        accel = system("/sbin/lsmod | grep -q kvm_hv") ? "tcg" : "kvm:tcg";
+        cmd_src = g_strdup_printf("-machine accel=%s -m 256M"
                                   " -name pcsource,debug-threads=on"
                                   " -serial file:%s/src_serial"
                                   " -drive file=%s,if=pflash,format=raw",
-                                  tmpfs, bootpath);
-        cmd_dst = g_strdup_printf("-machine accel=kvm:tcg -m 256M"
+                                  accel, tmpfs, bootpath);
+        cmd_dst = g_strdup_printf("-machine accel=%s -m 256M"
                                   " -name pcdest,debug-threads=on"
                                   " -serial file:%s/dest_serial"
                                   " -incoming %s",
-                                  tmpfs, uri);
+                                  accel, tmpfs, uri);
     } else {
         g_assert_not_reached();
     }

That way, accel=kvm:tcg is only used if the kvm_hv module is loaded,
otherwise it will use accel=tcg instead.

 Thomas

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1335 bytes --]

On 11/15/2016 12:48 PM, Thomas Huth wrote:

>> Even for Power, I'd prefer to keep KVM since the problem only happens with
>> KVM PR which isn't the preferred way to do KVM on bare metal... until this
>> get fixed, I'd rather suggest people to run make check with KVM HV.
> 
> OK ... what do you think about a patch like this:
> 
> diff --git a/tests/postcopy-test.c b/tests/postcopy-test.c
> --- a/tests/postcopy-test.c
> +++ b/tests/postcopy-test.c
> @@ -380,17 +380,19 @@ static void test_migrate(void)
>                                    " -incoming %s",
>                                    tmpfs, bootpath, uri);
>      } else if (strcmp(arch, "ppc64") == 0) {
> +        const char *accel;
>          init_bootfile_ppc(bootpath);
> -        cmd_src = g_strdup_printf("-machine accel=kvm:tcg -m 256M"
> +        accel = system("/sbin/lsmod | grep -q kvm_hv") ? "tcg" : "kvm:tcg";

Unsafe use of system() (all I have to do is stick a counterfeit 'grep'
earlier on my PATH to mess you up). Is there a safer way to grab that
information without having to call out to the shell?

> That way, accel=kvm:tcg is only used if the kvm_hv module is loaded,
> otherwise it will use accel=tcg instead.


-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Laurent Vivier]

On 15/11/2016 20:00, Eric Blake wrote:
> On 11/15/2016 12:48 PM, Thomas Huth wrote:
> 
>>> Even for Power, I'd prefer to keep KVM since the problem only happens with
>>> KVM PR which isn't the preferred way to do KVM on bare metal... until this
>>> get fixed, I'd rather suggest people to run make check with KVM HV.
>>
>> OK ... what do you think about a patch like this:
>>
>> diff --git a/tests/postcopy-test.c b/tests/postcopy-test.c
>> --- a/tests/postcopy-test.c
>> +++ b/tests/postcopy-test.c
>> @@ -380,17 +380,19 @@ static void test_migrate(void)
>>                                    " -incoming %s",
>>                                    tmpfs, bootpath, uri);
>>      } else if (strcmp(arch, "ppc64") == 0) {
>> +        const char *accel;
>>          init_bootfile_ppc(bootpath);
>> -        cmd_src = g_strdup_printf("-machine accel=kvm:tcg -m 256M"
>> +        accel = system("/sbin/lsmod | grep -q kvm_hv") ? "tcg" : "kvm:tcg";
> 
> Unsafe use of system() (all I have to do is stick a counterfeit 'grep'
> earlier on my PATH to mess you up). Is there a safer way to grab that
> information without having to call out to the shell?

I think trying to open "/dev/kvm" would be enough to know if kvm is
available or not.

Laurent

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Laurent Vivier]

On 15/11/2016 21:39, Laurent Vivier wrote:
> 
> 
> On 15/11/2016 20:00, Eric Blake wrote:
>> On 11/15/2016 12:48 PM, Thomas Huth wrote:
>>
>>>> Even for Power, I'd prefer to keep KVM since the problem only happens with
>>>> KVM PR which isn't the preferred way to do KVM on bare metal... until this
>>>> get fixed, I'd rather suggest people to run make check with KVM HV.
>>>
>>> OK ... what do you think about a patch like this:
>>>
>>> diff --git a/tests/postcopy-test.c b/tests/postcopy-test.c
>>> --- a/tests/postcopy-test.c
>>> +++ b/tests/postcopy-test.c
>>> @@ -380,17 +380,19 @@ static void test_migrate(void)
>>>                                    " -incoming %s",
>>>                                    tmpfs, bootpath, uri);
>>>      } else if (strcmp(arch, "ppc64") == 0) {
>>> +        const char *accel;
>>>          init_bootfile_ppc(bootpath);
>>> -        cmd_src = g_strdup_printf("-machine accel=kvm:tcg -m 256M"
>>> +        accel = system("/sbin/lsmod | grep -q kvm_hv") ? "tcg" : "kvm:tcg";
>>
>> Unsafe use of system() (all I have to do is stick a counterfeit 'grep'
>> earlier on my PATH to mess you up). Is there a safer way to grab that
>> information without having to call out to the shell?
> 
> I think trying to open "/dev/kvm" would be enough to know if kvm is
> available or not.

OK, I've missed you want to check KVM HV only...

Laurent

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Laurent Vivier]

On 15/11/2016 21:44, Laurent Vivier wrote:
> 
> 
> On 15/11/2016 21:39, Laurent Vivier wrote:
>>
>>
>> On 15/11/2016 20:00, Eric Blake wrote:
>>> On 11/15/2016 12:48 PM, Thomas Huth wrote:
>>>
>>>>> Even for Power, I'd prefer to keep KVM since the problem only happens with
>>>>> KVM PR which isn't the preferred way to do KVM on bare metal... until this
>>>>> get fixed, I'd rather suggest people to run make check with KVM HV.
>>>>
>>>> OK ... what do you think about a patch like this:
>>>>
>>>> diff --git a/tests/postcopy-test.c b/tests/postcopy-test.c
>>>> --- a/tests/postcopy-test.c
>>>> +++ b/tests/postcopy-test.c
>>>> @@ -380,17 +380,19 @@ static void test_migrate(void)
>>>>                                    " -incoming %s",
>>>>                                    tmpfs, bootpath, uri);
>>>>      } else if (strcmp(arch, "ppc64") == 0) {
>>>> +        const char *accel;
>>>>          init_bootfile_ppc(bootpath);
>>>> -        cmd_src = g_strdup_printf("-machine accel=kvm:tcg -m 256M"
>>>> +        accel = system("/sbin/lsmod | grep -q kvm_hv") ? "tcg" : "kvm:tcg";
>>>
>>> Unsafe use of system() (all I have to do is stick a counterfeit 'grep'
>>> earlier on my PATH to mess you up). Is there a safer way to grab that
>>> information without having to call out to the shell?
>>
>> I think trying to open "/dev/kvm" would be enough to know if kvm is
>> available or not.
> 
> OK, I've missed you want to check KVM HV only...

So, I think you can check for "/sys/module/kvm_hv"

Laurent

Re: [Qemu-devel] QEMU postcopy-test failing on ppc64

[Greg Kurz]

On Tue, 15 Nov 2016 19:48:20 +0100
Thomas Huth <thuth@redhat.com> wrote:

> On 15.11.2016 19:13, Greg Kurz wrote:
> > On Tue, 15 Nov 2016 17:43:39 +0000
> > "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:
> >   
> >> * Thomas Huth (thuth@redhat.com) wrote:  
> >>> On 15.11.2016 16:18, Stefan Hajnoczi wrote:    
> >>>> On Tue, Nov 15, 2016 at 2:56 PM, Greg Kurz <groug@kaod.org> wrote:    
> >>>>> On Tue, 15 Nov 2016 11:14:57 +0000
> >>>>> Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >>>>>    
> >>>>>> On Tue, Nov 15, 2016 at 10:09 AM, Greg Kurz <groug@kaod.org> wrote:    
> >>>>>>> On Tue, 15 Nov 2016 10:53:35 +0100
> >>>>>>> Laurent Vivier <lvivier@redhat.com> wrote:
> >>>>>>>    
> >>>>>>>> On 14/11/2016 21:52, Stefan Hajnoczi wrote:    
> >>>>>>>>> I hit a failure running "make check" on ppc64 for the first time.  Ideas?
> >>>>>>>>>
> >>>>>>>>> Stefan
> >>>>>>>>>
> >>>>>>>>> commit 682df581c65ed2c1b9e77093e332214ecaa1ee93
> >>>>>>>>>
> >>>>>>>>>   GTESTER check-qtest-ppc64
> >>>>>>>>> Memory content inconsistency at 5af4000 first_byte = 1b last_byte = 1a
> >>>>>>>>> current = 7c hit_edge = 1
> >>>>>>>>> Memory content inconsistency at 5af5000 first_byte = 1b last_byte = 7c
> >>>>>>>>> current = 1b hit_edge = 1
> >>>>>>>>> Memory content inconsistency at 5e59000 first_byte = 1b last_byte = 1b
> >>>>>>>>> current = 1a hit_edge = 1
> >>>>>>>>> **
> >>>>>>>>> ERROR:tests/postcopy-test.c:345:check_guests_ram: 'bad' should be FALSE
> >>>>>>>>> GTester: last random seed: R02S9d79166a1ca7e21940a0f4b0b1255d5b
> >>>>>>>>>    
> >>>>>>>>
> >>>>>>>> Are you using KVM PR?
> >>>>>>>>
> >>>>>>>> it  was working fine with TCG and KVM HV.
> >>>>>>>>
> >>>>>>>> Apparently, USERFAULTFD doesn't work with KVM PR.
> >>>>>>>>
> >>>>>>>> I've already seen this kind of error with nested KVM on Power:
> >>>>>>>> guest in guest with KVM PR in host.
> >>>>>>>>
> >>>>>>>> This problem was reported on IRC by Greg if I remember correctly (CC:)
> >>>>>>>>    
> >>>>>>>
> >>>>>>> Yeah I hit this when running make check in a PPC64 BE guest which
> >>>>>>> has kvm_pr loaded. I did not find time to investigate though... I've
> >>>>>>> switched to run make check on bare metal POWER7 instead.    
> >>>>>>
> >>>>>> Right, it's POWER7 PPC64 BE with kvm_pr.
> >>>>>>    
> >>>>>
> >>>>> And you cannot install a PPC64 BE distro on this POWER7 host and use
> >>>>> kvm_hv instead ?    
> >>>>
> >>>> Sure, I could get a non-kvm_pr environment.  I'm just concerned that
> >>>> QEMU 2.8-rc0 is being tagged today and there is a POWER environment
> >>>> where "make check" fails.
> >>>>
> >>>> Is kvm_pr supported by QEMU?    
> >>>
> >>> Basically yes, it's just like Greg said in another mail, it does not get
> >>> much attention these days.
> >>>     
> >>>> If it is supported then "make check" ought to pass.    
> >>>
> >>> OK, since nobody got a really, really good idea how to fix this so far:
> >>> What about changing the QEMU test to use only tcg for now, so we've got
> >>> a clean release with QEMU v2.8? And then afterwards, we can either fix
> >>> kvm-pr in the kernel, or introduce some other mechanism to select
> >>> whether the test should be run with KVM or not (either a detection of
> >>> the KVM type, or an environment variable or something similar).    
> >>
> >> I'm OK if you want to do that for Power; I'd prefer it kept preferring KVM on
> >> x86.  
> > 
> > Even for Power, I'd prefer to keep KVM since the problem only happens with
> > KVM PR which isn't the preferred way to do KVM on bare metal... until this
> > get fixed, I'd rather suggest people to run make check with KVM HV.  
> 
> OK ... what do you think about a patch like this:
> 
> diff --git a/tests/postcopy-test.c b/tests/postcopy-test.c
> --- a/tests/postcopy-test.c
> +++ b/tests/postcopy-test.c
> @@ -380,17 +380,19 @@ static void test_migrate(void)
>                                    " -incoming %s",
>                                    tmpfs, bootpath, uri);
>      } else if (strcmp(arch, "ppc64") == 0) {
> +        const char *accel;
>          init_bootfile_ppc(bootpath);
> -        cmd_src = g_strdup_printf("-machine accel=kvm:tcg -m 256M"
> +        accel = system("/sbin/lsmod | grep -q kvm_hv") ? "tcg" : "kvm:tcg";
> +        cmd_src = g_strdup_printf("-machine accel=%s -m 256M"
>                                    " -name pcsource,debug-threads=on"
>                                    " -serial file:%s/src_serial"
>                                    " -drive file=%s,if=pflash,format=raw",
> -                                  tmpfs, bootpath);
> -        cmd_dst = g_strdup_printf("-machine accel=kvm:tcg -m 256M"
> +                                  accel, tmpfs, bootpath);
> +        cmd_dst = g_strdup_printf("-machine accel=%s -m 256M"
>                                    " -name pcdest,debug-threads=on"
>                                    " -serial file:%s/dest_serial"
>                                    " -incoming %s",
> -                                  tmpfs, uri);
> +                                  accel, tmpfs, uri);
>      } else {
>          g_assert_not_reached();
>      }
> 
> That way, accel=kvm:tcg is only used if the kvm_hv module is loaded,
> otherwise it will use accel=tcg instead.
> 

I guess it's acceptable to assume that the test will more likely
pass when using KVM HV... but I'm not very comfortable as QEMU is
still supposed to work with KVM PR and we don't know yet what's
happening.

Cheers.

--
Greg

>  Thomas
>