pvgrub regression in xen.git 123c77937975

pvgrub regression in xen.git 123c77937975

[Ian Campbell]

There seems to be a pvgrub regression somewhere in the range
3a28f760508f..123c77937975 instead of booting the selected kernel I get:
        
        Booting 'Debian GNU/Linux, kernel 3.2.0-4-amd64'
        
        root  (hd0,0)
        
         Filesystem type is ext2fs, partition type 0x83
        
        kernel  /boot/vmlinuz-3.2.0-4-amd64 root=UUID=59fd4a5b-040d-4f59-8fc3-19324ddb4
        
        0b7 ro 
        
        initrd  /boot/initrd.img-3.2.0-4-amd64
        
        ============= Init TPM Front ================
        Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT
        Tpmfront:Info Shutting down tpmfront
        xc_dom_build_image returned -1
        
        close(3)
        
        Error 9: Unknown boot failure
        
        Press any key to continue...
        
This was spotted in ad-hoc testing on the Cambridge osstest instance of
my distro testing series[0] (which adds pvgrub tests to the main flights
too). i386 guests fail in the same way.

I've attempted to convince osstest to have a stab at bisecting, but
glancing at the log over tools/libxc tools/stubdom I suspect either
Konrad's cleanups to libxc error handling or Wei's "libxc: add p2m_size
to xc_dom_image".

Anyway, just a heads up in case something springs to mind, otherwise
I'll let the bisector churn at it over the weekend. There's been a few
false starts but if you are within Citrix you can see the progress (I
hope) at http://osstest.xs.citrite.net/~ianc/summary2.html (so far it's
just trying to reproduce the basis cases, and has been tripping over the
xen.git:5d4c0952f issue, now hacked around I hope)

Ian.

[0] http://mid.gmane.org/<1430905605.2660.187.camel@citrix.com>

Re: pvgrub regression in xen.git 123c77937975

[Konrad Rzeszutek Wilk]

On Fri, May 08, 2015 at 05:16:49PM +0100, Ian Campbell wrote:
> There seems to be a pvgrub regression somewhere in the range
> 3a28f760508f..123c77937975 instead of booting the selected kernel I get:
>         
>         Booting 'Debian GNU/Linux, kernel 3.2.0-4-amd64'
>         
>         root  (hd0,0)
>         
>          Filesystem type is ext2fs, partition type 0x83
>         
>         kernel  /boot/vmlinuz-3.2.0-4-amd64 root=UUID=59fd4a5b-040d-4f59-8fc3-19324ddb4
>         
>         0b7 ro 
>         
>         initrd  /boot/initrd.img-3.2.0-4-amd64
>         
>         ============= Init TPM Front ================
>         Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT
>         Tpmfront:Info Shutting down tpmfront
>         xc_dom_build_image returned -1
>         
>         close(3)
>         
>         Error 9: Unknown boot failure
>         
>         Press any key to continue...
>         
> This was spotted in ad-hoc testing on the Cambridge osstest instance of
> my distro testing series[0] (which adds pvgrub tests to the main flights
> too). i386 guests fail in the same way.
> 
> I've attempted to convince osstest to have a stab at bisecting, but
> glancing at the log over tools/libxc tools/stubdom I suspect either
> Konrad's cleanups to libxc error handling or Wei's "libxc: add p2m_size
> to xc_dom_image".

Hey Ian,

I am soo woefully behind emails - sorry for late reply.

I don't have anything on my mind which could be the fault of this - but
I am curious of the bisection results.
> 
> Anyway, just a heads up in case something springs to mind, otherwise
> I'll let the bisector churn at it over the weekend. There's been a few
> false starts but if you are within Citrix you can see the progress (I
> hope) at http://osstest.xs.citrite.net/~ianc/summary2.html (so far it's
> just trying to reproduce the basis cases, and has been tripping over the
> xen.git:5d4c0952f issue, now hacked around I hope)
> 
> Ian.
> 
> [0] http://mid.gmane.org/<1430905605.2660.187.camel@citrix.com>
> 

Re: pvgrub regression in xen.git 123c77937975

[Ian Campbell]

On Fri, 2015-05-08 at 17:16 +0100, Ian Campbell wrote:
> There seems to be a pvgrub regression somewhere in the range
> 3a28f760508f..123c77937975

The bisector has fingered:

        commit 840837907c6186307c19abbec926852ba448facd
          Author: Wei Liu <wei.liu2@citrix.com>
          Date:   Mon Mar 16 09:52:22 2015 +0000
          
              libxc: add p2m_size to xc_dom_image
        
Full report below. I've put a copy of the graph at:
http://xenbits.xen.org/people/ianc/tmp/201505/bisect.pvgrub.html

Wei, you can see an instance of the failure at
http://osstest.xs.citrite.net/~osstest/testlogs/logs/37344/

Ian.

branch xen-unstable
xen branch xen-unstable
job test-amd64-amd64-amd64-pvgrub
test guest-start

Tree: linux git://xenbits.xen.org/linux-pvops.git
Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-unstable.git
Tree: xen git://xenbits.xen.org/xen.git

*** Found and reproduced problem changeset ***

  Bug is in tree:  xen git://xenbits.xen.org/xen.git
  Bug introduced:  840837907c6186307c19abbec926852ba448facd
  Bug not present: 61e8e8079a4e984600713b492944d6427b9f77cf


  commit 840837907c6186307c19abbec926852ba448facd
  Author: Wei Liu <wei.liu2@citrix.com>
  Date:   Mon Mar 16 09:52:22 2015 +0000
  
      libxc: add p2m_size to xc_dom_image
      
      Add a new field p2m_size to keep track of the number of pages covered by
      p2m.  Change total_pages to p2m_size in functions which in fact need
      the size of p2m.
      
      This is needed because we are going to ditch the assumption that PV x86
      has only one contiguous ram region. Originally the p2m size was always
      equal to total_pages, but we will soon change that in later patch.
      
      This patch doesn't change the behaviour of libxc.
      
      Signed-off-by: Wei Liu <wei.liu2@citrix.com>
      Reviewed-by: Dario Faggioli <dario.faggioli@citrix.com>
      Cc: Ian Campbell <ian.campbell@citrix.com>
      Cc: Ian Jackson <ian.jackson@eu.citrix.com>
      Acked-by: Ian Campbell <ian.campbell@citrix.com>


For bisection revision-tuple graph see:
   http://osstest.xs.citrite.net/~osstest/testlogs/results-adhoc/bisect.xen-unstable.test-amd64-amd64-amd64-pvgrub..html
Revision IDs in each graph node refer, respectively, to the Trees above.

----------------------------------------
Searching for failure / basis pass:
 37344 fail [host=moss-bug] / 37345 ok.
Failure / basis pass flights: 37344 / 37345
(tree with no url: ovmf)
(tree with no url: seabios)
Tree: linux git://xenbits.xen.org/linux-pvops.git
Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-unstable.git
Tree: xen git://xenbits.xen.org/xen.git
Latest 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3b45fcf0c163b9cff4d8115f7b75b42918a9b1b5 727b998448e852a5e8eb570ac3a259ef62fbdacb 123c7793797502b222300eb710cd3873dcca41ee
Basis pass 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 3a28f760508fb35c430edac17a9efde5aff6d1d5
Generating revisions with ./adhoc-revtuple-generator  git://xenbits.xen.org/linux-pvops.git#8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9-8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 git://xenbits.xen.org/osstest/linux-firmware.git#c530a75c1e6a472b0eb9558310b518f0dfcd8860-c530a75c1e6a472b0eb9558310b518f0dfcd8860 git://xenbits.xen.org/staging/qemu-xen-unstable.git#a4b276b4ce49c8d70dd841ff885b900ec652b994-3b45fcf0c163b9cff4d8115f7b75b42918a9b1b5 git://xenbits.xen.org/staging/qemu-upstream-unstable.git#727b998448e852a5e8eb570ac3a259ef62fbdacb-727b998448e852a5e8eb570ac3a259ef62fbdacb git://xenbits.xen.org/xen.git#3a28f760508fb35c430edac17a9efde5aff6d1d5-123c7793797502b222300eb710cd3873dcca41ee
+ exec
+ sh -xe
+ cd /export/home/ianc/repos/qemu-xen-unstable
+ git remote set-url origin git://drall.uk.xensource.com:9419/git://xenbits.xen.org/staging/qemu-xen-unstable.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /export/home/ianc/repos/xen
+ git remote set-url origin git://drall.uk.xensource.com:9419/git://xenbits.xen.org/xen.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /export/home/ianc/repos/qemu-xen-unstable
+ git remote set-url origin git://drall.uk.xensource.com:9419/git://xenbits.xen.org/staging/qemu-xen-unstable.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /export/home/ianc/repos/xen
+ git remote set-url origin git://drall.uk.xensource.com:9419/git://xenbits.xen.org/xen.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
Loaded 2003 nodes in revision graph
Searching for test results:
 37347 fail 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3b45fcf0c163b9cff4d8115f7b75b42918a9b1b5 727b998448e852a5e8eb570ac3a259ef62fbdacb 123c7793797502b222300eb710cd3873dcca41ee
 37363 pass 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 61e8e8079a4e984600713b492944d6427b9f77cf
 37349 fail 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 ab42b4408cb4fc4f869d73218e3d2034e6f5e8ac 727b998448e852a5e8eb570ac3a259ef62fbdacb eb141d23a00a349dc1474bb982e7812fc091f845
 37350 fail 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 916735e1814d4d6df0d90b871b0666cdfaf134cd
 37364 fail 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 840837907c6186307c19abbec926852ba448facd
 37351 fail 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 51f2224552cc1e6cb22b8b169999cd9711bc8cbc
 37331 [host=lace-bug]
 37353 fail 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 24732f8605cb744783c2220089b2d5bef74a03ae
 37354 fail 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 50619a4734185da6740650c5affbfaf5785b792c
 37342 pass 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 3a28f760508fb35c430edac17a9efde5aff6d1d5
 37365 pass 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 61e8e8079a4e984600713b492944d6427b9f77cf
 37366 pass 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 61e8e8079a4e984600713b492944d6427b9f77cf
 37355 fail 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 840837907c6186307c19abbec926852ba448facd
 37344 fail 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3b45fcf0c163b9cff4d8115f7b75b42918a9b1b5 727b998448e852a5e8eb570ac3a259ef62fbdacb 123c7793797502b222300eb710cd3873dcca41ee
 37345 pass 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 3a28f760508fb35c430edac17a9efde5aff6d1d5
 37367 fail 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 840837907c6186307c19abbec926852ba448facd
 37357 pass 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb c9f8db9b1653400d0d11f56bed750e210e8f2360
 37358 pass 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 7e9ec50b0535bf2630da9d279a060775817d136d
 37359 pass 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 28b3ca49b0f1fbe945342973fbdc8c5e56481213
 37360 pass 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 61e8e8079a4e984600713b492944d6427b9f77cf
 37362 fail 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 840837907c6186307c19abbec926852ba448facd
Searching for interesting versions
 Result found: flight 37342 (pass), for basis pass
 Result found: flight 37344 (fail), for basis failure
 Repro found: flight 37345 (pass), for basis pass
 Repro found: flight 37347 (fail), for basis failure
 0 revisions at 8a5f782c33c04ea5c9b3ca6fb32d6039e2e5c0c9 c530a75c1e6a472b0eb9558310b518f0dfcd8860 a4b276b4ce49c8d70dd841ff885b900ec652b994 727b998448e852a5e8eb570ac3a259ef62fbdacb 61e8e8079a4e984600713b492944d6427b9f77cf
No revisions left to test, checking graph state.
 Result found: flight 37360 (pass), for last pass
 Result found: flight 37362 (fail), for first failure
 Repro found: flight 37363 (pass), for last pass
 Repro found: flight 37364 (fail), for first failure
 Repro found: flight 37365 (pass), for last pass
 Repro found: flight 37367 (fail), for first failure

*** Found and reproduced problem changeset ***

  Bug is in tree:  xen git://xenbits.xen.org/xen.git
  Bug introduced:  840837907c6186307c19abbec926852ba448facd
  Bug not present: 61e8e8079a4e984600713b492944d6427b9f77cf

+ exec
+ sh -xe
+ cd /export/home/ianc/repos/xen
+ git remote set-url origin git://drall.uk.xensource.com:9419/git://xenbits.xen.org/xen.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*

  commit 840837907c6186307c19abbec926852ba448facd
  Author: Wei Liu <wei.liu2@citrix.com>
  Date:   Mon Mar 16 09:52:22 2015 +0000
  
      libxc: add p2m_size to xc_dom_image
      
      Add a new field p2m_size to keep track of the number of pages covered by
      p2m.  Change total_pages to p2m_size in functions which in fact need
      the size of p2m.
      
      This is needed because we are going to ditch the assumption that PV x86
      has only one contiguous ram region. Originally the p2m size was always
      equal to total_pages, but we will soon change that in later patch.
      
      This patch doesn't change the behaviour of libxc.
      
      Signed-off-by: Wei Liu <wei.liu2@citrix.com>
      Reviewed-by: Dario Faggioli <dario.faggioli@citrix.com>
      Cc: Ian Campbell <ian.campbell@citrix.com>
      Cc: Ian Jackson <ian.jackson@eu.citrix.com>
      Acked-by: Ian Campbell <ian.campbell@citrix.com>

Revision graph left in /home/ianc/public_html/bisect.pvgrub.{dot,ps,png,html}.
----------------------------------------
37367: tolerable ALL FAIL

flight 37367 xen-unstable adhoc-bisect [real]
http://osstest.xs.citrite.net/~osstest/testlogs/logs/37367/

Failures :-/ but no regressions.

Tests which did not succeed,
including tests which could not be run:
 test-amd64-amd64-amd64-pvgrub 10 guest-start                  fail  never pass


jobs:
 test-amd64-amd64-amd64-pvgrub                                fail    


------------------------------------------------------------
sg-report-flight on osstest.xs.citrite.net
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images

Logs, config files, etc. are available at
    http://osstest.xs.citrite.net/~osstest/testlogs/logs

Test harness code can be found at
    http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary

Re: pvgrub regression in xen.git 123c77937975

[Wei Liu]

On Sun, May 10, 2015 at 07:19:09AM +0100, Ian Campbell wrote:
> On Fri, 2015-05-08 at 17:16 +0100, Ian Campbell wrote:
> > There seems to be a pvgrub regression somewhere in the range
> > 3a28f760508f..123c77937975
> 
> The bisector has fingered:
> 
>         commit 840837907c6186307c19abbec926852ba448facd
>           Author: Wei Liu <wei.liu2@citrix.com>
>           Date:   Mon Mar 16 09:52:22 2015 +0000
>           
>               libxc: add p2m_size to xc_dom_image
>         
> Full report below. I've put a copy of the graph at:
> http://xenbits.xen.org/people/ianc/tmp/201505/bisect.pvgrub.html
> 
> Wei, you can see an instance of the failure at
> http://osstest.xs.citrite.net/~osstest/testlogs/logs/37344/

Unfortunately Citrix access gateway doesn't work for me at the moment so
I couldn't see the log.

But I've looked at the code. PV grub's kexec function doesn't update the
newly added field. I will try to reproduce on my local test box.

Wei.

Re: pvgrub regression in xen.git 123c77937975

[Ian Campbell]

On Sun, 2015-05-10 at 13:20 +0100, Wei Liu wrote:
> On Sun, May 10, 2015 at 07:19:09AM +0100, Ian Campbell wrote:
> > On Fri, 2015-05-08 at 17:16 +0100, Ian Campbell wrote:
> > > There seems to be a pvgrub regression somewhere in the range
> > > 3a28f760508f..123c77937975
> > 
> > The bisector has fingered:
> > 
> >         commit 840837907c6186307c19abbec926852ba448facd
> >           Author: Wei Liu <wei.liu2@citrix.com>
> >           Date:   Mon Mar 16 09:52:22 2015 +0000
> >           
> >               libxc: add p2m_size to xc_dom_image
> >         
> > Full report below. I've put a copy of the graph at:
> > http://xenbits.xen.org/people/ianc/tmp/201505/bisect.pvgrub.html
> > 
> > Wei, you can see an instance of the failure at
> > http://osstest.xs.citrite.net/~osstest/testlogs/logs/37344/
> 
> Unfortunately Citrix access gateway doesn't work for me at the moment so
> I couldn't see the log.

FWIW http://xenbits.xen.org/people/ianc/tmp/201505/37344/ (no build-*
logs, just test-*), but it looks like you don't actually need them.

> But I've looked at the code. PV grub's kexec function doesn't update the
> newly added field. I will try to reproduce on my local test box.

I've seen your patch, will take a proper look tomorrow.

Thanks for the quick turnaround!

Ian.