Re: [adhoc osstest bisection] complete test-amd64-amd64-xl

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 21/05/15 10:31, Jan Beulich wrote:
>>>> On 21.05.15 at 11:16, <ian.campbell@citrix.com> wrote:
>> The osstest instance in Cambridge appears to have found a machine
>> specific failure which a bisect has blamed on the "x86/domctl: cleanup"
>> patch below.
>>
>> (NB: this is not the arm releated leak check thing which is currently
>> hitting the xen-unstable flights in the main osstest instance)
>>
>> Since the logs are internal I've put a copy of the bisection output at
>> http://xenbits.xen.org/people/ianc/tmp/bisect.osstest.test-amd64-amd64-xl..h 
>> tml
>>
>> Along with the logs of the first two passes from the last good revision:
>> http://xenbits.xen.org/people/ianc/tmp/37537/ 
>> http://xenbits.xen.org/people/ianc/tmp/37541/ 
>> and the logs from the first two fails from the first bad revision:
>> http://xenbits.xen.org/people/ianc/tmp/37540/ 
>> http://xenbits.xen.org/people/ianc/tmp/37542/ 
>>
>> The failure is in migration at e.g.
>>
>> http://xenbits.xen.org/people/ianc/tmp/37540/test-amd64-amd64-xl/17.ts-guest 
>> -localmigrate.log
>>
>> Anyone got any thoughts on this?
> Reviewing the change another time, I see nothing. Unless Andrew
> has an idea, I think we'd need to revert the patch in smaller chunks
> to get an understanding what's going on here. The only thing I
> could remotely imagine would be something writing over the
> boundaries of a stack variable, of which we may now have got a
> few more of.

Ian and I have just spent some time investigating this.

The failure is that libxl observes a PV guest failing to respond to a
xenstore-based request to suspend, although the guest logs themselves
appear to suggest that the guest is performing correctly.

The identified changeset clearly suggests that this is a toolstack side
behavioural change, although the actual toolstack actions using domctls
appear to be working fine.

However, it is intermittent and appears to be host specific, rather than
consistent across the board.  Also, XenServer has not experienced any
symptoms like this.

~Andrew