Xend transaction reduction breaks migration

Xend transaction reduction breaks migration

[Keir Fraser]

Hey Daniel,

It turns out that your changeset 16260 in xen-unstable breaks our localhost
non-live migration tests. What happens is that randomly (but fairly often)
xend will throw an assertion at line 'assert console_port' in
XendCheckpoint.py. This is in the domain_restore_fd context, so result is
that the restorer dies and then of course the saver fails soon after since
the migration socket goes away.

Any idea how your patch could cause this rather bizarre behaviour? I haven't
worked it out myself, though I'm 99% sure it *is* that changeset that has
caused this regression (based on tests with the changeset reverted).

It's currently the only blocker in our automated tests on the staging tree,
and blocks pushing to the main xen-unstable tree. :-)

 Thanks,
 Keir

Re: Xend transaction reduction breaks migration

[Daniel P. Berrange]

On Wed, Nov 14, 2007 at 11:44:42PM +0000, Keir Fraser wrote:
> Hey Daniel,
> 
> It turns out that your changeset 16260 in xen-unstable breaks our localhost
> non-live migration tests. What happens is that randomly (but fairly often)
> xend will throw an assertion at line 'assert console_port' in
> XendCheckpoint.py. This is in the domain_restore_fd context, so result is
> that the restorer dies and then of course the saver fails soon after since
> the migration socket goes away.
> 
> Any idea how your patch could cause this rather bizarre behaviour? I haven't
> worked it out myself, though I'm 99% sure it *is* that changeset that has
> caused this regression (based on tests with the changeset reverted).

Basically, instead of doing many small transactions, my change made some stuff
all happen in one large transaction. Oddly though, I tried to be careful so
that I only changed the codepath for the 'xm list' operation (well the SEXPR
call to the list operation). Other operations like create, save, restore
ought to be left using fine grained transactions as before.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: Xend transaction reduction breaks migration

[Keir Fraser]

On 15/11/07 18:39, "Daniel P. Berrange" <berrange@redhat.com> wrote:

>> Any idea how your patch could cause this rather bizarre behaviour? I haven't
>> worked it out myself, though I'm 99% sure it *is* that changeset that has
>> caused this regression (based on tests with the changeset reverted).
> 
> Basically, instead of doing many small transactions, my change made some stuff
> all happen in one large transaction. Oddly though, I tried to be careful so
> that I only changed the codepath for the 'xm list' operation (well the SEXPR
> call to the list operation). Other operations like create, save, restore
> ought to be left using fine grained transactions as before.

I'll have another look at nearby changesets in xen-unstable then. Yours
doesn't really look all that likely to cause this problem, except that the
empirical evidence does point at it.

 -- Keir

Re: Re: Xend transaction reduction breaks migration

[Keir Fraser]

On 15/11/07 18:46, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:

> On 15/11/07 18:39, "Daniel P. Berrange" <berrange@redhat.com> wrote:
> 
>> all happen in one large transaction. Oddly though, I tried to be careful so
>> that I only changed the codepath for the 'xm list' operation (well the SEXPR
>> call to the list operation). Other operations like create, save, restore
>> ought to be left using fine grained transactions as before.
> 
> I'll have another look at nearby changesets in xen-unstable then. Yours
> doesn't really look all that likely to cause this problem, except that the
> empirical evidence does point at it.

It is your patch. The readDomTxn() functions and friends should prepend the
dompath, not the vmpath.

We wouldn't notice this normally since self.console_port is not actually
really used after domain building. So we have to race the migration thread
to get the assertion to fire, and that happens fairly rarely.

 -- Keir

Re: Re: Xend transaction reduction breaks migration

[Daniel P. Berrange]

On Thu, Nov 15, 2007 at 10:11:29PM +0000, Keir Fraser wrote:
> On 15/11/07 18:46, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
> > On 15/11/07 18:39, "Daniel P. Berrange" <berrange@redhat.com> wrote:
> > 
> >> all happen in one large transaction. Oddly though, I tried to be careful so
> >> that I only changed the codepath for the 'xm list' operation (well the SEXPR
> >> call to the list operation). Other operations like create, save, restore
> >> ought to be left using fine grained transactions as before.
> > 
> > I'll have another look at nearby changesets in xen-unstable then. Yours
> > doesn't really look all that likely to cause this problem, except that the
> > empirical evidence does point at it.
> 
> It is your patch. The readDomTxn() functions and friends should prepend the
> dompath, not the vmpath.

Aha. Thanks for finding & fixing that.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|