xm-test domain creation delay

xm-test domain creation delay

[Ewan Mellor]

Here's an idea for xm-test.  Instead of waiting a full 20 seconds for the
guest domain to come up, we could instead write to the store from inside the
guest.

The easiest way to do this would be with the command line tools xenstore-read
and xenstore-write.  If you use these tools without the -s option, this should
mean that they write using the domain's implicit root, so if you don't use a
path with a / at the front, then the path will be unique per-domain.

All we would have to do is arrange so that the last thing that the busybox
guest did when it has booted up is to write a node xm-test/booted=1 or
something, and then the xm-test application could register a watch for that
path (i.e. /local/domain/<domid>/xm-test/booted), and would then be able to
detect when the domain was ready.  This should be a lot quicker than waiting
for a full 20 seconds each time, so the test should run a lot quicker in
total.

Thoughts anyone?

Ewan.

Re: xm-test domain creation delay

[Dan Smith]

EM> Here's an idea for xm-test.  Instead of waiting a full 20 seconds
EM> for the guest domain to come up, we could instead write to the
EM> store from inside the guest.

Yes, we definitely need to eliminate the 20-second delay for each
DomU.  The reason we do this is that when xm-test was being developed,
the block driver conversion to xenbus was happening, and the DomUs
were taking a long time to boot, especially if no block devices were
attached.  I was planning to lower this timeout, but it would
definitely be better if we had some sort of notification.

EM> The easiest way to do this would be with the command line tools
EM> xenstore-read and xenstore-write.  If you use these tools without
EM> the -s option, this should mean that they write using the domain's
EM> implicit root, so if you don't use a path with a / at the front,
EM> then the path will be unique per-domain.

This is not quite as easy as it sounds.  In order to simply copy the
user's xenstore-write binary into the ramdisk, we would need to also
copy several of their libraries like libc and libxenctrl.  We could
recompile a static version of xenstore-write to go into the ramdisk,
which may work.

EM> xm-test/booted=1 or something, and then the xm-test application
EM> could register a watch for that path

So, I wonder if this kind of functionality would be useful outside the
realm of xm-test?  Can anyone think of a reason why we might want to
have this be built-in to the DomU kernel or something?

If not, I think the best (and easiest) way to solve the immediate
problem in xm-test is to use the console as a way to poll the status
of the DomU.  We can eliminate the explicit 20 second wait time, and
instead have the console library make multiple attempts to attach,
with a timeout of it fails for too long.  This would give us a much
quicker DomU boot time, without requiring anything else to go into the
initial ramdisk.  I'm working on a patch for this right now.

Thoughts?

-- 
Dan Smith
IBM Linux Technology Center
Open Hypervisor Team
email: danms@us.ibm.com

Re: xm-test domain creation delay

[Ewan Mellor]

On Mon, Nov 14, 2005 at 08:00:06AM -0800, Dan Smith wrote:

> EM> The easiest way to do this would be with the command line tools
> EM> xenstore-read and xenstore-write.  If you use these tools without
> EM> the -s option, this should mean that they write using the domain's
> EM> implicit root, so if you don't use a path with a / at the front,
> EM> then the path will be unique per-domain.
> 
> This is not quite as easy as it sounds.  In order to simply copy the
> user's xenstore-write binary into the ramdisk, we would need to also
> copy several of their libraries like libc and libxenctrl.  We could
> recompile a static version of xenstore-write to go into the ramdisk,
> which may work.

Well it would be better if we could compile and link against the version of
libc in the ramdisk!  That would be a different version of xenstore-write to
the one that we compile for the user of course, which might be a pain to do,
but that's the right way to do it.  How do all the other applications in the
ramdisk get compiled now?

> EM> xm-test/booted=1 or something, and then the xm-test application
> EM> could register a watch for that path
> 
> So, I wonder if this kind of functionality would be useful outside the
> realm of xm-test?  Can anyone think of a reason why we might want to
> have this be built-in to the DomU kernel or something?
> 
> If not, I think the best (and easiest) way to solve the immediate
> problem in xm-test is to use the console as a way to poll the status
> of the DomU.  We can eliminate the explicit 20 second wait time, and
> instead have the console library make multiple attempts to attach,
> with a timeout of it fails for too long.  This would give us a much
> quicker DomU boot time, without requiring anything else to go into the
> initial ramdisk.  I'm working on a patch for this right now.

Is it sufficient to say that once you can connect the console, then the guest
is ready for commands?  That wouldn't be the case for a "real" guest, of
course, but if that's OK for xm-test (or close enough that we only need a 1
second timeout or something) then that sounds like a better idea.

Ewan.

Re: xm-test domain creation delay

[Sean Dague]

[-- Attachment #1.1: Type: text/plain, Size: 2039 bytes --]

On Mon, Nov 14, 2005 at 04:10:25PM +0000, Ewan Mellor wrote:
> On Mon, Nov 14, 2005 at 08:00:06AM -0800, Dan Smith wrote:
> 
> > EM> The easiest way to do this would be with the command line tools
> > EM> xenstore-read and xenstore-write.  If you use these tools without
> > EM> the -s option, this should mean that they write using the domain's
> > EM> implicit root, so if you don't use a path with a / at the front,
> > EM> then the path will be unique per-domain.
> > 
> > This is not quite as easy as it sounds.  In order to simply copy the
> > user's xenstore-write binary into the ramdisk, we would need to also
> > copy several of their libraries like libc and libxenctrl.  We could
> > recompile a static version of xenstore-write to go into the ramdisk,
> > which may work.
> 
> Well it would be better if we could compile and link against the version of
> libc in the ramdisk!  That would be a different version of xenstore-write to
> the one that we compile for the user of course, which might be a pain to do,
> but that's the right way to do it.  How do all the other applications in the
> ramdisk get compiled now?

I agree, building the xentools in build root would be ideal here.  I'm not
sure how hard it is to integrate it though.

I think we've got a more general issue of how to usefully get the xenstore
tools into a DomU which may have different build env than Dom0.  It would be
nice if there was a target to build *only* the xenstore tools expected for
DomU, that might help in this regard.

	-Sean

-- 
__________________________________________________________________

Sean Dague                                       Mid-Hudson Valley
sean at dague dot net                            Linux Users Group
http://dague.net                                 http://mhvlug.org

There is no silver bullet.  Plus, werewolves make better neighbors
than zombies, and they tend to keep the vampire population down.
__________________________________________________________________

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: xm-test domain creation delay

[Dan Smith]

EM> Well it would be better if we could compile and link against the
EM> version of libc in the ramdisk!

That's true for the libc case, but not for libxenctrl, right?

EM> How do all the other applications in the ramdisk get compiled now?

I believe that the only application on the ramdisk at the moment is
busybox, which is static.  uClibc is on there as well, since the
buildroot system can optionally download, compile, and link other
applications into the ramdisk automatically.

EM> Is it sufficient to say that once you can connect the console,
EM> then the guest is ready for commands? 

Well, not in the "xm console" sense, but in the xm-test console
library, this is true.  The xm-test console library isn't connected
and passed back to the test until it has seen the special shell
prompt.  So, this means the DomU is actually ready for commands.

EM> That wouldn't be the case for a "real" guest, of course, but if
EM> that's OK for xm-test (or close enough that we only need a 1
EM> second timeout or something) then that sounds like a better idea.

Right, which is why I asked if this sort of thing would be needed in a
generic sense.

I'm testing the modification to the console right now, which seems to
be working on my machine. 

-- 
Dan Smith
IBM Linux Technology Center
Open Hypervisor Team
email: danms@us.ibm.com

Re: xm-test domain creation delay

[Ewan Mellor]

On Mon, Nov 14, 2005 at 08:28:48AM -0800, Dan Smith wrote:

> 
> EM> Well it would be better if we could compile and link against the
> EM> version of libc in the ramdisk!
> 
> That's true for the libc case, but not for libxenctrl, right?

Well of course you'd need to compile libxenctrl (and libxenstore) for that
environment too.

> Well, not in the "xm console" sense, but in the xm-test console
> library, this is true.  The xm-test console library isn't connected
> and passed back to the test until it has seen the special shell
> prompt.  So, this means the DomU is actually ready for commands.

Of course!  I'd forgotten about the special shell prompt hack.  Well that
seems like a much easier way to do it.

> EM> That wouldn't be the case for a "real" guest, of course, but if
> EM> that's OK for xm-test (or close enough that we only need a 1
> EM> second timeout or something) then that sounds like a better idea.
> 
> Right, which is why I asked if this sort of thing would be needed in a
> generic sense.

Yes, it would be nice for more general cases, but that of course means
controlling the guest environment, and that would mean more interaction with
distro-specific aspects than I personally would like to take on at the moment.
It's not scary when we're doing it for xm-test, because we do control that
environment, but doing it for general guests is more fiddly.  However, I'm
sure somebody somewhere will want to know when the guest is actually booted,
as opposed to merely started, and writing to xenstore seems like a good way to
go about it.

I'd still go with the shell prompt hack though.

Ewan.

Re: xm-test domain creation delay

[Nivedita Singhvi]

Dan Smith wrote:
> EM> Here's an idea for xm-test.  Instead of waiting a full 20 seconds
> EM> for the guest domain to come up, we could instead write to the
> EM> store from inside the guest.
> 
> Yes, we definitely need to eliminate the 20-second delay for each

AMEN! We're hoping to add network tests into xm-test shortly,
and the overhead of domain creation for each trivial net test
is causing us some design conflicts.

Anything you can do on that front, Sean, will be much appreciated
and fully supported!

thanks,
Nivedita

Re: xm-test domain creation delay

[Nivedita Singhvi]

Nivedita Singhvi wrote:
> Dan Smith wrote:

> AMEN! We're hoping to add network tests into xm-test shortly,
> and the overhead of domain creation for each trivial net test
> is causing us some design conflicts.
> 
> Anything you can do on that front, Sean, will be much appreciated
                                     ^^^^^^^

Sorry, Dan, didn't mean to rename you,  s/Sean/Dan!


thanks,
Nivedita

RE: xm-test domain creation delay

[Ian Pratt]

 > > Right, which is why I asked if this sort of thing would be 
> needed in a 
> > generic sense.
> 
> Yes, it would be nice for more general cases, but that of 
> course means controlling the guest environment, and that 
> would mean more interaction with distro-specific aspects than 
> I personally would like to take on at the moment.
> It's not scary when we're doing it for xm-test, because we do 
> control that environment, but doing it for general guests is 
> more fiddly.  However, I'm sure somebody somewhere will want 
> to know when the guest is actually booted, as opposed to 
> merely started, and writing to xenstore seems like a good way 
> to go about it.

It's arguable that having a irtual filesystem inplemented in the kernel
for accessing xenstore would be good. Ideally, it would need to support
mkdir and creating new files which is slightly tricky. Allowing
read/write of existing nodes should be straightforward building on sys
or proc.

Ian

Re: xm-test domain creation delay

[Anthony Liguori]

Ian Pratt wrote:

> > > Right, which is why I asked if this sort of thing would be 
>  
>
>>needed in a 
>>    
>>
>>>generic sense.
>>>      
>>>
>>Yes, it would be nice for more general cases, but that of 
>>course means controlling the guest environment, and that 
>>would mean more interaction with distro-specific aspects than 
>>I personally would like to take on at the moment.
>>It's not scary when we're doing it for xm-test, because we do 
>>control that environment, but doing it for general guests is 
>>more fiddly.  However, I'm sure somebody somewhere will want 
>>to know when the guest is actually booted, as opposed to 
>>merely started, and writing to xenstore seems like a good way 
>>to go about it.
>>    
>>
>
>It's arguable that having a irtual filesystem inplemented in the kernel
>for accessing xenstore would be good.
>
2.6.14 now has FUSE which allows for virtual filesystems to be 
implemented in userspace.

Regards,

Anthony Liguori

>Ideally, it would need to support
>mkdir and creating new files which is slightly tricky. Allowing
>read/write of existing nodes should be straightforward building on sys
>or proc.
>
>Ian
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
>  
>