VMX device models not getting created anymore?

VMX device models not getting created anymore?

[Arun Sharma]

This changeset:

http://xen.bkbits.net:8080/xeno-unstable.bk/cset@42a47555156iGoOjUyZtvbuMFZHXkg

seems to have removed the call to creating device models.

@@ -773,43 +680,6 @@
                  ctrl.initController(reboot=True)
          else:
              self.create_configured_devices()
-        if self.is_vmx:
-            self.create_vmx_model()

I didn't see equivalent code getting added anywhere else.

	-Arun

RE: VMX device models not getting created anymore?

[Ian Pratt]

 > http://xen.bkbits.net:8080/xeno-unstable.bk/cset@42a47555156iG
> oOjUyZtvbuMFZHXkg
> 
> seems to have removed the call to creating device models.
> 
> @@ -773,43 +680,6 @@
>                   ctrl.initController(reboot=True)
>           else:
>               self.create_configured_devices()
> -        if self.is_vmx:
> -            self.create_vmx_model()
> 
> I didn't see equivalent code getting added anywhere else.

Yep, I just sync'ed my VT tree up to the head to get a couple of your
fixes, and inherited this breakage as well. There's been loads of
changes to xend to move its database over to xenstore, but I don't think
its going to be hard to fix up. 

I'm not going to get a chance to look at it before Monday evening
though.

Best,
Ian

Re: VMX device models not getting created anymore?

[Mike Wray]

Ian Pratt wrote:
>  > http://xen.bkbits.net:8080/xeno-unstable.bk/cset@42a47555156iG
> 
>>oOjUyZtvbuMFZHXkg
>>
>>seems to have removed the call to creating device models.
>>
>>@@ -773,43 +680,6 @@
>>                  ctrl.initController(reboot=True)
>>          else:
>>              self.create_configured_devices()
>>-        if self.is_vmx:
>>-            self.create_vmx_model()
>>
>>I didn't see equivalent code getting added anywhere else.

The code got moved into createDeviceModel in VmxImageHandler
in image.py, but it looks like the call to createDeviceModel got
lost. It should go at the end of create_devices in XendDomainInfo:

     def create_devices(self):
         """Create the devices for a vm.

         @raise: VmError for invalid devices
         """
         if self.rebooting():
             for ctrl in self.getDeviceControllers():
                 ctrl.initController(reboot=True)
         else:
             self.create_configured_devices()
         self.image.createDeviceModel()
> 
> 
> Yep, I just sync'ed my VT tree up to the head to get a couple of your
> fixes, and inherited this breakage as well. There's been loads of
> changes to xend to move its database over to xenstore, but I don't think
> its going to be hard to fix up. 
> 
> I'm not going to get a chance to look at it before Monday evening
> though.
> 
> Best,
> Ian
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: VMX device models not getting created anymore?

[Arun Sharma]

Mike Wray wrote:
>     def create_devices(self):
>         """Create the devices for a vm.
> 
>         @raise: VmError for invalid devices
>         """
>         if self.rebooting():
>             for ctrl in self.getDeviceControllers():
>                 ctrl.initController(reboot=True)
>         else:
>             self.create_configured_devices()
>         self.image.createDeviceModel()

I sent a patch: vmx-device-models-py.patch to the list to do just this, 
but the event channel numbers seem to be off by one for some reason 
(please see comments inside the patch).

	-Arun

Re: VMX device models not getting created anymore?

[Mike Wray]

Arun Sharma wrote:
> Mike Wray wrote:
> 
>>     def create_devices(self):
>>         """Create the devices for a vm.
>>
>>         @raise: VmError for invalid devices
>>         """
>>         if self.rebooting():
>>             for ctrl in self.getDeviceControllers():
>>                 ctrl.initController(reboot=True)
>>         else:
>>             self.create_configured_devices()
>>         self.image.createDeviceModel()
> 
> 
> I sent a patch: vmx-device-models-py.patch to the list to do just this, 
> but the event channel numbers seem to be off by one for some reason 
> (please see comments inside the patch).

OK, I see the patch. What makes you think the event channel port
numbers are off? Is it possible that it should be using port2 instead
of port1?

Mike

Re: VMX device models not getting created anymore?

[Arun Sharma]

Mike Wray wrote:

>> I sent a patch: vmx-device-models-py.patch to the list to do just 
>> this, but the event channel numbers seem to be off by one for some 
>> reason (please see comments inside the patch).
> 
> 
> OK, I see the patch. What makes you think the event channel port
> numbers are off? Is it possible that it should be using port2 instead
> of port1?

If I print self.device_channel.port1 and compare it to the output of

# xm list --long

It's off by one and device models don't get any events from the VMX domain.

When I add the -1, things work ok i.e. device models do receive events 
from the VMX domain.

port1 = dom0 end of the event channel. That's what the device models 
listen to. port2 = hypervisor/vmx domain end of the event channel.

	-Arun

Re: VMX device models not getting created anymore?

[Mike Wray]

Arun Sharma wrote:
> Mike Wray wrote:
> 
>>> I sent a patch: vmx-device-models-py.patch to the list to do just 
>>> this, but the event channel numbers seem to be off by one for some 
>>> reason (please see comments inside the patch).
>>
>>
>>
>> OK, I see the patch. What makes you think the event channel port
>> numbers are off? Is it possible that it should be using port2 instead
>> of port1?
> 
> 
> If I print self.device_channel.port1 and compare it to the output of
> 
> # xm list --long
> 
> It's off by one and device models don't get any events from the VMX domain.

Compared to which field is it off by 1?
The VMX code creates its own event channel using  channel.eventChannel(), which will
allocate new ports. Possibly it should be reusing the existing control channel port
instead? This might account for the off-by-one as ports are allocated sequentially.

> 
> When I add the -1, things work ok i.e. device models do receive events 
> from the VMX domain.
> 
> port1 = dom0 end of the event channel. That's what the device models 
> listen to. port2 = hypervisor/vmx domain end of the event channel.

Hope this helps,

Mike

Re: VMX device models not getting created anymore?

[Arun Sharma]

Mike Wray wrote:
> Compared to which field is it off by 1?
> The VMX code creates its own event channel using  
> channel.eventChannel(), which will
> allocate new ports. Possibly it should be reusing the existing control 
> channel port
> instead? This might account for the off-by-one as ports are allocated 
> sequentially.

That seems to be the problem. If I add some logging, I see:

[2005-06-17 12:06:07 xend] INFO (channel:32) created event channel: 
<EventChannel dom1:0:19 dom2:5:2>
[2005-06-17 12:06:07 xend] INFO (channel:32) created event channel: 
<EventChannel dom1:0:20 dom2:5:3>

The first one is created here:

File "/usr/lib/python/xen/xend/XendDomain.py", line 276, in domain_create
     dominfo = XendDomainInfo.create(self.dbmap, config)
   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 159, in create
     vm.construct(config)
   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 501, in construct
     self.construct_image()
   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 547, in 
construct_image
     self.create_channel()
   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 680, in 
create_channel
     self.store_channel = self.eventChannel("store_channel")
   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 674, in 
eventChannel
     return EventChannel.restoreFromDB(db, 0, self.id)
   File "/usr/lib/python/xen/xend/server/channel.py", line 59, in 
restoreFromDB
     evtchn = cls.interdomain(dom1, dom2, port1=port1, port2=port2)

The second is created here:

File "/usr/lib/python/xen/xend/XendDomain.py", line 276, in domain_create
     dominfo = XendDomainInfo.create(self.dbmap, config)
   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 159, in create
     vm.construct(config)
   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 502, in construct
     self.configure()
   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 884, in configure
     self.create_devices()
   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 704, in 
create_devices
     self.image.createDeviceModel()
   File "/usr/lib/python/xen/xend/image.py", line 300, in createDeviceModel
     self.device_channel = channel.eventChannel(0, self.vm.getDomain())
   File "/usr/lib/python/xen/xend/server/channel.py", line 116, in 
eventChannel
     return EventChannel.interdomain(dom1, dom2, port1=port1, port2=port2)

Things get more interesting, because self.device_channel['port1'] for 
the second channel returns 19 instead of 20.

Are there assumptions in the code that there must be only one 
interdomain event channel between two domains?

	-Arun

Re: VMX device models not getting created anymore?

[Arun Sharma]

Arun Sharma wrote:

> That seems to be the problem. If I add some logging, I see:
> 
> [2005-06-17 12:06:07 xend] INFO (channel:32) created event channel: 
> <EventChannel dom1:0:19 dom2:5:2>
> [2005-06-17 12:06:07 xend] INFO (channel:32) created event channel: 
> <EventChannel dom1:0:20 dom2:5:3>
[..]
> 
> Things get more interesting, because self.device_channel['port1'] for 
> the second channel returns 19 instead of 20.
> 

This statement is not true. I got confused because the only the first 
event channel shows up in xm list --long.

The real issue is the hard coding in:

xen/include/public/io/ioreq.h:

#define IOPACKET_PORT   2

This was true before your changes went in. After your changes, xen 
started sending IOPACKET events on:

<EventChannel dom1:0:19 dom2:5:2>

but the user space device models were listening on:

<EventChannel dom1:0:20 dom2:5:3>

However, if I subtract -1, everything magically works :)

I think the quick fix is to redefine IOPACKET_PORT to be 3. Will send a 
patch to remove the hard coding ASAP.

	-Arun

Re: VMX device models not getting created anymore?

[Mike Wray]

Arun Sharma wrote:
> Arun Sharma wrote:
> 
>> That seems to be the problem. If I add some logging, I see:
>>
>> [2005-06-17 12:06:07 xend] INFO (channel:32) created event channel: 
>> <EventChannel dom1:0:19 dom2:5:2>
>> [2005-06-17 12:06:07 xend] INFO (channel:32) created event channel: 
>> <EventChannel dom1:0:20 dom2:5:3>
> 
> [..]
> 
>>
>> Things get more interesting, because self.device_channel['port1'] for 
>> the second channel returns 19 instead of 20.
>>
> 
> This statement is not true. I got confused because the only the first 
> event channel shows up in xm list --long.
> 
> The real issue is the hard coding in:
> 
> xen/include/public/io/ioreq.h:
> 
> #define IOPACKET_PORT   2
> 
> This was true before your changes went in. After your changes, xen 
> started sending IOPACKET events on:
> 
> <EventChannel dom1:0:19 dom2:5:2>
> 
> but the user space device models were listening on:
> 
> <EventChannel dom1:0:20 dom2:5:3>
> 
> However, if I subtract -1, everything magically works :)

Having to add or subtract 1 to make something work is almost always
not the right way to fix a problem - because it dosen't address
the real issue.

> I think the quick fix is to redefine IOPACKET_PORT to be 3. Will send a 
> patch to remove the hard coding ASAP.

That's not the fix. That just replaces one hard-coded constant with another, when
it shouldn't be hard-coded at all. It's going to break again if any other interdomain
port is allocated.

The port to use should be passed to the domain as a parameter.
This is what is done with the domain controller port,
and the xenstore port.

The correct fix is to add the device model port as a parameter to
xc_vmx_build. In fact this already has control_evtchn as a parameter,
but ignores it - so you could use that. You could probably re-use
the start_info field for it too as vmx domains don't seem to be using
the control channel otherwise:

Set domain_controller_evtchn in xc_vmx_build from control_evtchn, and use
domain->start_info->domain_controller_evtchn intead of IOPACKET_PORT.

In xend the vmx code can simply pass the domain control evtchn like the
other build functions, since the existing control evtchn isn't being used.

Hope this helps,

Mike

Re: VMX device models not getting created anymore?

[Arun Sharma]

Mike Wray wrote:

>> However, if I subtract -1, everything magically works :)
> 
> 
> Having to add or subtract 1 to make something work is almost always
> not the right way to fix a problem - because it dosen't address
> the real issue.
> 

You probably missed the smiley :)

>> I think the quick fix is to redefine IOPACKET_PORT to be 3. Will send 
>> a patch to remove the hard coding ASAP.
> 
> 
> That's not the fix. That just replaces one hard-coded constant with 
> another, when
> it shouldn't be hard-coded at all. It's going to break again if any 
> other interdomain
> port is allocated.
> 
> The port to use should be passed to the domain as a parameter.
> This is what is done with the domain controller port,
> and the xenstore port.
> 

Yes of course, I promised to send a patch as well. The quick fix was for 
those who couldn't wait.

> The correct fix is to add the device model port as a parameter to
> xc_vmx_build. In fact this already has control_evtchn as a parameter,
> but ignores it - so you could use that. You could probably re-use
> the start_info field for it too as vmx domains don't seem to be using
> the control channel otherwise:
> 

The problem is that:

self.construct_image()
	...
		xc_vmx_build()
self.configure()
	...
		self.image.createDeviceModel()

The event channel hasn't been created at the time of xc_vmx_build().

	-Arun

Re: VMX device models not getting created anymore?

[Mike Wray]

Arun Sharma wrote:
> Mike Wray wrote:
> 
>>> However, if I subtract -1, everything magically works :)
>>
>>
>>
>> Having to add or subtract 1 to make something work is almost always
>> not the right way to fix a problem - because it dosen't address
>> the real issue.
>>
> 
> You probably missed the smiley :)
> 
>>> I think the quick fix is to redefine IOPACKET_PORT to be 3. Will send 
>>> a patch to remove the hard coding ASAP.
>>
>>
>>
>> That's not the fix. That just replaces one hard-coded constant with 
>> another, when
>> it shouldn't be hard-coded at all. It's going to break again if any 
>> other interdomain
>> port is allocated.
>>
>> The port to use should be passed to the domain as a parameter.
>> This is what is done with the domain controller port,
>> and the xenstore port.
>>
> 
> Yes of course, I promised to send a patch as well. The quick fix was for 
> those who couldn't wait.
> 
>> The correct fix is to add the device model port as a parameter to
>> xc_vmx_build. In fact this already has control_evtchn as a parameter,
>> but ignores it - so you could use that. You could probably re-use
>> the start_info field for it too as vmx domains don't seem to be using
>> the control channel otherwise:
>>
> 
> The problem is that:
> 
> self.construct_image()
>     ...
>         xc_vmx_build()
> self.configure()
>     ...
>         self.image.createDeviceModel()
> 
> The event channel hasn't been created at the time of xc_vmx_build().

All the other build fns need the channel, so it is created before the domain image:

     def construct_image(self):
         """Construct the boot image for the domain.
         """
         self.create_channel()
         self.image.createImage()
         self.exportToDB()
         self.storeConnect()

 From the linux build image:

     def buildDomain(self):
         control_evtchn = self.vm.channel.getRemotePort()
	...
         d  = xc.linux_build(dom            = self.vm.getDomain(),
                             image          = self.kernel,
                             control_evtchn = control_evtchn,
                             store_evtchn   = store_evtchn,
                             cmdline        = self.cmdline,
                             ramdisk        = self.ramdisk,
                             flags          = self.flags,
                             vcpus          = self.vm.vcpus)
         ...

Mike

Re: VMX device models not getting created anymore?

[Mike Wray]

Arun Sharma wrote:
> Mike Wray wrote:
> 
>> Compared to which field is it off by 1?
>> The VMX code creates its own event channel using  
>> channel.eventChannel(), which will
>> allocate new ports. Possibly it should be reusing the existing control 
>> channel port
>> instead? This might account for the off-by-one as ports are allocated 
>> sequentially.
> 
> 
> That seems to be the problem. If I add some logging, I see:
> 
> [2005-06-17 12:06:07 xend] INFO (channel:32) created event channel: 
> <EventChannel dom1:0:19 dom2:5:2>
> [2005-06-17 12:06:07 xend] INFO (channel:32) created event channel: 
> <EventChannel dom1:0:20 dom2:5:3>
> 
> The first one is created here:
> 
> File "/usr/lib/python/xen/xend/XendDomain.py", line 276, in domain_create
>     dominfo = XendDomainInfo.create(self.dbmap, config)
>   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 159, in create
>     vm.construct(config)
>   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 501, in construct
>     self.construct_image()
>   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 547, in 
> construct_image
>     self.create_channel()
>   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 680, in 
> create_channel
>     self.store_channel = self.eventChannel("store_channel")
>   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 674, in 
> eventChannel
>     return EventChannel.restoreFromDB(db, 0, self.id)
>   File "/usr/lib/python/xen/xend/server/channel.py", line 59, in 
> restoreFromDB
>     evtchn = cls.interdomain(dom1, dom2, port1=port1, port2=port2)
> 
> The second is created here:
> 
> File "/usr/lib/python/xen/xend/XendDomain.py", line 276, in domain_create
>     dominfo = XendDomainInfo.create(self.dbmap, config)
>   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 159, in create
>     vm.construct(config)
>   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 502, in construct
>     self.configure()
>   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 884, in configure
>     self.create_devices()
>   File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 704, in 
> create_devices
>     self.image.createDeviceModel()
>   File "/usr/lib/python/xen/xend/image.py", line 300, in createDeviceModel
>     self.device_channel = channel.eventChannel(0, self.vm.getDomain())
>   File "/usr/lib/python/xen/xend/server/channel.py", line 116, in 
> eventChannel
>     return EventChannel.interdomain(dom1, dom2, port1=port1, port2=port2)
> 
> Things get more interesting, because self.device_channel['port1'] for 
> the second channel returns 19 instead of 20.
> 
> Are there assumptions in the code that there must be only one 
> interdomain event channel between two domains?

It depends which code you are talking about.
The code using the domain control channel (i.e. the frontend and backend drivers)
gets upset if you don't use the first event channel on the domain.
Other code will use any interdomin channel, but you need to tell it the
port number somehow.

Mike

RE: VMX device models not getting created anymore?

[Ian Pratt]

> Mike Wray wrote:
> >     def create_devices(self):
> >         """Create the devices for a vm.
> > 
> >         @raise: VmError for invalid devices
> >         """
> >         if self.rebooting():
> >             for ctrl in self.getDeviceControllers():
> >                 ctrl.initController(reboot=True)
> >         else:
> >             self.create_configured_devices()
> >         self.image.createDeviceModel()
> 
> I sent a patch: vmx-device-models-py.patch to the list to do 
> just this, but the event channel numbers seem to be off by 
> one for some reason (please see comments inside the patch).

Yep, I think its even checked in, but the out by one needs to be tracked
down urgently. I couldn't spot anything obvious.

Ian

Re: VMX device models not getting created anymore?

[Arun Sharma]

Ian Pratt wrote:
>>I sent a patch: vmx-device-models-py.patch to the list to do 
>>just this, but the event channel numbers seem to be off by 
>>one for some reason (please see comments inside the patch).
> 
> 
> Yep, I think its even checked in, but the out by one needs to be tracked
> down urgently. I couldn't spot anything obvious.
> 

I didn't find the patch in yesterday's nightly snapshot.

	-Arun