Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[John Levon]

On Thu, Oct 04, 2007 at 08:40:16AM -0700, Xen patchbot-3.1-testing wrote:

> xend: fix server/netif.py so that it respects type=None.
> Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
> xen-unstable changeset:   15972:3a799196ff69d3a3d5e4a891c13434aa61ce60a9
> xen-unstable date:        Thu Sep 27 17:44:03 2007 +0100
>  
> -        if not typ:
> -            typ = xoptions.netback_type
> -

Grumble, this makes xoptions.netback_type now clearly unused. I'm not
sure whether this breaks us yet, but certainly there's either cleanup
needed or it needs fixing.

What does type=None do?

regards
john

Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[Daniel P. Berrange]

On Fri, Oct 05, 2007 at 01:21:24AM +0100, John Levon wrote:
> On Thu, Oct 04, 2007 at 08:40:16AM -0700, Xen patchbot-3.1-testing wrote:
> 
> > xend: fix server/netif.py so that it respects type=None.
> > Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
> > xen-unstable changeset:   15972:3a799196ff69d3a3d5e4a891c13434aa61ce60a9
> > xen-unstable date:        Thu Sep 27 17:44:03 2007 +0100
> >  
> > -        if not typ:
> > -            typ = xoptions.netback_type
> > -
> 
> Grumble, this makes xoptions.netback_type now clearly unused. I'm not
> sure whether this breaks us yet, but certainly there's either cleanup
> needed or it needs fixing.

It shouldn't break anything unless it was already broken.

Basically with type=None, then the in-memory VM would be booted with type
of None, but at the same time the VM config would be written out to disk
with type of 'netfront'. You could stop & start the HVM guest many times
and XenD would always use a type of None when configuring qemu/netback.
The moment you restarted XenD (or rebooted the machine), then it would read
the config in from disk & now see type=netfront. So your config which had
been working fine, suddenly changes when XenD is restarted & your qemu 
NICs disappear. So if this breaks anything for Solaris, then it must have
been broken all along, because this only changes the config written to disk,
not the in-memory one XenD is actuall using to start off with.

> What does type=None do?

The behaviour is

  - type == netfront  -> setup netback only
  - type == ioemu     -> setup qemu-dm only
  - type is None      -> setup netback & qemu-dm

The latter is the preferred behavour because it means the same VM config works
whether the guest is using the RTL8139 emulated NIC, or PV drivers.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[Daniel P. Berrange]

On Fri, Oct 05, 2007 at 01:38:40AM +0100, Daniel P. Berrange wrote:
> On Fri, Oct 05, 2007 at 01:21:24AM +0100, John Levon wrote:
> > On Thu, Oct 04, 2007 at 08:40:16AM -0700, Xen patchbot-3.1-testing wrote:
> > 
> > > xend: fix server/netif.py so that it respects type=None.
> > > Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
> > > xen-unstable changeset:   15972:3a799196ff69d3a3d5e4a891c13434aa61ce60a9
> > > xen-unstable date:        Thu Sep 27 17:44:03 2007 +0100
> > >  
> > > -        if not typ:
> > > -            typ = xoptions.netback_type
> > > -
> > 
> > Grumble, this makes xoptions.netback_type now clearly unused. I'm not
> > sure whether this breaks us yet, but certainly there's either cleanup
> > needed or it needs fixing.
> 
> It shouldn't break anything unless it was already broken.
> 
> Basically with type=None, then the in-memory VM would be booted with type
> of None, but at the same time the VM config would be written out to disk
> with type of 'netfront'. You could stop & start the HVM guest many times
> and XenD would always use a type of None when configuring qemu/netback.
> The moment you restarted XenD (or rebooted the machine), then it would read
> the config in from disk & now see type=netfront. So your config which had
> been working fine, suddenly changes when XenD is restarted & your qemu 
> NICs disappear. So if this breaks anything for Solaris, then it must have
> been broken all along, because this only changes the config written to disk,
> not the in-memory one XenD is actuall using to start off with.

Actually if your Dom0/DomU drivers are working differently to the Linux ones
it could conceivably cause an issue. To explain what goes on in XenD when
starting a VM:

 1 The netif.getDeviceDetails() method is called to get the dict for
   front and backend devices. If type == 'ioemu' then, frontend dict is an
   empty stub (ie no handle/mac fields present).

 2 DevController.createDevice() writes the front & back dicts into xenstore

       - frontend VIF info is written into /local/[DOMID]/device/vif/[DEVID]
       - backend VIF info is written info /local/0/backends/device/vif/[DEVID]

 3 netback sees the info in /local/0/backends/device/vif/[DEVID] and sets up
   the backend devices.

 4 if the frontend dict was a stub (ie no handle/mac fields present) then it
   doesn't do anything, otherwise it connects to backend.

 5 in image.py if type is None or 'ioemu' then qemu is given a -nic arg to
   add an rtl8139 nic.

 6 VM config is saved, by calling DevController.configurations(), which
   calls configuration() on netif, which reads info out of xenstore in the
   backend/frontend paths. 


So, when in Step 1, the getDeviceDetails() was converting a type of 'None'
into 'netfront', when we get to step 6 it reads the type of 'netfront' out of
xenstore and writes it to disk. So next time XenD restarts, the type is fixed
to be 'netfront', and not None. This means that in step '5', qemu nevers sets
up a -nic arg.

So the one place where Solaris could in theory find trouble is if the kernel
modules for back/front actually look at the 'type' field in xenstore. In the
Linux impl, neither the back/front drivers ever look at the 'type' field - it
is only ever interpreted by XenD.

> > What does type=None do?
> 
> The behaviour is
> 
>   - type == netfront  -> setup netback only
>   - type == ioemu     -> setup qemu-dm only
>   - type is None      -> setup netback & qemu-dm
> 
> The latter is the preferred behavour because it means the same VM config works
> whether the guest is using the RTL8139 emulated NIC, or PV drivers.


Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[David Edmondson]

On Fri, Oct 05, 2007 at 02:22:33AM +0100, Daniel P. Berrange wrote:
> So the one place where Solaris could in theory find trouble is if
> the kernel modules for back/front actually look at the 'type' field
> in xenstore. In the Linux impl, neither the back/front drivers ever
> look at the 'type' field - it is only ever interpreted by XenD.

We do look at the 'type' field, to determine how to connect up the
backend to a physical device.

"type=ioemu" and "type=netfront" both behave similarly to the Linux
code ("netfront" creates a pseudo-ethernet device that could be used
by IP or a bridge, "ioemu" prompts qemu-dm to do the right thing
(which is different on Solaris, but anyway)).

"type=SUNW_mac", the default on Solaris (well, previously the
default), results in different approach where we expect to open a
physical or pseudo-physical device directly and relay
packets. Typically it's a pseudo-physical device created by the
hotplug scripts using Crossbow[1] VNICs.

I'll look at the changes more closely and figure out what to do.

[1] http://opensolaris.org/os/project/crossbow/

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[Keir Fraser]

On 5/10/07 14:41, "David Edmondson" <dme@sun.com> wrote:

> "type=SUNW_mac", the default on Solaris (well, previously the
> default), results in different approach where we expect to open a
> physical or pseudo-physical device directly and relay
> packets. Typically it's a pseudo-physical device created by the
> hotplug scripts using Crossbow[1] VNICs.
> 
> I'll look at the changes more closely and figure out what to do.

Perhaps slightly gross, but could we perhaps:
    if not typ and xoptions.netback_type != "netfront":
        typ = xoptions.netback_type

Bear in mind that you need a fix quite quickly for 3.1.1...

 -- Keir

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[David Edmondson]

On Fri, Oct 05, 2007 at 03:00:31PM +0100, Keir Fraser wrote:
> Perhaps slightly gross, but could we perhaps:

Yes, "blech".

It seems that at the place where 'None' is interpreted to mean
'netfront' we need to use xoptions.netback_type rather than
'netfront'.

> Bear in mind that you need a fix quite quickly for 3.1.1...

What is "quite quickly"?

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[Daniel P. Berrange]

On Fri, Oct 05, 2007 at 03:33:01PM +0100, David Edmondson wrote:
> On Fri, Oct 05, 2007 at 03:00:31PM +0100, Keir Fraser wrote:
> > Perhaps slightly gross, but could we perhaps:
> 
> Yes, "blech".
> 
> It seems that at the place where 'None' is interpreted to mean
> 'netfront' we need to use xoptions.netback_type rather than
> 'netfront'.

That would be the kernel...   Can't your code which looks at xenstore
for the type field just treat the absence of it as 'SUNW_mac' or 
whatever default value you want to use ?

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[David Edmondson]

On Fri, Oct 05, 2007 at 04:29:06PM +0100, Daniel P. Berrange wrote:
> On Fri, Oct 05, 2007 at 03:33:01PM +0100, David Edmondson wrote:
> > On Fri, Oct 05, 2007 at 03:00:31PM +0100, Keir Fraser wrote:
> > > Perhaps slightly gross, but could we perhaps:
> > 
> > Yes, "blech".
> > 
> > It seems that at the place where 'None' is interpreted to mean
> > 'netfront' we need to use xoptions.netback_type rather than
> > 'netfront'.
> 
> That would be the kernel...  Can't your code which looks at xenstore
> for the type field just treat the absence of it as 'SUNW_mac' or
> whatever default value you want to use ?

Yes, that's what we'll probably do.

Will you remove the netback_type default stuff?

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[Keir Fraser]

On 5/10/07 15:33, "David Edmondson" <dme@sun.com> wrote:

>> Bear in mind that you need a fix quite quickly for 3.1.1...
> 
> What is "quite quickly"?

We plan to release within a week. Really this needs to be fixed by end of
Monday. Seems to me we can knock up something suitable in that timeframe
though.

Daniel: Could we simply make xoptions.netback_type = None by default (for
all but Solaris) instead of "netfront"? Then we could put:
 if not typ:
    typ = xoptions.netback_type

Back in as is and it'd do the right thing?

 -- Keir

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[Daniel P. Berrange]

On Fri, Oct 05, 2007 at 04:56:28PM +0100, Keir Fraser wrote:
> On 5/10/07 15:33, "David Edmondson" <dme@sun.com> wrote:
> 
> >> Bear in mind that you need a fix quite quickly for 3.1.1...
> > 
> > What is "quite quickly"?
> 
> We plan to release within a week. Really this needs to be fixed by end of
> Monday. Seems to me we can knock up something suitable in that timeframe
> though.
> 
> Daniel: Could we simply make xoptions.netback_type = None by default (for
> all but Solaris) instead of "netfront"? Then we could put:
>  if not typ:
>     typ = xoptions.netback_type
> 
> Back in as is and it'd do the right thing?

That would certainly work for Linux, though it would not enable the HVM
guests on a Solaris host to be configured with both PV drivers & QEMU at
the same time. Upto solaris guys if that's a good enough solution...

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[Keir Fraser]

On 5/10/07 17:06, "Daniel P. Berrange" <berrange@redhat.com> wrote:

>> Daniel: Could we simply make xoptions.netback_type = None by default (for
>> all but Solaris) instead of "netfront"? Then we could put:
>>  if not typ:
>>     typ = xoptions.netback_type
>> 
>> Back in as is and it'd do the right thing?
> 
> That would certainly work for Linux, though it would not enable the HVM
> guests on a Solaris host to be configured with both PV drivers & QEMU at
> the same time. Upto solaris guys if that's a good enough solution...

Not clear there's a perfect solution that doesn't involve modifying the
Solaris PV netback to interpret None as SunW_mac (or whatever it was).
That's probably the nicest solution of all, and get rid of netback_type
default altogether. But that may be unacceptable due to backward compat
issues with existing Solaris PV netback?

 -- Keir

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[David Edmondson]

On Fri, Oct 05, 2007 at 05:09:55PM +0100, Keir Fraser wrote:
> Not clear there's a perfect solution that doesn't involve modifying the
> Solaris PV netback to interpret None as SunW_mac (or whatever it was).
> That's probably the nicest solution of all, and get rid of netback_type
> default altogether. But that may be unacceptable due to backward compat
> issues with existing Solaris PV netback?

I don't believe that there are any backwards compatibility issues
here. The people using the current OpenSolaris bits are versed in
dealing with change.

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[Keir Fraser]

On 5/10/07 17:17, "David Edmondson" <dme@sun.com> wrote:

> On Fri, Oct 05, 2007 at 05:09:55PM +0100, Keir Fraser wrote:
>> Not clear there's a perfect solution that doesn't involve modifying the
>> Solaris PV netback to interpret None as SunW_mac (or whatever it was).
>> That's probably the nicest solution of all, and get rid of netback_type
>> default altogether. But that may be unacceptable due to backward compat
>> issues with existing Solaris PV netback?
> 
> I don't believe that there are any backwards compatibility issues
> here. The people using the current OpenSolaris bits are versed in
> dealing with change.

Okay, so what's the conclusion? It sounds like:
 1. What is currently in 3.1-testing and unstable is okay, and you will
modify your netback driver; but
 2. We should clean up by removing xoption.netback_type in a further patch,
because it's not used any longer.

Right?

 -- Keir

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[David Edmondson]

On Sat, Oct 06, 2007 at 07:41:44AM +0100, Keir Fraser wrote:
> Okay, so what's the conclusion? It sounds like:
>  1. What is currently in 3.1-testing and unstable is okay, and you will
> modify your netback driver; but
>  2. We should clean up by removing xoption.netback_type in a further patch,
> because it's not used any longer.
> 
> Right?

Yes, I believe that is correct.

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[David Edmondson]

On Fri, Oct 05, 2007 at 05:06:56PM +0100, Daniel P. Berrange wrote:
> That would certainly work for Linux, though it would not enable the
> HVM guests on a Solaris host to be configured with both PV drivers &
> QEMU at the same time. Upto solaris guys if that's a good enough
> solution...

We'll make 'None' mean 'our preferred default' in our implementation.

The whole "PV and IOEMU" at the same time makes me uncomfortable,
though it looks to be more problematic with disks than network
devices.

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[Daniel P. Berrange]

On Fri, Oct 05, 2007 at 05:16:35PM +0100, David Edmondson wrote:
> On Fri, Oct 05, 2007 at 05:06:56PM +0100, Daniel P. Berrange wrote:
> > That would certainly work for Linux, though it would not enable the
> > HVM guests on a Solaris host to be configured with both PV drivers &
> > QEMU at the same time. Upto solaris guys if that's a good enough
> > solution...
> 
> We'll make 'None' mean 'our preferred default' in our implementation.
> 
> The whole "PV and IOEMU" at the same time makes me uncomfortable,
> though it looks to be more problematic with disks than network
> devices.

The key is that the Dom0 configuration of the guest should allow the guest
kernel to choose the drivers. Obviously you don't want both drivers active
at once, so one idea is for the PV drivers to 'grab' the PCI resources
associated with the emulated NIC. So once you load the PV driver, there's 
no way for the RTL8139 driver to get access to the NIC, thus ensuring only
one is ever active at once. The same principle applies to disk adapters.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[David Edmondson]

On Fri, Oct 05, 2007 at 06:30:48PM +0100, Daniel P. Berrange wrote:
> > The whole "PV and IOEMU" at the same time makes me uncomfortable,
> > though it looks to be more problematic with disks than network
> > devices.
> 
> The key is that the Dom0 configuration of the guest should allow the
> guest kernel to choose the drivers.

Understood.

> Obviously you don't want both drivers active at once, so one idea is
> for the PV drivers to 'grab' the PCI resources associated with the
> emulated NIC.

That seems so wrong.

In Solaris drivers are loaded as a result of a mapping from the PCI ID
of the device to a driver name (I've no idea if it's the same in
Linux). What happens if the RTL8139 driver gets in first?

> So once you load the PV driver, there's no way for the RTL8139
> driver to get access to the NIC, thus ensuring only one is ever
> active at once. The same principle applies to disk adapters.

Re: Re: [Xen-changelog] [xen-3.1-testing] xend: fix server/netif.py so that it respects type=None.

[Daniel P. Berrange]

On Fri, Oct 05, 2007 at 06:36:22PM +0100, David Edmondson wrote:
> On Fri, Oct 05, 2007 at 06:30:48PM +0100, Daniel P. Berrange wrote:
> > > The whole "PV and IOEMU" at the same time makes me uncomfortable,
> > > though it looks to be more problematic with disks than network
> > > devices.
> > 
> > The key is that the Dom0 configuration of the guest should allow the
> > guest kernel to choose the drivers.
> 
> Understood.
> 
> > Obviously you don't want both drivers active at once, so one idea is
> > for the PV drivers to 'grab' the PCI resources associated with the
> > emulated NIC.
> 
> That seems so wrong.
> 
> In Solaris drivers are loaded as a result of a mapping from the PCI ID
> of the device to a driver name (I've no idea if it's the same in
> Linux). What happens if the RTL8139 driver gets in first?

If the PV driver is unable to grab the PCI resources of the RTL NIC
because RTL8139 is already loaded, then it wouldn't try to initialize 
the PV nic. So this basically provides the safety net against both
being active (accidentally or by delibrate admuin actions). Then you 
need to figure out a way to prioritization the driver you really want.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|