Re: Missing libvirt+libxl functionality (Was: Re: [Xen-users] Programmatic administration of Xen machines)

Re: Missing libvirt+libxl functionality (Was: Re: [Xen-users] Programmatic administration of Xen machines)

[Ian Campbell]

On Mon, 2016-01-04 at 13:31 +0100, Andreas Pflug wrote:
> Am 04.01.16 um 13:13 schrieb Ian Campbell:
> > On Mon, 2016-01-04 at 12:47 +0100, Andreas Pflug wrote:
> > > Am 04.01.16 um 12:36 schrieb Ian Campbell:
> > > > Sorry to hijack this thread.
> > > > 
> > > > On Mon, 2015-12-28 at 12:56 +0100, Andreas Pflug wrote:
> > > > > Actually, I find virsh a bad idea since its support for the new
> > > > > xl
> > > > > interface isn't feature complete as the old xm interface was. I
> > > > > had
> > > > > to
> > > > > realize this the hard way when my virsh based python tools didn't
> > > > > work
> > > > > as expected after upgrading from xm to xl.
> > > > Note that virsh speaks to libxl directly, not via xl.
> > > > 
> > > > Please can you list the functionality of libvirt's libxl backed
> > > > which
> > > > is
> > > > missing compared to the old xend based one? 
> > > libvirt only handles domains created through it, i.e. using xl at the
> > > same time is incompatible. Since dom0 can't be created using libvirt,
> > > it's missing from "virsh list" in any case.
> > This works for me with a reasonably modern set of bits:
> > 
> >     root@marilith-n0    :~# virsh list
> >      Id    Name                           State
> >     ----------------------------------------------------
> >      0     Domain-0                       running
> >      13    d                              running
> > 
> > It probably requires the correct running of xen-init-dom0 during boot
> > (likely via the xencommons initscript).
> > 
> > I could believe it was broken at some point in history though.
> > 
> > >  libvirt stores its own state, not using libxl/xenstore stuff.
> > Please can you elaborate.
> I tested on Debian with 4.4.1 and xl toolstack. See the author's blog
> post:
> https://blog.xenproject.org/2014/01/17/libvirt-support-for-xens-new-
> libxenlight-toolstack/

I think that particular aspect of the blog post is no longer valid, at
least AFAICT with modern libvirt.

> I had even more trouble, e.g. I wasn't able to use non-standard block
> scripts (neither via /etc/xen/scripts/block-XXX nor via a script
> parameter) which are mandatory for me.

Looking at the code it seems like this is still missing.

Copying the devel list and maintainer in case I've either missed something
or there is a reason for this (other than "still on the TODO list", which I
expect is most likely).

If you find other shortcomings of libvirt with libxl then I suppose the
best way to approach them would be to report them as bugs:
    http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen

(I don't know if Jim or libvirt prefers some other means of doing so).

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Missing libvirt+libxl functionality (Was: Re: [Xen-users] Programmatic administration of Xen machines)

[Jim Fehlig]

On 01/04/2016 05:37 AM, Ian Campbell wrote:
> On Mon, 2016-01-04 at 13:31 +0100, Andreas Pflug wrote:
>> Am 04.01.16 um 13:13 schrieb Ian Campbell:
>>> On Mon, 2016-01-04 at 12:47 +0100, Andreas Pflug wrote:
>>>> Am 04.01.16 um 12:36 schrieb Ian Campbell:
>>>>> Sorry to hijack this thread.
>>>>>
>>>>> On Mon, 2015-12-28 at 12:56 +0100, Andreas Pflug wrote:
>>>>>> Actually, I find virsh a bad idea since its support for the new
>>>>>> xl
>>>>>> interface isn't feature complete as the old xm interface was. I
>>>>>> had
>>>>>> to
>>>>>> realize this the hard way when my virsh based python tools didn't
>>>>>> work
>>>>>> as expected after upgrading from xm to xl.
>>>>> Note that virsh speaks to libxl directly, not via xl.
>>>>>
>>>>> Please can you list the functionality of libvirt's libxl backed
>>>>> which
>>>>> is
>>>>> missing compared to the old xend based one? 
>>>> libvirt only handles domains created through it, i.e. using xl at the
>>>> same time is incompatible. Since dom0 can't be created using libvirt,
>>>> it's missing from "virsh list" in any case.
>>> This works for me with a reasonably modern set of bits:
>>>
>>>     root@marilith-n0    :~# virsh list
>>>      Id    Name                           State
>>>     ----------------------------------------------------
>>>      0     Domain-0                       running
>>>      13    d                              running
>>>
>>> It probably requires the correct running of xen-init-dom0 during boot
>>> (likely via the xencommons initscript).
>>>
>>> I could believe it was broken at some point in history though.
>>>
>>>>  libvirt stores its own state, not using libxl/xenstore stuff.
>>> Please can you elaborate.
>> I tested on Debian with 4.4.1 and xl toolstack. See the author's blog
>> post:
>> https://blog.xenproject.org/2014/01/17/libvirt-support-for-xens-new-
>> libxenlight-toolstack/
> I think that particular aspect of the blog post is no longer valid, at
> least AFAICT with modern libvirt.

Commit 45697fe5 added dom0 management support to the libvirt libxl driver.
libvirt >= 1.2.18 contains the commit.

>
>> I had even more trouble, e.g. I wasn't able to use non-standard block
>> scripts (neither via /etc/xen/scripts/block-XXX nor via a script
>> parameter) which are mandatory for me.
> Looking at the code it seems like this is still missing.
>
> Copying the devel list and maintainer in case I've either missed something
> or there is a reason for this (other than "still on the TODO list", which I
> expect is most likely).

It is on a TODO list, but I'm not sure how to solve it :-/. Unlike the
libxl_device_disk struct, the corresponding libvirt struct does not support a
'script' field, and I doubt it ever will. Repeated attempts to add a <script>
element to the <disk> config have been rejected. The libvirt community prefers
all config needed to setup disks be provided in the XML, not left to a magic
script doing unknown things behind libvirtd's back.

In the old xend-based libvirt driver, some of the block-* scripts worked by
accident. E.g. the block-iscsi script might work with config such as

    <disk type='file' device='disk'>
      <driver name='file'/>
      <source
file='iscsi:iqn.2006-09.de.suse@0ac47ee2-216e-452a-a341-a12624cd0225'/>
      <target dev='hda'/>
    </disk>

The 'iscsi:iqn.2006-09.de.suse@0ac47ee2-216e-452a-a341-a12624cd0225' blob was
passed wholesale to xend, which would eat it and "do the right thing". AFAIK,
libxl is not that forgiving. I've cc'd Olaf on this thread since we recently
discussed how libvirt+libxl might support external block scripts. As I recall,
we didn't have an novel ideas.

>
> If you find other shortcomings of libvirt with libxl then I suppose the
> best way to approach them would be to report them as bugs:
>     http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen
>
> (I don't know if Jim or libvirt prefers some other means of doing so).

See http://libvirt.org/bugs.html for reporting libvirt bugs. Deficiencies in
libxl compared to the old xend toolstack should be reported against Xen - and
probably libvirt too since it most likely contains the deficiency as well.

Regards,
Jim

Re: Missing libvirt+libxl functionality (Was: Re: [Xen-users] Programmatic administration of Xen machines)

[Olaf Hering]

On Thu, Jan 07, Jim Fehlig wrote:

> The 'iscsi:iqn.2006-09.de.suse@0ac47ee2-216e-452a-a341-a12624cd0225' blob was
> passed wholesale to xend, which would eat it and "do the right thing". AFAIK,
> libxl is not that forgiving. I've cc'd Olaf on this thread since we recently
> discussed how libvirt+libxl might support external block scripts. As I recall,
> we didn't have an novel ideas.

I think the "raw" string is known to libvirts libxl driver. It could do
a strncmp() and fill also ->script. Then libxl would call that thing by
itself, libvirt does not need to know about such internals. Of course
that would mean an additional "script=$path" is still not supported  by
libvirt. But thats not a loss because whoever is relying on can just
copy its own file into XEN_SCRIPT_DIR. After all only our own dmmd thing
requires a script because that is its whole purpose (carry the block
configuration along with the vm configuration).

Olaf

Re: Missing libvirt+libxl functionality (Was: Re: [Xen-users] Programmatic administration of Xen machines)

[Ian Campbell]

On Thu, 2016-01-07 at 22:54 -0700, Jim Fehlig wrote:
> > > I had even more trouble, e.g. I wasn't able to use non-standard block
> > > scripts (neither via /etc/xen/scripts/block-XXX nor via a script
> > > parameter) which are mandatory for me.
> > Looking at the code it seems like this is still missing.
> > 
> > Copying the devel list and maintainer in case I've either missed something
> > or there is a reason for this (other than "still on the TODO list", which I
> > expect is most likely).
> 
> It is on a TODO list, but I'm not sure how to solve it :-/. Unlike the
> libxl_device_disk struct, the corresponding libvirt struct does not support a
> 'script' field, and I doubt it ever will. Repeated attempts to add a 
> element to the  config have been rejected. The libvirt community prefers
> all config needed to setup disks be provided in the XML, not left to a magic
> script doing unknown things behind libvirtd's back.

That's an understandable position for them to take, I think.

If someone does use the libvirt's XML based mechanisms to configure things
then does the libxl backend correctly plumb them through? I suppose
something in libvirt either produces a blk device (which can be trivially
converted to a suitable libxl disk phy thing) or a reference to some image
file (perhaps less trivially convertable but doable, most of the time?).

If the libvirt XML stuff works then I don't see any pressing need to plumb
Xen scripts through to libvirt if the libvirt maintainers do not want to
support that abstract concept. If folks want functionality which would only
be available via scripts then they should be encouraged to implement that
functionality generically in libvirt in a way which is acceptable to the
maintainers.

I appreciate that this might mean that some xl style cfg files cannot be
laundered through domxml-from-native to create a working xml config, but I
think that's a natural consequence of having two separate projects with
somewhat non-overlapping featuresets and design goals.

I see domxml-from-native as more of a convenience for configurations which
fall into the union of xl and libvirt's featuresets. I suppose if someone
was feeling adventurous then they could try and have domxml-from-native
spot some uses of well known block scripts and convert to the equivalent
libvirt XML e.g. spot block-iscsi and convert that into equivalent libvirt
XML objects (assuming libvirt supports iscsi, I didn't look).

Ian.

Re: Missing libvirt+libxl functionality (Was: Re: [Xen-users] Programmatic administration of Xen machines)

[Jim Fehlig]

On 01/08/2016 05:46 AM, Ian Campbell wrote:
> On Thu, 2016-01-07 at 22:54 -0700, Jim Fehlig wrote:
>>>> I had even more trouble, e.g. I wasn't able to use non-standard block
>>>> scripts (neither via /etc/xen/scripts/block-XXX nor via a script
>>>> parameter) which are mandatory for me.
>>> Looking at the code it seems like this is still missing.
>>>
>>> Copying the devel list and maintainer in case I've either missed something
>>> or there is a reason for this (other than "still on the TODO list", which I
>>> expect is most likely).
>> It is on a TODO list, but I'm not sure how to solve it :-/. Unlike the
>> libxl_device_disk struct, the corresponding libvirt struct does not support a
>> 'script' field, and I doubt it ever will. Repeated attempts to add a 
>> element to the  config have been rejected. The libvirt community prefers
>> all config needed to setup disks be provided in the XML, not left to a magic
>> script doing unknown things behind libvirtd's back.
> That's an understandable position for them to take, I think.
>
> If someone does use the libvirt's XML based mechanisms to configure things
> then does the libxl backend correctly plumb them through?

No, not at this time. At least not the interesting ones like nbd, iscsi, rbd
(and to some extent npiv).

>  I suppose
> something in libvirt either produces a blk device (which can be trivially
> converted to a suitable libxl disk phy thing) or a reference to some image
> file (perhaps less trivially convertable but doable, most of the time?).

That would be one approach. The other would be to pass the info to libxl. In
some use cases, it is desired/needed in libxl anyhow. I was recently
experimenting with qemu's integrated librados support, which provides native
access to block devices on ceph clusters. I hacked xl/libxl to support an 'rbd'
qdisk, adding server, port, etc. to xl.cfg and libxl_device_disk. The info was
needed when invoking qemu. nbd and iscsi qdisks could be similarly supported.
IMO, it would be unfortunate to reimplement qdisk features in the libvirt libxl
driver. Note that for many of these disk types, the libvirt qemu driver passes
the config along to qemu.

>
> If the libvirt XML stuff works then I don't see any pressing need to plumb
> Xen scripts through to libvirt if the libvirt maintainers do not want to
> support that abstract concept. If folks want functionality which would only
> be available via scripts then they should be encouraged to implement that
> functionality generically in libvirt in a way which is acceptable to the
> maintainers.

Right. That's exactly what has happened over the years as libvirt has gained
support for nbd, rbd, iscsi, and npiv disk devices.

>
> I appreciate that this might mean that some xl style cfg files cannot be
> laundered through domxml-from-native to create a working xml config, but I
> think that's a natural consequence of having two separate projects with
> somewhat non-overlapping featuresets and design goals.

Agreed, but we can strive to minimize that.

>
> I see domxml-from-native as more of a convenience for configurations which
> fall into the union of xl and libvirt's featuresets. I suppose if someone
> was feeling adventurous then they could try and have domxml-from-native
> spot some uses of well known block scripts and convert to the equivalent
> libvirt XML e.g. spot block-iscsi and convert that into equivalent libvirt
> XML objects (assuming libvirt supports iscsi, I didn't look).

Yes, that would be possible.  E.g. something like the following xl -> domxml
conversion

  disk = [ 'iscsi:iqn.2013-07.com.example:iscsi-nopool/2,xvda,w' ]

  <disk type='network' device='disk'>
    <driver name='qemu' type='raw'/>
    <source protocol='iscsi' name='iqn.2013-07.com.example:iscsi-nopool/2'>
      <host name='example.com'/>
    </source>
    <target dev='xvda'/>
  </disk>

But as mentioned above, I think there is some benefit in supporting some of
these disk types in libxl, and hence xl.cfg. I owe the list an RFC mail about
this topic and hope to post something in the next week or two - unless the idea
is immediately discouraged :-).

Regards,
Jim