Bugs in xl that affect stubdom+tapdisk2, and others

Bugs in xl that affect stubdom+tapdisk2, and others

[John Weekes]

I'm having these problems when using "xl" in the latest c/s of 4.2 
(25912:651d965ee2c0):

- It's not allowing multiple domains to use the same images, at least 
under tapdisk2. If I start one domain using an image (a read-only CD-ROM 
image, for instance), then start another, I can see in "tap-ctl list" 
that only one of the domains actually receives the proper access. The 
other either never gets it, or it is disconnected after it's created -- 
I can't readily tell which.

- When I use "xl destroy" on a domain that also has an associated 
stubdom, "xl" tries to destroy the tapdisk2 entries for both the domain 
and its stubdomain, resulting in errors like these:

# xl destroy testvds4
libxl: error: libxl_blktap2.c:73:libxl__device_destroy_tapdisk: Unable 
to find type aio disk /servers/isos/DummyCD.iso: No such file or directory
libxl: error: libxl_blktap2.c:73:libxl__device_destroy_tapdisk: Unable 
to find type aio disk /servers/customers/testvds4.img: No such file or 
directory
libxl: error: libxl_blktap2.c:73:libxl__device_destroy_tapdisk: Unable 
to find type aio disk /servers/isos/DummyCD-2.iso: No such file or directory
libxl: error: libxl.c:1466:devices_destroy_cb: libxl__devices_destroy 
failed for 19
libxl: error: libxl.c:1466:devices_destroy_cb: libxl__devices_destroy 
failed for 18

If I'm reading the source correctly, it seems as though the various 
destruction functions aren't passing down the domid and ultimately 
libxl__device_destroy_tapdisk is calling tap_ctl_find, which just 
globally matches based on the type and name of the image, causing the 
errors on destroy. I was trying to find a quick fix, but it looks like 
any proper one will require querying xenstore for the tapdisk entries 
specific to the domain, and I'm not immediately able to make that 
change, so I've personally just switched back to using "xm" for the time 
being. (I don't know if this relates to the first bug re: multiple 
domains using the same image files).

Another, much more minor, bug in "xl" that I've found is that "xl 
uptime" will not run without an argument; it is supposed to just show 
all uptimes in this circumstance. This is an easy one-liner fix of 
modifying line:

tools/libxl/xl_cmdimpl.c:5749:    while ((opt = def_getopt(argc, argv, 
"s", "uptime", 1)) != -1) {

into this:

tools/libxl/xl_cmdimpl.c:5749:    while ((opt = def_getopt(argc, argv, 
"s", "uptime", 0)) != -1) {

Also, a couple of small typos in "xl" --

libxl.c:556:        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "geting 
domain info list");
libxl.c:575:        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "geting 
domain info list");
libxl.c:678:        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "geting 
domain info list");
libxl.c:1393:        LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "non-existant 
domain %d", domid);

Also, running "xl sched-credit" always gives me an error, even though it 
seems to work. Maybe it should be silent if cpu pools aren't defined? 
Maybe it's supposed to be creating a second cpupool for the second CPU 
in this system, and isn't?

# xl sched-credit
libxl: error: libxl.c:596:cpupool_info: failed to get info for cpupool1
: No such file or directory
Cpupool Pool-0: tslice=5ms ratelimit=500us
Name                                ID Weight  Cap
Domain-0                             0  65535    0

As an additional note about 4.2 -- this isn't an "xl" issue, but rather 
a build problem -- even though I use "./configure --libdir=/usr/lib64", 
the build is still creating a "dist/install/usr/lib" directory, and 
because of this, running ./install.sh nukes my system's "/usr/lib" -> 
"/usr/lib64" softlink (which in turn causes all sorts of other problems 
until it's corrected). Files that are being put in that tree:

xen-4.2-testing.hg # find dist/install/usr/lib
dist/install/usr/lib
dist/install/usr/lib/xen
dist/install/usr/lib/xen/boot
dist/install/usr/lib/xen/boot/xenstore-stubdom.gz
dist/install/usr/lib/xen/boot/ioemu-stubdom.gz
dist/install/usr/lib/xen/boot/pv-grub-x86_64.gz
dist/install/usr/lib/xen/boot/hvmloader
dist/install/usr/lib/xen/boot/pv-grub-x86_32.gz
dist/install/usr/lib/xen/bin
dist/install/usr/lib/xen/bin/stubdom-dm
dist/install/usr/lib/xen/bin/qemu-io
dist/install/usr/lib/xen/bin/qemu-nbd
dist/install/usr/lib/xen/bin/qemu-img
dist/install/usr/lib/xen/bin/xenpaging
dist/install/usr/lib/xen/bin/qemu-dm
dist/install/usr/lib/xen/bin/qemu-system-i386
dist/install/usr/lib/xen/bin/qemu-ga
dist/install/usr/lib/xen/bin/stubdompath.sh

-John

Re: Bugs in xl that affect stubdom+tapdisk2, and others

[Ian Campbell]

On Thu, 2012-11-08 at 20:45 +0000, John Weekes wrote:

> - It's not allowing multiple domains to use the same images, at least 
> under tapdisk2. If I start one domain using an image (a read-only CD-ROM 
> image, for instance), then start another, I can see in "tap-ctl list" 
> that only one of the domains actually receives the proper access. The 
> other either never gets it, or it is disconnected after it's created -- 
> I can't readily tell which.
> 
> - When I use "xl destroy" on a domain that also has an associated 
> stubdom, "xl" tries to destroy the tapdisk2 entries for both the domain 
> and its stubdomain, resulting in errors like these:
> 
> # xl destroy testvds4
> libxl: error: libxl_blktap2.c:73:libxl__device_destroy_tapdisk: Unable 
> to find type aio disk /servers/isos/DummyCD.iso: No such file or directory
> libxl: error: libxl_blktap2.c:73:libxl__device_destroy_tapdisk: Unable 
> to find type aio disk /servers/customers/testvds4.img: No such file or 
> directory
> libxl: error: libxl_blktap2.c:73:libxl__device_destroy_tapdisk: Unable 
> to find type aio disk /servers/isos/DummyCD-2.iso: No such file or directory
> libxl: error: libxl.c:1466:devices_destroy_cb: libxl__devices_destroy 
> failed for 19
> libxl: error: libxl.c:1466:devices_destroy_cb: libxl__devices_destroy 
> failed for 18
> 
> If I'm reading the source correctly, it seems as though the various 
> destruction functions aren't passing down the domid and ultimately 
> libxl__device_destroy_tapdisk is calling tap_ctl_find, which just 
> globally matches based on the type and name of the image, causing the 
> errors on destroy.

Both of these sound like genuine bugs which should be fixed, thanks for
reporting. Are either of them something you might be interested in
tackling? I'm nore than happy to give guidance etc.

For clarity it might be useful to have the config files for the various
domains involved here.

> Another, much more minor, bug in "xl" that I've found is that "xl 
> uptime" will not run without an argument; it is supposed to just show 
> all uptimes in this circumstance. This is an easy one-liner fix of 
> modifying line:
> 
> tools/libxl/xl_cmdimpl.c:5749:    while ((opt = def_getopt(argc, argv, 
> "s", "uptime", 1)) != -1) {
> 
> into this:
> 
> tools/libxl/xl_cmdimpl.c:5749:    while ((opt = def_getopt(argc, argv, 
> "s", "uptime", 0)) != -1) {
> 
> Also, a couple of small typos in "xl" --
> 
> libxl.c:556:        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "geting 
> domain info list");
> libxl.c:575:        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "geting 
> domain info list");
> libxl.c:678:        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "geting 
> domain info list");
> libxl.c:1393:        LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "non-existant 
> domain %d", domid);

Would you mind submitting patches for these two issues please?

See http://wiki.xen.org/wiki/Submitting_Xen_Patches for some
hints/tip/guidance.

> Also, running "xl sched-credit" always gives me an error, even though it 
> seems to work. Maybe it should be silent if cpu pools aren't defined? 

George (CCd) submitted a fix for this a while back (see
http://marc.info/?l=xen-devel&m=134502346714773). There is an
outstanding question though.

> As an additional note about 4.2 -- this isn't an "xl" issue, but rather 
> a build problem -- even though I use "./configure --libdir=/usr/lib64", 
> the build is still creating a "dist/install/usr/lib" directory, and 
> because of this, running ./install.sh nukes my system's "/usr/lib" -> 
> "/usr/lib64" softlink (which in turn causes all sorts of other problems 
> until it's corrected). Files that are being put in that tree:
> 
> xen-4.2-testing.hg # find dist/install/usr/lib
> dist/install/usr/lib

This is an interesting one. I'm not sure how we can avoid this
behaviour, perhaps there is a tar option to cause it to follow rather
than clobber synlinks?

The only other choice I can think of is to change install.sh to use a
(long) explicit list of $(FOO_DIR)/* entries instead of the single * to
avoid including system directories, or to otherwise make install.sh more
fragile and prone to bitrot :-(

I must confess I didn't realise anyone used install.sh -- is there some
advantage to it over make foo-install?

OOI which distro has this lib->lib64 link and what issues does removing
it create?

> dist/install/usr/lib/xen

This is $(XENFIRMWAREDIR). I have a feeling there is a reason this is
not using $(LIBDIR), but I don't recall what it was -- Roget/Matt do you
remember?

Thanks again for reporting all these.

Ian.

> dist/install/usr/lib/xen/boot
> dist/install/usr/lib/xen/boot/xenstore-stubdom.gz
> dist/install/usr/lib/xen/boot/ioemu-stubdom.gz
> dist/install/usr/lib/xen/boot/pv-grub-x86_64.gz
> dist/install/usr/lib/xen/boot/hvmloader
> dist/install/usr/lib/xen/boot/pv-grub-x86_32.gz
> dist/install/usr/lib/xen/bin
> dist/install/usr/lib/xen/bin/stubdom-dm
> dist/install/usr/lib/xen/bin/qemu-io
> dist/install/usr/lib/xen/bin/qemu-nbd
> dist/install/usr/lib/xen/bin/qemu-img
> dist/install/usr/lib/xen/bin/xenpaging
> dist/install/usr/lib/xen/bin/qemu-dm
> dist/install/usr/lib/xen/bin/qemu-system-i386
> dist/install/usr/lib/xen/bin/qemu-ga
> dist/install/usr/lib/xen/bin/stubdompath.sh
> 
> -John
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Bugs in xl that affect stubdom+tapdisk2, and others

[Matt Wilson]

On Fri, Nov 09, 2012 at 01:29:04PM +0000, Ian Campbell wrote:
> On Thu, 2012-11-08 at 20:45 +0000, John Weekes wrote:
> > As an additional note about 4.2 -- this isn't an "xl" issue, but rather 
> > a build problem -- even though I use "./configure --libdir=/usr/lib64", 
> > the build is still creating a "dist/install/usr/lib" directory, and 
> > because of this, running ./install.sh nukes my system's "/usr/lib" -> 
> > "/usr/lib64" softlink (which in turn causes all sorts of other problems 
> > until it's corrected). Files that are being put in that tree:
> > 
> > xen-4.2-testing.hg # find dist/install/usr/lib
> > dist/install/usr/lib
> 
> This is an interesting one. I'm not sure how we can avoid this
> behaviour, perhaps there is a tar option to cause it to follow rather
> than clobber synlinks?
> 
> The only other choice I can think of is to change install.sh to use a
> (long) explicit list of $(FOO_DIR)/* entries instead of the single * to
> avoid including system directories, or to otherwise make install.sh more
> fragile and prone to bitrot :-(

Oh dear. If we find that there's not a good reason for it anymore, I'd
think it would be better to remove install.sh instead.

> I must confess I didn't realise anyone used install.sh -- is there some
> advantage to it over make foo-install?
> 
> OOI which distro has this lib->lib64 link and what issues does removing
> it create?
> 
> > dist/install/usr/lib/xen
> 
> This is $(XENFIRMWAREDIR). I have a feeling there is a reason this is
> not using $(LIBDIR), but I don't recall what it was -- Roget/Matt do you
> remember?

There's no need for these files to be multilib compatible, as they are
not runtime DSOs. They're internal to Xen and I think that they should
be moved to $LIBEXECDIR instead, like /usr/libexec/xen/. Unfortunately
there's not much agreement on the usefulness of /usr/libexec, so we'll
need to make sure that packagers can adjust the paths for their
systems' standards.

For 4.2 we didn't want to change where these files were being placed,
and I believe they've bin in $PREFIX/lib/ for quite some time.

Matt

> Thanks again for reporting all these.
> 
> Ian.
> 
> > dist/install/usr/lib/xen/boot
> > dist/install/usr/lib/xen/boot/xenstore-stubdom.gz
> > dist/install/usr/lib/xen/boot/ioemu-stubdom.gz
> > dist/install/usr/lib/xen/boot/pv-grub-x86_64.gz
> > dist/install/usr/lib/xen/boot/hvmloader
> > dist/install/usr/lib/xen/boot/pv-grub-x86_32.gz
> > dist/install/usr/lib/xen/bin
> > dist/install/usr/lib/xen/bin/stubdom-dm
> > dist/install/usr/lib/xen/bin/qemu-io
> > dist/install/usr/lib/xen/bin/qemu-nbd
> > dist/install/usr/lib/xen/bin/qemu-img
> > dist/install/usr/lib/xen/bin/xenpaging
> > dist/install/usr/lib/xen/bin/qemu-dm
> > dist/install/usr/lib/xen/bin/qemu-system-i386
> > dist/install/usr/lib/xen/bin/qemu-ga
> > dist/install/usr/lib/xen/bin/stubdompath.sh

Re: Bugs in xl that affect stubdom+tapdisk2, and others

[Ian Campbell]

On Fri, 2012-11-09 at 19:55 +0000, Matt Wilson wrote:
> On Fri, Nov 09, 2012 at 01:29:04PM +0000, Ian Campbell wrote:
> > On Thu, 2012-11-08 at 20:45 +0000, John Weekes wrote:
> > > As an additional note about 4.2 -- this isn't an "xl" issue, but rather 
> > > a build problem -- even though I use "./configure --libdir=/usr/lib64", 
> > > the build is still creating a "dist/install/usr/lib" directory, and 
> > > because of this, running ./install.sh nukes my system's "/usr/lib" -> 
> > > "/usr/lib64" softlink (which in turn causes all sorts of other problems 
> > > until it's corrected). Files that are being put in that tree:
> > > 
> > > xen-4.2-testing.hg # find dist/install/usr/lib
> > > dist/install/usr/lib
> > 
> > This is an interesting one. I'm not sure how we can avoid this
> > behaviour, perhaps there is a tar option to cause it to follow rather
> > than clobber synlinks?
> > 
> > The only other choice I can think of is to change install.sh to use a
> > (long) explicit list of $(FOO_DIR)/* entries instead of the single * to
> > avoid including system directories, or to otherwise make install.sh more
> > fragile and prone to bitrot :-(
> 
> Oh dear. If we find that there's not a good reason for it anymore, I'd
> think it would be better to remove install.sh instead.

That's one option. If the list could be autogenerated (e.g configure
doing install.sh.in -> install.sh) then that might make it less prone to
problems.

> > I must confess I didn't realise anyone used install.sh -- is there some
> > advantage to it over make foo-install?
> > 
> > OOI which distro has this lib->lib64 link and what issues does removing
> > it create?
> > 
> > > dist/install/usr/lib/xen
> > 
> > This is $(XENFIRMWAREDIR). I have a feeling there is a reason this is
> > not using $(LIBDIR), but I don't recall what it was -- Roget/Matt do you
> > remember?
> 
> There's no need for these files to be multilib compatible, as they are
> not runtime DSOs. They're internal to Xen and I think that they should
> be moved to $LIBEXECDIR instead, like /usr/libexec/xen/. Unfortunately
> there's not much agreement on the usefulness of /usr/libexec, so we'll
> need to make sure that packagers can adjust the paths for their
> systems' standards.

Our configure seem to provide --libexecdir and I think packagers on
platforms which don't use /usr/libexec (e.g. Debian & derivatives) are
probably pretty used to passing that option.

> For 4.2 we didn't want to change where these files were being placed,
> and I believe they've bin in $PREFIX/lib/ for quite some time.

I remember now, yes deferring from 4.2 was the right thing to do but now
that 4.3 is in swing this seems like as good a time as any to be making
a change like this.

Ian.

Re: Bugs in xl that affect stubdom+tapdisk2, and others

[John Weekes]

[-- Attachment #1.1: Type: text/plain, Size: 6175 bytes --]

I took a look at the below concerns again in 4.3. The summarized 
concerns were:

1. Multiple domains not being able to use the same read-only images.
2. "xl destroy" giving error messages when tapdisk2 and stub domains are 
used.
3. "xl uptime" not working without an argument.
4. Minor typos in xl output.
5. A mistaken warning printed by "xl sched-credit".
6. Build issues related to /usr/lib being used even though ./configure 
was run with "--libdir=/usr/lib64"

The current situation:

1. Worked around: It appears that tapdisk2 has been disabled for 
read-only images, with qdisk used instead, presumably to work around 
this concern.
2. Still an issue: These errors still appear. But, since tapdisk2 is no 
longer used for CD images, they are mostly harmless now.
3. Still an issue: I am submitting a patch to address this.
4. Still an issue: I am submitting a patch to address this.
5. Fixed: I am no longer seeing the warning.
6. Still an issue: But, not a major one.

I also noticed a newly-introduced bug:

7. rtc_timeoffset is not accepting negative values. Attempting to use a 
negative value causes an error to be printed, with domain creation halted.

I am submitting a patch separately to address #7 along with #3 and #4. 
I've tested the patch and it works properly for me.

-John


On 11/8/2012 12:45 PM, John Weekes wrote:
> I'm having these problems when using "xl" in the latest c/s of 4.2 
> (25912:651d965ee2c0):
>
> - It's not allowing multiple domains to use the same images, at least 
> under tapdisk2. If I start one domain using an image (a read-only 
> CD-ROM image, for instance), then start another, I can see in "tap-ctl 
> list" that only one of the domains actually receives the proper 
> access. The other either never gets it, or it is disconnected after 
> it's created -- I can't readily tell which.
>
> - When I use "xl destroy" on a domain that also has an associated 
> stubdom, "xl" tries to destroy the tapdisk2 entries for both the 
> domain and its stubdomain, resulting in errors like these:
>
> # xl destroy testvds4
> libxl: error: libxl_blktap2.c:73:libxl__device_destroy_tapdisk: Unable 
> to find type aio disk /servers/isos/DummyCD.iso: No such file or 
> directory
> libxl: error: libxl_blktap2.c:73:libxl__device_destroy_tapdisk: Unable 
> to find type aio disk /servers/customers/testvds4.img: No such file or 
> directory
> libxl: error: libxl_blktap2.c:73:libxl__device_destroy_tapdisk: Unable 
> to find type aio disk /servers/isos/DummyCD-2.iso: No such file or 
> directory
> libxl: error: libxl.c:1466:devices_destroy_cb: libxl__devices_destroy 
> failed for 19
> libxl: error: libxl.c:1466:devices_destroy_cb: libxl__devices_destroy 
> failed for 18
>
> If I'm reading the source correctly, it seems as though the various 
> destruction functions aren't passing down the domid and ultimately 
> libxl__device_destroy_tapdisk is calling tap_ctl_find, which just 
> globally matches based on the type and name of the image, causing the 
> errors on destroy. I was trying to find a quick fix, but it looks like 
> any proper one will require querying xenstore for the tapdisk entries 
> specific to the domain, and I'm not immediately able to make that 
> change, so I've personally just switched back to using "xm" for the 
> time being. (I don't know if this relates to the first bug re: 
> multiple domains using the same image files).
>
> Another, much more minor, bug in "xl" that I've found is that "xl 
> uptime" will not run without an argument; it is supposed to just show 
> all uptimes in this circumstance. This is an easy one-liner fix of 
> modifying line:
>
> tools/libxl/xl_cmdimpl.c:5749:    while ((opt = def_getopt(argc, argv, 
> "s", "uptime", 1)) != -1) {
>
> into this:
>
> tools/libxl/xl_cmdimpl.c:5749:    while ((opt = def_getopt(argc, argv, 
> "s", "uptime", 0)) != -1) {
>
> Also, a couple of small typos in "xl" --
>
> libxl.c:556:        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "geting 
> domain info list");
> libxl.c:575:        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "geting 
> domain info list");
> libxl.c:678:        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "geting 
> domain info list");
> libxl.c:1393:        LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "non-existant 
> domain %d", domid);
>
> Also, running "xl sched-credit" always gives me an error, even though 
> it seems to work. Maybe it should be silent if cpu pools aren't 
> defined? Maybe it's supposed to be creating a second cpupool for the 
> second CPU in this system, and isn't?
>
> # xl sched-credit
> libxl: error: libxl.c:596:cpupool_info: failed to get info for cpupool1
> : No such file or directory
> Cpupool Pool-0: tslice=5ms ratelimit=500us
> Name                                ID Weight  Cap
> Domain-0                             0  65535    0
>
> As an additional note about 4.2 -- this isn't an "xl" issue, but 
> rather a build problem -- even though I use "./configure 
> --libdir=/usr/lib64", the build is still creating a 
> "dist/install/usr/lib" directory, and because of this, running 
> ./install.sh nukes my system's "/usr/lib" -> "/usr/lib64" softlink 
> (which in turn causes all sorts of other problems until it's 
> corrected). Files that are being put in that tree:
>
> xen-4.2-testing.hg # find dist/install/usr/lib
> dist/install/usr/lib
> dist/install/usr/lib/xen
> dist/install/usr/lib/xen/boot
> dist/install/usr/lib/xen/boot/xenstore-stubdom.gz
> dist/install/usr/lib/xen/boot/ioemu-stubdom.gz
> dist/install/usr/lib/xen/boot/pv-grub-x86_64.gz
> dist/install/usr/lib/xen/boot/hvmloader
> dist/install/usr/lib/xen/boot/pv-grub-x86_32.gz
> dist/install/usr/lib/xen/bin
> dist/install/usr/lib/xen/bin/stubdom-dm
> dist/install/usr/lib/xen/bin/qemu-io
> dist/install/usr/lib/xen/bin/qemu-nbd
> dist/install/usr/lib/xen/bin/qemu-img
> dist/install/usr/lib/xen/bin/xenpaging
> dist/install/usr/lib/xen/bin/qemu-dm
> dist/install/usr/lib/xen/bin/qemu-system-i386
> dist/install/usr/lib/xen/bin/qemu-ga
> dist/install/usr/lib/xen/bin/stubdompath.sh
>
> -John
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


[-- Attachment #1.2: Type: text/html, Size: 8528 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel